Chore running - general question
-
- Posts: 47
- Joined: Thu Jun 07, 2012 10:20 am
- OLAP Product: TM1
- Version: Various
- Excel Version: Various
- Location: UK
Chore running - general question
Hi,
I am working on a model where the requirement is to update a cube from base data (CSV) "instantly" when a new CSV becomes available. I am looking at whether to have a Chore running a simple TI every 30 seconds that checks for the existence of the CSV file, if it finds one then it runs an EXECUTEPROCESS to load the data.
I realise this is a massive generalisation and I will need to test this approach but I wanted to find out if anyone has had any experience of running chores to this level of repetition and have seen any performance overhead or issues in doing this? Alternatively is there a better way of doing it? - eg I've looked at running a Windows Scheduled Task and using this to kick off a Tm1RunTI but I have concerns over Task Scheduler's reliability.
Thanks
I am working on a model where the requirement is to update a cube from base data (CSV) "instantly" when a new CSV becomes available. I am looking at whether to have a Chore running a simple TI every 30 seconds that checks for the existence of the CSV file, if it finds one then it runs an EXECUTEPROCESS to load the data.
I realise this is a massive generalisation and I will need to test this approach but I wanted to find out if anyone has had any experience of running chores to this level of repetition and have seen any performance overhead or issues in doing this? Alternatively is there a better way of doing it? - eg I've looked at running a Windows Scheduled Task and using this to kick off a Tm1RunTI but I have concerns over Task Scheduler's reliability.
Thanks
-
- MVP
- Posts: 2832
- Joined: Tue Feb 16, 2010 2:39 pm
- OLAP Product: TM1, Palo
- Version: Beginning of time thru 10.2
- Excel Version: 2003-2007-2010-2013
- Location: Atlanta, GA
- Contact:
Re: Chore running - general question
A chore with a TI behind it that looks for the existence of a file is a tried and true method. I don't know about the every 30 seconds though. It depends on exactly how reliable your network is. Any hiccup and you can get in trouble with only 30 seconds between attempts. I've been burned before with a small window like that. I personally would never do anything like that with an interval less than two or three minutes.
- George Regateiro
- MVP
- Posts: 326
- Joined: Fri May 16, 2008 3:35 pm
- OLAP Product: TM1
- Version: 10.1.1
- Excel Version: 2007 SP3
- Location: Tampa FL USA
Re: Chore running - general question
I don't know if you have experience with .NET but there is a file monitor class that will monitor a directory for operations (add/modify/delete) and can execute code based on those triggers. You can then use the REST API or command line operations to kick off that process. I prefer this method just because it gets around the file not being there or possibly being locked when the TI kicks off. Also if you have locking in your system a 30 second window is short enough that you can have stacking processes and potentially make your locking worse.
If you go the chore route I would look in the forum. There are plenty of examples of building a resilient TI process that checks of the files existence and availability so that you are not generating unneeded errors.
If you go the chore route I would look in the forum. There are plenty of examples of building a resilient TI process that checks of the files existence and availability so that you are not generating unneeded errors.
-
- Community Contributor
- Posts: 217
- Joined: Thu Aug 15, 2013 9:05 am
- OLAP Product: TM1
- Version: 10.2.1.1
- Excel Version: 14.0.6129.5000
Re: Chore running - general question
You also need to be aware of the logs when you run chores that frequently. We use to have two chores running every minute, with each chore running a TI Process. This meant that every time we looked in the transaction log all we saw a lot of the time was the repeating logs of the minute chores. This made it more difficult when looking for transaction errors.
What is the source that is creating the csv? That same source, if at all possible, could fire the TM1RunTI.exe tool that could fire the TI that picks up the flat file.
What is the source that is creating the csv? That same source, if at all possible, could fire the TM1RunTI.exe tool that could fire the TI that picks up the flat file.
- macsir
- MVP
- Posts: 785
- Joined: Wed May 30, 2012 6:50 am
- OLAP Product: TM1
- Version: PAL 2.0.9
- Excel Version: Office 365
- Contact:
Re: Chore running - general question
Another way is to use Java to kick off TI process if you use latest TM1.
-
- Community Contributor
- Posts: 217
- Joined: Thu Aug 15, 2013 9:05 am
- OLAP Product: TM1
- Version: 10.2.1.1
- Excel Version: 14.0.6129.5000
Re: Chore running - general question
How exactly would you use Java to kick off a TI Process? By latest TM1 do you mean 10.2 fix pack 3?Another way is to use Java to kick off TI process if you use latest TM1.
Thanks.
Trevor.
- Alan Kirk
- Site Admin
- Posts: 6610
- Joined: Sun May 11, 2008 2:30 am
- OLAP Product: TM1
- Version: PA2.0.9.18 Classic NO PAW!
- Excel Version: 2013 and Office 365
- Location: Sydney, Australia
- Contact:
Re: Chore running - general question
There seems to have been a bit of a misunderstanding about that; I've heard this a couple of times now.TrevorGoss wrote:How exactly would you use Java to kick off a TI Process? By latest TM1 do you mean 10.2 fix pack 3?Another way is to use Java to kick off TI process if you use latest TM1.
TM1 has had a client side Java API for many, many years, and many, many versions. That API could be used to do many things that any of the other APIs can do, and that includes triggering off chores and processes via the TM1Chore class which I didn't discuss in my article, but can if you really want me to.
What, no-one? {Scans the room looking at all of the upturned chairs, the hastily slammed door and hearing dozens of footsteps running for the exit.} OK, take my word for it then.
Java applications, like most languages, can also monitor directories, though in Java's case this became easier when the nio API was introduced.
Consequently it would be possible for a Java application to be running in the background (on either the server or a client), scanning a folder for new data files and triggering the upload when the file appears. There's no need for the latest TM1 version to do this.
The change that has been made in the latest version has been to allow TI extensions, which is where Java code is called by TI, not where Java calls TI.
The other thing that has changed in the latest versions has of course been the RESTful API... but that is not Java-specific and is in fact language agnostic. Yes, a Java application could use it to trigger a chore just as it could trigger the chore through the classic Java API as I described above... but a program that uses the RESTful API need not be only Java.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
- macsir
- MVP
- Posts: 785
- Joined: Wed May 30, 2012 6:50 am
- OLAP Product: TM1
- Version: PAL 2.0.9
- Excel Version: Office 365
- Contact:
Re: Chore running - general question
Yes, sorry for the confusion. As Alan said, you can use Java to run TI via REST API, or use TI to run Java via TI Java extension.TrevorGoss wrote:How exactly would you use Java to kick off a TI Process? By latest TM1 do you mean 10.2 fix pack 3?Another way is to use Java to kick off TI process if you use latest TM1.
Thanks.
Trevor.
-
- Community Contributor
- Posts: 217
- Joined: Thu Aug 15, 2013 9:05 am
- OLAP Product: TM1
- Version: 10.2.1.1
- Excel Version: 14.0.6129.5000
Re: Chore running - general question
TI Java extension? I am yet to hear of this? Can you provide more information on this please? such as links? Or again, is this just the API? lolor use TI to run Java via TI Java extension
To run a TI outside of TM1 I just use the TM1RunTI.exe tool, but yes you can also use the APIs.
conflax, did you ever solve this?Hi,
I am working on a model where the requirement is to update a cube from base data (CSV) "instantly" when a new CSV becomes available. I am looking at whether to have a Chore running a simple TI every 30 seconds that checks for the existence of the CSV file, if it finds one then it runs an EXECUTEPROCESS to load the data.
I realise this is a massive generalisation and I will need to test this approach but I wanted to find out if anyone has had any experience of running chores to this level of repetition and have seen any performance overhead or issues in doing this? Alternatively is there a better way of doing it? - eg I've looked at running a Windows Scheduled Task and using this to kick off a Tm1RunTI but I have concerns over Task Scheduler's reliability.
Thanks
Thanks
- Alan Kirk
- Site Admin
- Posts: 6610
- Joined: Sun May 11, 2008 2:30 am
- OLAP Product: TM1
- Version: PA2.0.9.18 Classic NO PAW!
- Excel Version: 2013 and Office 365
- Location: Sydney, Australia
- Contact:
Re: Chore running - general question
No, it's not the API. It's on the server side. The APIs are on the client side.TrevorGoss wrote:TI Java extension? I am yet to hear of this? Can you provide more information on this please? such as links? Or again, is this just the API? lolor use TI to run Java via TI Java extension
It's been discussed here before but you may be unaware of it because the process for implementing them has been so extensively "documented" in the way that IBM "documents" so many things. But start here and here.
I'll be writing up an instruction guide similar to, but I expect rather shorter than, my Java API guide once my fingers recover from the latter, and any readers' eyeballs recover.
Quite so.TrevorGoss wrote:To run a TI outside of TM1 I just use the TM1RunTI.exe tool, but yes you can also use the APIs.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
-
- Community Contributor
- Posts: 217
- Joined: Thu Aug 15, 2013 9:05 am
- OLAP Product: TM1
- Version: 10.2.1.1
- Excel Version: 14.0.6129.5000
Re: Chore running - general question
Excellent, thank you.No, it's not the API. It's on the server side. The APIs are on the client side.
It's been discussed here before but you may be unaware of it because the process for implementing them has been so extensively "documented" in the way that IBM "documents" so many things. But start here and here.