RunProcess and Process Ends msg

User avatar
roei61
Posts: 39
Joined: Wed Aug 22, 2012 5:44 am
OLAP Product: PAX
Version: 2.0.7
Excel Version: 2016
Location: Israel

RunProcess and Process Ends msg

Post by roei61 » Sun Oct 13, 2019 5:57 am

Hi,

I have process that being executed by users (few users can execute the process on the same time).
I'm using few runprocess commands in this process to run other processes.
The main process ends after few seconds and the user gets msg of "Process Ends". However, there are still threats that running from the runprocess command.
I want to delay the "Process Ends" msg until all the runprocesses are done also.

Any ideas?
Keep in mind that there can be also runprocesses that being executed by other users

Thanks

declanr
MVP
Posts: 1638
Joined: Mon Dec 05, 2011 11:51 am
OLAP Product: Cognos TM1
Version: PA2.0 and most of the old ones
Excel Version: All of em
Location: Manchester, United Kingdom
Contact:

Re: RunProcess and Process Ends msg

Post by declanr » Sun Oct 13, 2019 11:23 am

I handle this by having the master process create a text file in a folder and pass the name of that file to the sub process (doing this for ever sub process it runs.) The last line of the sub process deletes the file.

At the end of the master process it loops the folder counting the files that exist until it finds all the files created have been deleted (indicating all sub processes have completed.) I also put a backup option in where after a set amount of time it will finish with an error if there are still files that exist (e.g if I expect it to take 20 seconds total I would maybe set a failure time of 60 seconds.)
Declan Rodger

User avatar
roei61
Posts: 39
Joined: Wed Aug 22, 2012 5:44 am
OLAP Product: PAX
Version: 2.0.7
Excel Version: 2016
Location: Israel

Re: RunProcess and Process Ends msg

Post by roei61 » Sun Oct 13, 2019 11:39 am

I thought on something similar: to create an element \ view and delete in the end.
In this way I can add an attribute that counts how many processes completed since not always the last process will ends the latest.

What would you do if one of the process has error? I don't want to show "Process Completed" msg.
How can I count how many process not completed correctly?

However, I hope there is more elegant way.

lotsaram
MVP
Posts: 3348
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: RunProcess and Process Ends msg

Post by lotsaram » Mon Oct 14, 2019 7:49 am

I second Declan's suggestion. File based semaphores are really the only way to go when running processes on independent threads. Reading a cell value (or attribute value) is vey tricky to get right because of TM1's commit model and which version of the data your monitoring process is rteading versus the separate commits of each worker thread, file based semaphores are just much easier top get right and better for one's sanity.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
tiagoblauth
Posts: 20
Joined: Wed Jul 08, 2015 11:50 am
OLAP Product: TM1, Cognos BI
Version: 10.2.1
Excel Version: 2010

Re: RunProcess and Process Ends msg

Post by tiagoblauth » Mon Oct 14, 2019 3:30 pm

I used to create control cubes, so I store start date and end dates and completion. You can provide read access to users, so they can monitor it.
You can also check for errors by during the process when you run trigger processes this way:

return_value = ExecuteProcess('create_sales_cube');

return_value can be compared with these functions:
ProcessExitByChoreQuit() - indicates that the process exited due to execution of the ChoreQuit function
ProcessExitNormal() - indicates that the process executed normally
ProcessExitMinorError() - indicates that the process executed successfully but encountered minor errors
ProcessExitByQuit() - indicates that the process exited because of an explicit "quit" command
ProcessExitWithMessage() - indicates that the process exited normally, with a message written to tm1server.log
ProcessExitSeriousError() - indicates that the process exited because of a serious error
ProcessExitOnInit() - indicates that the process aborted during initialization
ProcessExitByBreak() - indicates that the process exited because it encountered a ProcessBreak function

Based on comparison you can place message on the control cube or do externa actions like send an email to support.

User avatar
ykud
Posts: 64
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: RunProcess and Process Ends msg

Post by ykud » Mon Oct 14, 2019 11:26 pm

lotsaram wrote:
Mon Oct 14, 2019 7:49 am
I second Declan's suggestion. File based semaphores are really the only way to go when running processes on independent threads. Reading a cell value (or attribute value) is vey tricky to get right because of TM1's commit model and which version of the data your monitoring process is rteading versus the separate commits of each worker thread, file based semaphores are just much easier top get right and better for one's sanity.

Just make sure you don't run too many things too fast with file semaphores :)
Any modern OS has it's own file system cache (and any storage system has another one as well), so if you have a lot of processes doing things in parallel a finished file write can be not 'flushed' to disk and another process can not see the completion and, for example, restart it. This is where you start seeing things like 'Write a lock file & Sleep 500ms' in TI code. Very nasty to trace :(
In 90% of cases files should suffice, but I'm starting to think that a sqllite instance to store the flags with guaranteed read / write might be better at large scale.

What would really be great is for IBM to come up with 'have X finished' call and then we'll all forget all these file / cube locking solutions as a bad dream they are.

Cheers,
Yuri

lotsaram
MVP
Posts: 3348
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: RunProcess and Process Ends msg

Post by lotsaram » Tue Oct 15, 2019 6:42 am

ykud wrote:
Mon Oct 14, 2019 11:26 pm
I'm starting to think that a sqllite instance to store the flags with guaranteed read / write might be better at large scale.
I find this idea quite interesting. But in exactly the kind of scenario where immediacy of file writes might be problemantic with lots of multi-threadig and dozens of TIs running at once wouldn't this have it's own locking problems as sqlite has an exclusive write access model?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

Wim Gielis
MVP
Posts: 2339
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: RunProcess and Process Ends msg

Post by Wim Gielis » Thu Oct 17, 2019 9:47 pm

declanr wrote:
Sun Oct 13, 2019 11:23 am
I handle this by having the master process create a text file in a folder and pass the name of that file to the sub process (doing this for ever sub process it runs.) The last line of the sub process deletes the file.

At the end of the master process it loops the folder counting the files that exist until it finds all the files created have been deleted (indicating all sub processes have completed.) I also put a backup option in where after a set amount of time it will finish with an error if there are still files that exist (e.g if I expect it to take 20 seconds total I would maybe set a failure time of 60 seconds.)
To check whether there are still certain files lurking around in the TM1 data directory (or a subfolder thereof), this condensed code could be used.
Different processes that are running at the same time should be taking away the text files.

Code: Select all

# do we still have files lurking around ?
While( 1 > 0 ) ;
   If( WildcardFileSearch( 'D:\TM1Data\*.txt', '' ) @= '' );
      Break;
   Else;
      Sleep( 100 );
   EndIf;
End;
Use a better filemask than just "all TXT files" and/or use a subfolder or a different folder.

Note the 2 undocumented functions Break and Sleep but they prove to be very useful in here ! Usually I like to condense the code to the essence ;-)
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

User avatar
ykud
Posts: 64
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: RunProcess and Process Ends msg

Post by ykud » Sat Oct 19, 2019 11:34 pm

lotsaram wrote:
Tue Oct 15, 2019 6:42 am
ykud wrote:
Mon Oct 14, 2019 11:26 pm
I'm starting to think that a sqllite instance to store the flags with guaranteed read / write might be better at large scale.
I find this idea quite interesting. But in exactly the kind of scenario where immediacy of file writes might be problemantic with lots of multi-threadig and dozens of TIs running at once wouldn't this have it's own locking problems as sqlite has an exclusive write access model?
Yep, but it's not as much of problem as a solution from my point of view.
Write locks in SQLLite would guarantee that you'd never have a 'dirty' read as with file system (when you think there's no file present because it's still cached on OS level). And sqllite transactions are really fast, so you're sacrificing way less speed than the usual Sleep a second after a file write.
And you'd also be able to 'store' the execution timeline properly in a db table, making error tracing way simpler than trying to guess what thread created / deleted what file when.

Y

User avatar
roei61
Posts: 39
Joined: Wed Aug 22, 2012 5:44 am
OLAP Product: PAX
Version: 2.0.7
Excel Version: 2016
Location: Israel

Re: RunProcess and Process Ends msg

Post by roei61 » Sun Oct 20, 2019 5:59 am

Hi,

Thank you all for the answers.
However, I'm still missing solution to know if there was a error \ minor error while running those processes.
I don't want the user to get the process ends msg if there were errors.

Since RunProcess brings no result I need Idea how can I register the process result.

Wim Gielis
MVP
Posts: 2339
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: RunProcess and Process Ends msg

Post by Wim Gielis » Sun Oct 20, 2019 1:33 pm

Did you bother to check the documentation on the RunProcess function ?
https://www.ibm.com/support/knowledgece ... ocess.html

There is a section on “Return values”.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

User avatar
roei61
Posts: 39
Joined: Wed Aug 22, 2012 5:44 am
OLAP Product: PAX
Version: 2.0.7
Excel Version: 2016
Location: Israel

Re: RunProcess and Process Ends msg

Post by roei61 » Sun Oct 20, 2019 2:01 pm

Yes, I saw it.
1. This part is not clear to me, how can I get an error from the running process if it did not ended yet since main process already ends?
2. How can I recognize "minor errors", "Error" etc. ?

Wim Gielis
MVP
Posts: 2339
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: RunProcess and Process Ends msg

Post by Wim Gielis » Sun Oct 20, 2019 2:32 pm

Do you stretch the main process with the file semaphore, as suggested above ?
You can only see whether there was an error or not, AFAIK, looking at the documentation. Or the subprocess should be augmented with your ow custom logging and based on that, retrieve the process outcome.

As always, it would help if you post the exact code in TI.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

Wim Gielis
MVP
Posts: 2339
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: RunProcess and Process Ends msg

Post by Wim Gielis » Sun Oct 20, 2019 2:34 pm

tiagoblauth wrote:
Mon Oct 14, 2019 3:30 pm
I used to create control cubes, so I store start date and end dates and completion. You can provide read access to users, so they can monitor it.
You can also check for errors by during the process when you run trigger processes this way:

return_value = ExecuteProcess('create_sales_cube');

return_value can be compared with these functions:
ProcessExitByChoreQuit() - indicates that the process exited due to execution of the ChoreQuit function
ProcessExitNormal() - indicates that the process executed normally
ProcessExitMinorError() - indicates that the process executed successfully but encountered minor errors
ProcessExitByQuit() - indicates that the process exited because of an explicit "quit" command
ProcessExitWithMessage() - indicates that the process exited normally, with a message written to tm1server.log
ProcessExitSeriousError() - indicates that the process exited because of a serious error
ProcessExitOnInit() - indicates that the process aborted during initialization
ProcessExitByBreak() - indicates that the process exited because it encountered a ProcessBreak function

Based on comparison you can place message on the control cube or do externa actions like send an email to support.
This works for ExecuteProcess but not RunProcess.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

Wim Gielis
MVP
Posts: 2339
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: RunProcess and Process Ends msg

Post by Wim Gielis » Wed Oct 23, 2019 10:17 pm

declanr wrote:
Sun Oct 13, 2019 11:23 am
I also put a backup option in where after a set amount of time it will finish with an error if there are still files that exist (e.g if I expect it to take 20 seconds total I would maybe set a failure time of 60 seconds.)
Hi Declan

Could you give some more information about the backup setup ?
Let's say there is a mother process that kicks off 12 times the same process (once for every month), and the whole process is taking 5 minutes where you except 2 minutes. So after 2 minutes, the mother process is stopped, but the subprocess keeps running for months 4, 5, 6, ..., 12 for example ?
The output is a message in the TM1 message log and the mother process that stops but you do not take action to stop the subprocess (runs) ?
If it's that, I could do it, but I would be interested in knowing if you also stop the subprocess in one what or another. For example with a different text file ?
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

User avatar
tiagoblauth
Posts: 20
Joined: Wed Jul 08, 2015 11:50 am
OLAP Product: TM1, Cognos BI
Version: 10.2.1
Excel Version: 2010

Re: RunProcess and Process Ends msg

Post by tiagoblauth » Thu Oct 24, 2019 8:18 pm

To monitor multiple processes running at the same time and having multiple users running it, I used to create control cubes. It lists the processes needed, parameters, start / end date, process status (queued / running / completed / error), dependency. So, you can monitor what is going on and also the process did not start running in the middle of another execution.

Drg
Posts: 139
Joined: Fri Aug 12, 2016 10:02 am
OLAP Product: tm1
Version: 10.2.0 - 10.3.0
Excel Version: 2010

Re: RunProcess and Process Ends msg

Post by Drg » Fri Oct 25, 2019 11:45 am

Our manager in ibm temm us you need use rest api (pass them you QUEID) to retrive status execution process.

We still wait norml function retriving status in TI.

HyunaHyuna
Posts: 11
Joined: Thu Jun 16, 2016 7:57 pm
OLAP Product: Planning Analytics
Version: 2.0
Excel Version: 2016

Re: RunProcess and Process Ends msg

Post by HyunaHyuna » Fri Feb 07, 2020 2:09 pm

Hi,

I have built a bunch of new processes to replace our current solution with Rest API, using a lot of the solutions discussed above.
declanr wrote:
Sun Oct 13, 2019 11:23 am
I handle this by having the master process create a text file in a folder and pass the name of that file to the sub process (doing this for ever sub process it runs.) The last line of the sub process deletes the file.

Code: Select all

 
StringGlobalVariable( 'vGlobalTxtFileName' );
n_calcA = 1;  
WHILE( n_calcA <= nASubsetSize );
 sA = SubsetGetElementName( sDimA, cTemp, n_calcA );  
 vGlobalTxtFileName = cFolder | sProcess | '_' | sA | '_' |  sVersion | '.txt';
 ASCIIOUTPUT( vGlobalTxtFileName, 'Started' );
 RunProcess( sProcess, 'pA', sA, 'pB', sB, 'pVersion', sVersion );
 #This sleep is here to give time to the subset to register the current global variable into a local variable
 Sleep( 100 );
 ENDIF;
 n_calcA = n_calcA + 1;         
END;
While( 
 WildcardFileSearch( cFolder | sProcess | '*.txt', '' ) @<> '' 
 % WildcardFileSearch( cFolder | sProcess | '*.txt$', '' ) @<> ''
 % n_FailSafe >= 6 * 30
 % n_FailSafe = 1 
) ;
 Sleep( 10000 );
 n_FailSafe = n_FailSafe + 1;
End;
Our master process does almost everything in the prologue tab, there are just some cubelogging turned back on in the epilogue tab and deleting temp subsets. All of the code above is in Prologue. I have ran into the following issue:

I don't want to have a bunch of parameters to give information from one process to its sub, so to avoid this, I have implemented a set of global variables ("is this a chore?", "which chore?", "path to folder containing txt files that have to be wildcardsearched", etc). During testing, however, I have found that while everything works well when I use ExecuteProcess, when I switch it out for RunProcess, the global variables all end up being empty strings or 0s.

In the above code: when I replace RunProcess with ExecuteProcess, the global variable gets transferred and recognized. When I turn it back into RunProcess, the subprocess sees vGlobalTxtFileName as empty.

Has anyone else run into this issue? How do you send the name of the text file that has to be deleted to the subprocess? Do you use a parameter, do global variables work for you, is there a third option you use? (I have now created a small admin cube to write this info into, and read it from, but I still feel global variables would be more elegant)

Thanks in advance!

User avatar
PavoGa
Community Contributor
Posts: 409
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.05
Excel Version: 2013 PAW
Location: Cleveland, Tennessee

Re: RunProcess and Process Ends msg

Post by PavoGa » Fri Feb 07, 2020 7:00 pm

HyunaHyuna wrote:
Fri Feb 07, 2020 2:09 pm

I don't want to have a bunch of parameters to give information from one process to its sub, so to avoid this, I have implemented a set of global variables ("is this a chore?", "which chore?", "path to folder containing txt files that have to be wildcardsearched", etc). During testing, however, I have found that while everything works well when I use ExecuteProcess, when I switch it out for RunProcess, the global variables all end up being empty strings or 0s.

In the above code: when I replace RunProcess with ExecuteProcess, the global variable gets transferred and recognized. When I turn it back into RunProcess, the subprocess sees vGlobalTxtFileName as empty.
Since RunProcess operates by opening a separate thread, it has no connection to the Session or Global variables declared in the master process.
Ty
Cleveland, TN

EvgenyT
Community Contributor
Posts: 316
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: 2016
Location: Sydney, Australia

Re: RunProcess and Process Ends msg

Post by EvgenyT » Sun Feb 09, 2020 1:52 am

Wim Gielis wrote:
Sun Oct 20, 2019 1:33 pm
Did you bother to check the documentation on the RunProcess function ?
https://www.ibm.com/support/knowledgece ... ocess.html

There is a section on “Return values”.
The return value is JobID which I understand is only possible to get via REST API? So you still need something like POSTMAN etc. to execute and capture the value

Post Reply