Page 1 of 2

RunProcess and threads

Posted: Wed Apr 24, 2019 3:38 pm
by Wim Gielis
Hi,

I run a simple TI process that copies data a number of times:

Code: Select all

i = 1;
While( i <= 10 );
   RunProcess( 'CopyData_HoursSpent', 'pSourceMonth', '1', 'pSourceYear', '2019', 'pTargetMonth', '2', 'pTargetYear', '2019', 'pPerson', 'Wim' );
   i = i + 1;
End;
Yes, this means running 10 times the same process.

With ExecuteProcess I see that 1 thread is used in TM1top.
With RunProcess I see multiple threads. That's fine but after the process finishes the threads are still visible in TM1Top.

Shouldn't these threads be killed properly at the end of the process ?

I am logged in as admin and I start the process using admin.
01.png
01.png (19.85 KiB) Viewed 1888 times

Re: RunProcess and threads

Posted: Wed Apr 24, 2019 8:12 pm
by qml
Nah, these threads stick around in the exact same way as any other REST API threads do (because RunProcess uses REST API calls behind the scenes). Ultimately these threads do go away, perhaps when IdleConnectionTimeOutSeconds is reached (this needs to be confirmed). Not sure if this is really a problem beyond the visual clutter - the resource use of each thread is minimal.

Re: RunProcess and threads

Posted: Wed Apr 24, 2019 10:34 pm
by EvgenyT
Hi Guys

Although I have not used RunProcess just yet, but if I understand it correctly by using Rest API, TM1 does not need to log the user in and out (causing a lock) anymore as it used to happen with TM1RunTI? Typical workaround used to be to create a tone of admin accounts...

Thanks

Evgeny

Re: RunProcess and threads

Posted: Thu Apr 25, 2019 5:51 am
by lotsaram
EvgenyT wrote:
Wed Apr 24, 2019 10:34 pm
Although I have not used RunProcess just yet, but if I understand it correctly by using Rest API, TM1 does not need to log the user in and out (causing a lock) anymore as it used to happen with TM1RunTI? Typical workaround used to be to create a tone of admin accounts...
Yes, the user ID of the parent process is used, like the asynchronous request model of the Rest API.
To be fair though about tm1runti the locking that you *sometimes* get during the login phase only happens when using CAM. Never seen it happen when using vanilla TM1 security.

Re: RunProcess and threads

Posted: Thu Apr 25, 2019 9:28 am
by qml
lotsaram wrote:
Thu Apr 25, 2019 5:51 am
To be fair though about tm1runti the locking that you *sometimes* get during the login phase only happens when using CAM. Never seen it happen when using vanilla TM1 security.
That's right. CAM log-on consists of more than just the log-on itself. And CAM log-off consists of more than just the log-off. Security needs to be (potentially) updated (as group membership is pulled from the Cognos cookie) and, depending on your CreateNewCAMClients setting, the }Clients dimension might even need to be updated. And so there are additional locks requested by these actions that are not needed during native authentication.

Re: RunProcess and threads

Posted: Fri Apr 26, 2019 1:39 am
by mce
Thanks for the notes about RunProcess. Here are some questions about RunProcess. Any help will be appreciated. Thanks in advance.

When you consider Synchronized() function or when you consider batchupdate functions, are the threads treated as completely independent process executions similar to the way it happens when executed via RunTI or are they still dependent on the main process that triggered them similar to the way it happens when executed via ExecuteProcess function?

I mean, are the locked objects shared/inherited from the main process or are they completely treated as if they are triggered via RunTI as independent sessions?

Can we use RunProcess instead of RunTI in all cases, if we do not need to use credentials of a different user in executing the processes?

Re: RunProcess and threads

Posted: Fri Apr 26, 2019 4:11 am
by ykud
Can I also jump on with some questions on RunProcess (don't have 2.0.6 to test yet): what's with transaction isolation in RunProcess?
A simple test is to start with an empty cube with 1 cell.
Process A:
1) CellPutN 1 into that cell
2) RunProcess (B)
3) CellGetN that cell in a loop for a minute (does it see the changes done by B or A works in isolation and only sees it's own changes)

Process B:
1) CellGetN cell (does it see 0 or 1 -- do we inherit the parent TI transaction context)?
2) CellPutN 2 into that cell

And once they both are done, what do we have in the cube? 1 or 2? Run TIs would end up with 1, hence I'm a big fan of using files for thread control. If we get 2 with RunProcess, chances are we can move to a cube with 'Thread status' that could be polled from A to determine whether B has finished.

Thanks,
Y

Re: RunProcess and threads

Posted: Fri Apr 26, 2019 2:35 pm
by Bakkone
Wait?

Is runprocess already in 2.0.6? I thought I had to wait until 2.0.7 which is getting constant delays.

Or is 2.0.7 already on the cloud plattform?

Re: RunProcess and threads

Posted: Fri Apr 26, 2019 2:42 pm
by Wim Gielis
I can use it in 2.0.6. on my laptop. It's 11.4 (tm1s.exe = 11.4.0.21)
01.png
01.png (19.43 KiB) Viewed 1772 times


2.0.7 should be out either today, either early next week - so I have been told by someone from IBM.

Wim

Re: RunProcess and threads

Posted: Fri Apr 26, 2019 3:17 pm
by Bakkone
Thx for the info. Turns out I wasnt on 06 at the server I was thinking about. Guess a few more months went by faster than I thought.

Re: RunProcess and threads

Posted: Fri Apr 26, 2019 3:33 pm
by lotsaram
Bakkone wrote:
Fri Apr 26, 2019 2:35 pm
Wait?

Is runprocess already in 2.0.6? I thought I had to wait until 2.0.7 which is getting constant delays.

Or is 2.0.7 already on the cloud plattform?
It's in 2.0.6 (11.4) it just isn't either announced, documented, nor officially supported until 2.0.7.

Re: RunProcess and threads

Posted: Mon Apr 29, 2019 3:04 pm
by mce
I know RunProcess is not supported yet, but seems to work in 2.0.5.
However I found a case where it did not work.
Those processes that connects to Oracle DB as data source, could not connect to source DB with the following error, although they work all Ok when executed using ExecuteProcess or using RunTI.

Code: Select all

TM1.SQLAPI   S1000[Oracle][ODBC][Ora]ORA-12541: TNS:no listener
I hope this issue has been resolved in a next release or at least when it is officially supported.

Regards,

Re: RunProcess and threads

Posted: Tue Apr 30, 2019 7:56 pm
by Wim Gielis
Hello

Would it be possible to check it in version 2.0.7 please ?

Re: RunProcess and threads

Posted: Wed May 01, 2019 2:12 pm
by qml
mce wrote:
Fri Apr 26, 2019 1:39 am
When you consider Synchronized() function or when you consider batchupdate functions, are the threads treated as completely independent process executions similar to the way it happens when executed via RunTI or are they still dependent on the main process that triggered them similar to the way it happens when executed via ExecuteProcess function?

I mean, are the locked objects shared/inherited from the main process or are they completely treated as if they are triggered via RunTI as independent sessions?
Each thread created by RunProcess is a separate, independent transaction and has its own locks.
mce wrote: Can we use RunProcess instead of RunTI in all cases, if we do not need to use credentials of a different user in executing the processes?
I cannot think of a scenario where you couldn't use it. Just remember RunProcess is fully asynchronous i.e. there is no equivalent of the Wait flag of ExecuteCommand.
ykud wrote:
Fri Apr 26, 2019 4:11 am
Can I also jump on with some questions on RunProcess (don't have 2.0.6 to test yet): what's with transaction isolation in RunProcess?
A simple test is to start with an empty cube with 1 cell.
Process A:
1) CellPutN 1 into that cell
2) RunProcess (B)
3) CellGetN that cell in a loop for a minute (does it see the changes done by B or A works in isolation and only sees it's own changes)

Process B:
1) CellGetN cell (does it see 0 or 1 -- do we inherit the parent TI transaction context)?
2) CellPutN 2 into that cell

And once they both are done, what do we have in the cube? 1 or 2? Run TIs would end up with 1, hence I'm a big fan of using files for thread control. If we get 2 with RunProcess, chances are we can move to a cube with 'Thread status' that could be polled from A to determine whether B has finished.
As I said, process A and B would have their own transaction sets, so would not see the changes made by the other process. The process that finishes last commits its changes last.

Re: RunProcess and threads

Posted: Fri May 03, 2019 7:32 am
by lotsaram
I agree with what qml said, it's not really about the parallelization of processes ruining on different threads but much more about the commit model used in TI where only the TI processes belonging to the same transaction have access to changes they have made prior to the commit and merge with the base model.

To get back to Yuri's point about controlling or orchestrating batch processes. flags in cubes are just inherently problematic. The surest way to manage batches and process dependencies (e.g. process B is not allowed to start because process A is still running) has always been via file flags or else a control table in RDBMS. Maybe RunProcess could change this ... I could imagine a solution where all calls to update a process status cube are encapsulated in a RunProcess (which should do nothing more than CellPut therefore run & commit should be more or less instant). If the commit to the status cube is outside the transaction of the process then maybe keeping track of status within TM1 is possible. I would want to test this pretty thoroughly tho before abandoning tried and true methods with file flags.

Re: RunProcess and threads

Posted: Mon May 06, 2019 12:17 am
by EvgenyT
How about sequence orchestration by using Synchronized() ?

Re: RunProcess and threads

Posted: Mon May 06, 2019 1:49 am
by ykud
qml wrote:
Wed May 01, 2019 2:12 pm
As I said, process A and B would have their own transaction sets, so would not see the changes made by the other process. The process that finishes last commits its changes last.
Thank you very much for confirming this.
lotsaram wrote:
Fri May 03, 2019 7:32 am
To get back to Yuri's point about controlling or orchestrating batch processes. flags in cubes are just inherently problematic. The surest way to manage batches and process dependencies (e.g. process B is not allowed to start because process A is still running) has always been via file flags or else a control table in RDBMS. Maybe RunProcess could change this
I agree wholeheartedly, but I see the 'cube based' solutions on the client sites every now and then and they sometimes 'almost' work and it takes me a while to convince people to go to good old files way.
If we could get the job guid from RunProcess (we already) and there was an asynchronous CheckJobStatus(JobGuid) TI function, we could find out whether the child process completed successfully or not and it would eliminate the need to for files.

Overall it's just inconvenient to implement parallelisation primitives with files, would be easier to have a mutex / semaphore / thread status checking logic built in TM1 (synchronised is doing a sorta/kinda mutex already, I know). Too many ways to implement this and make mistakes otherwise and it's a notoriously hard to test area, some race conditions can only happen in production :)

Re: RunProcess and threads

Posted: Tue May 07, 2019 10:20 am
by Drg
ykud wrote:
Mon May 06, 2019 1:49 am
qml wrote:
Wed May 01, 2019 2:12 pm
As I said, process A and B would have their own transaction sets, so would not see the changes made by the other process. The process that finishes last commits its changes last.
Thank you very much for confirming this.
lotsaram wrote:
Fri May 03, 2019 7:32 am
To get back to Yuri's point about controlling or orchestrating batch processes. flags in cubes are just inherently problematic. The surest way to manage batches and process dependencies (e.g. process B is not allowed to start because process A is still running) has always been via file flags or else a control table in RDBMS. Maybe RunProcess could change this
I agree wholeheartedly, but I see the 'cube based' solutions on the client sites every now and then and they sometimes 'almost' work and it takes me a while to convince people to go to good old files way.
If we could get the job guid from RunProcess (we already) and there was an asynchronous CheckJobStatus(JobGuid) TI function, we could find out whether the child process completed successfully or not and it would eliminate the need to for files.

Overall it's just inconvenient to implement parallelisation primitives with files, would be easier to have a mutex / semaphore / thread status checking logic built in TM1 (synchronised is doing a sorta/kinda mutex already, I know). Too many ways to implement this and make mistakes otherwise and it's a notoriously hard to test area, some race conditions can only happen in production :)

Yes, I think in the new versions we will see the function of checking job status.
But I do not agree that the processes of orchestration are not solved.
In our production environment there are several solutions for sequential queues, one based on files, the other based on a cube of queues, both work (I note that a solution based on a cube is rather complicated, but it works!)

Re: RunProcess and threads

Posted: Wed May 08, 2019 6:09 am
by ykud
Drg wrote:
Tue May 07, 2019 10:20 am
But I do not agree that the processes of orchestration are not solved.
In our production environment there are several solutions for sequential queues, one based on files, the other based on a cube of queues, both work (I note that a solution based on a cube is rather complicated, but it works!)
Why having both? To make support more intellectually stimulating? :-)

How do you do check for task completion via cubes, do a runTI that reads the cube contents? What happens if this 'check' process itself fails, would you be able to differentiate between it running successfully or failing silently (you ran an ExecuteCommand and it returned all good)?

Re: RunProcess and threads

Posted: Wed May 08, 2019 6:20 am
by Drg
ykud wrote:
Wed May 08, 2019 6:09 am
Why having both? To make support more intellectually stimulating? :-)

How do you do check for task completion via cubes, do a runTI that reads the cube contents? What happens if this 'check' process itself fails, would you be able to differentiate between it running successfully or failing silently (you ran an ExecuteCommand and it returned all good)?
Yuri historically it turned out that there are two different logics used for different processes; in the second, some java is used.

We chose a different concept, we use runti to start the wrapper process, inside which we run the executeprocess (here we had to use a little magic) and read the execution status of the process and describe the status in a cube. there is also the only process that polls the status cube and determines how many threads are free and so on based on it.