RunProcess and threads

Wim Gielis
MVP
Posts: 2087
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.7
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

RunProcess and threads

Post by Wim Gielis » Wed Apr 24, 2019 3:38 pm

Hi,

I run a simple TI process that copies data a number of times:

Code: Select all

i = 1;
While( i <= 10 );
   RunProcess( 'CopyData_HoursSpent', 'pSourceMonth', '1', 'pSourceYear', '2019', 'pTargetMonth', '2', 'pTargetYear', '2019', 'pPerson', 'Wim' );
   i = i + 1;
End;
Yes, this means running 10 times the same process.

With ExecuteProcess I see that 1 thread is used in TM1top.
With RunProcess I see multiple threads. That's fine but after the process finishes the threads are still visible in TM1Top.

Shouldn't these threads be killed properly at the end of the process ?

I am logged in as admin and I start the process using admin.
01.png
01.png (19.85 KiB) Viewed 1920 times
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 service with AutoHotKey

User avatar
qml
MVP
Posts: 1073
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.6 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: RunProcess and threads

Post by qml » Wed Apr 24, 2019 8:12 pm

Nah, these threads stick around in the exact same way as any other REST API threads do (because RunProcess uses REST API calls behind the scenes). Ultimately these threads do go away, perhaps when IdleConnectionTimeOutSeconds is reached (this needs to be confirmed). Not sure if this is really a problem beyond the visual clutter - the resource use of each thread is minimal.
Kamil Arendt

EvgenyT
Community Contributor
Posts: 275
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: Cognos TM 1
Version: 10.2.2
Excel Version: 2013
Location: Sydney, Australia

Re: RunProcess and threads

Post by EvgenyT » Wed Apr 24, 2019 10:34 pm

Hi Guys

Although I have not used RunProcess just yet, but if I understand it correctly by using Rest API, TM1 does not need to log the user in and out (causing a lock) anymore as it used to happen with TM1RunTI? Typical workaround used to be to create a tone of admin accounts...

Thanks

Evgeny

lotsaram
MVP
Posts: 3251
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: RunProcess and threads

Post by lotsaram » Thu Apr 25, 2019 5:51 am

EvgenyT wrote:
Wed Apr 24, 2019 10:34 pm
Although I have not used RunProcess just yet, but if I understand it correctly by using Rest API, TM1 does not need to log the user in and out (causing a lock) anymore as it used to happen with TM1RunTI? Typical workaround used to be to create a tone of admin accounts...
Yes, the user ID of the parent process is used, like the asynchronous request model of the Rest API.
To be fair though about tm1runti the locking that you *sometimes* get during the login phase only happens when using CAM. Never seen it happen when using vanilla TM1 security.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
qml
MVP
Posts: 1073
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.6 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: RunProcess and threads

Post by qml » Thu Apr 25, 2019 9:28 am

lotsaram wrote:
Thu Apr 25, 2019 5:51 am
To be fair though about tm1runti the locking that you *sometimes* get during the login phase only happens when using CAM. Never seen it happen when using vanilla TM1 security.
That's right. CAM log-on consists of more than just the log-on itself. And CAM log-off consists of more than just the log-off. Security needs to be (potentially) updated (as group membership is pulled from the Cognos cookie) and, depending on your CreateNewCAMClients setting, the }Clients dimension might even need to be updated. And so there are additional locks requested by these actions that are not needed during native authentication.
Kamil Arendt

User avatar
mce
Community Contributor
Posts: 305
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: RunProcess and threads

Post by mce » Fri Apr 26, 2019 1:39 am

Thanks for the notes about RunProcess. Here are some questions about RunProcess. Any help will be appreciated. Thanks in advance.

When you consider Synchronized() function or when you consider batchupdate functions, are the threads treated as completely independent process executions similar to the way it happens when executed via RunTI or are they still dependent on the main process that triggered them similar to the way it happens when executed via ExecuteProcess function?

I mean, are the locked objects shared/inherited from the main process or are they completely treated as if they are triggered via RunTI as independent sessions?

Can we use RunProcess instead of RunTI in all cases, if we do not need to use credentials of a different user in executing the processes?

User avatar
ykud
Posts: 53
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: RunProcess and threads

Post by ykud » Fri Apr 26, 2019 4:11 am

Can I also jump on with some questions on RunProcess (don't have 2.0.6 to test yet): what's with transaction isolation in RunProcess?
A simple test is to start with an empty cube with 1 cell.
Process A:
1) CellPutN 1 into that cell
2) RunProcess (B)
3) CellGetN that cell in a loop for a minute (does it see the changes done by B or A works in isolation and only sees it's own changes)

Process B:
1) CellGetN cell (does it see 0 or 1 -- do we inherit the parent TI transaction context)?
2) CellPutN 2 into that cell

And once they both are done, what do we have in the cube? 1 or 2? Run TIs would end up with 1, hence I'm a big fan of using files for thread control. If we get 2 with RunProcess, chances are we can move to a cube with 'Thread status' that could be polled from A to determine whether B has finished.

Thanks,
Y

Bakkone
Posts: 94
Joined: Mon Oct 27, 2014 10:50 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2013

Re: RunProcess and threads

Post by Bakkone » Fri Apr 26, 2019 2:35 pm

Wait?

Is runprocess already in 2.0.6? I thought I had to wait until 2.0.7 which is getting constant delays.

Or is 2.0.7 already on the cloud plattform?

Wim Gielis
MVP
Posts: 2087
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.7
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: RunProcess and threads

Post by Wim Gielis » Fri Apr 26, 2019 2:42 pm

I can use it in 2.0.6. on my laptop. It's 11.4 (tm1s.exe = 11.4.0.21)
01.png
01.png (19.43 KiB) Viewed 1804 times


2.0.7 should be out either today, either early next week - so I have been told by someone from IBM.

Wim
Last edited by Wim Gielis on Fri Apr 26, 2019 3:25 pm, edited 1 time in total.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 service with AutoHotKey

Bakkone
Posts: 94
Joined: Mon Oct 27, 2014 10:50 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2013

Re: RunProcess and threads

Post by Bakkone » Fri Apr 26, 2019 3:17 pm

Thx for the info. Turns out I wasnt on 06 at the server I was thinking about. Guess a few more months went by faster than I thought.

lotsaram
MVP
Posts: 3251
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: RunProcess and threads

Post by lotsaram » Fri Apr 26, 2019 3:33 pm

Bakkone wrote:
Fri Apr 26, 2019 2:35 pm
Wait?

Is runprocess already in 2.0.6? I thought I had to wait until 2.0.7 which is getting constant delays.

Or is 2.0.7 already on the cloud plattform?
It's in 2.0.6 (11.4) it just isn't either announced, documented, nor officially supported until 2.0.7.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
mce
Community Contributor
Posts: 305
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: RunProcess and threads

Post by mce » Mon Apr 29, 2019 3:04 pm

I know RunProcess is not supported yet, but seems to work in 2.0.5.
However I found a case where it did not work.
Those processes that connects to Oracle DB as data source, could not connect to source DB with the following error, although they work all Ok when executed using ExecuteProcess or using RunTI.

Code: Select all

TM1.SQLAPI   S1000[Oracle][ODBC][Ora]ORA-12541: TNS:no listener
I hope this issue has been resolved in a next release or at least when it is officially supported.

Regards,

Wim Gielis
MVP
Posts: 2087
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.7
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: RunProcess and threads

Post by Wim Gielis » Tue Apr 30, 2019 7:56 pm

Hello

Would it be possible to check it in version 2.0.7 please ?
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 service with AutoHotKey

User avatar
qml
MVP
Posts: 1073
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.6 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: RunProcess and threads

Post by qml » Wed May 01, 2019 2:12 pm

mce wrote:
Fri Apr 26, 2019 1:39 am
When you consider Synchronized() function or when you consider batchupdate functions, are the threads treated as completely independent process executions similar to the way it happens when executed via RunTI or are they still dependent on the main process that triggered them similar to the way it happens when executed via ExecuteProcess function?

I mean, are the locked objects shared/inherited from the main process or are they completely treated as if they are triggered via RunTI as independent sessions?
Each thread created by RunProcess is a separate, independent transaction and has its own locks.
mce wrote: Can we use RunProcess instead of RunTI in all cases, if we do not need to use credentials of a different user in executing the processes?
I cannot think of a scenario where you couldn't use it. Just remember RunProcess is fully asynchronous i.e. there is no equivalent of the Wait flag of ExecuteCommand.
ykud wrote:
Fri Apr 26, 2019 4:11 am
Can I also jump on with some questions on RunProcess (don't have 2.0.6 to test yet): what's with transaction isolation in RunProcess?
A simple test is to start with an empty cube with 1 cell.
Process A:
1) CellPutN 1 into that cell
2) RunProcess (B)
3) CellGetN that cell in a loop for a minute (does it see the changes done by B or A works in isolation and only sees it's own changes)

Process B:
1) CellGetN cell (does it see 0 or 1 -- do we inherit the parent TI transaction context)?
2) CellPutN 2 into that cell

And once they both are done, what do we have in the cube? 1 or 2? Run TIs would end up with 1, hence I'm a big fan of using files for thread control. If we get 2 with RunProcess, chances are we can move to a cube with 'Thread status' that could be polled from A to determine whether B has finished.
As I said, process A and B would have their own transaction sets, so would not see the changes made by the other process. The process that finishes last commits its changes last.
Kamil Arendt

lotsaram
MVP
Posts: 3251
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: RunProcess and threads

Post by lotsaram » Fri May 03, 2019 7:32 am

I agree with what qml said, it's not really about the parallelization of processes ruining on different threads but much more about the commit model used in TI where only the TI processes belonging to the same transaction have access to changes they have made prior to the commit and merge with the base model.

To get back to Yuri's point about controlling or orchestrating batch processes. flags in cubes are just inherently problematic. The surest way to manage batches and process dependencies (e.g. process B is not allowed to start because process A is still running) has always been via file flags or else a control table in RDBMS. Maybe RunProcess could change this ... I could imagine a solution where all calls to update a process status cube are encapsulated in a RunProcess (which should do nothing more than CellPut therefore run & commit should be more or less instant). If the commit to the status cube is outside the transaction of the process then maybe keeping track of status within TM1 is possible. I would want to test this pretty thoroughly tho before abandoning tried and true methods with file flags.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

EvgenyT
Community Contributor
Posts: 275
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: Cognos TM 1
Version: 10.2.2
Excel Version: 2013
Location: Sydney, Australia

Re: RunProcess and threads

Post by EvgenyT » Mon May 06, 2019 12:17 am

How about sequence orchestration by using Synchronized() ?

User avatar
ykud
Posts: 53
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: RunProcess and threads

Post by ykud » Mon May 06, 2019 1:49 am

qml wrote:
Wed May 01, 2019 2:12 pm
As I said, process A and B would have their own transaction sets, so would not see the changes made by the other process. The process that finishes last commits its changes last.
Thank you very much for confirming this.
lotsaram wrote:
Fri May 03, 2019 7:32 am
To get back to Yuri's point about controlling or orchestrating batch processes. flags in cubes are just inherently problematic. The surest way to manage batches and process dependencies (e.g. process B is not allowed to start because process A is still running) has always been via file flags or else a control table in RDBMS. Maybe RunProcess could change this
I agree wholeheartedly, but I see the 'cube based' solutions on the client sites every now and then and they sometimes 'almost' work and it takes me a while to convince people to go to good old files way.
If we could get the job guid from RunProcess (we already) and there was an asynchronous CheckJobStatus(JobGuid) TI function, we could find out whether the child process completed successfully or not and it would eliminate the need to for files.

Overall it's just inconvenient to implement parallelisation primitives with files, would be easier to have a mutex / semaphore / thread status checking logic built in TM1 (synchronised is doing a sorta/kinda mutex already, I know). Too many ways to implement this and make mistakes otherwise and it's a notoriously hard to test area, some race conditions can only happen in production :)

Drg
Posts: 117
Joined: Fri Aug 12, 2016 10:02 am
OLAP Product: tm1
Version: 10.2.0 - 10.3.0
Excel Version: 2010

Re: RunProcess and threads

Post by Drg » Tue May 07, 2019 10:20 am

ykud wrote:
Mon May 06, 2019 1:49 am
qml wrote:
Wed May 01, 2019 2:12 pm
As I said, process A and B would have their own transaction sets, so would not see the changes made by the other process. The process that finishes last commits its changes last.
Thank you very much for confirming this.
lotsaram wrote:
Fri May 03, 2019 7:32 am
To get back to Yuri's point about controlling or orchestrating batch processes. flags in cubes are just inherently problematic. The surest way to manage batches and process dependencies (e.g. process B is not allowed to start because process A is still running) has always been via file flags or else a control table in RDBMS. Maybe RunProcess could change this
I agree wholeheartedly, but I see the 'cube based' solutions on the client sites every now and then and they sometimes 'almost' work and it takes me a while to convince people to go to good old files way.
If we could get the job guid from RunProcess (we already) and there was an asynchronous CheckJobStatus(JobGuid) TI function, we could find out whether the child process completed successfully or not and it would eliminate the need to for files.

Overall it's just inconvenient to implement parallelisation primitives with files, would be easier to have a mutex / semaphore / thread status checking logic built in TM1 (synchronised is doing a sorta/kinda mutex already, I know). Too many ways to implement this and make mistakes otherwise and it's a notoriously hard to test area, some race conditions can only happen in production :)

Yes, I think in the new versions we will see the function of checking job status.
But I do not agree that the processes of orchestration are not solved.
In our production environment there are several solutions for sequential queues, one based on files, the other based on a cube of queues, both work (I note that a solution based on a cube is rather complicated, but it works!)

User avatar
ykud
Posts: 53
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: RunProcess and threads

Post by ykud » Wed May 08, 2019 6:09 am

Drg wrote:
Tue May 07, 2019 10:20 am
But I do not agree that the processes of orchestration are not solved.
In our production environment there are several solutions for sequential queues, one based on files, the other based on a cube of queues, both work (I note that a solution based on a cube is rather complicated, but it works!)
Why having both? To make support more intellectually stimulating? :-)

How do you do check for task completion via cubes, do a runTI that reads the cube contents? What happens if this 'check' process itself fails, would you be able to differentiate between it running successfully or failing silently (you ran an ExecuteCommand and it returned all good)?

Drg
Posts: 117
Joined: Fri Aug 12, 2016 10:02 am
OLAP Product: tm1
Version: 10.2.0 - 10.3.0
Excel Version: 2010

Re: RunProcess and threads

Post by Drg » Wed May 08, 2019 6:20 am

ykud wrote:
Wed May 08, 2019 6:09 am
Why having both? To make support more intellectually stimulating? :-)

How do you do check for task completion via cubes, do a runTI that reads the cube contents? What happens if this 'check' process itself fails, would you be able to differentiate between it running successfully or failing silently (you ran an ExecuteCommand and it returned all good)?
Yuri historically it turned out that there are two different logics used for different processes; in the second, some java is used.

We chose a different concept, we use runti to start the wrapper process, inside which we run the executeprocess (here we had to use a little magic) and read the execution status of the process and describe the status in a cube. there is also the only process that polls the status cube and determines how many threads are free and so on based on it.

Post Reply