NoWait command for TI processes

Suggest and discuss enhancements for TM1
Post Reply
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

NoWait command for TI processes

Post by lotsaram »

Even with all the enhancements made to the locking model over the last few versions no one would say it is easy to run jobs in parallel TM1 to better use available cores and generate faster application response time. Currently we are limited to 3 options
1/ schedule chores to run concurrently
2/ use an external 3rd party scheduling tool to call tm1runti to run processes in parallel via additional logins
3/ Use ExecuteCommand combined with batch files and tm1runti to run processes in parallel via additional logins

I suspect option 3 is the most common.

BUT adding a NoWait command option to TI script would be much, much simpler. Imagine if a no wait command gave the ability to call TI processes in parallel using ExecuteProcess and extra worker threads were added without needing a full additional external login to generate the extra thread. How simple would life be?

To run a load process in parallel all you would need to do is this

NoWait;
ExecuteProcess( sProc, 'pCube', 'SalesOrders', 'pFile', cLoadDir | 'File1.txt' );
ExecuteProcess( sProc, 'pCube', 'SalesOrders', 'pFile', cLoadDir | 'File2.txt' );
ExecuteProcess( sProc, 'pCube', 'SalesOrders', 'pFile', cLoadDir | 'File3.txt' );
ExecuteProcess( sProc, 'pCube', 'SalesOrders', 'pFile', cLoadDir | 'File4.txt' );

Does anyone else share my enthusiasm for this idea?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
Paul Segal
Community Contributor
Posts: 306
Joined: Mon May 12, 2008 8:11 am
OLAP Product: TM1
Version: TM1 11 and up
Excel Version: Too many to count

Re: NoWait command for TI processes

Post by Paul Segal »

Yes, that would be very useful.
Paul
Duncan P
MVP
Posts: 600
Joined: Wed Aug 17, 2011 1:19 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2
Excel Version: 2003 2007
Location: York, UK

Re: NoWait command for TI processes

Post by Duncan P »

I really like this.

However a couple questions need to be asked about expected behaviour in this scenario.
  • Which of the uncommitted changes in the current process can the called "no wait" process see? Are we happy with "none"?
  • What happens about global variables? Again the obvious answer is "no visibility".
  • Do we anticipate any requirement for synchronisation for clean-up? If so how might that work? We would need to be very careful of deadlocks in this case.
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: NoWait command for TI processes

Post by TableManagerOne »

Rather than diving into general asynchronous programming, would a split/join in the scheduler (the chore) suffice? A multi-commit chore could be extended to branch (\) and merge (/) arbitrarily. At any given branch point, the called processes have the potential to run in parallel, as new threads running in the context of the parent process. However, if a lock conflict/deadlock is encountered, the sub-process rolls back (but only to the branch point) and waits for its siblings to complete. The chore-branch waits for all children to complete one way or another before continuing on to call the next process/set of branching processes, or merge itself back into whichever stream it branched from.

P1, P2, -------------------, P7
____\ P3a /
____\ P3b------------ /
______\ P4a, P5a /
______\ P4b, P5a /

Each branch and merge includes a commit, making changes of the branching stream visible to spawned streams, and changes of the spawned stream visible to the branching steam (after all spawned streams commit.)

Not as powerful of course, but perhaps easier to wrap one's head around (mine at least.)
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: NoWait command for TI processes

Post by lotsaram »

  • Which of the uncommitted changes in the current process can the called "no wait" process see? Are we happy with "none"?
Given the current way of achieving parallelization is with independent logins and therefore each processes uncommitted changes aren't visible to any other thread I think this would be just fine.
  • What happens about global variables? Again the obvious answer is "no visibility".
No visibility also fine. From a practical perspective any global variables that were needed could be passed to processes as parameters anyway.
  • Do we anticipate any requirement for synchronisation for clean-up? If so how might that work? We would need to be very careful of deadlocks in this case.
To me the mechanism used be parallel interaction seems just fine, if there do happen to be any overlapping data writes then it is last commit wins. i.e. programmer beware, but in the scenarios where people would want to use this by rights some care should be taken to make sure that data sources and targets don't overlap
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
rmackenzie
MVP
Posts: 733
Joined: Wed May 14, 2008 11:06 pm

Re: NoWait command for TI processes

Post by rmackenzie »

lotsaram wrote:
  • Which of the uncommitted changes in the current process can the called "no wait" process see? Are we happy with "none"?
Given the current way of achieving parallelization is with independent logins and therefore each processes uncommitted changes aren't visible to any other thread I think this would be just fine.
  • What happens about global variables? Again the obvious answer is "no visibility".
No visibility also fine. From a practical perspective any global variables that were needed could be passed to processes as parameters anyway.
  • Do we anticipate any requirement for synchronisation for clean-up? If so how might that work? We would need to be very careful of deadlocks in this case.
To me the mechanism used be parallel interaction seems just fine, if there do happen to be any overlapping data writes then it is last commit wins. i.e. programmer beware, but in the scenarios where people would want to use this by rights some care should be taken to make sure that data sources and targets don't overlap
Sounds good - just a simple command to automate the effects of option 2 or 3 by the sounds of it. This would be very useful.
Robin Mackenzie
jstrygner
MVP
Posts: 195
Joined: Wed Jul 22, 2009 10:35 pm
OLAP Product: TM1
Version: 9.5.2 FP3
Excel Version: 2010

Re: NoWait command for TI processes

Post by jstrygner »

Agree this would be very handy and I would love to have such an option.

I think the major difficulty in implementing it would be with running more than one action under the same thread ID.
The TM1RunTI and Chores will have own threads.
Probably running more than one TI within same login session would require serious change in the TM1 engine.
Question would be if such NoWait; could kind of clone login session and run parallel TIs under those clone threads (same user login, different thread ID). After that it should be good to handle closing those clones as well.
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: NoWait command for TI processes

Post by lotsaram »

jstrygner wrote:Agree this would be very handy and I would love to have such an option.

I think the major difficulty in implementing it would be with running more than one action under the same thread ID.
I figure if they can spawn new threads for running view calculation in parallel then it must be possible to do the same from a TI. I just think this would be a great feature to have, so much easier than external calls to TM1RunTI.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
Post Reply