Using ExecuteProcess to call other processes within a Master Process - What are the caveats?

[7Zark7]

Hi Guys,

I have been reluctant to use ExecuteProcess to process multiple processes due to me being partly paranoid and use chores instead. However due to the LOCK that
a chore places on the system while it's in process I am heavily leaning towards utilising a Master Process that will make multiple calls via the ExecuteProcess function. I use

What I wanted to know is, what will stop a Mega Process embedded with ExecuteProcess calls to fail? I am concerned that in the event a process fails along the way then the rest of the process will then not fire. The chore that we fires off every morning runs 120 processes ( we obviously donot want to hiccup along the way ), hence the reason for my caution. I use ExecuteProcess extensively however not for 100 odd processes chained up.Per the documentation the only time an ExecuteProcess call with fail is:

*****************************************************************************************************************************************************************************************************************
The parameter names passed in the ExecuteProcess function are matched at runtime against the parameter names specified in the process to be executed. If the passed names cannot be found in the parameter list of the process to be executed, a serious error results, causing the immediate termination of the process from which ExecuteProcess is called
*****************************************************************************************************************************************************************************************************************

My question is "Are there any OTHER scenarios that would prevent the rest of the processes running within a super process to run besides from what has been stated in the documentation above?"

Is it also possible to skip process where return_value = ProcessExitSeriousError(). I guess this is not possible based on the documentation.

Cheers

[lotsaram]

In terms of locking there's no difference. A chore and a process chain from ExecuteProcess calls within a master process are the same in that the TM1 server sees both these as a single transaction and a lock held is still for the duration of the transaction. The reason for using a master process versus a chore is that it is much easier to manage and maintain within a process as parameter values can be programatically manipulated and you can also determin in the code based on conditions whether to run some processes or not. If you have a lot of processes to include as part of a single job then it is clearly the better interface as the chore editor is really from the dark ages.

In the scenario you describe of a chore with 120 processes, ... well that would be a nightmare. In your position I would definitely implement a master process and then just have a chore calling only that process, much easier.

The other benefit of ExecuteProcess is that you don't need to explicitly pass all parameters with values. Any parameters not named just pick up the default value. This is a big advantage over chores where each parameter must be passed a value.

Your question

Is it also possible to skip process where return_value = ProcessExitSeriousError().

I think the answer to that should be pretty obvious. You only know the numeric return value of a process AFTER it has run. Based on a value > 0 you could stop the main execution or write a value to a control cube that will stop the process running next time but clearly it's too late to skip a process after it has already run.

[TrevorGoss]

I agree with all of what lotsaram has said, have you also thought about using the TM1RunTI.exe program? You can call this from a Master ti, which can pass arguments to an ExecuteCommand function call, which in turn will call the process you want, and as long as you avoid locking contention, you can run multiple ti's at the same time.

Also, 120 processes does indeed sound like a nightmare, what causes you to have so many?

[7Zark7]

Thanks Guys. Sucks about the lock duration. But agree that under my current situation this is a much easier way to manage data being loaded into the system. Yep agree on RUNTI, we utilize RUNTI extensively BUT you have a be careful of LOCK contention and also need to by mindful of dependencies where one process may rely on another. At least this way we can reduce over-all run time and hence lock duration by taking advantage of the non utilized cores by spawning multiple RUNIT scripts.

In regards to having 120 processes. There are alot of independent cubes and multiple meta data updates built seperately to avoid system locking.

So just to confirm, unless someone botches the syntax on a parameter name then the master master should execute each process from start to end?

Cheers

Zark

[BrianL]

If you're trying to shorten the amount of time locks are held, have you looked into using multiple commit mode for your chore?

With the default single commit mode, all changes are kept private to the chore and all locks are held until the entire chore finishes executing. In multiple commit mode, the changes for each process are kept private to the process and all locks are held until the individual process finishes executing.

Multiple commit mode is a way to release locks sooner, but you should exercise care, since it also publishes changes sooner. Often a hybrid approach of using multiple commit mode with some master TI processes can achieve the desired behavior.

[Gabor]

Depending on what sources your TI's are using, it's sometimes better to first build a temporary structure in TM1 to keep the "external" uploads separate from your main data and then finally you can go and update your dimensions and cubes.
This 2nd step is usally much faster because you have already everything in TM1, which will minimize system locks by nature.
Another optimization is to combine multiple updates of a large dimension to just have a single event and so on.