TM1RunTI / Runprocess issues

Post Reply
prameson
Posts: 16
Joined: Wed May 11, 2011 11:36 am
OLAP Product: TM1
Version: 2.09
Excel Version: Office 365 E5
Location: London

TM1RunTI / Runprocess issues

Post by prameson » Wed Feb 05, 2020 12:05 pm

Hi All

We recently upgraded to PA (we are now on 2.08) and to Security Mode 3 (from 2) to simplify user login are having some issues with some processes using TM1RunTI (which worked fine in the previous versions).  We created an admin user account with the name of the Server service account in order to get the RunTis to work.

The processes are to create static versions of a planning model and we use TM1RunTI to copy various years in parallel to speed up the process.

When run as is they now seem to either randomly just run for one year or run for all years but not in parallel.

I have looked to change the processes to use the RunProcess function in the new version.  I can get this to work for the first time run the process, but then in the second run I get errors - "Error: Prolog procedure line (0): Unable to open data source" and the cube name.  The third time I run the process I see "Execution rolling back due to lock exception" errors in the server log but the copy processes seem to re-run and the copy does happen correctly

Here is the logic I have:

Control Process with parameters (Years, Version from/to)
Prolog - uses ExecuteProcess to call a create view process for each year specified  - a source view and a zero out view are created (using ViewConstruct) and the ZeroOut view is zeroed out.

Epilog - uses RunProcess to call a DataLoad process for each year and do the copy using the previously created views (and destroy the views in the Epilog (the source and the zeroout for that year).


Any help or advice would be greatly appreciated - I'm not sure what the best practice is now for parallel processing - should I be looking to use Batch processing / maybe looking to use something like RushTI or is there just some logic issue with my code?

Thanks
Peter

tomok
MVP
Posts: 2758
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: TM1RunTI / Runprocess issues

Post by tomok » Wed Feb 05, 2020 12:24 pm

It sounds like you are dealing with lock contentions. You are creating an all encompassing view, for use in the ViewZeroOut, and then calling sub-processes that load into portions of that same area, while the original process is still active. I cannot guarantee that is the problem but it looks very suspicious to me. If it were me I would do the create view, zero it out, create the source view, assign it as the data source and then do the data load all in the sub-processes. I would also set the temporary parameter of the ViewCreate to 1 so that they are destroyed at the end of each run of the sub-process. Probably a little less efficient than your current way but you shouldn't run into any locking issues.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

prameson
Posts: 16
Joined: Wed May 11, 2011 11:36 am
OLAP Product: TM1
Version: 2.09
Excel Version: Office 365 E5
Location: London

Re: TM1RunTI / Runprocess issues

Post by prameson » Wed Feb 05, 2020 1:05 pm

Thanks Tom

I'm creating separate zero out views for each year and then zeroing out in the prolog of the control process. The epilog then calls the DataLoad process.

If I create the views as temporary then surely they won't exist when I call the load processes using RunProcess?

tomok
MVP
Posts: 2758
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: TM1RunTI / Runprocess issues

Post by tomok » Wed Feb 05, 2020 1:22 pm

prameson wrote:
Wed Feb 05, 2020 1:05 pm
I'm creating separate zero out views for each year and then zeroing out in the prolog of the control process. The epilog then calls the DataLoad process.
You are creating a view to zero out each year which puts a write lock on that view. Now, before that process has a chance to finish and thus release the lock, you are calling a sub-process to load to the same area that the original process has a write lock for. Do you understand?
prameson wrote:
Wed Feb 05, 2020 1:05 pm
If I create the views as temporary then surely they won't exist when I call the load processes using RunProcess?
What I am saying is to do the viewzeroout AND the load all in the subprocess. I am not guaranteeing that this is the issue, especially if the sub-process is called by RunProcess, since that is different than ExecuteProcess, but I would not do what your are doing on principle because of the potential for write lock issues.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

prameson
Posts: 16
Joined: Wed May 11, 2011 11:36 am
OLAP Product: TM1
Version: 2.09
Excel Version: Office 365 E5
Location: London

Re: TM1RunTI / Runprocess issues

Post by prameson » Wed Feb 05, 2020 2:44 pm

You are creating a view to zero out each year which puts a write lock on that view. Now, before that process has a chance to finish and thus release the lock, you are calling a sub-process to load to the same area that the original process has a write lock for. Do you understand?
What I am saying is to do the viewzeroout AND the load all in the subprocess. I am not guaranteeing that this is the issue, especially if the sub-process is called by RunProcess, since that is different than ExecuteProcess, but I would not do what your are doing on principle because of the potential for write lock issues.
Yes, I understand there is a lock issue but the zeroout happens in the viewcreate sub process before I call the load subprocess. I don't see how putting the zero out and load into the sub process could possibly help (I've tried - it doesn't).

It seems like I need a way to be sure the lock has been released before the load processes start but not sure how to do this.

User avatar
Ajay
Regular Participant
Posts: 159
Joined: Wed May 14, 2008 8:27 am
OLAP Product: TM1
Version: 10.2.0, PA 2.0.9
Excel Version: 2016
Location: London

Re: TM1RunTI / Runprocess issues

Post by Ajay » Wed Feb 05, 2020 6:24 pm

So, I use the following and it hasn't caused any issues for me....not that I have noticed any :)

Have a master process, that you can from TM1RunTi, or anything else, and then "daisy chain" from it, so below the process called "Master_Process" calls each of the others, which complete before advancing to the next one

Master_Process
-Process_A_Create_View
-Process_B_Zero_View
-Process_C_Load_Data


Note that "Master_Process" contains the following in it's Epilog:

ExecuteProcess('Process_A_Create_View');
ExecuteProcess('Process_B_Zero_View');
ExecuteProcess('Process_C_Load_Data');

Hope this helps.....as I said above, I do this and I've not noticed any problems.

Ajay

Wim Gielis
MVP
Posts: 2552
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1, Jedox
Version: PAL 2.0.8
Excel Version: Microsoft 365
Location: Brussels, Belgium
Contact:

Re: TM1RunTI / Runprocess issues

Post by Wim Gielis » Wed Feb 05, 2020 7:57 pm

Ajay wrote:
Wed Feb 05, 2020 6:24 pm
So, I use the following and it hasn't caused any issues for me....not that I have noticed any :)

Have a master process, that you can from TM1RunTi, or anything else, and then "daisy chain" from it, so below the process called "Master_Process" calls each of the others, which complete before advancing to the next one

Master_Process
-Process_A_Create_View
-Process_B_Zero_View
-Process_C_Load_Data


Note that "Master_Process" contains the following in it's Epilog:

ExecuteProcess('Process_A_Create_View');
ExecuteProcess('Process_B_Zero_View');
ExecuteProcess('Process_C_Load_Data');

Hope this helps.....as I said above, I do this and I've not noticed any problems.

Ajay
And where do you execute processes in parallel and/or use RunProcess instead of ExecuteProcess ?
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 112 TM1 articles and a lot of custom code
Newest blog article: Updating process line numbers with AutoHotKey

prameson
Posts: 16
Joined: Wed May 11, 2011 11:36 am
OLAP Product: TM1
Version: 2.09
Excel Version: Office 365 E5
Location: London

Re: TM1RunTI / Runprocess issues

Post by prameson » Thu Feb 06, 2020 10:28 am

Thanks Everyone for your help.

I have managed to solve the problem using the following steps:

1. Changed the ExecuteProcess comands that I used for the View create sub process calls to RunProcess comands
2. Removed the ZeroOut view creation from the View create sub process and put it in the prolog of the Data Load sub process

Hope this helps anyone else having similar issues.

Peter

Post Reply