BatchUpdateStart and BatchUpdateFinishWait in master process

Post Reply
buncikp
Posts: 16
Joined: Tue Feb 12, 2013 7:31 am
OLAP Product: TM1+Cognos Express
Version: 9.5.2+10.1+10.2
Excel Version: 2007+2010
Location: Prague

BatchUpdateStart and BatchUpdateFinishWait in master process

Post by buncikp » Thu Feb 19, 2015 12:02 pm

Hello,
Is it ok to use BatchUpdateStart and BatchUpdateFinishWait in a master process that calls several child processes? Or is it necessary to put them in each of child process? When using batch updates in master process I get inconistent results in target cubes for each child process.
Thanks for help

mvaspal
Community Contributor
Posts: 319
Joined: Wed Nov 03, 2010 9:16 pm
OLAP Product: tm1
Version: 9 5 1 - 10 2 2 - 10.3 PA
Excel Version: From 2007 to 2013
Location: Earth

Re: BatchUpdateStart and BatchUpdateFinishWait in master pro

Post by mvaspal » Thu Feb 19, 2015 12:09 pm

Hi
It is ok to use as long as in the second child TI you do not reference results from the first child TI

So if the executed TIs are independent from each other, this should work. If TI 1 writes a value to a cell that TI2 reads then, I assume it won't work.

buncikp
Posts: 16
Joined: Tue Feb 12, 2013 7:31 am
OLAP Product: TM1+Cognos Express
Version: 9.5.2+10.1+10.2
Excel Version: 2007+2010
Location: Prague

Re: BatchUpdateStart and BatchUpdateFinishWait in master pro

Post by buncikp » Thu Feb 19, 2015 12:43 pm

mvaspal wrote:Hi
It is ok to use as long as in the second child TI you do not reference results from the first child TI

So if the executed TIs are independent from each other, this should work. If TI 1 writes a value to a cell that TI2 reads then, I assume it won't work.
Thank you mvaspal!
The above seems logical, however is the isolation of data provided per each child process? Just hypothetically:
1) We have master process that calls BatchupdateStart, then spawns child process A and then process B. Finally it will fire BatchUpdateFinishWait.
2) Child process A modifies cube C and writes data to slice S1.
3) Child process B modifies cube C and writes data to slice S2.
4) Slices S1 and S2 are independent.
Do I understand correctly that BatchUpdate will create sort of working space that will be later commited by BatchUpdateFinish? I'm still lost how these spaces are distributed per child processes. If only one working space is created in master process I would assume both child processes are contributing to the same space. I understand that reading from such space would provide inconsistent results because missing commit action.
Thank you

EDIT: It seems that the child processes can't see any changes provided by other ones as they will see just the state before BatchUpdateStart. Will try it to confirm.

mvaspal
Community Contributor
Posts: 319
Joined: Wed Nov 03, 2010 9:16 pm
OLAP Product: tm1
Version: 9 5 1 - 10 2 2 - 10.3 PA
Excel Version: From 2007 to 2013
Location: Earth

Re: BatchUpdateStart and BatchUpdateFinishWait in master pro

Post by mvaspal » Fri Feb 20, 2015 8:53 am

Hi

I would think that if the data source is odbc or txt or another cube, and Process B does not use any data from the target cube's S1 slice then it should work, but this is just an assumption, I have only ever used batch update in a master process when on its Data tab I went through the }Cubes dimension and called the child processes one by one for each cube, and these child processes only copied/cleared/froze versions within the same cube (btw I called the same child process for each cube, just with diffferent parameters), so my case is a bit different from yours
Do I understand correctly that BatchUpdate will create sort of working space that will be later commited by BatchUpdateFinish?
I would think so, yes

buncikp
Posts: 16
Joined: Tue Feb 12, 2013 7:31 am
OLAP Product: TM1+Cognos Express
Version: 9.5.2+10.1+10.2
Excel Version: 2007+2010
Location: Prague

Re: BatchUpdateStart and BatchUpdateFinishWait in master pro

Post by buncikp » Fri Feb 20, 2015 10:38 am

mvaspal wrote:Hi

I would think that if the data source is odbc or txt or another cube, and Process B does not use any data from the target cube's S1 slice then it should work, but this is just an assumption, I have only ever used batch update in a master process when on its Data tab I went through the }Cubes dimension and called the child processes one by one for each cube, and these child processes only copied/cleared/froze versions within the same cube (btw I called the same child process for each cube, just with diffferent parameters), so my case is a bit different from yours
Do I understand correctly that BatchUpdate will create sort of working space that will be later commited by BatchUpdateFinish?
I would think so, yes
Hello mvaspal,
I gave it a short try and results are interesting:
I have created simple cube X with dimension element A in one dimension and element Value in the second. I filled manually value as 100.
I have created 3 simple processes:

Code: Select all

### Master Process
asciioutput('test3.txt', 'Master Check 1', numbertostring(cellgetn('X', 'A','value')));
batchupdatestart;
executeprocess('Child 1');
asciioutput('test3.txt', 'Master Check 2', numbertostring(cellgetn('X', 'A','value')));
executeprocess('Child 2');
asciioutput('test3.txt', 'Master Check 3', numbertostring(cellgetn('X', 'A','value')));
batchupdatefinishwait(0);
asciioutput('test3.txt', 'Master Check 4', numbertostring(cellgetn('X', 'A','value')));

Code: Select all

### Child process 1
asciioutput('test1.txt', 'Child 1, Check 1', numbertostring(cellgetn('X', 'A','value')));
cellputn(200, 'X', 'A', 'value');
asciioutput('test1.txt', 'Child 1, Check 2', numbertostring(cellgetn('X', 'A','value')));

Code: Select all

### Child process 2
asciioutput('test2.txt', 'Child 2, Check 1', numbertostring(cellgetn('X', 'A','value')));
cellputn(300, 'X', 'A', 'value');
asciioutput('test2.txt', 'Child 2, Check 2', numbertostring(cellgetn('X', 'A','value')));
Then I ran the master process and here are the results:

Code: Select all

test3.txt
"Master Check 1","100"
"Master Check 2","200"
"Master Check 3","300"
"Master Check 4","300"

Code: Select all

test1.txt
"Child 1, Check 1","100"
"Child 1, Check 2","200"

Code: Select all

test2.txt
"Child 2, Check 1","200"
"Child 2, Check 2","300"
It looks pretty consistent -- each process sees the working space and changes of data. I guess there must be something wrong with our code as BatchUpdates seem not to do any consistency issues. Or I might be wrong with my assumption (as failure is always an option).
Best
Petr

EDIT2:
I have updated the processes to be able to run in parallel, I have added new element B for parallel test and let two master processes run in parallel, each one writing independently to element A, resp. B. Result is again values are consistent across each parallel run.

HyunaHyuna
Posts: 9
Joined: Thu Jun 16, 2016 7:57 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2016

Re: BatchUpdateStart and BatchUpdateFinishWait in master process

Post by HyunaHyuna » Tue Mar 20, 2018 3:32 pm

Hi, I hope you don't mind me bringing this back.

I am trying to have a master process execute other processes in parallel, and some of those need BatchUpdate. Here is what I am doing now:

Master Process:

Code: Select all

ExecuteCommand('"(...)Hustle.exe" "\\(...)\FileThatParallelRunsProcessA.txt" 20', 1);
BatchUpdateStart;
   ExecuteCommand('"(...)Hustle.exe" "\\(...)\FileThatParallelRunsProcessB.txt" 20', 1);
BatchUpdateFinishWait(0);
Content of FileThatParallelRunsProcessA.txt (ProcessB is a separate .txt file with the same lines, just with the process name changed) (this is the first line, repeated n times over pCountry):

Code: Select all

 \\(location on server)\tm1runti.exe -process ProcessA -adminhost host -server servername -user username
  -Camnamespace CNS -pwd password pCountry="xxx1" pMonth="yyy" pVersion="zzz" pParallel=1
Both ProcessA and ProcessB create the necessary source and target views, set the sourcedata as the source view, viewzeroout the target view, then do some CellPutNs in the data tab (ProcessB a bunch more than ProcessA), and then delete temporary objects in epilogue. ProcessB in addition has BatchUpdateStart in Prologue and BatchUpdateFinishWait(0) in Epilogue. Both of them load into the same cube, but to different measures, and from different source cubes.

ProcessA has no BatchUpdate in the process, and it runs fine.
ProcessB has an insane amount of CellPutNs, so it needs BatchUpdate, or it runs for days.

Here is what I found: When I have BatchUpdateFinish in Epilogue of ProcessB, this causes the parallel processes to get stuck in an eternal "Wait" cycle. I guess the first process that is ready tries to finish batch updating, but all the others are still updating the cube. Removing the BatchUpdateFinish for parallel loads solved the infinite loop, but here came the problem I am struggling with now.

When I remove BatchUpdateStart from the Prologue of ProcessB, and have it outside in the master process, it doesn't seem to work, I don't think BatchUpdate is turned on for the case. It runs for ages, when it should be done in minutes. When I have BatchUpdateStart in the Prologue tab of the process AND the master process, but only have BatchUpdateFinishWait(0) in the master process, the data isn't saved. All the calculations run (textoutput in the data tab confirms that data is fine) and they run fast, but I get an empty view as a result, because the BatchUpdate didn't seem to finish. (Strange that I get the textoutputs, though)

It seems to me from all this that BatchUpdateStart does not work outside in a master process with how I am doing it, and I wondered whether you could offer some advice on what could I be doing wrong. My other option (I think, I haven't tested it) is to somehow determine which country runs for the longest, and have the BatchUpdateFInish turned on for that one in the Epilogue of ProcessB, but that kind of changes with the data input in the cells, so it's a wonky solution.

lotsaram
MVP
Posts: 3126
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: BatchUpdateStart and BatchUpdateFinishWait in master process

Post by lotsaram » Tue Mar 20, 2018 4:34 pm

You say you need BatchUpdate but do you really?

BatchUpdate more or less went the way of the dinosaurs when Parallel Interaction was introduced. So why do you think you need it?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

HyunaHyuna
Posts: 9
Joined: Thu Jun 16, 2016 7:57 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2016

Re: BatchUpdateStart and BatchUpdateFinishWait in master process

Post by HyunaHyuna » Wed Mar 21, 2018 8:44 am

Thank you for your reply :)

I have edited out the detailed description of my problem, as I believe I have found a solution. Instead of using CellPutNs, I TextOutPut everything into a .csv file, and then import everything from there using a separate process. I just had to be sure to clarify the data source name for parallel runs in the prologue of the second, import process:

Code: Select all

sExportFile = sExportLocation | sFileName;

DataSourceType = 'CHARACTERDELIMITED';
DatasourceNameForServer = sExportFile;
DatasourceASCIIDelimiter=',';
DatasourceASCIIDecimalSeparator='.';
DatasourceASCIIThousandSeparator=',';
Thank you for suggesting that BatchUpdate was a thing of the dinosaurs, it was what motivated me to look for a completely different solution!

mvaspal
Community Contributor
Posts: 319
Joined: Wed Nov 03, 2010 9:16 pm
OLAP Product: tm1
Version: 9 5 1 - 10 2 2 - 10.3 PA
Excel Version: From 2007 to 2013
Location: Earth

Re: BatchUpdateStart and BatchUpdateFinishWait in master process

Post by mvaspal » Mon Apr 09, 2018 12:22 pm

BatchUpdate more or less went the way of the dinosaurs when Parallel Interaction was introduced.
Correct me if I'm wrong but it could be still useful in a TI that loads data within a cube where the source view has lot of rule calc'd cells...at least as long as the new feature of invalidating only a slice of a cube's cache is under development

lotsaram
MVP
Posts: 3126
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: BatchUpdateStart and BatchUpdateFinishWait in master process

Post by lotsaram » Mon Apr 09, 2018 3:03 pm

mvaspal wrote:
Mon Apr 09, 2018 12:22 pm
Correct me if I'm wrong but it could be still useful in a TI that loads data within a cube where the source view has lot of rule calc'd cells...at least as long as the new feature of invalidating only a slice of a cube's cache is under development
Yes batchupdate could be useful in that scenario as long as you don't want to query any of the new data within the process. The batch updates are kind of like a pre-sandbox sandbox where the new input data is held totally outside the model and not merged at all until the commit. So any CellGetN performed during the process will read data from the status of the cube BEFORE the process. I guess that not having to build a merged model pre-commit is one reason why batchupdate is able to be so much faster. But the lack of ability to query any consolidated cells post update until AFTER the batch update has finished really limits the usefulness and potential use cases IMO.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
tomok
MVP
Posts: 2483
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: BatchUpdateStart and BatchUpdateFinishWait in master process

Post by tomok » Mon Apr 09, 2018 4:23 pm

lotsaram wrote:
Mon Apr 09, 2018 3:03 pm
mvaspal wrote:
Mon Apr 09, 2018 12:22 pm
Correct me if I'm wrong but it could be still useful in a TI that loads data within a cube where the source view has lot of rule calc'd cells...at least as long as the new feature of invalidating only a slice of a cube's cache is under development
Yes batchupdate could be useful in that scenario as long as you don't want to query any of the new data within the process. The batch updates are kind of like a pre-sandbox sandbox where the new input data is held totally outside the model and not merged at all until the commit. So any CellGetN performed during the process will read data from the status of the cube BEFORE the process. I guess that not having to build a merged model pre-commit is one reason why batchupdate is able to be so much faster. But the lack of ability to query any consolidated cells post update until AFTER the batch update has finished really limits the usefulness and potential use cases IMO.
Interesting. Does that mean the the function CellIncrementN would not work in a BatchUpdate process?
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

lotsaram
MVP
Posts: 3126
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: BatchUpdateStart and BatchUpdateFinishWait in master process

Post by lotsaram » Mon Apr 09, 2018 6:45 pm

tomok wrote:
Mon Apr 09, 2018 4:23 pm
Interesting. Does that mean the the function CellIncrementN would not work in a BatchUpdate process?
Been a long while since I used it but pretty sure the answer is yes, exactly this problem.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

mvaspal
Community Contributor
Posts: 319
Joined: Wed Nov 03, 2010 9:16 pm
OLAP Product: tm1
Version: 9 5 1 - 10 2 2 - 10.3 PA
Excel Version: From 2007 to 2013
Location: Earth

Re: BatchUpdateStart and BatchUpdateFinishWait in master process

Post by mvaspal » Tue Apr 10, 2018 7:40 am

http://tm1forum.com/viewtopic.php?f=18& ... ntN#p49881

Last time I tried CellIncrement did not work, but the old method did, which is surprising

I think in some general TIs like archiving a rule-intensive dynamic scenario to a static scenario - batch update is still a great feature

Post Reply