Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post Reply
jjramirez
Posts: 31
Joined: Fri Jan 13, 2017 12:33 pm
OLAP Product: IBM Planning Analytics/TM1
Version: 2.0.9.15
Excel Version: Excel 2019

Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by jjramirez »

Hi!

I am trying parallel execution for the first time and i'm living with a problem that i don't understand:

i have a master process that call some child process in batch using Tm1RunTi. The problem is, sometimes, aleatory, the master process just stop execution with any action completed.

The master process is calling 94 TI Process in Batch. Some times the master get finish successfully but othe times begin execution (i can see many childs process running in the TM1 Operations Console ) and then just stop with any results and no error messages in log file.

When i look in log file for more details i just find that:
  • Only 25 child process was successfully completed (i was expecting 94)
  • 50 records of TM1.Login (25 attempt and 25 success)
  • and eventually this messages in tm1lockexeptiondebug.log:
17580 [44] DEBUG 2017-09-15 17:41:26.897 TM1.Lock.Exception Contention encountered attempting to acquire lock (0x00000011811C5410) on object [Client "CAMID("ldap_sprc:u:d3fd2c09f5e2e74da348fa0160ae90ed")", addr=0x00000011808DF010, index=R-2255] in READONLY mode at ..\tm1_r7s\TM1ClientImpl.cpp:1296 during function 'SystemServerConnectWithCAMNamespace'.
Entering wait state 'IXC'.
Blocked by the following 1 thread:
Thread 7908 holds the lock in IX mode
sometime it fails with no lock exeption messages, so i am not sure this is the problem.

i was reviewing the code in the master process and in the child process but this is ok and i pretty sure because sometime it runs complete and successfully as i said before.

i have other master process that works fine but just calling 12 or 24 process a time.
As each TI process that i call in batch is a new Server Connection, i was wondering if the number of server connections is limited maybe by number of cors in the server or maybe by license...

i hear some people send more than 2000 process at the same time with Tm1RunTi... and i wasn't expecting to deal with any problem with just 94.

I have TM1 10.2.2 with FP7 installed, the server has 2 processors and MTQ is set as "MTQ = 2" in configuration file.

in anycase i can't explain why sometimes works pretty fine and sometimes just crash. I am making the testing always with the same Data and if i execute the childs one by one there's no problem... that's why i think is not a coding or Data problem. The only thing out of normal state is the lock message in log file that i have shown above.

I hope you can help me to understand more the parallel execution and to clarify if the problem is in the loggin step.

Thank you very much in advanced.
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by paulsimon »

Hi

Are all the processes running under the same user id? Have you checked that the maximum connections for that user is either 0 no maximum or is set high enough to allow this?

If the data source is the same in all cases then you may not get much benefit from this degree of parallelism since the duration will determine more on the speed of reading than the speed of writing.

Are all the processes writing to the same cube? Even then with PI this should be OK. However, are you certain that all they are doing is writing data, no subset or view creation?

Are they all writing to distinct areas of the cube?

The general guidelines on max concurrent writers per server is around 25 so at 94 you may just be going too far. TM1 is not a transaction processing system.

Have you tried splitting the load up into batches of 24? Presumably this will work. It might be worth checking on the overall duration of this approach to see if the 94 parallel load approach is worthwhile.

Regards

Paul Simon
jjramirez
Posts: 31
Joined: Fri Jan 13, 2017 12:33 pm
OLAP Product: IBM Planning Analytics/TM1
Version: 2.0.9.15
Excel Version: Excel 2019

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by jjramirez »

Hi Paul

first of all, thanks for your answer.
  • Yes, All process are running under the same user ID and MaximumPorts parameter in }ClientProporties is cero for all clients and i didn't find another parameter in TM1 configuration file that specify this number. Could you please tell me where i can confirm this data ? (
  • The data sourse is not the same, the cube sourse and the cube destiny are the same, but every Child TI Process is processing different cube sections. This Process consist only in copy all data from one version and then write them to another version in the same cube but i am sending one process for each month of the year (months are independent and theres no any rule in this cube). this process creates a subset, a view source and a view destiny but i make sure there's not locks contention among them naming every object as unique, i put a random number in the name and i haven't seem any lock contention in the TM1Server.Log:

Code: Select all

vRandom = NumberToString ( Int ( Rand () * 100000000) ) ;
# Views:
vueSource = 'TI.' | sProcess  | pCube | pDimension2 | pSliceFirstElement | '.Source.' | cTimeStamp | '.' | vRandom ;
vueClear = 'TI.' | sProcess | pCube | pDimension2 | pSliceFirstElement | '.Clear.' | cTimeStamp | '.' | vRandom  ;

# Subsets:
subSource = vueSource ;
subClear = vueClear ;
subLeaf = 'SYS.Leaf' ;
  • one child process reads "January" from "bugget Version" and wite it in "January" from "Backup Version" and so on. I have 3 years ( 2018, 2019, 2020) for 3 cubes. so i am talking about (3*12*3) 108 childs making the same operation but in different sections of each cube. (I said 94 cause i am ignoring some months).
When i execute the process for one month it works perfectly, and on the other hand the log file is writing 50 successfully login operation for this process. I think i can reduce the batch to 25 childs and in this case there will be no big problem with processing time. But i'd like to understand exactly what happend because i have to create more TI process with more than 25 child to perform anothe calculations and i need to make sure this will not happen.

If there are a limit about connection number, why sometimes it works fine...? if a limit is set in the server, I would expect the TI process to always fail ...

Thanks Paul!
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by paulsimon »

Hi

There are various possible factors. When you process asynchronously you have no control over the order so possibly some times it is going to work and other times it might hit a memory or lock contention or some other limit.

I am not convinced that you are going to gain anything from this degree of parallelism.

Presumably you have something like one cube oer country or something like that and you are copying from one version to another. However, I suggest that you try doing all months on one go rather than one month at a time.

Regards

Paul SImon
jjramirez
Posts: 31
Joined: Fri Jan 13, 2017 12:33 pm
OLAP Product: IBM Planning Analytics/TM1
Version: 2.0.9.15
Excel Version: Excel 2019

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by jjramirez »

Hi Paul

yes, but in the case of contention i would expect some messages in "tm1lockexceptiondebug" file , but the only message it appears is about a lock in object client, no other objects.

Please, it would be great if you can check the ti process in attachments:
  • MasterProcess = Traslado.Presupuesto
  • ChildProcess: OS.Data.Copy.Element.Slice2
  • generic process to call child in batch = OS.Procesar.Batch
also i put the loskexecption file

maybe you can take a look...it would be great.

regards.
Attachments
Traslado.Presupuesto.pro
Master_Process
(65.93 KiB) Downloaded 312 times
OS.Procesar.Batch.pro
Generic_process_for_TM1RunTi
(9.25 KiB) Downloaded 327 times
OS.Data.Copy.Element.Slice2.pro
Child_Process
(25.08 KiB) Downloaded 314 times
Drg
Regular Participant
Posts: 159
Joined: Fri Aug 12, 2016 10:02 am
OLAP Product: tm1
Version: 10.2.0 - 10.3.0
Excel Version: 2010

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by Drg »

Do you use any anchors (cells of the queue cubes, attributes, global variables) for the process that you run through the Runti?
It is important to understand that each RUNTI process is isolated and has access to the data relevant for different times.
Suppose we run a 3 process instance that reads the value from the cell and writes it to a file and then increments it by n.
in our case, all three processes will return the same number.
jjramirez
Posts: 31
Joined: Fri Jan 13, 2017 12:33 pm
OLAP Product: IBM Planning Analytics/TM1
Version: 2.0.9.15
Excel Version: Excel 2019

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by jjramirez »

Yes, i am using some global parameters and attributes in the child processes that run with TM1RunTi.
but the case is when i send 12 processes for a year i see in logfile only 11 executions... why this happens?
i have verified the MaximumPorts parameter in }CLientProporties control cube and is set to cero for every process and server admin send me a screenshot from license where it says :
MAX_USERS=999999
MAX_READONLY_USERS=99999

i dont understand why in some cases all the child processes get login...
Drg
Regular Participant
Posts: 159
Joined: Fri Aug 12, 2016 10:02 am
OLAP Product: tm1
Version: 10.2.0 - 10.3.0
Excel Version: 2010

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by Drg »

jjramirez wrote: Thu Sep 21, 2017 4:29 pm Yes, i am using some global parameters and attributes in the child processes that run with TM1RunTi.
but the case is when i send 12 processes for a year i see in logfile only 11 executions... why this happens?
i have verified the MaximumPorts parameter in }CLientProporties control cube and is set to cero for every process and server admin send me a screenshot from license where it says :
MAX_USERS=999999
MAX_READONLY_USERS=99999

i dont understand why in some cases all the child processes get login...
i think you see 11 process bacause you have a bad logic in processes chain
Try create simple chain proc1 Prepare data - proc 2 RunTI, for counting you TI threads.

P/s/
My expirence runti in parralel, so painfull.
Many params may affect to result execution( when you are cooking view, whether you use in view (sabsets) mdx, ...) and bugs tm1, need be ready to all it/
jjramirez
Posts: 31
Joined: Fri Jan 13, 2017 12:33 pm
OLAP Product: IBM Planning Analytics/TM1
Version: 2.0.9.15
Excel Version: Excel 2019

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by jjramirez »

Hi Drg
P/s/
My expirence runti in parralel, so painfull.
Many params may affect to result execution( when you are cooking view, whether you use in view (sabsets) mdx, ...) and bugs tm1, need be ready to all it/
Yes you're right. I think found that Tm1RunTi only reads until 9 parameters and my process has 15...
i confirmed this adding a LogOut instruction in the prolog of child process, the i realize that parameters from 10 to 15 was in blank:

Code: Select all

12768   [1d4a]   INFO   2017-09-22 10:01:11.212   TM1.TILogOutput   OS.Data.Copy.Element.Slice2-pElementSource=Real-pElementTarget=RecalculopCube=R.Estado.Financiero-pDimension=R_Version-pSliceDimension=Periodo-pSliceFirstElement=2017.06-pSliceLastElement=2017.06-pSkipZero=-pSkipConsolidated=-pSkipRules=-pClearDestOnly=-pDimension2=-pElement2=-pElement2b=
[img]
BatFile_For_Tm1RunTi
BatFile_For_Tm1RunTi
.BatFile_for_TM1RunTi.PNG (47.72 KiB) Viewed 24659 times
[img]

i changed the child process to ask only for 8 parameters and it is working fine now. Even though Tm1Server.Log never show me an execution error before...


On the other hand,

All process i am running with RunTi always create a SourceView and a TargetView in the prolog and they create subsets using MDX to filter de data before processing. Now taking into account your last comment i dont know if it is better to create all views and subsets before to run the master process in parallel to avoid contentions. this way i could create 12 sourceviews, one for each month: view.2017.01 , view.2017.02 and so on...
and then when i run the batch with runTi, every child has to call their specific view: Process.2017.01 will use view.2017.01 that already exists and so on

Do you think it would work or... ??

Thank you so much for your answers!
Drg
Regular Participant
Posts: 159
Joined: Fri Aug 12, 2016 10:02 am
OLAP Product: tm1
Version: 10.2.0 - 10.3.0
Excel Version: 2010

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by Drg »

My process use queCube to create logs wich view created(or need be create run ti proc) and 2 level process(es) read this cube(but this no good idea to store large text fields).

my logic looks like:

Code: Select all

p1 prepare data (i use savecubedata to save my prepared view and subsets )
p1 log runti params to self log file  and runti p2     
   p2    executeprocess('') and log params run process           
       p3    copy data, trasfer data and etc
       p3    log flags,params in each (prolog metadata data epilog)            
   p2   get error code from p3 
p1 clear temp data
After read you logs and get error string or bug in tm1(find it and workaround it)

you run over 15 process on one thread?
maybe you should run one process by one user thread?
jjramirez
Posts: 31
Joined: Fri Jan 13, 2017 12:33 pm
OLAP Product: IBM Planning Analytics/TM1
Version: 2.0.9.15
Excel Version: Excel 2019

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by jjramirez »

Hi!

First of all, i want to correct my last post...
Yes you're right. I think found that Tm1RunTi only reads until 9 parameters and my process has 15...
RunTi can send to the .Bat File as many parameters as the TI Process has, but the CMD batch script only can read until 9 parameters. There's some ways to get more than 9 parameters as descripted in the link below, but i didn't consider this so that's why my process always faild as i show you in my last post. (I realized it until i put the LogOutput instruction) and i decided edit the TI process to ask only for 9 parameters... i was easy for me...

https://stackoverflow.com/questions/832 ... -batch-scr


On the other hand,
you run over 15 process on one thread?
maybe you should run one process by one user thread?
yes, i run over 15 processes but all in Batch, i mean i am sending each one as a new connection with RunTi...
User avatar
mce
Community Contributor
Posts: 352
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by mce »

Hello,
I have a similar problem in executing paralel processes via RunTI with the same CAM user.
TM1TopScreen.PNG
TM1TopScreen.PNG (8.34 KiB) Viewed 22576 times

Code: Select all

9372   [23e]   DEBUG   2019-06-21 17:58:08.023   TM1.Lock.Exception   Contention encountered attempting to acquire lock (0x000000000BEA1C10) on object [Client "CAMID("CognosEx:u:uid=myrunti_user")", addr=0x0000000009BF5E10, index=R-4440] in READONLY mode at ..\tm1_r7s\TM1ClientImpl.cpp:1479 during function 'SystemServerConnectWithCAMNamespace'. Entering wait state 'IXC'.
Blocked by the following 1 thread:
Thread 8992 holds the lock in IX mode
CAM user authentication keeps causing contention lock and not able to login via RunTI.
Do you know if this solution is resolved in any version? This issue is happening in PA 2.0.5 now.

Regards,
John Hammond
Community Contributor
Posts: 295
Joined: Mon Mar 23, 2009 10:50 am
OLAP Product: PAW/PAX 2.0.72 Perspectives
Version: TM1 Server 11.8.003
Excel Version: 365 and 2016
Location: South London

Re: Tm1RunTi, what is the Maximum TM1 Server Conections allowed in parallel execution?

Post by John Hammond »

tm1runTI doesnt like \ in the parameters. Sometimes you have to change to double backslash. That might cause your missing parameters...
Post Reply