Hi all,
I am currently working on parallelizing processes using the RunProcess function.
I do have a lot of processes to run (around 2000 every time) and to avoid CPU exceeding, I use the synchronized function to force serial execution in the same way as explained in this Cubewise article : https://www.cubewise.com/blog/how-to-us ... f-threads/.
So I have limited the number of processes running simultaneously to 50 and everything works pretty find until the end when certain processes start to queue up i.e there is only one process running and a lot are in wait mode. I think that it may be because of the synchronized function when the subprocesses all share the same lock object name, they need to wait for the one currently running to finish. In the end, I have to wait for each process with this lock object to finish individually before the next one can be launched (even if there are less than 50 subprocesses remaining!) which could be parallelized instead.
Is there any solution to avoid this problem of queuing and to start again parallization in this case ? By modifying the lock object name or any other means ?
Thanks in avance
How to avoid queuing at the end of processes parallelization ?
-
- Community Contributor
- Posts: 154
- Joined: Tue Apr 02, 2013 1:41 pm
- OLAP Product: tm1, cognos bi
- Version: from TM1 9.4 to PA 2.0.9.6
- Excel Version: 2010
- Location: Toronto, ON
Re: How to avoid queuing at the end of processes parallelization ?
Do yourself a favor, use RushTI. Before I started using RushTI, process parallelization was always causing challenges.
Ardian Alikaj
-
- Posts: 98
- Joined: Mon Jul 29, 2019 5:02 am
- OLAP Product: Planning Analytics
- Version: 2.0.9.x
- Excel Version: Office 365 16
Re: How to avoid queuing at the end of processes parallelization ?
There is a setting where locking is reported in TM1Server.log. This is a good place to start to gain insight.
I used RunProcess in the same manner in my prior role and had a lot of problems with locking. When these specific locks were encountered the parallel processes would go into serial mode. And the locking was not just internal to the RunProcess collection, it was due to users updating a cube which had a dimension which was an attribute of one of the dimensions of the cube being updated by the RunProcess collection. There's not much joy in this!
I used RunProcess in the same manner in my prior role and had a lot of problems with locking. When these specific locks were encountered the parallel processes would go into serial mode. And the locking was not just internal to the RunProcess collection, it was due to users updating a cube which had a dimension which was an attribute of one of the dimensions of the cube being updated by the RunProcess collection. There's not much joy in this!
Re: How to avoid queuing at the end of processes parallelization ?
Data commit (the last step of the process) is single threaded / locking as it updates the cube index structure, so it might be the case when a lot of Tis are committing at the same time and are queuing due to this. It might be better to split data less evenly, so that they're committing at different times.
-
- MVP
- Posts: 3664
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: How to avoid queuing at the end of processes parallelization ?
This! Don't make all your processing chunks equal sized. This will inevitably lead to a commit queue bottleneck.ykud wrote: ↑Wed Mar 27, 2024 11:26 pm Data commit (the last step of the process) is single threaded / locking as it updates the cube index structure, so it might be the case when a lot of Tis are committing at the same time and are queuing due to this. It might be better to split data less evenly, so that they're committing at different times.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
-
- MVP
- Posts: 3126
- Joined: Mon Dec 29, 2008 6:26 pm
- OLAP Product: TM1, Jedox
- Version: PAL 2.0.9.18
- Excel Version: Microsoft 365
- Location: Brussels, Belgium
- Contact:
Re: How to avoid queuing at the end of processes parallelization ?
Also, 2000 processes doing less work is most of the time less optimal than less processes doing more work (on average).
Best regards,
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly