TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post Reply
User avatar
mce
Community Contributor
Posts: 310
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by mce » Wed Nov 20, 2019 4:00 pm

Hello,

We run the same TI process in parallel with different parameters to update different portions of a cube. However, we observe some locking contentions on some dimension objects causing the processes to roll back and re-execute. There is no dimension operations in this process that is running in parallel and there is no public MDX subsets in those dimensions. There are many dimensions in the model but this contention occurs only in few dimensions which has no different usage than other dimensions in the process.

In summary, Lock contentions occur where there is no reason for it to occur in TM1. This is causing lots of trouble and our concurrent processes that are updating data in parallel are all failing to commit and keeps rolling back and hence finish in very long time at the end since they rolled back.

We believe this is a defect in TM1's locking behaviour as per our observations.
Any comments would be helpful to guide us how we can resolve or work around this issue?
Thanks in advance.

Regards

User avatar
qml
MVP
Posts: 1079
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.6 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by qml » Wed Nov 20, 2019 5:21 pm

Do you have any functions in this TI which would operate on these locking dimensions, even as read-only? DIMIX/DIMNM/ATTRS...?

Are any of the locking dimensions ever updated by any of the *Direct functions - not necessarily in this process, but in general?
Kamil Arendt

User avatar
mce
Community Contributor
Posts: 310
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by mce » Wed Nov 20, 2019 8:50 pm

qml wrote:
Wed Nov 20, 2019 5:21 pm
Do you have any functions in this TI which would operate on these locking dimensions, even as read-only? DIMIX/DIMNM/ATTRS...?
Yes, we use DIMIX a lot. In an occasion, I replaced DIMIX by SubsetElementGetIndex, and it was still locking on that dimension.
We also use CellGetS for a cube that use the dimension that is being locked with contention.
qml wrote:
Wed Nov 20, 2019 5:21 pm
Are any of the locking dimensions ever updated by any of the *Direct functions - not necessarily in this process, but in general?
The process that is running in parallel, does not make any dimension updates. It only updates data and parallel threads all hit different slices of the same cube.
In some occasions, we use direct function when updating dimensions, but this happens at a very different time than data update. After updating dimensions, we also make save to disk before we start updating cubes. Hence this should not impact. Are there any known issues with Direct functions?

User avatar
ykud
Posts: 61
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by ykud » Wed Nov 20, 2019 10:21 pm

We saw something similar in one of the models and unfortunately, there wasn't any real solution, we just had to 'commit' locks in different parts of the process earlier by running each portion as it's own RunTi with ExecuteCommand(...,1).

Can you try running this process in the TIDebugger and see what Locks it gets? If you enable Lock=Debug logging, you'd see what object causes contention before rollback and you'd be able to see where this object get's locked by the TI?

User avatar
qml
MVP
Posts: 1079
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.6 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by qml » Fri Nov 22, 2019 9:51 am

I still don't trust the *Direct metadata functions very much. The fact you need to remember to run DimensionUpdateDirect after any such updates in order for the new elements to be fully functional tells you a bit about the downsides of overusing these functions. But I digress.

When I had issues with similar contention locking in the past, I discovered that the first DIMIX after a (regular, not necessarily Direct) dimension update would require an IX/W lock rather than the regular R lock. I worked around it by adding a dummy DIMIX in the Epilog of the processes that were updating the dimensions in question and that solved the issue. My hypothesis is that the first DIMIX call triggers some internal index caching or something similar.
Kamil Arendt

User avatar
mce
Community Contributor
Posts: 310
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by mce » Sun Nov 24, 2019 11:39 am

ykud wrote:
Wed Nov 20, 2019 10:21 pm
We saw something similar in one of the models and unfortunately, there wasn't any real solution, we just had to 'commit' locks in different parts of the process earlier by running each portion as it's own RunTi with ExecuteCommand(...,1).
We are already running the threads in different sessions using RunTI.
ykud wrote:
Wed Nov 20, 2019 10:21 pm
If you enable Lock=Debug logging, you'd see what object causes contention before rollback and you'd be able to see where this object get's locked by the TI?
It is already enabled, and we see the name of the dimension that is causing contention. But this dimension is never updated at all during the process. It is only ready using DIMIX or ELPAR or SubsetElementGetIndex. The same operations are done in all other dimensions, but it is causing contention only in this dimension. The stranger point about it, it does not happen in all executions, it happen only in some executions. I also observe that contention is more likely to happen when DIMIX function returns 0 in a lot more records.

User avatar
mce
Community Contributor
Posts: 310
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by mce » Sun Nov 24, 2019 11:46 am

qml wrote:
Fri Nov 22, 2019 9:51 am
I still don't trust the *Direct metadata functions very much. The fact you need to remember to run DimensionUpdateDirect after any such updates in order for the new elements to be fully functional tells you a bit about the downsides of overusing these functions. But I digress.

When I had issues with similar contention locking in the past, I discovered that the first DIMIX after a (regular, not necessarily Direct) dimension update would require an IX/W lock rather than the regular R lock. I worked around it by adding a dummy DIMIX in the Epilog of the processes that were updating the dimensions in question and that solved the issue. My hypothesis is that the first DIMIX call triggers some internal index caching or something similar.
Thanks for the note. I will check this.

PlanningDev
Community Contributor
Posts: 340
Joined: Tue Aug 17, 2010 6:31 am
OLAP Product: Planning Analytics
Version: 2.0.5
Excel Version: 2016

Re: TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by PlanningDev » Tue Dec 03, 2019 9:58 pm

Couldn't quite tell from the post but do you turn logging on/off in your TI's?

If so it turns out that CubeGetLogChanges/CubeSetLogChanges is locking. You would need to use CellPutS to the }CubeProperties cube for Logging instead. It is definetly causing locking for us especially when Chores start to run at the same time.

https://code.cubewise.com/blog/10-tips- ... pplication

EvgenyT
Community Contributor
Posts: 277
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: Cognos TM 1
Version: 10.2.2
Excel Version: 2013
Location: Sydney, Australia

Re: TM1.Lock Contention happens on some dimension objects due to read operations in concurrent TI processes

Post by EvgenyT » Wed Dec 04, 2019 10:26 am

On the subject of CubeGetLogChanges/CubeSetLogChange
Had an issue with new }bedrock processes where it would cause a lock when executed in parallel via RushTI or RunTI.
There is a new parameter for logging in those processes (I can't remember exactly which }bedrock TIs were they). In the end, I had to hash out that code in }berock TIs to avoid lock contention.

Post Reply