How to cancel a thread that cannot be cancelled in TM1 Top
- paulsimon
- MVP
- Posts: 808
- Joined: Sat Sep 03, 2011 11:10 pm
- OLAP Product: TM1
- Version: PA 2.0.5
- Excel Version: 2016
- Contact:
How to cancel a thread that cannot be cancelled in TM1 Top
Hi
I thought I would share this discovery in case it helps anyone else.
On TM1 10.1.1, we had a situation where a user's thread was running a ChangeSetUndo. This had been running for 5000 seconds and was clearly never going to stop.
There were other threads in a Commit Wait state that were presumably unable to Commit because of something the running process was locking.
New users were unable to login
We had one Admin who had already verified Admin access in TM1 Top
The long running thread could not be cancelled in TM1 Top
The Commit Wait threads could not be cancelled in TM1 Top
New users could not login - Login attempt appearing as ???
New attempts to Verify additional Admin access in TM1 Top did not work
It appeared that the only answer was going to be to restart the service. From prior experience the service would almost certainly not shut down and we would therefore have to kill the TM1SD.EXE. This would lose 6 hours of activity on the server since the last backup. Some would be recovered from logs. Other items happen with logging turned off. Effectively the standard recovery would given nothing useful. Therefore having to terminate the process was not an appealing option.
The only remaining way I could think of to affect the server was to change a dynamic TM1S.CFG parameter.
I searched for a TM1S.CFG parameter that was dynamic and found IdleConnectionTimeOutSeconds. I changed since from 3600 to 3601.
After a few seconds TM1 Top suddenly cleared leaving some very happy bunnies.
It seems that perhaps changing this parameter cleared the running and waiting threads without the need to terminate the service.
I noticed that in TM1 Top there seems to be two special threads Th:Pseudo and Th:DynamicConfig. I would guess that the latter watches for changes to the TM1S.CFG on dynamic parameters and does the magic.
Regards
Paul Simon
I thought I would share this discovery in case it helps anyone else.
On TM1 10.1.1, we had a situation where a user's thread was running a ChangeSetUndo. This had been running for 5000 seconds and was clearly never going to stop.
There were other threads in a Commit Wait state that were presumably unable to Commit because of something the running process was locking.
New users were unable to login
We had one Admin who had already verified Admin access in TM1 Top
The long running thread could not be cancelled in TM1 Top
The Commit Wait threads could not be cancelled in TM1 Top
New users could not login - Login attempt appearing as ???
New attempts to Verify additional Admin access in TM1 Top did not work
It appeared that the only answer was going to be to restart the service. From prior experience the service would almost certainly not shut down and we would therefore have to kill the TM1SD.EXE. This would lose 6 hours of activity on the server since the last backup. Some would be recovered from logs. Other items happen with logging turned off. Effectively the standard recovery would given nothing useful. Therefore having to terminate the process was not an appealing option.
The only remaining way I could think of to affect the server was to change a dynamic TM1S.CFG parameter.
I searched for a TM1S.CFG parameter that was dynamic and found IdleConnectionTimeOutSeconds. I changed since from 3600 to 3601.
After a few seconds TM1 Top suddenly cleared leaving some very happy bunnies.
It seems that perhaps changing this parameter cleared the running and waiting threads without the need to terminate the service.
I noticed that in TM1 Top there seems to be two special threads Th:Pseudo and Th:DynamicConfig. I would guess that the latter watches for changes to the TM1S.CFG on dynamic parameters and does the magic.
Regards
Paul Simon
-
- MVP
- Posts: 2831
- Joined: Tue Feb 16, 2010 2:39 pm
- OLAP Product: TM1, Palo
- Version: Beginning of time thru 10.2
- Excel Version: 2003-2007-2010-2013
- Location: Atlanta, GA
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Wow! This almost seems too good to be true!
-
- MVP
- Posts: 1815
- Joined: Mon Dec 05, 2011 11:51 am
- OLAP Product: Cognos TM1
- Version: PA2.0 and most of the old ones
- Excel Version: All of em
- Location: Manchester, United Kingdom
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
This is a brilliant discovery - one I wish I had heard of years ago - lost count of the amount of times I've had to kill services now.
Does present a worry though in the fact that a change to a dynamic config parameter would presumably also kill any genuine threads that are running at the same point in time; not that I often change any dynamic parameters in a live model but still worth being wary of.
I will do some of my own testing in various versions.
Thanks for sharing.
Does present a worry though in the fact that a change to a dynamic config parameter would presumably also kill any genuine threads that are running at the same point in time; not that I often change any dynamic parameters in a live model but still worth being wary of.
I will do some of my own testing in various versions.
Thanks for sharing.
Declan Rodger
-
- MVP
- Posts: 3652
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: How to cancel a thread that cannot be cancelled in TM1 Top
This is indeed an amazing discovery, kudos.
Would it be bad karma t say I'm looking forward to trying this out?
Would it be bad karma t say I'm looking forward to trying this out?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
- paulsimon
- MVP
- Posts: 808
- Joined: Sat Sep 03, 2011 11:10 pm
- OLAP Product: TM1
- Version: PA 2.0.5
- Excel Version: 2016
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Hi Declan
That was one of my thoughts too. I did a search for any information on the side effects of changing a dynamic config parameter but I couldn't find anything. It does mention that it only polls for changes to the TM1S.CFG every 60 seconds, so I was probably lucky that I changed it on about the 55th seconds of its cycle.
It was a little late in the day but I can check with the user to find out if the journal they were posting to TM1 did get stored in the cube, or whether the change got rolled back. The indication was that they were trying to undo changes. I don't think that they were consciously trying to do that but they might have clicked the undo button by accident. However, I doubt that, as in the system I inherited, most of the TM1 Web sheets are launched from BI, and the TM1 toolbar is usually hidden.
It would be interesting to see if it was the fact that I chose IdleConnectionTimeOut as perhaps the change made it check, although the thread that seemed to be the problem was in a running rather than an idle state.
Regards
Paul Simon
That was one of my thoughts too. I did a search for any information on the side effects of changing a dynamic config parameter but I couldn't find anything. It does mention that it only polls for changes to the TM1S.CFG every 60 seconds, so I was probably lucky that I changed it on about the 55th seconds of its cycle.
It was a little late in the day but I can check with the user to find out if the journal they were posting to TM1 did get stored in the cube, or whether the change got rolled back. The indication was that they were trying to undo changes. I don't think that they were consciously trying to do that but they might have clicked the undo button by accident. However, I doubt that, as in the system I inherited, most of the TM1 Web sheets are launched from BI, and the TM1 toolbar is usually hidden.
It would be interesting to see if it was the fact that I chose IdleConnectionTimeOut as perhaps the change made it check, although the thread that seemed to be the problem was in a running rather than an idle state.
Regards
Paul Simon
-
- Community Contributor
- Posts: 219
- Joined: Mon Jul 23, 2012 8:31 am
- OLAP Product: TM1
- Version: 10.2.2
- Excel Version: 2010
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Hmmm... I wonder if this would also clean out those pesky duplicate REST calls from CAFE Users that tend to lock up TM1.
- Steve Rowe
- Site Admin
- Posts: 2415
- Joined: Wed May 14, 2008 4:25 pm
- OLAP Product: TM1
- Version: TM1 v6,v7,v8,v9,v10,v11+PAW
- Excel Version: Nearly all of them
Re: How to cancel a thread that cannot be cancelled in TM1 Top
+1 from me, post of the decade!
Technical Director
www.infocat.co.uk
www.infocat.co.uk
- Elessar
- Community Contributor
- Posts: 338
- Joined: Mon Nov 21, 2011 12:33 pm
- OLAP Product: PA 2
- Version: 2.0.9
- Excel Version: 2016
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Brilliant!
Best regards, Alexander Dvoynev
TM1 and Data Science blog: 6th article - PAfE + VBA: Commit each cell without pressing “Commit” button.
TM1 and Data Science blog: 6th article - PAfE + VBA: Commit each cell without pressing “Commit” button.
- jim wood
- Site Admin
- Posts: 3951
- Joined: Wed May 14, 2008 1:51 pm
- OLAP Product: TM1
- Version: PA 2.0.7
- Excel Version: Office 365
- Location: 37 East 18th Street New York
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
+1 from me too. I bet even IBM are happy about this. The amount of support calls they must get with issues like this.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
-
- MVP
- Posts: 264
- Joined: Mon Nov 03, 2014 8:23 pm
- OLAP Product: TM1
- Version: 9.5.2 10.1 10.2 PA2
- Excel Version: 2016
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Not to be a Debbie Downer, but has behavior been replicated more than once?
Since the only document I can find from IBM that makes an attempt at defining IdleConnectionTimeOut states the parameter only applies to idle client connections, I'm a bit skeptical that this has any impact on thread cancellation.
Since the only document I can find from IBM that makes an attempt at defining IdleConnectionTimeOut states the parameter only applies to idle client connections, I'm a bit skeptical that this has any impact on thread cancellation.
http://www-01.ibm.com/support/docview.w ... wg21407112This parameter specifies a timeout limit for idle client connections, in seconds.
For example, if you include the line IdleConnectionTimeOutSeconds=900 in TM1s.cfg,
the server disconnects idle client connections after 900 seconds
- Steve Rowe
- Site Admin
- Posts: 2415
- Joined: Wed May 14, 2008 4:25 pm
- OLAP Product: TM1
- Version: TM1 v6,v7,v8,v9,v10,v11+PAW
- Excel Version: Nearly all of them
Re: How to cancel a thread that cannot be cancelled in TM1 Top
I'm guessing not....
but I think what Paul is saying is that the act of refreshing the parameter is what triggers the clearance of locked threads, rather than the value parameter itself. It is not impossible to imagine that when the parameter changes the instance checks for overtime threads in a different enough way to normal that locked threads are dropped.
Of course it could also be a big coincidence as well and TM1 would of sorted it self out any why.
Testing required....
but I think what Paul is saying is that the act of refreshing the parameter is what triggers the clearance of locked threads, rather than the value parameter itself. It is not impossible to imagine that when the parameter changes the instance checks for overtime threads in a different enough way to normal that locked threads are dropped.
Of course it could also be a big coincidence as well and TM1 would of sorted it self out any why.
Testing required....
Technical Director
www.infocat.co.uk
www.infocat.co.uk
- gtonkin
- MVP
- Posts: 1198
- Joined: Thu May 06, 2010 3:03 pm
- OLAP Product: TM1
- Version: Latest and greatest
- Excel Version: Office 365 64-bit
- Location: JHB, South Africa
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
We have used this technique over the years infrequently but repeatedly-always worked when we needed it too-cannot remember having an issue yet. May be jinxed now however.BrianL wrote:Not to be a Debbie Downer, but has behavior been replicated more than once?
- paulsimon
- MVP
- Posts: 808
- Joined: Sat Sep 03, 2011 11:10 pm
- OLAP Product: TM1
- Version: PA 2.0.5
- Excel Version: 2016
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Hi
I am not claiming that this is a cure all for all TM1 Server lockup situations. However, in this particular case it worked. As Steve said I believe that it is the act of changing a dynamic parameter rather than which dynamic parameter you change that has the effect.
On the system that I am working with at the moment we can go months without a lockup issue, so it may be some time before I need to find out whether this works in other cases.
Unfortunately we have had other issues today with the Content Store going down which affected CAM and then with intermittent Time Outs in TM1 Web, often happening as soon as the user launches a subset editor from either a Web Sheet or a View, and often shortly after the user logs in, ie not normal time outs. Because of this I have not had a chance to follow up on whether the change to the dynamic parameter had any effect on Threads that were running at the time. Initial investigations appear to indicate that this was not the case, but I need to do more investigation and this may not be possible given the passage of time.
Regards
Paul Simon
I am not claiming that this is a cure all for all TM1 Server lockup situations. However, in this particular case it worked. As Steve said I believe that it is the act of changing a dynamic parameter rather than which dynamic parameter you change that has the effect.
On the system that I am working with at the moment we can go months without a lockup issue, so it may be some time before I need to find out whether this works in other cases.
Unfortunately we have had other issues today with the Content Store going down which affected CAM and then with intermittent Time Outs in TM1 Web, often happening as soon as the user launches a subset editor from either a Web Sheet or a View, and often shortly after the user logs in, ie not normal time outs. Because of this I have not had a chance to follow up on whether the change to the dynamic parameter had any effect on Threads that were running at the time. Initial investigations appear to indicate that this was not the case, but I need to do more investigation and this may not be possible given the passage of time.
Regards
Paul Simon
- qml
- MVP
- Posts: 1094
- Joined: Mon Feb 01, 2010 1:01 pm
- OLAP Product: TM1 / Planning Analytics
- Version: 2.0.9 and all previous
- Excel Version: 2007 - 2016
- Location: London, UK, Europe
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Kamil Arendt
- jim wood
- Site Admin
- Posts: 3951
- Joined: Wed May 14, 2008 1:51 pm
- OLAP Product: TM1
- Version: PA 2.0.7
- Excel Version: Office 365
- Location: 37 East 18th Street New York
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
-
- Posts: 119
- Joined: Mon Oct 27, 2014 10:50 am
- OLAP Product: TM1
- Version: 10.2.2
- Excel Version: 2013
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Thanks for the tip!
- Steve Rowe
- Site Admin
- Posts: 2415
- Joined: Wed May 14, 2008 4:25 pm
- OLAP Product: TM1
- Version: TM1 v6,v7,v8,v9,v10,v11+PAW
- Excel Version: Nearly all of them
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Wasn't it just because we didn't want to introduce another method for us to annoy each other......jim wood wrote:Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Technical Director
www.infocat.co.uk
www.infocat.co.uk
- jim wood
- Site Admin
- Posts: 3951
- Joined: Wed May 14, 2008 1:51 pm
- OLAP Product: TM1
- Version: PA 2.0.7
- Excel Version: Office 365
- Location: 37 East 18th Street New York
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Sounds about right. I'd like your post but I can't.Steve Rowe wrote:Wasn't it just because we didn't want to introduce another method for us to annoy each other......jim wood wrote:Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
- macsir
- MVP
- Posts: 782
- Joined: Wed May 30, 2012 6:50 am
- OLAP Product: TM1
- Version: PAL 2.0.9
- Excel Version: Office 365
- Contact:
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Brilliant, i will test it next time in 10.2.2!
- Oratia623
- Posts: 40
- Joined: Mon Apr 27, 2009 5:36 am
- OLAP Product: TM1/PA/CA
- Version: V7.x to 2.0.9+
- Excel Version: All
- Location: Sydney, Australia
Re: How to cancel a thread that cannot be cancelled in TM1 Top
Awesome!!!
We have exactly this situation in the past, and will give this a try next time.
@paulsimon You are THE MAN!
Proof that this community and forum rocks!
We have exactly this situation in the past, and will give this a try next time.
@paulsimon You are THE MAN!
Proof that this community and forum rocks!
Paul Williamson
____________________________________________________________________________________
I came. I saw. I did not concur.
____________________________________________________________________________________
I came. I saw. I did not concur.