How to cancel a thread that cannot be cancelled in TM1 Top

User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

How to cancel a thread that cannot be cancelled in TM1 Top

Post by paulsimon »

Hi

I thought I would share this discovery in case it helps anyone else.

On TM1 10.1.1, we had a situation where a user's thread was running a ChangeSetUndo. This had been running for 5000 seconds and was clearly never going to stop.

There were other threads in a Commit Wait state that were presumably unable to Commit because of something the running process was locking.

New users were unable to login

We had one Admin who had already verified Admin access in TM1 Top

The long running thread could not be cancelled in TM1 Top

The Commit Wait threads could not be cancelled in TM1 Top

New users could not login - Login attempt appearing as ???

New attempts to Verify additional Admin access in TM1 Top did not work

It appeared that the only answer was going to be to restart the service. From prior experience the service would almost certainly not shut down and we would therefore have to kill the TM1SD.EXE. This would lose 6 hours of activity on the server since the last backup. Some would be recovered from logs. Other items happen with logging turned off. Effectively the standard recovery would given nothing useful. Therefore having to terminate the process was not an appealing option.

The only remaining way I could think of to affect the server was to change a dynamic TM1S.CFG parameter.

I searched for a TM1S.CFG parameter that was dynamic and found IdleConnectionTimeOutSeconds. I changed since from 3600 to 3601.

After a few seconds TM1 Top suddenly cleared leaving some very happy bunnies.

It seems that perhaps changing this parameter cleared the running and waiting threads without the need to terminate the service.

I noticed that in TM1 Top there seems to be two special threads Th:Pseudo and Th:DynamicConfig. I would guess that the latter watches for changes to the TM1S.CFG on dynamic parameters and does the magic.

Regards

Paul Simon
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by tomok »

Wow! This almost seems too good to be true!
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
declanr
MVP
Posts: 1815
Joined: Mon Dec 05, 2011 11:51 am
OLAP Product: Cognos TM1
Version: PA2.0 and most of the old ones
Excel Version: All of em
Location: Manchester, United Kingdom
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by declanr »

This is a brilliant discovery - one I wish I had heard of years ago - lost count of the amount of times I've had to kill services now.

Does present a worry though in the fact that a change to a dynamic config parameter would presumably also kill any genuine threads that are running at the same point in time; not that I often change any dynamic parameters in a live model but still worth being wary of.
I will do some of my own testing in various versions.

Thanks for sharing.
Declan Rodger
lotsaram
MVP
Posts: 3648
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by lotsaram »

This is indeed an amazing discovery, kudos.
Would it be bad karma t say I'm looking forward to trying this out?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by paulsimon »

Hi Declan

That was one of my thoughts too. I did a search for any information on the side effects of changing a dynamic config parameter but I couldn't find anything. It does mention that it only polls for changes to the TM1S.CFG every 60 seconds, so I was probably lucky that I changed it on about the 55th seconds of its cycle.

It was a little late in the day but I can check with the user to find out if the journal they were posting to TM1 did get stored in the cube, or whether the change got rolled back. The indication was that they were trying to undo changes. I don't think that they were consciously trying to do that but they might have clicked the undo button by accident. However, I doubt that, as in the system I inherited, most of the TM1 Web sheets are launched from BI, and the TM1 toolbar is usually hidden.

It would be interesting to see if it was the fact that I chose IdleConnectionTimeOut as perhaps the change made it check, although the thread that seemed to be the problem was in a running rather than an idle state.

Regards

Paul Simon
RJ!
Community Contributor
Posts: 219
Joined: Mon Jul 23, 2012 8:31 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2010

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by RJ! »

Hmmm... I wonder if this would also clean out those pesky duplicate REST calls from CAFE Users that tend to lock up TM1.
User avatar
Steve Rowe
Site Admin
Posts: 2407
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Steve Rowe »

+1 from me, post of the decade!
Technical Director
www.infocat.co.uk
User avatar
Elessar
Community Contributor
Posts: 328
Joined: Mon Nov 21, 2011 12:33 pm
OLAP Product: PA 2
Version: 2.0.9
Excel Version: 2016
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Elessar »

Brilliant!
Best regards, Alexander Dvoynev

TM1 and Data Science blog: 6th article - PAfE + VBA: Commit each cell without pressing “Commit” button.
User avatar
jim wood
Site Admin
Posts: 3951
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: PA 2.0.7
Excel Version: Office 365
Location: 37 East 18th Street New York
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by jim wood »

+1 from me too. I bet even IBM are happy about this. The amount of support calls they must get with issues like this.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
BrianL
MVP
Posts: 264
Joined: Mon Nov 03, 2014 8:23 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2 PA2
Excel Version: 2016

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by BrianL »

Not to be a Debbie Downer, but has behavior been replicated more than once?

Since the only document I can find from IBM that makes an attempt at defining IdleConnectionTimeOut states the parameter only applies to idle client connections, I'm a bit skeptical that this has any impact on thread cancellation.
This parameter specifies a timeout limit for idle client connections, in seconds.
For example, if you include the line IdleConnectionTimeOutSeconds=900 in TM1s.cfg,
the server disconnects idle client connections after 900 seconds
http://www-01.ibm.com/support/docview.w ... wg21407112
User avatar
Steve Rowe
Site Admin
Posts: 2407
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Steve Rowe »

I'm guessing not....
but I think what Paul is saying is that the act of refreshing the parameter is what triggers the clearance of locked threads, rather than the value parameter itself. It is not impossible to imagine that when the parameter changes the instance checks for overtime threads in a different enough way to normal that locked threads are dropped.

Of course it could also be a big coincidence as well and TM1 would of sorted it self out any why.

Testing required....
Technical Director
www.infocat.co.uk
User avatar
gtonkin
MVP
Posts: 1192
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: Latest and greatest
Excel Version: Office 365 64-bit
Location: JHB, South Africa
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by gtonkin »

BrianL wrote:Not to be a Debbie Downer, but has behavior been replicated more than once?
We have used this technique over the years infrequently but repeatedly-always worked when we needed it too-cannot remember having an issue yet. May be jinxed now however.
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by paulsimon »

Hi

I am not claiming that this is a cure all for all TM1 Server lockup situations. However, in this particular case it worked. As Steve said I believe that it is the act of changing a dynamic parameter rather than which dynamic parameter you change that has the effect.

On the system that I am working with at the moment we can go months without a lockup issue, so it may be some time before I need to find out whether this works in other cases.

Unfortunately we have had other issues today with the Content Store going down which affected CAM and then with intermittent Time Outs in TM1 Web, often happening as soon as the user launches a subset editor from either a Web Sheet or a View, and often shortly after the user logs in, ie not normal time outs. Because of this I have not had a chance to follow up on whether the change to the dynamic parameter had any effect on Threads that were running at the time. Initial investigations appear to indicate that this was not the case, but I need to do more investigation and this may not be possible given the passage of time.

Regards

Paul Simon
User avatar
qml
MVP
Posts: 1094
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.9 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by qml »

Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Kamil Arendt
User avatar
jim wood
Site Admin
Posts: 3951
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: PA 2.0.7
Excel Version: Office 365
Location: 37 East 18th Street New York
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by jim wood »

qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
Bakkone
Posts: 119
Joined: Mon Oct 27, 2014 10:50 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2013

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Bakkone »

Thanks for the tip!
User avatar
Steve Rowe
Site Admin
Posts: 2407
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Steve Rowe »

jim wood wrote:
qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....
Wasn't it just because we didn't want to introduce another method for us to annoy each other...... :roll:
Technical Director
www.infocat.co.uk
User avatar
jim wood
Site Admin
Posts: 3951
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: PA 2.0.7
Excel Version: Office 365
Location: 37 East 18th Street New York
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by jim wood »

Steve Rowe wrote:
jim wood wrote:
qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....
Wasn't it just because we didn't want to introduce another method for us to annoy each other...... :roll:
Sounds about right. I'd like your post but I can't.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
User avatar
macsir
MVP
Posts: 782
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1
Version: PAL 2.0.9
Excel Version: Office 365
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by macsir »

Brilliant, i will test it next time in 10.2.2!
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/
User avatar
Oratia623
Posts: 40
Joined: Mon Apr 27, 2009 5:36 am
OLAP Product: TM1/PA/CA
Version: V7.x to 2.0.9+
Excel Version: All
Location: Sydney, Australia

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Oratia623 »

Awesome!!!
We have exactly this situation in the past, and will give this a try next time.
@paulsimon You are THE MAN!

Proof that this community and forum rocks!
Paul Williamson
____________________________________________________________________________________
I came. I saw. I did not concur.
Post Reply