How to cancel a thread that cannot be cancelled in TM1 Top

User avatar
paulsimon
MVP
Posts: 635
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: 10.1.1 and 10.2.2
Excel Version: 2013

How to cancel a thread that cannot be cancelled in TM1 Top

Post by paulsimon » Tue Apr 26, 2016 3:48 pm

Hi

I thought I would share this discovery in case it helps anyone else.

On TM1 10.1.1, we had a situation where a user's thread was running a ChangeSetUndo. This had been running for 5000 seconds and was clearly never going to stop.

There were other threads in a Commit Wait state that were presumably unable to Commit because of something the running process was locking.

New users were unable to login

We had one Admin who had already verified Admin access in TM1 Top

The long running thread could not be cancelled in TM1 Top

The Commit Wait threads could not be cancelled in TM1 Top

New users could not login - Login attempt appearing as ???

New attempts to Verify additional Admin access in TM1 Top did not work

It appeared that the only answer was going to be to restart the service. From prior experience the service would almost certainly not shut down and we would therefore have to kill the TM1SD.EXE. This would lose 6 hours of activity on the server since the last backup. Some would be recovered from logs. Other items happen with logging turned off. Effectively the standard recovery would given nothing useful. Therefore having to terminate the process was not an appealing option.

The only remaining way I could think of to affect the server was to change a dynamic TM1S.CFG parameter.

I searched for a TM1S.CFG parameter that was dynamic and found IdleConnectionTimeOutSeconds. I changed since from 3600 to 3601.

After a few seconds TM1 Top suddenly cleared leaving some very happy bunnies.

It seems that perhaps changing this parameter cleared the running and waiting threads without the need to terminate the service.

I noticed that in TM1 Top there seems to be two special threads Th:Pseudo and Th:DynamicConfig. I would guess that the latter watches for changes to the TM1S.CFG on dynamic parameters and does the magic.

Regards

Paul Simon

User avatar
tomok
MVP
Posts: 2483
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by tomok » Tue Apr 26, 2016 3:53 pm

Wow! This almost seems too good to be true!
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

declanr
MVP
Posts: 1588
Joined: Mon Dec 05, 2011 11:51 am
OLAP Product: Cognos TM1
Version: PA2.0 and most of the old ones
Excel Version: All of em
Location: Manchester, United Kingdom
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by declanr » Tue Apr 26, 2016 4:28 pm

This is a brilliant discovery - one I wish I had heard of years ago - lost count of the amount of times I've had to kill services now.

Does present a worry though in the fact that a change to a dynamic config parameter would presumably also kill any genuine threads that are running at the same point in time; not that I often change any dynamic parameters in a live model but still worth being wary of.
I will do some of my own testing in various versions.

Thanks for sharing.

lotsaram
MVP
Posts: 3126
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by lotsaram » Tue Apr 26, 2016 8:25 pm

This is indeed an amazing discovery, kudos.
Would it be bad karma t say I'm looking forward to trying this out?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
paulsimon
MVP
Posts: 635
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: 10.1.1 and 10.2.2
Excel Version: 2013

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by paulsimon » Tue Apr 26, 2016 8:41 pm

Hi Declan

That was one of my thoughts too. I did a search for any information on the side effects of changing a dynamic config parameter but I couldn't find anything. It does mention that it only polls for changes to the TM1S.CFG every 60 seconds, so I was probably lucky that I changed it on about the 55th seconds of its cycle.

It was a little late in the day but I can check with the user to find out if the journal they were posting to TM1 did get stored in the cube, or whether the change got rolled back. The indication was that they were trying to undo changes. I don't think that they were consciously trying to do that but they might have clicked the undo button by accident. However, I doubt that, as in the system I inherited, most of the TM1 Web sheets are launched from BI, and the TM1 toolbar is usually hidden.

It would be interesting to see if it was the fact that I chose IdleConnectionTimeOut as perhaps the change made it check, although the thread that seemed to be the problem was in a running rather than an idle state.

Regards

Paul Simon

RJ!
Community Contributor
Posts: 219
Joined: Mon Jul 23, 2012 8:31 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2010

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by RJ! » Tue Apr 26, 2016 10:49 pm

Hmmm... I wonder if this would also clean out those pesky duplicate REST calls from CAFE Users that tend to lock up TM1.

User avatar
Steve Rowe
Site Admin
Posts: 1802
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: 10.2.2., PAW
Excel Version: Nearly all of them

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Steve Rowe » Wed Apr 27, 2016 10:10 am

+1 from me, post of the decade!

User avatar
Elessar
Posts: 147
Joined: Mon Nov 21, 2011 12:33 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2010
Location: Russia

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Elessar » Wed Apr 27, 2016 12:45 pm

Brilliant!
Sorry for my English ;)

User avatar
jim wood
Site Admin
Posts: 3601
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by jim wood » Wed Apr 27, 2016 1:03 pm

+1 from me too. I bet even IBM are happy about this. The amount of support calls they must get with issues like this.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 10 64-bit. TM1 Version: 10.2.2

BrianL
MVP
Posts: 264
Joined: Mon Nov 03, 2014 8:23 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2 PA2
Excel Version: 2016

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by BrianL » Wed Apr 27, 2016 2:41 pm

Not to be a Debbie Downer, but has behavior been replicated more than once?

Since the only document I can find from IBM that makes an attempt at defining IdleConnectionTimeOut states the parameter only applies to idle client connections, I'm a bit skeptical that this has any impact on thread cancellation.
This parameter specifies a timeout limit for idle client connections, in seconds.
For example, if you include the line IdleConnectionTimeOutSeconds=900 in TM1s.cfg,
the server disconnects idle client connections after 900 seconds
http://www-01.ibm.com/support/docview.w ... wg21407112

User avatar
Steve Rowe
Site Admin
Posts: 1802
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: 10.2.2., PAW
Excel Version: Nearly all of them

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Steve Rowe » Wed Apr 27, 2016 3:13 pm

I'm guessing not....
but I think what Paul is saying is that the act of refreshing the parameter is what triggers the clearance of locked threads, rather than the value parameter itself. It is not impossible to imagine that when the parameter changes the instance checks for overtime threads in a different enough way to normal that locked threads are dropped.

Of course it could also be a big coincidence as well and TM1 would of sorted it self out any why.

Testing required....

User avatar
gtonkin
MVP
Posts: 630
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by gtonkin » Wed Apr 27, 2016 3:56 pm

BrianL wrote:Not to be a Debbie Downer, but has behavior been replicated more than once?
We have used this technique over the years infrequently but repeatedly-always worked when we needed it too-cannot remember having an issue yet. May be jinxed now however.

User avatar
paulsimon
MVP
Posts: 635
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: 10.1.1 and 10.2.2
Excel Version: 2013

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by paulsimon » Wed Apr 27, 2016 10:01 pm

Hi

I am not claiming that this is a cure all for all TM1 Server lockup situations. However, in this particular case it worked. As Steve said I believe that it is the act of changing a dynamic parameter rather than which dynamic parameter you change that has the effect.

On the system that I am working with at the moment we can go months without a lockup issue, so it may be some time before I need to find out whether this works in other cases.

Unfortunately we have had other issues today with the Content Store going down which affected CAM and then with intermittent Time Outs in TM1 Web, often happening as soon as the user launches a subset editor from either a Web Sheet or a View, and often shortly after the user logs in, ie not normal time outs. Because of this I have not had a chance to follow up on whether the change to the dynamic parameter had any effect on Threads that were running at the time. Initial investigations appear to indicate that this was not the case, but I need to do more investigation and this may not be possible given the passage of time.

Regards

Paul Simon

User avatar
qml
MVP
Posts: 1059
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.4 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by qml » Thu Apr 28, 2016 9:52 am

Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Kamil Arendt

User avatar
jim wood
Site Admin
Posts: 3601
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by jim wood » Thu Apr 28, 2016 12:45 pm

qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 10 64-bit. TM1 Version: 10.2.2

Bakkone
Posts: 66
Joined: Mon Oct 27, 2014 10:50 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2013

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Bakkone » Thu Apr 28, 2016 2:28 pm

Thanks for the tip!

User avatar
Steve Rowe
Site Admin
Posts: 1802
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: 10.2.2., PAW
Excel Version: Nearly all of them

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Steve Rowe » Thu Apr 28, 2016 2:47 pm

jim wood wrote:
qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....
Wasn't it just because we didn't want to introduce another method for us to annoy each other...... :roll:

User avatar
jim wood
Site Admin
Posts: 3601
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by jim wood » Fri Apr 29, 2016 12:58 pm

Steve Rowe wrote:
jim wood wrote:
qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....
Wasn't it just because we didn't want to introduce another method for us to annoy each other...... :roll:
Sounds about right. I'd like your post but I can't.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 10 64-bit. TM1 Version: 10.2.2

User avatar
macsir
Community Contributor
Posts: 548
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1, SSAS
Version: 9.5 - 10.2
Excel Version: 97 - 2013
Contact:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by macsir » Fri May 06, 2016 2:02 am

Brilliant, i will test it next time in 10.2.2!
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/

Oratia623
Posts: 39
Joined: Mon Apr 27, 2009 5:36 am
OLAP Product: TM1
Version: V7.x to v10.2.2
Excel Version: 2003 2007
Location: Sydney, Australia

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Post by Oratia623 » Tue May 10, 2016 2:27 am

Awesome!!!
We have exactly this situation in the past, and will give this a try next time.
@paulsimon You are THE MAN!

Proof that this community and forum rocks!
Paul Williamson
____________________________________________________________________________________
I came. I saw. I did not concur.

Post Reply