Page 1 of 2

How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Tue Apr 26, 2016 3:48 pm
by paulsimon
Hi

I thought I would share this discovery in case it helps anyone else.

On TM1 10.1.1, we had a situation where a user's thread was running a ChangeSetUndo. This had been running for 5000 seconds and was clearly never going to stop.

There were other threads in a Commit Wait state that were presumably unable to Commit because of something the running process was locking.

New users were unable to login

We had one Admin who had already verified Admin access in TM1 Top

The long running thread could not be cancelled in TM1 Top

The Commit Wait threads could not be cancelled in TM1 Top

New users could not login - Login attempt appearing as ???

New attempts to Verify additional Admin access in TM1 Top did not work

It appeared that the only answer was going to be to restart the service. From prior experience the service would almost certainly not shut down and we would therefore have to kill the TM1SD.EXE. This would lose 6 hours of activity on the server since the last backup. Some would be recovered from logs. Other items happen with logging turned off. Effectively the standard recovery would given nothing useful. Therefore having to terminate the process was not an appealing option.

The only remaining way I could think of to affect the server was to change a dynamic TM1S.CFG parameter.

I searched for a TM1S.CFG parameter that was dynamic and found IdleConnectionTimeOutSeconds. I changed since from 3600 to 3601.

After a few seconds TM1 Top suddenly cleared leaving some very happy bunnies.

It seems that perhaps changing this parameter cleared the running and waiting threads without the need to terminate the service.

I noticed that in TM1 Top there seems to be two special threads Th:Pseudo and Th:DynamicConfig. I would guess that the latter watches for changes to the TM1S.CFG on dynamic parameters and does the magic.

Regards

Paul Simon

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Tue Apr 26, 2016 3:53 pm
by tomok
Wow! This almost seems too good to be true!

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Tue Apr 26, 2016 4:28 pm
by declanr
This is a brilliant discovery - one I wish I had heard of years ago - lost count of the amount of times I've had to kill services now.

Does present a worry though in the fact that a change to a dynamic config parameter would presumably also kill any genuine threads that are running at the same point in time; not that I often change any dynamic parameters in a live model but still worth being wary of.
I will do some of my own testing in various versions.

Thanks for sharing.

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Tue Apr 26, 2016 8:25 pm
by lotsaram
This is indeed an amazing discovery, kudos.
Would it be bad karma t say I'm looking forward to trying this out?

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Tue Apr 26, 2016 8:41 pm
by paulsimon
Hi Declan

That was one of my thoughts too. I did a search for any information on the side effects of changing a dynamic config parameter but I couldn't find anything. It does mention that it only polls for changes to the TM1S.CFG every 60 seconds, so I was probably lucky that I changed it on about the 55th seconds of its cycle.

It was a little late in the day but I can check with the user to find out if the journal they were posting to TM1 did get stored in the cube, or whether the change got rolled back. The indication was that they were trying to undo changes. I don't think that they were consciously trying to do that but they might have clicked the undo button by accident. However, I doubt that, as in the system I inherited, most of the TM1 Web sheets are launched from BI, and the TM1 toolbar is usually hidden.

It would be interesting to see if it was the fact that I chose IdleConnectionTimeOut as perhaps the change made it check, although the thread that seemed to be the problem was in a running rather than an idle state.

Regards

Paul Simon

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Tue Apr 26, 2016 10:49 pm
by RJ!
Hmmm... I wonder if this would also clean out those pesky duplicate REST calls from CAFE Users that tend to lock up TM1.

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Wed Apr 27, 2016 10:10 am
by Steve Rowe
+1 from me, post of the decade!

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Wed Apr 27, 2016 12:45 pm
by Elessar
Brilliant!

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Wed Apr 27, 2016 1:03 pm
by jim wood
+1 from me too. I bet even IBM are happy about this. The amount of support calls they must get with issues like this.

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Wed Apr 27, 2016 2:41 pm
by BrianL
Not to be a Debbie Downer, but has behavior been replicated more than once?

Since the only document I can find from IBM that makes an attempt at defining IdleConnectionTimeOut states the parameter only applies to idle client connections, I'm a bit skeptical that this has any impact on thread cancellation.
This parameter specifies a timeout limit for idle client connections, in seconds.
For example, if you include the line IdleConnectionTimeOutSeconds=900 in TM1s.cfg,
the server disconnects idle client connections after 900 seconds
http://www-01.ibm.com/support/docview.w ... wg21407112

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Wed Apr 27, 2016 3:13 pm
by Steve Rowe
I'm guessing not....
but I think what Paul is saying is that the act of refreshing the parameter is what triggers the clearance of locked threads, rather than the value parameter itself. It is not impossible to imagine that when the parameter changes the instance checks for overtime threads in a different enough way to normal that locked threads are dropped.

Of course it could also be a big coincidence as well and TM1 would of sorted it self out any why.

Testing required....

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Wed Apr 27, 2016 3:56 pm
by gtonkin
BrianL wrote:Not to be a Debbie Downer, but has behavior been replicated more than once?
We have used this technique over the years infrequently but repeatedly-always worked when we needed it too-cannot remember having an issue yet. May be jinxed now however.

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Wed Apr 27, 2016 10:01 pm
by paulsimon
Hi

I am not claiming that this is a cure all for all TM1 Server lockup situations. However, in this particular case it worked. As Steve said I believe that it is the act of changing a dynamic parameter rather than which dynamic parameter you change that has the effect.

On the system that I am working with at the moment we can go months without a lockup issue, so it may be some time before I need to find out whether this works in other cases.

Unfortunately we have had other issues today with the Content Store going down which affected CAM and then with intermittent Time Outs in TM1 Web, often happening as soon as the user launches a subset editor from either a Web Sheet or a View, and often shortly after the user logs in, ie not normal time outs. Because of this I have not had a chance to follow up on whether the change to the dynamic parameter had any effect on Threads that were running at the time. Initial investigations appear to indicate that this was not the case, but I need to do more investigation and this may not be possible given the passage of time.

Regards

Paul Simon

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Thu Apr 28, 2016 9:52 am
by qml
Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Thu Apr 28, 2016 12:45 pm
by jim wood
qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Thu Apr 28, 2016 2:28 pm
by Bakkone
Thanks for the tip!

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Thu Apr 28, 2016 2:47 pm
by Steve Rowe
jim wood wrote:
qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....
Wasn't it just because we didn't want to introduce another method for us to annoy each other...... :roll:

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Fri Apr 29, 2016 12:58 pm
by jim wood
Steve Rowe wrote:
jim wood wrote:
qml wrote:Where is the like/upvote/give coins button when you need one? This is a historic finding and it would be good to pool our resources to figure out what this technique can and what it can't do.
Agreed. A like button or something like that has been discussed on a few occasions before. There was a reason why we never added it but I can't remember what it was. May be Alan or Martin Remember....
Wasn't it just because we didn't want to introduce another method for us to annoy each other...... :roll:
Sounds about right. I'd like your post but I can't.

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Fri May 06, 2016 2:02 am
by macsir
Brilliant, i will test it next time in 10.2.2!

Re: How to cancel a thread that cannot be cancelled in TM1 Top

Posted: Tue May 10, 2016 2:27 am
by Oratia623
Awesome!!!
We have exactly this situation in the past, and will give this a try next time.
@paulsimon You are THE MAN!

Proof that this community and forum rocks!