TM1 Server Instance Stops unexpectedly

[Mark RMBC]

I am using TM1 10.1.1

I have recently created a new TM1 server instance in our live/production environment.

I did this by basically manually copying the files from test to live. I created the folder structure in windows explorer first, copied over the CFG file from test, changed the relevant parameters, for example ensured it was pointing to the live environment. Then I copied in the data files and source files from test to the folders in windows explorer live.

Then in Cognos configuration I created the TM1 Server instance and switched it on.

All seemed well and good, data looked correct etc.

However I have a problem, this new TM1 Server instance periodically and unexpectedly shuts down/stops and I have to restart it in Cognos configuration. When looking at the server log I can't see any error messages.

It should be noted that there is no problem in our test environment

Now I understand the above is not enough to pinpoint an exact cause but has anyone encountered this problem and/or does anyone know of any 'usual' culprits that may cause this issue.

regards, Mark

[Paul Segal]

Is it busting memory?

[Neil Watson]

Are you running out of disk space?

Has the server recently had a "Security update?"

[Alan Kirk]

When you said that you changed the relevant parameters, I trust that includes the port number? (I never use the same number series for both dev and production).

More importantly you said that the server logs didn't show anything but have you looked in the Windows Application logs on the server's box?

[Mark RMBC]

Hi,

thanks for the responses

not ignoring you! Just waiting for some relevant feedback from our service providers before providing an adequate response.

I don't think this is busting memory but can't say for certain as yet. Disk space seems adequate.

I do have the same port numbers for test and live but we do this for all our models and this is the only one that crashes. The test and live environments are on completely different remote desktop platforms. And we use different URL's to access them.

Re the logs files the service providers have provided the following:

"There are no errors being reported in the TM1 logs, but they are mentioned in the Windows event logs (albeit nothing useful)."

I did notice a WERFAULT.exe in the task manager the last time it crashed but I also noticed this another time and it didn't crash.

Funnily enough I went onto the server instance last night and this morning, did lots of tasks and it hasn't crashed yet. So it is very difficult for me to recreate the issue. The only times I noticed it crashing was when I logged onto the web or view display control objects in perspectives. However on other occasions I do this it doesn't crash at all.

One thing that happened when I first started the new instance in live is that when i tried to open one of the views I got an error view array message. But when I restarted the server again that problem disappeared.

regards, mark

[Neil Watson]

Hi Mark,

The unrepeatable crashes are the most infuriating.

Assuming the models are ok, and it appears they are, then any error hints towards a possible hardware issue and a corrupt file / memory block.

Can you get IT to run deep diagnostics over both the disk and the RAM?
If any of the files are in a bad sector then this can cause issues, equally if the server tries to address a bad memory sector then a similar thing can happen.

Clutching at straws a bit here

Best of luck
Cheers
Neil

[qml]

Set up Windows Error Reporting to generate a full (type 2) crash dump next time a TM1 service crashes. This is something that will likely be requested by IBM if you open a service request anyway.

[Steve Rowe]

You're almost definitely going to need to upgrade your server side software, rather than chew through a lot of time getting to root cause and then upgrading anyway I'd look at upgrading your systems to a more current release.

Also consider that you are on 10.1.1, this goes off active support April 2017 (I'm told!), so you need / should upgrade soon anyway.

Cheers,

[Mark RMBC]

Hi all,

thanks for taking the time to respond to my query.

I am now almost 99% sure I have resolved this problem but am 1% sure I know why the problem happened in the first place!

I resolved it by basically deleting the server instance and starting again, only this time avoiding certain things I did in the original copy from test to live.

One thing I avoided this time is my theory for why it happened, which I will share:

We have processes to update the chart of accounts dimensions, so cost centre, nominal etc. In test for these processes I ODBC to our test data warehouse.
When I copied over to live one of my first jobs was to link these processes to our live data warehouse.
When I did this for our nominal dimension and ran the process I noticed an orphaned element, i.e. it did not have a parent code. I discovered this was due to the fact that we had set up some nominals in test that did not exist in live. I then decided to delete all the dimension elements except the very top level. However when I went to save the dimension an error message was displayed saying one of the parent elements was used in a rule (in the feeder part). I then hashed out that part of the rule, saved the rule and rebuilt the dimension. Then took the hash off the rule and saved the rule.

Not sure if that caused the issue but can't think of anything else I did that could have caused the problem.

I avoided doing all that second time around by just deleting the single orphaned element

I have also taken Alan's advice and am now keeping separate port numbers for test and live (though I am convinced this wasn't the actual problem).

regards, Mark