Server freezes in SaveData

Post Reply
upali
Posts: 38
Joined: Thu Oct 11, 2012 6:15 am
OLAP Product: TM1
Version: 10.2.2.4
Excel Version: 2010
Location: Melbourne, Australia

Server freezes in SaveData

Post by upali » Tue Jun 21, 2016 1:15 am

Hi all,

We've got this very strange behaviour in our Windows Prod box, which is not apparent in the Dev box.

Prod: Windows Server 2012, TM1 10.2.2.4, 1.5Tb RAM, 2Tb D:\, 32Cores
Dev: Windows Server 2012, TM1 10.2.2.4, 512Gb RAM, 1Tb D:\, 16Cores (x2 threads)

TM1 Models uses about 450-490Gb of RAM. C:\ Only Windows and System software. D:\ TM1 application and Models.

We are noticing that during SaveDataAll (or CubeSaveData) the server freezes after the following steps of operation.
Saving X.CUB
- Create X.CUB$ and start writing Transaction log entries
- Complete X.CUB$ - All fine
- Delete X.CUB
- Rename X.CUB$ to X.CUB – I’m not sure of the order of the last two steps, but the server freezes during the delete operation.

The bigger the CUB or FEEDER files, the loner the time the server takes to "get back". For example, a 50Gb CUB take about 10mins to write to disk, but another 6-10mins of freeze, during which all access to the server is lost (i.e. no RDP, can't remotely browse the folders, no FTP etc...).

We have excluded the TM1 Application and Data directories from Antivirus - So that is ruled out. If we load the same models into Dev and perform the Save operations, everything is fine. The big boy Prod has the issue, while 'lil Dev works fine on identical software.

I would like to know if you have seen similar issues (and how they were solved), especially if there is some contention of resources between RAM, CPU and HDD during a TM1 Save operation. Any recommendations on the application setup?

Thanks...

BrianL
MVP
Posts: 264
Joined: Mon Nov 03, 2014 8:23 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2 PA2
Excel Version: 2016

Re: Server freezes in SaveData

Post by BrianL » Tue Jun 21, 2016 1:35 am

While I haven't heard of this exact problem, is this network storage that's being written to in prod? Theoretically at least the large file write could saturate that network connection and prevent any of those operations you're describing. Maybe IT could check if a direct console login is also frozen during this time?

upali
Posts: 38
Joined: Thu Oct 11, 2012 6:15 am
OLAP Product: TM1
Version: 10.2.2.4
Excel Version: 2010
Location: Melbourne, Australia

Re: Server freezes in SaveData

Post by upali » Tue Jun 21, 2016 2:07 am

Yes Brian... Storages are in SAN, and we have done diagnostics on Prod and Dev. During the freeze no connections are available to the server, unfortunately.

User avatar
yyi
Community Contributor
Posts: 117
Joined: Thu Aug 28, 2008 4:42 am
Location: Sydney, Australia

Re: Server freezes in SaveData

Post by yyi » Tue Jun 21, 2016 5:54 am

Hmm, I wonder if the x2 threads have something to do with it.
win2012 svr: tm1 v10.2.2 x64, client v10.2.2 x64/PA 2.0, win10, excel 365.

PlanningDev
Community Contributor
Posts: 340
Joined: Tue Aug 17, 2010 6:31 am
OLAP Product: Planning Analytics
Version: 2.0.5
Excel Version: 2016

Re: Server freezes in SaveData

Post by PlanningDev » Thu Jun 23, 2016 5:29 pm

Have you looked to see if you have any Anti Virus Software or any specialized logging software? I know its not TM1 but we had an issue with IBM Watson Content Analytics where some kind of logging program would get triggered when we crawled data for a collection and the server would attempt to build log files that were hundreds of gigs in size and essentially bring down the server.

upali
Posts: 38
Joined: Thu Oct 11, 2012 6:15 am
OLAP Product: TM1
Version: 10.2.2.4
Excel Version: 2010
Location: Melbourne, Australia

Re: Server freezes in SaveData

Post by upali » Fri Jun 24, 2016 7:22 am

PlanningDev wrote:Have you looked to see if you have any Anti Virus Software or any specialized logging software? I know its not TM1 but we had an issue with IBM Watson Content Analytics where some kind of logging program would get triggered when we crawled data for a collection and the server would attempt to build log files that were hundreds of gigs in size and essentially bring down the server.
AV has an exclusion on all TM1 related directories, an no other logging programs are installed. We have also raised an incident with IBM, but still no solution.

Post Reply