Server Locked after save data process and save data is running for a long time .

Post Reply
Analytics123
Posts: 127
Joined: Tue May 23, 2017 12:45 pm
OLAP Product: Tm1
Version: 9.5
Excel Version: 2010

Server Locked after save data process and save data is running for a long time .

Post by Analytics123 » Mon Apr 23, 2018 9:29 pm

Hi ,

We recently upgraded from 10.1 to planning analytics 2.0.4 .

We have an issue where save data is taking a long time to run only in the afternoon and locks up all users . Save data is scheduled to run every 6 hours .

We have confirmed that there is no virus scan being run on the system during that time .

So here is what we have in the logs .

8160 [] ERROR 2018-04-23 17:30:35.766 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:37.778 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:39.791 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:41.803 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:43.816 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:45.828 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:47.840 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:49.853 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:51.865 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:53.878 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:55.890 TM1.Server sf_SafeDeleteFile() could not delete file, "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$" after 10 retrys in 20 seconds
8160 [] ERROR 2018-04-23 17:58:47.867 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Global Sales}vues\z_vwZeroOut-APAC.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Global Sales}vues\z_vwZeroOut-APAC.vue$"

Server comes back after restart , but half of the day , we have users locked up .


Any suggestions to trouble shoot the issue is appreciated .

User avatar
macsir
Community Contributor
Posts: 507
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1, SSAS
Version: 9.5 - 10.2
Excel Version: 97 - 2013
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by macsir » Mon Apr 23, 2018 9:46 pm

You probably can use process explorer to check what process is locking up your files at that time and go from there.
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/

lotsaram
MVP
Posts: 3052
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TM1, CX
Version: TM1 10.2.2 PA 2.0x
Excel Version: 2010 2013 365
Location: Switzerland

Re: Server Locked after save data process and save data is running for a long time .

Post by lotsaram » Tue Apr 24, 2018 5:44 am

Your issue is real. On PA instances I have also seen a lot of sf_DeleteFile and sf_SafeDelete errors, always to do with sub and/or vue files. It doesn't always occur on SaveData, sometimes it happens during TI process execution where a process that should execute in less than a second can drag on for minutes before finally completing. On some models this causes massive performance issues. I am however currently failing to convince IBM support that this is an issue so I don't know that anyone from engineering who could actually diagnose and fix it would actually know about it. I wish you better luck with your support ticket.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
Steve Rowe
Site Admin
Posts: 1751
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: 10.2.2., PAW
Excel Version: Nearly all of them

Re: Server Locked after save data process and save data is running for a long time .

Post by Steve Rowe » Tue Apr 24, 2018 2:43 pm

I'd have a think about how those views are used and created.

Going from the naming I'd guess they are part of the data load.
How are they created (i.e. permanent, old style built in a TI view, new temporary views built in a TI)
How are they deleted (no need, at the end of the job, we don't bother as they are temporary )
Did the job involved complete without error?
How (timewise) close is the save data to the creation / usage of the views?
Are the $ files left behind in the DD due to some legacy error and if they get deleted does the issue resolve?

Pretty sure I'm on this release (latest as of around 4 weeks ago) and I've not encountered this issue (touch wood, yes my desk is made entirely of wood). We always create old style permanent views and delete them and their subsets as soon as we're finished with them. In general we don't do save data at the end of a load job.

HTH

kangkc
Community Contributor
Posts: 181
Joined: Fri Oct 17, 2008 2:40 am
OLAP Product: TM1
Version: 9.x
Excel Version: 200x
Location: Singapore

Re: Server Locked after save data process and save data is running for a long time .

Post by kangkc » Wed Apr 25, 2018 1:12 pm

lotsaram wrote:
Tue Apr 24, 2018 5:44 am
Your issue is real. On PA instances I have also seen a lot of sf_DeleteFile and sf_SafeDelete errors, always to do with sub and/or vue files. It doesn't always occur on SaveData, sometimes it happens during TI process execution where a process that should execute in less than a second can drag on for minutes before finally completing. On some models this causes massive performance issues. I am however currently failing to convince IBM support that this is an issue so I don't know that anyone from engineering who could actually diagnose and fix it would actually know about it. I wish you better luck with your support ticket.
Very real indeed, I have seen two PA 2.0.3 and 2.0.1 environments having similar sf_DeleteFile errors but seems to be during savedata run. Support ticket to IBM didn't really help as we need to have a consistent test case but such problem comes randomly.
I don't recall seeing such issue in 10.2.2 era.

Analytics123
Posts: 127
Joined: Tue May 23, 2017 12:45 pm
OLAP Product: Tm1
Version: 9.5
Excel Version: 2010

TM1 Save Data Issue 2.0.4

Post by Analytics123 » Thu May 10, 2018 2:16 pm

Hi ,

We recently upgraded from 10.1 to planning analytics 2.0.4 .

We have an issue where save data is taking a long time to run only in the afternoon and locks up all users . Save data is scheduled to run every 6 hours .

We have confirmed that there is no virus scan being run on the system during that time .


so we were able to finally trouble shoot the issue and it happens when there is a overlap in chore timings. When one schedule chore starts before the previous one completes . This was never an issue in 10.1 . so this locks up the server on save data all with errror code 32 .
Can anyone help with this . we are modifiying the chore timings as that they dont overlap . but sometimes it happens out of our control .

Can I have one TI and run these chores in an order synchronously . so that one dont start when the previous still runs. So is this possible to do



So here is what we have in the logs .

8160 [] ERROR 2018-04-23 17:30:35.766 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:37.778 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"

Admin : Moved this new post onto the this thread as it is all part of the same problem and hopefully solution

User avatar
tomok
MVP
Posts: 2427
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: TM1 Save Data Issue 2.0.4

Post by tomok » Thu May 10, 2018 2:31 pm

Analytics123 wrote:
Thu May 10, 2018 2:16 pm
Can I have one TI and run these chores in an order synchronously . so that one dont start when the previous still runs. So is this possible to do [/b]
You can't run a chore from a TI, only other processes. What stopping you from combining everything into one chore so that they are run consecutively? You could also put everything into a master TI process that calls all the others from ExecuteProcess and then put that single process into a chore.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

Post Reply