Server Locked after save data process and save data is running for a long time .

Post Reply
Analytics123
Posts: 128
Joined: Tue May 23, 2017 12:45 pm
OLAP Product: Tm1
Version: 9.5
Excel Version: 2010

Server Locked after save data process and save data is running for a long time .

Post by Analytics123 »

Hi ,

We recently upgraded from 10.1 to planning analytics 2.0.4 .

We have an issue where save data is taking a long time to run only in the afternoon and locks up all users . Save data is scheduled to run every 6 hours .

We have confirmed that there is no virus scan being run on the system during that time .

So here is what we have in the logs .

8160 [] ERROR 2018-04-23 17:30:35.766 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:37.778 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:39.791 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:41.803 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:43.816 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:45.828 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:47.840 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:49.853 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:51.865 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:53.878 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:55.890 TM1.Server sf_SafeDeleteFile() could not delete file, "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$" after 10 retrys in 20 seconds
8160 [] ERROR 2018-04-23 17:58:47.867 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Global Sales}vues\z_vwZeroOut-APAC.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Global Sales}vues\z_vwZeroOut-APAC.vue$"

Server comes back after restart , but half of the day , we have users locked up .


Any suggestions to trouble shoot the issue is appreciated .
User avatar
macsir
MVP
Posts: 782
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1
Version: PAL 2.0.9
Excel Version: Office 365
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by macsir »

You probably can use process explorer to check what process is locking up your files at that time and go from there.
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Server Locked after save data process and save data is running for a long time .

Post by lotsaram »

Your issue is real. On PA instances I have also seen a lot of sf_DeleteFile and sf_SafeDelete errors, always to do with sub and/or vue files. It doesn't always occur on SaveData, sometimes it happens during TI process execution where a process that should execute in less than a second can drag on for minutes before finally completing. On some models this causes massive performance issues. I am however currently failing to convince IBM support that this is an issue so I don't know that anyone from engineering who could actually diagnose and fix it would actually know about it. I wish you better luck with your support ticket.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
User avatar
Steve Rowe
Site Admin
Posts: 2410
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: Server Locked after save data process and save data is running for a long time .

Post by Steve Rowe »

I'd have a think about how those views are used and created.

Going from the naming I'd guess they are part of the data load.
How are they created (i.e. permanent, old style built in a TI view, new temporary views built in a TI)
How are they deleted (no need, at the end of the job, we don't bother as they are temporary )
Did the job involved complete without error?
How (timewise) close is the save data to the creation / usage of the views?
Are the $ files left behind in the DD due to some legacy error and if they get deleted does the issue resolve?

Pretty sure I'm on this release (latest as of around 4 weeks ago) and I've not encountered this issue (touch wood, yes my desk is made entirely of wood). We always create old style permanent views and delete them and their subsets as soon as we're finished with them. In general we don't do save data at the end of a load job.

HTH
Technical Director
www.infocat.co.uk
kangkc
Community Contributor
Posts: 206
Joined: Fri Oct 17, 2008 2:40 am
OLAP Product: TM1, PA , TMVGate
Version: 2.x
Excel Version: 36x
Location: Singapore
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by kangkc »

lotsaram wrote: Tue Apr 24, 2018 5:44 am Your issue is real. On PA instances I have also seen a lot of sf_DeleteFile and sf_SafeDelete errors, always to do with sub and/or vue files. It doesn't always occur on SaveData, sometimes it happens during TI process execution where a process that should execute in less than a second can drag on for minutes before finally completing. On some models this causes massive performance issues. I am however currently failing to convince IBM support that this is an issue so I don't know that anyone from engineering who could actually diagnose and fix it would actually know about it. I wish you better luck with your support ticket.
Very real indeed, I have seen two PA 2.0.3 and 2.0.1 environments having similar sf_DeleteFile errors but seems to be during savedata run. Support ticket to IBM didn't really help as we need to have a consistent test case but such problem comes randomly.
I don't recall seeing such issue in 10.2.2 era.
Analytics123
Posts: 128
Joined: Tue May 23, 2017 12:45 pm
OLAP Product: Tm1
Version: 9.5
Excel Version: 2010

TM1 Save Data Issue 2.0.4

Post by Analytics123 »

Hi ,

We recently upgraded from 10.1 to planning analytics 2.0.4 .

We have an issue where save data is taking a long time to run only in the afternoon and locks up all users . Save data is scheduled to run every 6 hours .

We have confirmed that there is no virus scan being run on the system during that time .


so we were able to finally trouble shoot the issue and it happens when there is a overlap in chore timings. When one schedule chore starts before the previous one completes . This was never an issue in 10.1 . so this locks up the server on save data all with errror code 32 .
Can anyone help with this . we are modifiying the chore timings as that they dont overlap . but sometimes it happens out of our control .

Can I have one TI and run these chores in an order synchronously . so that one dont start when the previous still runs. So is this possible to do



So here is what we have in the logs .

8160 [] ERROR 2018-04-23 17:30:35.766 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:37.778 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"

Admin : Moved this new post onto the this thread as it is all part of the same problem and hopefully solution
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: TM1 Save Data Issue 2.0.4

Post by tomok »

Analytics123 wrote: Thu May 10, 2018 2:16 pm Can I have one TI and run these chores in an order synchronously . so that one dont start when the previous still runs. So is this possible to do [/b]
You can't run a chore from a TI, only other processes. What stopping you from combining everything into one chore so that they are run consecutively? You could also put everything into a master TI process that calls all the others from ExecuteProcess and then put that single process into a chore.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
pcj
Posts: 1
Joined: Wed May 23, 2018 6:28 pm
OLAP Product: TM1 / PA
Version: 2.0.4
Excel Version: 2016

Re: Server Locked after save data process and save data is running for a long time .

Post by pcj »

Update: I work on the same team as the OP, and we're still having this issue despite chores NOT overlapping anymore, so this is not the cause. It is however, always triggered by a SAVE DATA ALL.

To answer macsir, Process Explorer only reveals that TM1 itself is locking the object for deletion, nothing else. So it has proven useless. :|

IBM refuses to do anything more unless we hand over our entire model and data, which we do not do. We're unable to replicate the issue in a sample server, since we do not know the cause of it.

One thing we have noticed is the issue does not occur early in the AM after the chores ran and we do a SAVE DATA ALL, when users are not active in the system, nor does it occur in the late PM after users are no longer in the system. To add to the confusion, however, the objects that are causing the issues (vue, sub) are all TI Process related objects, not user objects. So this would reduce possible correlation of user activity possibly impacting the SAVE DATA ALL.

For the time being, to avoid this issue we're running SAVE DATA ALL only outside normal business hours. Obviously this is not our preference but we're out of solutions.

Thank you all for your input thus far and confirming our issue is real.
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by tomok »

pcj wrote: Fri May 25, 2018 1:32 pm IBM refuses to do anything more unless we hand over our entire model and data, which we do not do.
Why would you not? "Hand over" is probably not good wording. You do know that their license agreement with you precludes them from disclosing any of your data, anywhere, for any reason? I guess your models must store the launch codes for the US nuclear missile arsenal. :?
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
User avatar
jim wood
Site Admin
Posts: 3951
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: PA 2.0.7
Excel Version: Office 365
Location: 37 East 18th Street New York
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by jim wood »

tomok wrote: Fri May 25, 2018 3:12 pm
pcj wrote: Fri May 25, 2018 1:32 pm IBM refuses to do anything more unless we hand over our entire model and data, which we do not do.
Why would you not? "Hand over" is probably not good wording. You do know that their license agreement with you precludes them from disclosing any of your data, anywhere, for any reason? I guess your models must store the launch codes for the US nuclear missile arsenal. :?
If it were only that simple eh? This level of security is not unusual regardless of agreements in place. Next you'll be telling me industrial espionage is just a myth? Oh let's not forget about hackers. If somebody hacks in to IBM and steals the info from them who do you think within the original company would get the blame. Personally I wouldn't care what agreement is in place. If I was to hand over a model I'd make sure it was wiped and populated with random data first. When I used to work for BAe we had to sign the official secrets act. I very doubt many IBM employees have done so. An extreme example I know, but you asked why not and I've given you several.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by tomok »

jim wood wrote: Fri May 25, 2018 4:51 pm Next you'll be telling me industrial espionage is just a myth? Oh let's not forget about hackers. If somebody hacks in to IBM and steals the info from them who do you think within the original company would get the blame.
I guess if your model and data is "espionage-worthy" then this would be a valid reason. I personally have never worked on anything in TM1 that I would classify as such. Perhaps a few that might give the hacker some insider information, depending on timing of the hack, but certainly nothing that would really be earth shattering if others got their hands on it.

My point is that IBM's request for the model and data is valid. If you choose not to share it with them then you forfeit your right to whine about them not being able to help you with a randomly occurring issue.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
User avatar
PavoGa
MVP
Posts: 616
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.9.1
Excel Version: 2013 PAW
Location: Charleston, Tennessee

Re: Server Locked after save data process and save data is running for a long time .

Post by PavoGa »

tomok wrote: Fri May 25, 2018 3:12 pm
pcj wrote: Fri May 25, 2018 1:32 pm IBM refuses to do anything more unless we hand over our entire model and data, which we do not do.
Why would you not? "Hand over" is probably not good wording. You do know that their license agreement with you precludes them from disclosing any of your data, anywhere, for any reason? I guess your models must store the launch codes for the US nuclear missile arsenal. :?
Not to be piling on, but contractual obligations may also prevent handing over a model to IBM, which is exactly the case with the current models I'm working on.
Ty
Cleveland, TN
User avatar
jim wood
Site Admin
Posts: 3951
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: PA 2.0.7
Excel Version: Office 365
Location: 37 East 18th Street New York
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by jim wood »

tomok wrote: Fri May 25, 2018 5:22 pm My point is that IBM's request for the model and data is valid. If you choose not to share it with them then you forfeit your right to whine about them not being able to help you with a randomly occurring issue.
If only everything was that black and white. The day you're stuck on a client that does have data that has controls in place and you're stuck not being able to send it over I'll be first one pointing at you and laughing. Just because you haven't been in this is this situation it doesn't mean it's not valid (Well obviously it does in Tomok land mind) and the challenge when dealing with the politics surrounding this is not easy. Keep in mind if you do populate it with random data, you have to try and make sure it's not so random that all of sudden numbers don't make sense.

So please Tomok don't just point fingers and throw mud, any old hack can do that. I just did it above, it's not nice and it achieves nothing. As you'll have heard many times, if you don't have anything good to say, don't say it.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by tomok »

jim wood wrote: Tue May 29, 2018 11:49 am So please Tomok don't just point fingers and throw mud, any old hack can do that. I just did it above, it's not nice and it achieves nothing. As you'll have heard many times, if you don't have anything good to say, don't say it.
[hack in]Wow! I'm sorry that I made you feel it necessary to raise the level of vitriol so high and turn this into a personal attack on me. I just presented a viewpoint different than yours and made a generic statement (not necessarily directed at you) that someone who doesn't provide their model to IBM forfeits their right to complain about IBM support not helping them.[/hack out]
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
User avatar
jim wood
Site Admin
Posts: 3951
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: PA 2.0.7
Excel Version: Office 365
Location: 37 East 18th Street New York
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by jim wood »

tomok wrote: Tue May 29, 2018 12:58 pm
jim wood wrote: Tue May 29, 2018 11:49 am So please Tomok don't just point fingers and throw mud, any old hack can do that. I just did it above, it's not nice and it achieves nothing. As you'll have heard many times, if you don't have anything good to say, don't say it.
[hack in]Wow! I'm sorry that I made you feel it necessary to raise the level of vitriol so high and turn this into a personal attack on me. I just presented a viewpoint different than yours and made a generic statement (not necessarily directed at you) that someone who doesn't provide their model to IBM forfeits their right to complain about IBM support not helping them.[/hack out]
That's better. A reasonable point well made. The attack was a little aggressive I give you that. The reason is you don't just offer an opinion. You offer it wrapped in a way that belittles people, which is personal. So do me and the rest of us a favor and take a step back before posting something in that way.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by Alan Kirk »

I don't believe this; they STILL haven't fixed this as of 2.0.6.

I kept getting:

Code: Select all

12608   [648]   ERROR   2019-08-20 12:15:01.891   TM1.Server   sf_DeleteFile: Error on file "Path:\CubeName}vues\zRBXZeroOut.vue$". Error = error code:32  reason:"The process cannot access the file because it is being used by another process."   file:"Path:\CubeName}vues\zRBXZeroOut.vue$" 
12608   [648]   ERROR   2019-08-20 12:15:03.903   TM1.Server   sf_SafeDeleteFile() could not delete file, "Path:\CubeName}vues\zRBXZeroOut.vue$" after 10 retrys in 20 seconds
12608   [648]   ERROR   2019-08-20 12:15:03.903   TM1.Server   sf_DeleteFile: Error on file "Path:\CubeName}vues\zRPLZeroOut.vue$". Error = error code:32  reason:"The process cannot access the file because it is being used by another process."   file:"Path:\CubeName}vues\zRPLZeroOut.vue$" 
12608   [648]   ERROR   2019-08-20 12:15:05.916   TM1.Server   sf_DeleteFile: Error on file "Path:\CubeName}vues\zRPLZeroOut.vue$". Error = error code:32  reason:"The process cannot access the file because it is being used by another process."   file:"Path:\CubeName}vues\zRPLZeroOut.vue$" 
12608   [648]   ERROR   2019-08-20 12:15:07.928   TM1.Server   sf_DeleteFile: Error on file "Path:\CubeName}vues\zRPLZeroOut.vue$". Error = error code:32  reason:"The process cannot access the file because it is being used by another process."   file:"Path:\CubeName}vues\zRPLZeroOut.vue$" 
12608   [648]   ERROR   2019-08-20 12:15:09.941   TM1.Server   sf_DeleteFile: Error on file "Path:\CubeName}vues\zRPLZeroOut.vue$". Error = error code:32  reason:"The process cannot access the file because it is being used by another process."   file:"Path:\CubeName}vues\zRPLZeroOut.vue$" 
scattered all through the logs.

What the expletive deleted, I'm not deleting that view. It's supposed to stay put.

I did a search, found this thread, and no I am not handing over my frapping model and yes I WILL complain about basic file system operations not working.

In my chore the master process calls two sub-processes which use the view to zero out the existing values prior to uploading. (Yes, it's a short term hack to cover a transitional period which is why I'm not using temporary objects for this.)

The SaveDataAll is done in the epilog of the master process.

I did a quick and dirty test in a dev environment to try to partially replicate this (the subordinate processes do a ViewZeroOut but don't do an actual upload), but that didn't trigger it. Nonetheless I suspect that the issue lays somewhere down this path. If I had the time which at the moment I don't, I'd try to replicate a generic situation where it fails, but for the moment I just commented out the Save Data All.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
Post Reply