Server Locked after save data process and save data is running for a long time .

Post Reply
Analytics123
Posts: 128
Joined: Tue May 23, 2017 12:45 pm
OLAP Product: Tm1
Version: 9.5
Excel Version: 2010

Server Locked after save data process and save data is running for a long time .

Post by Analytics123 » Mon Apr 23, 2018 9:29 pm

Hi ,

We recently upgraded from 10.1 to planning analytics 2.0.4 .

We have an issue where save data is taking a long time to run only in the afternoon and locks up all users . Save data is scheduled to run every 6 hours .

We have confirmed that there is no virus scan being run on the system during that time .

So here is what we have in the logs .

8160 [] ERROR 2018-04-23 17:30:35.766 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:37.778 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:39.791 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:41.803 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:43.816 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:45.828 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:47.840 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:49.853 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:51.865 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:53.878 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:55.890 TM1.Server sf_SafeDeleteFile() could not delete file, "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$" after 10 retrys in 20 seconds
8160 [] ERROR 2018-04-23 17:58:47.867 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Global Sales}vues\z_vwZeroOut-APAC.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Global Sales}vues\z_vwZeroOut-APAC.vue$"

Server comes back after restart , but half of the day , we have users locked up .


Any suggestions to trouble shoot the issue is appreciated .

User avatar
macsir
Community Contributor
Posts: 520
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1, SSAS
Version: 9.5 - 10.2
Excel Version: 97 - 2013
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by macsir » Mon Apr 23, 2018 9:46 pm

You probably can use process explorer to check what process is locking up your files at that time and go from there.
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/

lotsaram
MVP
Posts: 3093
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TM1, CX
Version: TM1 10.2.2 PA 2.0x
Excel Version: 2010 2013 365
Location: Switzerland

Re: Server Locked after save data process and save data is running for a long time .

Post by lotsaram » Tue Apr 24, 2018 5:44 am

Your issue is real. On PA instances I have also seen a lot of sf_DeleteFile and sf_SafeDelete errors, always to do with sub and/or vue files. It doesn't always occur on SaveData, sometimes it happens during TI process execution where a process that should execute in less than a second can drag on for minutes before finally completing. On some models this causes massive performance issues. I am however currently failing to convince IBM support that this is an issue so I don't know that anyone from engineering who could actually diagnose and fix it would actually know about it. I wish you better luck with your support ticket.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
Steve Rowe
Site Admin
Posts: 1783
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: 10.2.2., PAW
Excel Version: Nearly all of them

Re: Server Locked after save data process and save data is running for a long time .

Post by Steve Rowe » Tue Apr 24, 2018 2:43 pm

I'd have a think about how those views are used and created.

Going from the naming I'd guess they are part of the data load.
How are they created (i.e. permanent, old style built in a TI view, new temporary views built in a TI)
How are they deleted (no need, at the end of the job, we don't bother as they are temporary )
Did the job involved complete without error?
How (timewise) close is the save data to the creation / usage of the views?
Are the $ files left behind in the DD due to some legacy error and if they get deleted does the issue resolve?

Pretty sure I'm on this release (latest as of around 4 weeks ago) and I've not encountered this issue (touch wood, yes my desk is made entirely of wood). We always create old style permanent views and delete them and their subsets as soon as we're finished with them. In general we don't do save data at the end of a load job.

HTH

kangkc
Community Contributor
Posts: 181
Joined: Fri Oct 17, 2008 2:40 am
OLAP Product: TM1
Version: 9.x
Excel Version: 200x
Location: Singapore

Re: Server Locked after save data process and save data is running for a long time .

Post by kangkc » Wed Apr 25, 2018 1:12 pm

lotsaram wrote:
Tue Apr 24, 2018 5:44 am
Your issue is real. On PA instances I have also seen a lot of sf_DeleteFile and sf_SafeDelete errors, always to do with sub and/or vue files. It doesn't always occur on SaveData, sometimes it happens during TI process execution where a process that should execute in less than a second can drag on for minutes before finally completing. On some models this causes massive performance issues. I am however currently failing to convince IBM support that this is an issue so I don't know that anyone from engineering who could actually diagnose and fix it would actually know about it. I wish you better luck with your support ticket.
Very real indeed, I have seen two PA 2.0.3 and 2.0.1 environments having similar sf_DeleteFile errors but seems to be during savedata run. Support ticket to IBM didn't really help as we need to have a consistent test case but such problem comes randomly.
I don't recall seeing such issue in 10.2.2 era.

Analytics123
Posts: 128
Joined: Tue May 23, 2017 12:45 pm
OLAP Product: Tm1
Version: 9.5
Excel Version: 2010

TM1 Save Data Issue 2.0.4

Post by Analytics123 » Thu May 10, 2018 2:16 pm

Hi ,

We recently upgraded from 10.1 to planning analytics 2.0.4 .

We have an issue where save data is taking a long time to run only in the afternoon and locks up all users . Save data is scheduled to run every 6 hours .

We have confirmed that there is no virus scan being run on the system during that time .


so we were able to finally trouble shoot the issue and it happens when there is a overlap in chore timings. When one schedule chore starts before the previous one completes . This was never an issue in 10.1 . so this locks up the server on save data all with errror code 32 .
Can anyone help with this . we are modifiying the chore timings as that they dont overlap . but sometimes it happens out of our control .

Can I have one TI and run these chores in an order synchronously . so that one dont start when the previous still runs. So is this possible to do



So here is what we have in the logs .

8160 [] ERROR 2018-04-23 17:30:35.766 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"
8160 [] ERROR 2018-04-23 17:30:37.778 TM1.Server sf_DeleteFile: Error on file "d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$". Error = error code:32 reason:"The process cannot access the file because it is being used by another process." file:"d:\inc-internal\data\Cost Sets}vues\z_vwZeroOut.vue$"

Admin : Moved this new post onto the this thread as it is all part of the same problem and hopefully solution

User avatar
tomok
MVP
Posts: 2464
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: TM1 Save Data Issue 2.0.4

Post by tomok » Thu May 10, 2018 2:31 pm

Analytics123 wrote:
Thu May 10, 2018 2:16 pm
Can I have one TI and run these chores in an order synchronously . so that one dont start when the previous still runs. So is this possible to do [/b]
You can't run a chore from a TI, only other processes. What stopping you from combining everything into one chore so that they are run consecutively? You could also put everything into a master TI process that calls all the others from ExecuteProcess and then put that single process into a chore.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

pcj
Posts: 1
Joined: Wed May 23, 2018 6:28 pm
OLAP Product: TM1 / PA
Version: 2.0.4
Excel Version: 2016

Re: Server Locked after save data process and save data is running for a long time .

Post by pcj » Fri May 25, 2018 1:32 pm

Update: I work on the same team as the OP, and we're still having this issue despite chores NOT overlapping anymore, so this is not the cause. It is however, always triggered by a SAVE DATA ALL.

To answer macsir, Process Explorer only reveals that TM1 itself is locking the object for deletion, nothing else. So it has proven useless. :|

IBM refuses to do anything more unless we hand over our entire model and data, which we do not do. We're unable to replicate the issue in a sample server, since we do not know the cause of it.

One thing we have noticed is the issue does not occur early in the AM after the chores ran and we do a SAVE DATA ALL, when users are not active in the system, nor does it occur in the late PM after users are no longer in the system. To add to the confusion, however, the objects that are causing the issues (vue, sub) are all TI Process related objects, not user objects. So this would reduce possible correlation of user activity possibly impacting the SAVE DATA ALL.

For the time being, to avoid this issue we're running SAVE DATA ALL only outside normal business hours. Obviously this is not our preference but we're out of solutions.

Thank you all for your input thus far and confirming our issue is real.

User avatar
tomok
MVP
Posts: 2464
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by tomok » Fri May 25, 2018 3:12 pm

pcj wrote:
Fri May 25, 2018 1:32 pm
IBM refuses to do anything more unless we hand over our entire model and data, which we do not do.
Why would you not? "Hand over" is probably not good wording. You do know that their license agreement with you precludes them from disclosing any of your data, anywhere, for any reason? I guess your models must store the launch codes for the US nuclear missile arsenal. :?
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

User avatar
jim wood
Site Admin
Posts: 3577
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by jim wood » Fri May 25, 2018 4:51 pm

tomok wrote:
Fri May 25, 2018 3:12 pm
pcj wrote:
Fri May 25, 2018 1:32 pm
IBM refuses to do anything more unless we hand over our entire model and data, which we do not do.
Why would you not? "Hand over" is probably not good wording. You do know that their license agreement with you precludes them from disclosing any of your data, anywhere, for any reason? I guess your models must store the launch codes for the US nuclear missile arsenal. :?
If it were only that simple eh? This level of security is not unusual regardless of agreements in place. Next you'll be telling me industrial espionage is just a myth? Oh let's not forget about hackers. If somebody hacks in to IBM and steals the info from them who do you think within the original company would get the blame. Personally I wouldn't care what agreement is in place. If I was to hand over a model I'd make sure it was wiped and populated with random data first. When I used to work for BAe we had to sign the official secrets act. I very doubt many IBM employees have done so. An extreme example I know, but you asked why not and I've given you several.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 10 64-bit. TM1 Version: 10.2.2

User avatar
tomok
MVP
Posts: 2464
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by tomok » Fri May 25, 2018 5:22 pm

jim wood wrote:
Fri May 25, 2018 4:51 pm
Next you'll be telling me industrial espionage is just a myth? Oh let's not forget about hackers. If somebody hacks in to IBM and steals the info from them who do you think within the original company would get the blame.
I guess if your model and data is "espionage-worthy" then this would be a valid reason. I personally have never worked on anything in TM1 that I would classify as such. Perhaps a few that might give the hacker some insider information, depending on timing of the hack, but certainly nothing that would really be earth shattering if others got their hands on it.

My point is that IBM's request for the model and data is valid. If you choose not to share it with them then you forfeit your right to whine about them not being able to help you with a randomly occurring issue.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

User avatar
PavoGa
Community Contributor
Posts: 203
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0
Excel Version: 2013
Location: Cleveland, Tennessee

Re: Server Locked after save data process and save data is running for a long time .

Post by PavoGa » Tue May 29, 2018 10:32 am

tomok wrote:
Fri May 25, 2018 3:12 pm
pcj wrote:
Fri May 25, 2018 1:32 pm
IBM refuses to do anything more unless we hand over our entire model and data, which we do not do.
Why would you not? "Hand over" is probably not good wording. You do know that their license agreement with you precludes them from disclosing any of your data, anywhere, for any reason? I guess your models must store the launch codes for the US nuclear missile arsenal. :?
Not to be piling on, but contractual obligations may also prevent handing over a model to IBM, which is exactly the case with the current models I'm working on.
Ty
Cleveland, TN

User avatar
jim wood
Site Admin
Posts: 3577
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by jim wood » Tue May 29, 2018 11:49 am

tomok wrote:
Fri May 25, 2018 5:22 pm
My point is that IBM's request for the model and data is valid. If you choose not to share it with them then you forfeit your right to whine about them not being able to help you with a randomly occurring issue.
If only everything was that black and white. The day you're stuck on a client that does have data that has controls in place and you're stuck not being able to send it over I'll be first one pointing at you and laughing. Just because you haven't been in this is this situation it doesn't mean it's not valid (Well obviously it does in Tomok land mind) and the challenge when dealing with the politics surrounding this is not easy. Keep in mind if you do populate it with random data, you have to try and make sure it's not so random that all of sudden numbers don't make sense.

So please Tomok don't just point fingers and throw mud, any old hack can do that. I just did it above, it's not nice and it achieves nothing. As you'll have heard many times, if you don't have anything good to say, don't say it.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 10 64-bit. TM1 Version: 10.2.2

User avatar
tomok
MVP
Posts: 2464
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by tomok » Tue May 29, 2018 12:58 pm

jim wood wrote:
Tue May 29, 2018 11:49 am
So please Tomok don't just point fingers and throw mud, any old hack can do that. I just did it above, it's not nice and it achieves nothing. As you'll have heard many times, if you don't have anything good to say, don't say it.
[hack in]Wow! I'm sorry that I made you feel it necessary to raise the level of vitriol so high and turn this into a personal attack on me. I just presented a viewpoint different than yours and made a generic statement (not necessarily directed at you) that someone who doesn't provide their model to IBM forfeits their right to complain about IBM support not helping them.[/hack out]
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

User avatar
jim wood
Site Admin
Posts: 3577
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: Server Locked after save data process and save data is running for a long time .

Post by jim wood » Wed May 30, 2018 12:31 pm

tomok wrote:
Tue May 29, 2018 12:58 pm
jim wood wrote:
Tue May 29, 2018 11:49 am
So please Tomok don't just point fingers and throw mud, any old hack can do that. I just did it above, it's not nice and it achieves nothing. As you'll have heard many times, if you don't have anything good to say, don't say it.
[hack in]Wow! I'm sorry that I made you feel it necessary to raise the level of vitriol so high and turn this into a personal attack on me. I just presented a viewpoint different than yours and made a generic statement (not necessarily directed at you) that someone who doesn't provide their model to IBM forfeits their right to complain about IBM support not helping them.[/hack out]
That's better. A reasonable point well made. The attack was a little aggressive I give you that. The reason is you don't just offer an opinion. You offer it wrapped in a way that belittles people, which is personal. So do me and the rest of us a favor and take a step back before posting something in that way.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 10 64-bit. TM1 Version: 10.2.2

Post Reply