Order of loading files in memory

Post Reply
Wim Gielis
MVP
Posts: 2278
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Order of loading files in memory

Post by Wim Gielis » Wed Nov 27, 2019 9:50 am

Hello,

Just a question that popped up with me.

I (re)start a TM1 model, let's say 20 GB and 5 minutes of loading time.
Does anybody know in what order, more or less, files are read into memory ?
The question is, after starting a model, can I still change the contents of a PRO file or copy/paste a PRO file in the TM1 data directory such that it is still picked during loading ? IF processes are "loaded" at the end of the transaction, this would avoid starting all over again with the load.

My real user case is a TM1 model of 115 GB where I needed to import a couple of TI processes but I forgot... until 80 GB was loaded and half an hour was gone :o

Thanks
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

Napshot
Posts: 6
Joined: Thu Aug 02, 2018 9:13 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: Perspective

Re: Order of loading files in memory

Post by Napshot » Wed Nov 27, 2019 11:22 am

Hey
I did some small test on this and from what i can see processes are some of the last that are loaded in. I could copy/delete past *.pro in to the data library at around 90% (might be abit higher than that) and see the effects at startup.

So in a order like this, based on the log file:

Cube headers > Dimension > Rules > Cube body > feeders >processes > chores

Morten

User avatar
gtonkin
MVP
Posts: 742
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: Order of loading files in memory

Post by gtonkin » Wed Nov 27, 2019 11:47 am

Wim Gielis wrote:
Wed Nov 27, 2019 9:50 am
...
My real user case is a TM1 model of 115 GB where I needed to import a couple of TI processes but I forgot... until 80 GB was loaded and half an hour was gone...
Would be useful to be able to use AutoHotKey to effectively retype the necessary details from your non-prod to prod environment then save/compile.
A nice project for those long weekends!

Wim Gielis
MVP
Posts: 2278
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Order of loading files in memory

Post by Wim Gielis » Wed Nov 27, 2019 11:49 am

gtonkin wrote:
Wed Nov 27, 2019 11:47 am
Wim Gielis wrote:
Wed Nov 27, 2019 9:50 am
...
My real user case is a TM1 model of 115 GB where I needed to import a couple of TI processes but I forgot... until 80 GB was loaded and half an hour was gone...
Would be useful to be able to use AutoHotKey to effectively retype the necessary details from your non-prod to prod environment then save/compile.
A nice project for those long weekends!
Indeed, use the TM1 REST API within tools like AutoHotKey :-)
Hmmm, I already do that :lol: :lol:
As in: you have a big daily zipped file of a TM1 directory and you specify a cube name. Then the AHK script will grab all necessary files (*.cub, dim, feeder, rux, ...). It will see what are the dimensions of the cube (through REST API), and retrieve those dim and }ElementAttributes and }ElementSecurity files. The output is a folder of objects :-) Selecting multiple cubes is possible too.

This works fine for TM1 authentication mode = 1. If someone wants to cooperate with me to get it working for CAM security, let me know. I would be interested in getting the REST API calls from AHK or PowerShell or ... working in other security modes.

The benefit is of course gaining time. It takes a lot of time to find cub and dim and other files in a (big) zip file. With this method, we don't need to unzip the file first. In bigger TM1 models I support, unzipping such a file of about 100 GB takes about 2 hours.
Last edited by Wim Gielis on Thu Nov 28, 2019 11:08 am, edited 2 times in total.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

Wim Gielis
MVP
Posts: 2278
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Order of loading files in memory

Post by Wim Gielis » Wed Nov 27, 2019 11:51 am

Napshot wrote:
Wed Nov 27, 2019 11:22 am
Hey
I did some small test on this and from what i can see processes are some of the last that are loaded in. I could copy/delete past *.pro in to the data library at around 90% (might be abit higher than that) and see the effects at startup.

So in a order like this, based on the log file:

Cube headers > Dimension > Rules > Cube body > feeders >processes > chores

Morten
Many thanks Morten. Next time I restart a model that is not loaded very quickly, I will test it too.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

Post Reply