Order of loading files in memory

Post Reply
Wim Gielis
MVP
Posts: 2467
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Order of loading files in memory

Post by Wim Gielis » Wed Nov 27, 2019 9:50 am

Hello,

Just a question that popped up with me.

I (re)start a TM1 model, let's say 20 GB and 5 minutes of loading time.
Does anybody know in what order, more or less, files are read into memory ?
The question is, after starting a model, can I still change the contents of a PRO file or copy/paste a PRO file in the TM1 data directory such that it is still picked during loading ? IF processes are "loaded" at the end of the transaction, this would avoid starting all over again with the load.

My real user case is a TM1 model of 115 GB where I needed to import a couple of TI processes but I forgot... until 80 GB was loaded and half an hour was gone :o

Thanks
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 112 TM1 articles and a lot of custom code
Newest blog article: Selecting elements in the Subset Editor quickly

Napshot
Posts: 6
Joined: Thu Aug 02, 2018 9:13 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: Perspective

Re: Order of loading files in memory

Post by Napshot » Wed Nov 27, 2019 11:22 am

Hey
I did some small test on this and from what i can see processes are some of the last that are loaded in. I could copy/delete past *.pro in to the data library at around 90% (might be abit higher than that) and see the effects at startup.

So in a order like this, based on the log file:

Cube headers > Dimension > Rules > Cube body > feeders >processes > chores

Morten

User avatar
gtonkin
MVP
Posts: 804
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: Order of loading files in memory

Post by gtonkin » Wed Nov 27, 2019 11:47 am

Wim Gielis wrote:
Wed Nov 27, 2019 9:50 am
...
My real user case is a TM1 model of 115 GB where I needed to import a couple of TI processes but I forgot... until 80 GB was loaded and half an hour was gone...
Would be useful to be able to use AutoHotKey to effectively retype the necessary details from your non-prod to prod environment then save/compile.
A nice project for those long weekends!

Wim Gielis
MVP
Posts: 2467
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Order of loading files in memory

Post by Wim Gielis » Wed Nov 27, 2019 11:49 am

gtonkin wrote:
Wed Nov 27, 2019 11:47 am
Wim Gielis wrote:
Wed Nov 27, 2019 9:50 am
...
My real user case is a TM1 model of 115 GB where I needed to import a couple of TI processes but I forgot... until 80 GB was loaded and half an hour was gone...
Would be useful to be able to use AutoHotKey to effectively retype the necessary details from your non-prod to prod environment then save/compile.
A nice project for those long weekends!
Indeed, use the TM1 REST API within tools like AutoHotKey :-)
Hmmm, I already do that :lol: :lol:
As in: you have a big daily zipped file of a TM1 directory and you specify a cube name. Then the AHK script will grab all necessary files (*.cub, dim, feeder, rux, ...). It will see what are the dimensions of the cube (through REST API), and retrieve those dim and }ElementAttributes and }ElementSecurity files. The output is a folder of objects :-) Selecting multiple cubes is possible too.

This works fine for TM1 authentication mode = 1. If someone wants to cooperate with me to get it working for CAM security, let me know. I would be interested in getting the REST API calls from AHK or PowerShell or ... working in other security modes.

The benefit is of course gaining time. It takes a lot of time to find cub and dim and other files in a (big) zip file. With this method, we don't need to unzip the file first. In bigger TM1 models I support, unzipping such a file of about 100 GB takes about 2 hours.
Last edited by Wim Gielis on Thu Nov 28, 2019 11:08 am, edited 2 times in total.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 112 TM1 articles and a lot of custom code
Newest blog article: Selecting elements in the Subset Editor quickly

Wim Gielis
MVP
Posts: 2467
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Order of loading files in memory

Post by Wim Gielis » Wed Nov 27, 2019 11:51 am

Napshot wrote:
Wed Nov 27, 2019 11:22 am
Hey
I did some small test on this and from what i can see processes are some of the last that are loaded in. I could copy/delete past *.pro in to the data library at around 90% (might be abit higher than that) and see the effects at startup.

So in a order like this, based on the log file:

Cube headers > Dimension > Rules > Cube body > feeders >processes > chores

Morten
Many thanks Morten. Next time I restart a model that is not loaded very quickly, I will test it too.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 112 TM1 articles and a lot of custom code
Newest blog article: Selecting elements in the Subset Editor quickly

Wim Gielis
MVP
Posts: 2467
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Order of loading files in memory

Post by Wim Gielis » Thu Dec 12, 2019 6:15 pm

Wim Gielis wrote:
Wed Nov 27, 2019 11:49 am
This works fine for TM1 authentication mode = 1. If someone wants to cooperate with me to get it working for CAM security, let me know. I would be interested in getting the REST API calls from AHK or PowerShell or ... working in other security modes.
It works by exploiting the log on information specified in this thread:
https://www.tm1forum.com/viewtopic.php?f=3&t=15063
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 112 TM1 articles and a lot of custom code
Newest blog article: Selecting elements in the Subset Editor quickly

Post Reply