TM1 Usage of more Memory each time a Process is used

Post Reply
Poffl
Posts: 2
Joined: Tue Dec 15, 2015 3:00 pm
OLAP Product: TM1, DynamicCubes
Version: 10.2.2 FP4
Excel Version: 2013

TM1 Usage of more Memory each time a Process is used

Post by Poffl »

Hello,

I've a question about the memory usage of TM1.

I don't understand the memory usage while executing a TM1 process. And I absolutly don't find anything with Google etc.
Every time I repeat my process to load data into a cube, it uses more and more memory from Windows. After some time I need to restart the TM1-server to release all allocated memory.
It is not a big deal, but I still want to know why TM1 needs more memory, even if i delete the data from the cube and write exactly the same data again.

As an example:

Right now, I have three TM1-servers. Every department has its own server (fail-over etc.).

A specific example is a TM1-Server with only one Cube called "PPV".
In this Cube, there are 13 dimensions. The biggest one has ~520k elements. The second largest only ~400.
There is no rule for the cube. All calculations are made in the database.

We are loading ~10m data rows into the cube.
If I restart the TM1-Server, the cube only needs about 3,2GB. The Data is already in the cube in this time. If I repeat the process now to load Data into the Cube, the TM1-Server needs about 5,7GB. Everytime the process is used, TM1 takes about 2,7GB - 3,2GB for each iteration. That means, the server will be using ~25GB in 10 days. (Every night runs a process to delete all data from the cube and then loads the actual data).

For a better understanding, there are attachements with pictures about the TM1 memory usage.
Another question: Why is the "number of populated numeric cells" not constant ? Why is this even getting bigger, even if I use a "CubeClearData"?


Here is the code from the process to load the data:

Prologue:

Code: Select all

#****Begin: Generated Statements***
#****End: Generated Statements****

CubeClearData('PPV');
Meta-Data - nothing;

Data:

Code: Select all

#****Begin: Generated Statements***
#****End: Generated Statements****

#CellPutN(MENGE_VOLLZUG,'PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge Vollzug');
#CellPutN(MENGE_PLAN,'PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge Plan');
#CellPutN(MENGE_FE,'PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge FE');


CellPutN(CellGetN('PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge Vollzug')+MENGE_VOLLZUG,'PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge Vollzug');
CellPutN(CellGetN('PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge Plan')+MENGE_PLAN,'PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge Plan');
CellPutN(CellGetN('PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge FE')+MENGE_FE,'PPV',ID_ORGANISATION,ID_ZEIT,ID_ZEIT_AUFGELADENER_STAND,ID_VORGANGSSCHLUESSEL,ID_EINHEIT,ID_EINSCHLAGSURSACHE,ID_BAUMART,ID_HERKUNFTSGEBIET,ID_HERKUNFTSSICCHERHEIT,ID_BESITZART,ID_BESTANDESINDEX,ID_STANDORTSREGION,'Menge FE');
In this part, I tested it without the cumulation of the data. Still the same result with the memory.

Epilogue - Nothing;


The Attachements are in German - sorry about that!

I don't have a problem to restart the TM1 Server. But right now, we use three TM1 Servers and after about 10 days all three are using about 100GB from the 500GB on the Windows server. We restart the TM1 servers every month to free the allocated memory.
But also we want to expand our TM1 Usage. That means we want more TM1 Servers.
In my opinion, that means I need to restart the TM1 servers more often. (Every 15 days etc...)

But before we are doing that, I really want to know why this is even happening.
I understand, that a process needs memory to work. But why is the cube getting bigger? Why isn't it just rolling into the garbage?
And why is TM1 not using the garbage for the process again?

Information about server / TM1:

IBM Cognos TM1 10.2.2 FP7
Windows Server 2008 R2 Enterprise
512 GB RAM
16 cores.

Sorry if you have trouble to read my questions. I'm not a native english speaker.
Thanks a lot,

Patrick
Attachments
TM1_memory_firstrepeat.JPG
TM1_memory_firstrepeat.JPG (97.6 KiB) Viewed 2574 times
TM1_memory.JPG
TM1_memory.JPG (81.5 KiB) Viewed 2574 times
User avatar
jim wood
Site Admin
Posts: 3951
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: PA 2.0.7
Excel Version: Office 365
Location: 37 East 18th Street New York
Contact:

Re: TM1 Usage of more Memory each time a Process is used

Post by jim wood »

Hello there,

Points about this are spread across this board but difficult to find. Basically TM1 caches values for users to come back to. Theory being that if you cache the value rather than calculate it every time you get better performance. When a data change comes in however, TM1 no longer uses that cache as any calculations will also need to be updated. The cache that was used however is not cleared properly. If memory serves (no pun intended, well may be a little) this is a restriction of both TM1 and windows, especially windows. DOS is not great at memory management. Generally it is advised that you should restart your TM1 service once a week to remove the cache and reduce junk memory build up.

I hope this helps. I'm sure others will chime in with a more in depth analysis of this problem but hopefully this should give you a general idea of what you're looking at,

Jim.
Struggling through the quagmire of life to reach the other side of who knows where.
Shop at Amazon
Jimbo PC Builds on YouTube
OS: Mac OS 11 PA Version: 2.0.7
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: TM1 Usage of more Memory each time a Process is used

Post by tomok »

I don't read or speak German so your TI code doesn't help any but is it possible one of these dimensions is for time, more specifically, days? You say you are reloading the same data every day but are you loading the data to different intersections in the cube? For example, yesterdays data would have been written to June 4, the cube is cleared and then today's data is written to June 5? Here's why I ask: CubeClearData does not release memory. So, if you loaded 2.5GB of data to June 4, clear, and then load 2.5GB of data to June 5 you're going to be using 5GB of memory. Executing the CubeClearData command only makes the June 4 intersections zero, it does not release the memory. The only way I know to release the memory is the take the service down, although it may be released if you unload the cube each day. I'm not sure on that so you would need to test but what you are experiencing makes sense if you are indeed loading to different intersections each day.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
Poffl
Posts: 2
Joined: Tue Dec 15, 2015 3:00 pm
OLAP Product: TM1, DynamicCubes
Version: 10.2.2 FP4
Excel Version: 2013

Re: TM1 Usage of more Memory each time a Process is used

Post by Poffl »

Hi Tomok, Hi Jim,

thank you for the answer.
As expected, I need to restart.

We will change interval for the restart now to 1 week.
Post Reply