About memory stacks and executeprocess

[Jeroen Eynikel]

Hi,

I have the following situation:

I have a TI process that works fine when launched by itself but when launched as part of another process (so by using executeprocess) it will fail with the message that the server was unable to allocate sufficient memory. Since the server itself is still under 1GB at this point I assume this to mean memory stack rather than pure RAM.

I have the same issue with this process on versions 9.5.1 and 9.5.2 32 Bit. On a 64 Bit server it works fine regardless of how you execute it.

Is there anyone who can clarify how exactly TM-1 is managing memory when TI processes are running. I.e. I always assumed it was caching in RAM untill it gets to the end of the tab (as it then commits) and then starts another cache.

Now since my experience above, this clearly is not the case. It seems that some sort of cache is created when starting a TI process and that this cache is 1) limited and 2) only released / recreated at the very end of the process. (As I see no other possibility to explain why the process works by itself but not when called over ExecuteProcess on 32 Bit).

Thanks for any info

Jeroen

[lotsaram]

Hi,

I have the following situation:

I have a TI process that works fine when launched by itself but when launched as part of another process (so by using executeprocess) it will fail with the message that the server was unable to allocate sufficient memory. Since the server itself is still under 1GB at this point I assume this to mean memory stack rather than pure RAM.

I have the same issue with this process on versions 9.5.1 and 9.5.2 32 Bit. On a 64 Bit server it works fine regardless of how you execute it.

Is there anyone who can clarify how exactly TM-1 is managing memory when TI processes are running. I.e. I always assumed it was caching in RAM untill it gets to the end of the tab (as it then commits) and then starts another cache.

Now since my experience above, this clearly is not the case. It seems that some sort of cache is created when starting a TI process and that this cache is 1) limited and 2) only released / recreated at the very end of the process. (As I see no other possibility to explain why the process works by itself but not when called over ExecuteProcess on 32 Bit).

Thanks for any info

Jeroen

I think this belongs in a separate thread as it is a totally different subject and question.

Sounds a bit weird. I have seen memory issues with TIs that recursively call themselves, TM1 doesn't seem to handle this very well. But splitting this into a "master" process to do the "calling" and a "slave" process to do the "doing" has always fixed this for me.

[Jeroen Eynikel]

Well I certainly agree with you that it is weird behaviour. I haven't ever had this problem before in 8 years of TM1 experience right now.

Maybe it will help a bit if I describe what the process does.

Basically it processes a source view and for every record in it it will cycle over several other dimensions (in while loops) to determine whether the combination has a value in a second cube.

So say my source view contains positions V, W, X, Y and Z. The process is then going to do multiple nested loops over dimensions A, B and C checking whether the combination of V, W, X, A, B, C has a data value in another cube. If so it writes a value to V,W,X,A,B,C in a third cube.

I have seen very weird behaviour with this so far. I.e. launching it directly: finishes succesfully and does what it is supposed to do.

Calling it from within another process over executeprocess => BigAlloc() error on one particular element in the 'V' range.

No problem on 64 Bit however.

My assumption right now is that when launching a TI process some sort of memory stack is reserved for it. And that launching another process from within this process does not create a new memory stack but rather keeps on using the old one. That would explain - to me at least - how what I am seeing is possible. It would also explain why I only have the problem on a 32 Bit server if one assumes the stack to be considerably bigger on a 64 Bit system which I think is a fair assumption.

Then again maybe it's a bug that manifests itself in both version 9.5.1 and 9.5.2 32 Bit, which is a possibility I cannot exclude either I guess.

If anyone knows how the memory allocation really works I'd be most gratefull for the info. If not I guess I will log it as a bug and see what comes of it.

[AnthonyT]

Not sure if I am off beam here, but I had a problem with memory when I ran a process that was running a loop that created a new view with each pass

Each view was destroyed after use, but the memory consumption went up

I added the following to the cfg file and all has been well since

Code: Select all

MaximumViewSize=2048

Hope this helps