Grabage memory runs away

[pablo]

Hallo,

I have this issue at a client where the memory increases till where everything hangs.

My stats_by_cube = 4.85 GIG
Stas_by_server = 21 GIG where the garabage memory equals 16 GIG.

I have inputted the Log file to prevent garabage memory to increase above 2 GIG.

See below:
#
# Enable INFO level logging through the shared memory appender, by default. The server
# will write informational messages, as well as errors and warnings to the log file.
#
log4j.logger.TM1=INFO, S1
# S1 is set to be a SharedMemoryAppender
log4j.appender.S1=org.apache.log4j.SharedMemoryAppender
# Specify the size of the shared memory segment
log4j.appender.S1.MemorySize=5 MB

# Specify the max filesize
log4j.appender.S1.MaxFileSize=100 MB

# Specify the max backup index
log4j.appender.S1.MaxBackupIndex=20

# Specify GMT or Local timezone
log4j.appender.S1.TimeZone=GMT


# limits the garbage memory to 2GB
log4j.appender.S1.MemorySize=2000 MB

What can cause the garbage memory to increase by so much?
If my understanding is correct the Stats by cube and Stats by server should more or less be the same?

Thank you in advance for any assistance.

Thank you

[David Usherwood]

# limits the garbage memory to 2GB
log4j.appender.S1.MemorySize=2000 MB

It doesn't - logging is nothing to do with garbage memory.

What can cause the garbage memory to increase by so much?

Creating and destroying large calculated views.

If my understanding is correct the Stats by cube and Stats by server should more or less be the same?

It's not (correct) - the difference is the garbage memory.

[pablo]

thank you

but is there any way to control the garbage memory?

all the views that the client will open is large calculated views.

any suggestions.

to restart the server every night?

[David Usherwood]

a Get more RAM - your client's system is that size
or
b Introduce conditionalised feeders to suppress calculation of irrelevant (eg) versions.

If you do a daily restart you are guaranteeing slow performance as the views recalculate at 9am.

[Jeroen Eynikel]

As I understand it opening and closing large views in itself does not create garbage memory.

It only creates garbage memory when the results in these views are no longer valid. (i.e. when the underlying data or what TM1 thinks is the underlying data anyways) changes.

Restarting the service would give this memory back to the OS.

But you may also want to check this link

http://www-01.ibm.com/support/docview.w ... wg21430011

I don't necessarily agree about the slow performance either. As everything it is a tradeoff but if you are updating your system with new data all the time anyways it makes very little difference if you restart or not performance wise. In both cases things will be recalculated from scratch.

Finally you could always have a chore precalculate certain views for the users.

[Wim Gielis]

As I understand it opening and closing large views in itself does not create garbage memory.

It only creates garbage memory when the results in these views are no longer valid. (i.e. when the underlying data or what TM1 thinks is the underlying data anyways) changes.

Restarting the service would give this memory back to the OS.

That was my opinion too, but a colleague of mine says that it's only by rebooting the server machine (not only TM1) that garbage memory will be given back.
What do the others think?

[Alan Kirk]

As I understand it opening and closing large views in itself does not create garbage memory.

It only creates garbage memory when the results in these views are no longer valid. (i.e. when the underlying data or what TM1 thinks is the underlying data anyways) changes.

Restarting the service would give this memory back to the OS.

That was my opinion too, but a colleague of mine says that it's only by rebooting the server machine (not only TM1) that garbage memory will be given back.
What do the others think?

I think I'd like to know where your collegue got that idea from given that garbage memory is, by definition, part of the memory that the TM1 server is claiming for its own use. Once the TM1 server stops, once the service is no longer running, once it has ceased to exist from the point of view of Windows... short of some kind of major glitch in the design of Windows (don't start, I'm being serious here) how could Windows NOT be able to access memory that a now non-existent process once used? Why is it that in my experience the garbage memory is always much lower after a server restart than it was before one? But the killer question is... if this garbage memory isn't being released back to the operating system until a server box reboot, and it isn't being "preserved" somewhere for the TM1 server when it restarts (which is what the "lower at startup" experience suggests), then... where is it going? And we're not talking about trivial amounts of memory here. Being on 32 bit I do a reboot every night. (As an aside I agree with Jeroen; the speed issue will only bite you on static data. However with "live" data, which is constantly being refreshed anyway, a reboot makes no difference.) I think I'd probably notice if tens of megabytes were just "vapourising" each and every night, given that the Windows server normally runs for weeks if not months at a time and I don't see any kind of creeping expansion of memory use over that period.

[Gregor Koch]

That was my opinion too, but a colleague of mine says that it's only by rebooting the server machine (not only TM1) that garbage memory will be given back.
What do the others think?

Once the TM1 server stops, once the service is no longer running, once it has ceased to exist from the point of view of Windows...

Agree with Alan, the comment about the machine restart is, sorry, garbage and must be a misunderstanding. Can be easily tested as well, unless there is absolutely no confidence in what Windows Task Manager has to say about it. Btw, the part about the (de)ceased (to exist) service made me think of the Dead Parrot sketch from Monty Python, whether intentional or not, thanks Alan.

On why garbage memory gets 'created':
http://publib.boulder.ibm.com/infocente ... olCub.html

To put it in layman's terms, TM1 at times needs more memory and sometimes it needs less. If it needs less it does not give it back to Windows unless you recycle the TM1 Server (Database, not machine).

But back to the OP.

What can cause the garbage memory to increase by so much?

To give you an answer you would have to give more information about when this increase happens?

Right on start up - not very likely although you might see memory in garbage after a clean start, just not that much.

Running overnight processes - can happen but again it is usually (again usually) not that much.

Lots of users doing stuff - what are they doing?

Other situations - donno.

The most extreme increase I've seen (actually very comparable to what you see size wise) in both 9.4 (dot.something, cant' remember but that is the version it started happening) and 9.5.2 was due to a medium to large Excel report that was not using a proper View function. Doesn't even need to be many users, one is enough who changes the parameters of the report and re runs it several times. I have lodged this behaviour with example model to IBM some time ago and the answer was to use View functions and ... well, to put up with it.
Now, before you run off and check all the reports for the View functions you might want to observe the server after a restart and use TM1 Top to see what users are doing and compare that to the memory usage increase ([Memory Used]) and the subsequent increase memory in garbage. At the same time monitor what the memory is used for in }StatsbyCube cube, particularly [Memory Used for Calculations]. If you see that one going up like crazy then I think my guess form above is spot on.
Now, A View function is not A proper View function. So checking whether there are View functions on all the cubes you get data from is not enough, they need to be the correct ones. How that works - please read manual.

Cheers

[Wim Gielis]

I think I'd like to know where your collegue got that idea from given that garbage memory is, by definition, part of the memory that the TM1 server is claiming for its own use.

I will ask him when time permits, and revert.

[pablo]

thanks for all the replies.

appreciated.

i took out the ConsolidateChildren function and no more issues.

iam now calculating a weighted average cost and not a weighted cost anymore.

thank you