Page 1 of 1
TM1 web crashes and needs to be restarted
Posted: Wed Jun 10, 2015 7:36 pm
by Mems
Gents,
Ok running TM1 10.2.2 FP 2.
We are running 7 application from the server and mostly use TM1web as the UI.
Somewhere during the day (usually once a day). TM1 web stops working (or actually, I have no idea what happens) but users cant log into TM1web anymore.
Perspectives/Architect still works fine and it is not specific to any application. - System wide problem
If I restart the application server everything works fine.
The error itself just feels like an out of memory issue so I Googled possible solutions for out of memory issues.
I did find the following solution.
http://www-01.ibm.com/support/docview. ... 21651888
I tried to find some documentation on what this does, can't find anything. I am happy to give it a try, but would like to understand it.
Also my log files doen't give me an OutofMemory issue every time, but it does give the occasional one.
Once a week and as mentioned it happens once a day
Extratct from log file
Code: Select all
2015-05-29 14:21:38,268 [Thread-11] ERROR (jmxupdate) com.ibm.cognos.fpmsvc.jmx.JMXUpdateThread - An exception occurred while updating jmx bean:
com.ibm.cognos.fpmsvc.exception.FPMSVCException: An exception occurred on the server. Please contact your system administrator. :java.lang.OutOfMemoryError: Java heap space :EXCEPTION
at com.ibm.cognos.fpmsvc.jmx.SessionInfoAgent.setSessionInfoMBean(SessionInfoAgent.java:60)
at com.ibm.cognos.fpmsvc.jmx.JMXUpdateThread.run(JMXUpdateThread.java:36)
Caused by:
java.lang.OutOfMemoryError
:
Java heap space
at
java.util.ArrayList
.
iterator
(
ArrayList.java
:
8
2
5
)
Any ideas
No errors display in web just says "Error".
Hopefully this configuration fixes it, hoping someone can point me into a direction to why this is happening and if i can monitor the usages somewhere.
Re: TM1 web crashes and needs to be restarted
Posted: Wed Jun 10, 2015 7:51 pm
by jim wood
I don't know if this will help at all but have you considered scheduling a nightly restart of the server or services? (I went for server first as with windows it's the only way to be sure that all memory is being released.) Have you also considered moving TM1Web on to separate sever if it isn't already on one?
Re: TM1 web crashes and needs to be restarted
Posted: Wed Jun 10, 2015 8:01 pm
by Mems
@Jim Wood
I can't really restart the server as we have a worldwide audience - no real system down time.
To restart the application server only takes a second so it's not a train smash, but I would just like to be able to monitor it or at least find a logfile that says, ok you ran out of memory.
As it happens everyday, the once a week ones doesn't really help me and makes it even more confusing.
I also feel that it might no be a build up, but it might be crashing with to many users hitting the server at the same time? I can make an argument for both.
With 'nothing' to go by, I'm not really sure where to go and fiddle.
I can move TM1web to another server, not really sure what this will accomplish as we have 96GB RAM server and we are only using 15GB.
I will also find it hard to justify without having a clear reason why this is happening.
Thanks for the response though
Re: TM1 web crashes and needs to be restarted
Posted: Wed Jun 10, 2015 8:51 pm
by jim wood
I hear you. Log files that aren't consistent can be very annoying. While I realize that moving TM1 Web (and even application server) to a different server won't help from a memory point of view, it may help with the number of users hitting the service etc. It'll also lower the number of threads being used on the TM1 server by either TM1Web or application server.
Re: TM1 web crashes and needs to be restarted
Posted: Wed Jun 10, 2015 8:59 pm
by BrianL
It's worth noting that TM1Web/Application server is limited in memory consumption by the JVM. You'd most likely hit the JVM memory limit well before the system runs out of memory. The IBM technote you referenced provides the instructions for upping the maximum memory the JVM is allowed to consume.
Re: TM1 web crashes and needs to be restarted
Posted: Wed Jun 10, 2015 9:16 pm
by lotsaram
Brian may have just beaten me to the punch but here it is in clear text.
You did of course follow TM1 Web basic setup 101 and increase the memory setting for TM1 Web to a sensible value? (something more than 4GB, more than 10 GB for a server with heavy use). The default value is an anaemic 768 MB which is guaranteed to have the web server choke and die with even moderate use.
- Tm1WebMemory.jpg (128.8 KiB) Viewed 15502 times
Re: TM1 web crashes and needs to be restarted
Posted: Wed Jun 10, 2015 10:02 pm
by Mems
I thought this was the issue and already changed it before i posted the thread.
Hoping it will fix the problem.
Just thought I would be able to at least find a log file somewhere?
I changed it to the recommended 4GB.
Is there any way I can track the usages?
Is it a number of users hitting it at once or a slow build up?
Any information on how I can track this to put it to a suitable level will be great.
For all I know it might be 20GB needed, if it is a build-up and I need to cycle it once a week.
Once again thanks for the response
Re: TM1 web crashes and needs to be restarted
Posted: Thu Jun 11, 2015 3:11 am
by nick_leeson
Log files in TM1Web are located here dir\\webapps\tm1web\WEB-INF\logs. using these logs you could point out the last websheet that was the cause of the crash or the last last slice which tipped the server over.
Good luck !!
Re: TM1 web crashes and needs to be restarted
Posted: Thu Jun 11, 2015 8:52 am
by lotsaram
It's not always possible to get useful insights out of the logs. The first thing I would do is to look at task manager at the time that TM1 Web becomes unresponsive. If the memory being consumed by tomcat6.exe is approximately equal to what the maximum memory has been set to then you have your answer, the parameter value is insufficient and needs to be increased.
Re: TM1 web crashes and needs to be restarted
Posted: Thu Jun 11, 2015 5:10 pm
by Mems
@Nick
nick_leeson wrote:Log files in TM1Web are located here dir\\webapps\tm1web\WEB-INF\logs. using these logs you could point out the last websheet that was the cause of the crash or the last last slice which tipped the server over.
Good luck !!
Yeah thats where i found that 'once a week' log I was referring to.
For future reference:
I found the log file, in the bin file.
Name of the file is
"javacore.20150610.112257.7260.0012.txt"
Obviously the date will change
Code: Select all
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TICHARSET 437
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received
1TIDATETIME Date: 2015/06/10 at 11:23:03
1TIFILENAME Javacore filename: D:\Program Files\ibm\cognos\tm1_64\bin64\javacore.20150610.112257.7260.0012.txt
1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)
1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)
Re: TM1 web crashes and needs to be restarted
Posted: Thu Jun 11, 2015 5:14 pm
by Mems
Thanks everyone
Re: TM1 web crashes and needs to be restarted
Posted: Thu Feb 16, 2017 5:13 pm
by schang
@Mems
Did you ever resolve this issue? If you did, what was your solution?
Re: TM1 web crashes and needs to be restarted
Posted: Mon Jul 31, 2017 5:34 am
by PeteB
Two weeks ago we upgraded from 10.2 to 11.0 (PAL 2.0) and we started to encounter TM1 Web crashing - java.lang.OutOfMemoryError: Java heap space. This upgrade involved the change from Tomcat to Websphere Liberty.
We have made below changes and currently are not experiencing any more crashes, but we do have outstanding questions. Fingers crossed!
We believe the main culprit is TM1 Web pages with embed macros each xlsm used up 1Gb of memory when running on Websphere (not previously an issue with Tomcat). When the unrequired macro was removed and the spreadsheet and then saved as an xlsx the memory required by the web server decreased to less than 100mb.
The following points are what we believe to be true from observing the logs.
• Each individual TM1 websheet drives the increase use of memory.
• The number of additional users has no impact on memory required by the web service.
• We believe you can calculate the maximum amount of memory required simply by opening all of your websheets at the one time.
IBM tells us “Websphere memory is returned to the O/S during a garbage collection phase as defined in Java.”
Q1. Can we define this garbage collection setting in Java? If so how?
Here is what we have done so far:
IBM said
However we believe a different setting is actually changes the max java heap size.
Q2. What is the difference between these following settings? (the description is exactly the same). Do these settings have any relationship or impact to each other?
• Local Configuration -> Environment -> IBM Cognos TM1 services -> IBM Cognos TM1
Maximum Memory for Websphere Liberty
• Local Configuration -> Environment -> IBM Applications -> TM1 Clients
Maximum Memory in MB
Currently we have increased the memory setting to 28Gb, but this is the maximum we are able to set it. We are currently using 17Gb of memory after removing the macros. IBM said
There are no limitation as to what the Java_Heap size can be defined at. I've seen 64GB as the largest I've witnessed, but its theoretically unlimited. When defining the JavaHeap size keep in mind this is pre-allocating the memory, so it has to be there, and be free as that chunk is grabbed upfront. If its not present it'll page, which would either take a ridiculously long time to load, or not load at all.
Q3. Just in case we do require more memory does anyone have any thoughts on why we are hitting this 28Gb limit setting the Websphere memory?
For Information only
Switched on the logging
Logging properties are stored in the log4j.properties file in the following location:
<TM1 install location>\webapps\tm1web\WEB-INF\configuration
https://www.ibm.com/support/knowledgece ... 51964.html
Q4 What is the best practice for tuning websphere.
https://www.ibm.com/support/knowledgece ... ofile.html
Additional information
We are running 64 bit physical server with TM1, Websphere and Cognos Analytics for AD security being the only applications on the server.
Re: TM1 web crashes and needs to be restarted
Posted: Thu Aug 30, 2018 4:16 am
by ykud
PeteB wrote: ↑Mon Jul 31, 2017 5:34 am
Q3. Just in case we do require more memory does anyone have any thoughts on why we are hitting this 28Gb limit setting the Websphere memory?
Just in case somebody stumbles on this thread, here's the answer & solution to this question:
http://www-01.ibm.com/support/docview.w ... wg22008559