TM1 web crashes and needs to be restarted

Post Reply
Mems
Posts: 58
Joined: Thu Apr 14, 2011 12:27 pm
OLAP Product: TM1
Version: v10.2.2
Excel Version: 2010
Location: South Africa

TM1 web crashes and needs to be restarted

Post by Mems » Wed Jun 10, 2015 7:36 pm

Gents,

Ok running TM1 10.2.2 FP 2.

We are running 7 application from the server and mostly use TM1web as the UI.

Somewhere during the day (usually once a day). TM1 web stops working (or actually, I have no idea what happens) but users cant log into TM1web anymore.
Perspectives/Architect still works fine and it is not specific to any application. - System wide problem

If I restart the application server everything works fine.

The error itself just feels like an out of memory issue so I Googled possible solutions for out of memory issues.
I did find the following solution.

http://www-01.ibm.com/support/docview. ... 21651888

I tried to find some documentation on what this does, can't find anything. I am happy to give it a try, but would like to understand it.

Also my log files doen't give me an OutofMemory issue every time, but it does give the occasional one.
Once a week and as mentioned it happens once a day

Extratct from log file

Code: Select all

2015-05-29 14:21:38,268 [Thread-11] ERROR (jmxupdate) com.ibm.cognos.fpmsvc.jmx.JMXUpdateThread - An exception occurred while updating jmx bean: 
com.ibm.cognos.fpmsvc.exception.FPMSVCException: An exception occurred on the server. Please contact your system administrator. :java.lang.OutOfMemoryError: Java heap space :EXCEPTION
	at com.ibm.cognos.fpmsvc.jmx.SessionInfoAgent.setSessionInfoMBean(SessionInfoAgent.java:60)
	at com.ibm.cognos.fpmsvc.jmx.JMXUpdateThread.run(JMXUpdateThread.java:36)
Caused by: 
java.lang.OutOfMemoryError
: 
Java heap space
	at 
java.util.ArrayList
.
iterator
(
ArrayList.java
:
8
2
5
)
Any ideas

No errors display in web just says "Error".

Hopefully this configuration fixes it, hoping someone can point me into a direction to why this is happening and if i can monitor the usages somewhere.
MEMS

User avatar
jim wood
Site Admin
Posts: 3609
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: TM1 web crashes and needs to be restarted

Post by jim wood » Wed Jun 10, 2015 7:51 pm

I don't know if this will help at all but have you considered scheduling a nightly restart of the server or services? (I went for server first as with windows it's the only way to be sure that all memory is being released.) Have you also considered moving TM1Web on to separate sever if it isn't already on one?
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 10 64-bit. TM1 Version: 10.2.2

Mems
Posts: 58
Joined: Thu Apr 14, 2011 12:27 pm
OLAP Product: TM1
Version: v10.2.2
Excel Version: 2010
Location: South Africa

Re: TM1 web crashes and needs to be restarted

Post by Mems » Wed Jun 10, 2015 8:01 pm

@Jim Wood

I can't really restart the server as we have a worldwide audience - no real system down time.
To restart the application server only takes a second so it's not a train smash, but I would just like to be able to monitor it or at least find a logfile that says, ok you ran out of memory.
As it happens everyday, the once a week ones doesn't really help me and makes it even more confusing.

I also feel that it might no be a build up, but it might be crashing with to many users hitting the server at the same time? I can make an argument for both.
With 'nothing' to go by, I'm not really sure where to go and fiddle.

I can move TM1web to another server, not really sure what this will accomplish as we have 96GB RAM server and we are only using 15GB.
I will also find it hard to justify without having a clear reason why this is happening.

Thanks for the response though
MEMS

User avatar
jim wood
Site Admin
Posts: 3609
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: TM1 web crashes and needs to be restarted

Post by jim wood » Wed Jun 10, 2015 8:51 pm

I hear you. Log files that aren't consistent can be very annoying. While I realize that moving TM1 Web (and even application server) to a different server won't help from a memory point of view, it may help with the number of users hitting the service etc. It'll also lower the number of threads being used on the TM1 server by either TM1Web or application server.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 10 64-bit. TM1 Version: 10.2.2

BrianL
MVP
Posts: 264
Joined: Mon Nov 03, 2014 8:23 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2 PA2
Excel Version: 2016

Re: TM1 web crashes and needs to be restarted

Post by BrianL » Wed Jun 10, 2015 8:59 pm

It's worth noting that TM1Web/Application server is limited in memory consumption by the JVM. You'd most likely hit the JVM memory limit well before the system runs out of memory. The IBM technote you referenced provides the instructions for upping the maximum memory the JVM is allowed to consume.

lotsaram
MVP
Posts: 3140
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: TM1 web crashes and needs to be restarted

Post by lotsaram » Wed Jun 10, 2015 9:16 pm

Brian may have just beaten me to the punch but here it is in clear text.

You did of course follow TM1 Web basic setup 101 and increase the memory setting for TM1 Web to a sensible value? (something more than 4GB, more than 10 GB for a server with heavy use). The default value is an anaemic 768 MB which is guaranteed to have the web server choke and die with even moderate use.
Tm1WebMemory.jpg
Tm1WebMemory.jpg (128.8 KiB) Viewed 5497 times
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

Mems
Posts: 58
Joined: Thu Apr 14, 2011 12:27 pm
OLAP Product: TM1
Version: v10.2.2
Excel Version: 2010
Location: South Africa

Re: TM1 web crashes and needs to be restarted

Post by Mems » Wed Jun 10, 2015 10:02 pm

I thought this was the issue and already changed it before i posted the thread.
Hoping it will fix the problem.
Just thought I would be able to at least find a log file somewhere?

I changed it to the recommended 4GB.
Is there any way I can track the usages?

Is it a number of users hitting it at once or a slow build up?
Any information on how I can track this to put it to a suitable level will be great.
For all I know it might be 20GB needed, if it is a build-up and I need to cycle it once a week.

Once again thanks for the response
MEMS

nick_leeson
Posts: 98
Joined: Sat Feb 11, 2012 11:13 am
OLAP Product: TM1 9x, BPC, Hyperion, HANA
Version: TM1 10
Excel Version: Excel 2003 - 2010

Re: TM1 web crashes and needs to be restarted

Post by nick_leeson » Thu Jun 11, 2015 3:11 am

Log files in TM1Web are located here dir\\webapps\tm1web\WEB-INF\logs. using these logs you could point out the last websheet that was the cause of the crash or the last last slice which tipped the server over.

Good luck !!

lotsaram
MVP
Posts: 3140
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: TM1 web crashes and needs to be restarted

Post by lotsaram » Thu Jun 11, 2015 8:52 am

It's not always possible to get useful insights out of the logs. The first thing I would do is to look at task manager at the time that TM1 Web becomes unresponsive. If the memory being consumed by tomcat6.exe is approximately equal to what the maximum memory has been set to then you have your answer, the parameter value is insufficient and needs to be increased.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

Mems
Posts: 58
Joined: Thu Apr 14, 2011 12:27 pm
OLAP Product: TM1
Version: v10.2.2
Excel Version: 2010
Location: South Africa

Re: TM1 web crashes and needs to be restarted

Post by Mems » Thu Jun 11, 2015 5:10 pm

@Nick
nick_leeson wrote:Log files in TM1Web are located here dir\\webapps\tm1web\WEB-INF\logs. using these logs you could point out the last websheet that was the cause of the crash or the last last slice which tipped the server over.

Good luck !!
Yeah thats where i found that 'once a week' log I was referring to.


For future reference:

I found the log file, in the bin file.

Name of the file is

"javacore.20150610.112257.7260.0012.txt"

Obviously the date will change

Code: Select all

0SECTION       TITLE subcomponent dump routine
NULL           ===============================
1TICHARSET     437
1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received 
1TIDATETIME    Date:                 2015/06/10 at 11:23:03
1TIFILENAME    Javacore filename:    D:\Program Files\ibm\cognos\tm1_64\bin64\javacore.20150610.112257.7260.0012.txt
1TIREQFLAGS    Request Flags: 0x81 (exclusive+preempt)
1TIPREPSTATE   Prep State: 0x104 (exclusive_vm_access+trace_disabled)
MEMS

Mems
Posts: 58
Joined: Thu Apr 14, 2011 12:27 pm
OLAP Product: TM1
Version: v10.2.2
Excel Version: 2010
Location: South Africa

Re: TM1 web crashes and needs to be restarted

Post by Mems » Thu Jun 11, 2015 5:14 pm

Thanks everyone
MEMS

schang
Posts: 1
Joined: Thu Feb 16, 2017 4:30 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: Excel 2013

Re: TM1 web crashes and needs to be restarted

Post by schang » Thu Feb 16, 2017 5:13 pm

@Mems

Did you ever resolve this issue? If you did, what was your solution?

PeteB
Posts: 21
Joined: Sun Sep 23, 2012 2:30 am
OLAP Product: Planning Analytics Local, PAW
Version: PAL 2.0.4, PAW 2.0.32
Excel Version: 2010

Re: TM1 web crashes and needs to be restarted

Post by PeteB » Mon Jul 31, 2017 5:34 am

Two weeks ago we upgraded from 10.2 to 11.0 (PAL 2.0) and we started to encounter TM1 Web crashing - java.lang.OutOfMemoryError: Java heap space. This upgrade involved the change from Tomcat to Websphere Liberty.

We have made below changes and currently are not experiencing any more crashes, but we do have outstanding questions. Fingers crossed!

We believe the main culprit is TM1 Web pages with embed macros each xlsm used up 1Gb of memory when running on Websphere (not previously an issue with Tomcat). When the unrequired macro was removed and the spreadsheet and then saved as an xlsx the memory required by the web server decreased to less than 100mb.
The following points are what we believe to be true from observing the logs.
• Each individual TM1 websheet drives the increase use of memory.
• The number of additional users has no impact on memory required by the web service.
• We believe you can calculate the maximum amount of memory required simply by opening all of your websheets at the one time.

IBM tells us “Websphere memory is returned to the O/S during a garbage collection phase as defined in Java.”

Q1. Can we define this garbage collection setting in Java? If so how?

Here is what we have done so far:
IBM said
The JVM for TM1Web needs to be increased. The default is set to 768mb which is usually too low. Most of our customers set to around 8GB. Here's where you update this parameter: http://www-01.ibm.com/support/docview. ... g27041876
However we believe a different setting is actually changes the max java heap size.

Q2. What is the difference between these following settings? (the description is exactly the same). Do these settings have any relationship or impact to each other?
• Local Configuration -> Environment -> IBM Cognos TM1 services -> IBM Cognos TM1
Maximum Memory for Websphere Liberty
Image
• Local Configuration -> Environment -> IBM Applications -> TM1 Clients
Maximum Memory in MB
Image

Currently we have increased the memory setting to 28Gb, but this is the maximum we are able to set it. We are currently using 17Gb of memory after removing the macros. IBM said
There are no limitation as to what the Java_Heap size can be defined at. I've seen 64GB as the largest I've witnessed, but its theoretically unlimited. When defining the JavaHeap size keep in mind this is pre-allocating the memory, so it has to be there, and be free as that chunk is grabbed upfront. If its not present it'll page, which would either take a ridiculously long time to load, or not load at all.
Q3. Just in case we do require more memory does anyone have any thoughts on why we are hitting this 28Gb limit setting the Websphere memory?

For Information only
Switched on the logging
Logging properties are stored in the log4j.properties file in the following location:
<TM1 install location>\webapps\tm1web\WEB-INF\configuration

https://www.ibm.com/support/knowledgece ... 51964.html

Q4 What is the best practice for tuning websphere.
https://www.ibm.com/support/knowledgece ... ofile.html

Additional information
We are running 64 bit physical server with TM1, Websphere and Cognos Analytics for AD security being the only applications on the server.
Attachments
2.png
2.png (284 Bytes) Viewed 2865 times
1.png
1.png (377 Bytes) Viewed 2865 times

User avatar
ykud
Posts: 35
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: TM1 web crashes and needs to be restarted

Post by ykud » Thu Aug 30, 2018 4:16 am

PeteB wrote:
Mon Jul 31, 2017 5:34 am

Q3. Just in case we do require more memory does anyone have any thoughts on why we are hitting this 28Gb limit setting the Websphere memory?
Just in case somebody stumbles on this thread, here's the answer & solution to this question:
http://www-01.ibm.com/support/docview.w ... wg22008559

Post Reply