TM1 Java Process - CPU and Memory Utilization

Post Reply
dharav9
Posts: 72
Joined: Wed Aug 15, 2018 3:18 pm
OLAP Product: TM1
Version: 10.3
Excel Version: 2016

TM1 Java Process - CPU and Memory Utilization

Post by dharav9 »

Hi, All

1) I want o understand how CPU utilization works in resource monitor (through task manager). We have 24 logical processor and CPU activity shows more then that. Please review following image for it. We on windows 12.
CPU Utilization Reading.PNG
CPU Utilization Reading.PNG (136.56 KiB) Viewed 6134 times
2) When java process increase the CPU utilization (around 80% or higher we have to restart services for the java and this would hurt the user experience) and also increases memory consumption. I attached the word document with further information.
Debug CPU.docx
(387.96 KiB) Downloaded 289 times
3) Planning Analytics version 11.0.4 and Pax version .26. Total memory on the server 512 gb and total memory utilized by 2 tm1 instance = 350-400 gb.

Can anyone helps us to identify the bottle neck issue?

Please feel free if you require further information.

Thank You

Dharav
dharav9
Posts: 72
Joined: Wed Aug 15, 2018 3:18 pm
OLAP Product: TM1
Version: 10.3
Excel Version: 2016

Re: TM1 Java Process - CPU and Memory Utilization

Post by dharav9 »

Hi, All

System: 11.2.4 (PA)
Cognos Analytics : 11.06
Pax: 38

We found the cause behind increase in the CPU Spike for Java TM Platform Binary Process and it never reverts back to zero.

Cause:

We have two server instance. Finance and Accounting. When user exports (quick reports/Exploration) from both TM1 server instance on a single workbook, Java process spikes up CPU utilization.

Can anyone having two TM1 server instance, replicate following in development environment and update us with findings? (Just want to know whether it is bug or not),

Step 1: Export the quick report from one TM1 instance in Workbook A.
Step 2: Check the Java (TM) Platform SE Binary Process in Task Manager. It should be around 0% CPU utilization.
Step 3: Export quick report from 2nd TM1 server instance in workbook A (in different sheet).
Step 4: Check the CPU utilization for the Java (TM) Platform SE Binary Process in Task Manager. It should spike up and it won't come back to 0% again.

Please share your experience if you have two TM1 server instance are up and running. Just want to confirm whether it is a bug or not.

Thank You

DHARAV
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: TM1 Java Process - CPU and Memory Utilization

Post by paulsimon »

Hi

We had similar issues with CPU usage for Java on the TM1 Web server hitting 100% to the point where no one could login.

However, in our scenario we only have a single PA Cube Server instance. When you say that you have Finance and Accounting servers, do you mean two Web Servers or two PA Cube Servers? Although we only have a single PA Cube Server instance in Production, this is being accessed by a CDM Server which uses the Rest API to talk directly to the Cube Server, a WebSphere Liberty running TM1 Web, and an IIS Server which acts as a front end for some non-PA applications, and also for CAM. We suspect that CAM may have something to do with this since it is causing duplicate logins. We have a PMR on this with IBM.

We traced the CPU problem to one particular workbook. Something in that was causing the CPU to spike. Unfortunately, there is no easy way to pin point which workbook is causing the problem from the logs, nor apparently can this even be determined from analysis of the Crash Dump that we sent to IBM. We only found out by asking the users what they were doing at the time and narrowing it down (we have a large user community at multiple sites so it is not easy to find out what was being done at any one time).

There may have been something in that workbook that relates to your problem as we found hard-coded links in it to a Test Server. However, removing those links did not cure the problem. The IP address it was accessing no longer exists. It was a very old workbook that was converted from an xls to xlsx format. It seems that something in this caused Excel to think that the workbook had hundreds of columns. Any attempt to delete columns or remove data did not cure this. A Ctrl Shift End still shot way over to the right We ended up having to copy all the formula in that workbook to a new workbook and that seems to have cured the 100% CPU issue.

However, since getting over that problem we now have the problem that the WebSphere Server simply stops functioning. CPU is 0-1% but it will not process requests. This problem has been with IBM for a while. They have asked us to turn on Garbage Collection logs and send those over. However, turning on Garbage Collection can exacerbate the problem since it adds additional overhead. For the moment we have increased the RAM on the virtual server to mitigate that, and we have sent over the logs.

It is probably best to raise a PMR with IBM. Several users are having similar problems, and IBM have a series of debug steps to try to pinpoint the problem.

Regards

Paul Simon
dharav9
Posts: 72
Joined: Wed Aug 15, 2018 3:18 pm
OLAP Product: TM1
Version: 10.3
Excel Version: 2016

Re: TM1 Java Process - CPU and Memory Utilization

Post by dharav9 »

Hi, Paul

We have raised PMR with IBM already. I will keep you all posted if i find solution from them.

I mentioned about cube servers. Based on my testing output, CAM security is not culprit. In Mode 1, i also able to replicate the error.
I had enabled the verbose log to track the issue but it seems default settings work best and verbose log increase the overhead. I turned it off after identifying the issue.
However, since getting over that problem we now have the problem that the WebSphere Server simply stops functioning. CPU is 0-1% but it will not process requests. This problem has been with IBM for a while.
- Have you tried to re-install the environment in test server and to see output thereafter?

Thank You for the response and sharing the business case you have.

Dharav
dharav9
Posts: 72
Joined: Wed Aug 15, 2018 3:18 pm
OLAP Product: TM1
Version: 10.3
Excel Version: 2016

Re: TM1 Java Process - CPU and Memory Utilization

Post by dharav9 »

Hi, Paul

I found the solution for me. I am still waiting on IBM though.

Work Around:
Integration of VBA function:
CognosOfficeAutomationObject.ClearCache() BEFORE going to next worksheet.

I still believe IBM has to
Why it was increasing CPU?

We have two dimension name same in Cube Server A and Cube Server B but elements for both dimension are different.

E.g.:
Cube Server A
Dim:TIME=>Year,Jan,Feb, Mar etc
Cube Server B
Dim:TIME=>FY2019,JAN FY2019, FRB FY2019 etc

so once we export QR from cube server A, it holds cache memory. Now when we export QR from cube server B having same dimension name but different elements (dim:TIME)=> it tries to find TIME dimension in cache, once it finds (as previous exported view had same dimension but different elements) dimension then tries to retrieve the value based on elements for TIME dimension exists in the cache. As these elements are not part of cube server B, it throws bad tuple error and does not release CPU spike. Once it finds bad tuple then it tries to reach out to server to fetch data and QR exports comes out with performance lagging.

Clearing cache before connecting to different server helps and no increase in CPU spike and CPU comes back to zero % when no activity is there.

Hope this explanation may help you.

Thank You

Dharav
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: TM1 Java Process - CPU and Memory Utilization

Post by paulsimon »

Hi

Thanks for the reply. Glad you found a work around for your problem.

However, I think the problem that we are getting is different. Our problem is in TM1 Web rather than PAX. The underlying sheet is a normal Perspectives sheet. We are only connected to a single server.

I will let you know if we get any resolution to our problem from IBM.

Regards

Paul Simon
jwafro
Posts: 25
Joined: Thu Sep 07, 2017 8:31 am
OLAP Product: TM1
Version: 10.2.2 / 9.5.1
Excel Version: 2003 2010

Re: TM1 Java Process - CPU and Memory Utilization

Post by jwafro »

Hi Paul Simon

We've just had a similar problem as this - the java.exe process running at 100% allocated RAM and 100% CPU for a while before hanging.
eventually the system produced a massive dmp file in Bin64 (core.dmp) (24gb)

Tried increasing the allocated RAM way beyond recommended settings but this increased the size of the DMP file.

Seemingly, the solution to our problem has been related to the applications that were being used - a new application had been loaded which accidentally had 1million rows - something incredibly easy to do in excel. By reducing it to a sensible amount and restarting the service the system appears to have stabilised.

You can search the of excel apps under db\}externals\ - search by size/recently added.

Was inspired by this:
https://www.tm1forum.com/viewtopic.php?t=13907

Hopefully that helps!
Post Reply