Multi-threaded data save

Suggest and discuss enhancements for TM1
Post Reply
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Multi-threaded data save

Post by lotsaram »

We have had multi-threaded server loading for some time now and can now also multi-thread data loads. Together with 64 bit servers this has facilitated a huge expansion with the volume of data TM1 can handle.

However this all comes unstuck when a data save is executed following batch updates. As the data save is single threaded it can take a considerable amount of time to save data in a very large model.

My suggestion is to allow saving of .cub files to be multi-threaded in much the same way that loading of cubes is multi-threaded. Possibly with an additional config file parameter or possibly using the same settings as MaximumCubeLoadThreads.

I don't see why this shouldn't be possible ...
jstrygner
MVP
Posts: 195
Joined: Wed Jul 22, 2009 10:35 pm
OLAP Product: TM1
Version: 9.5.2 FP3
Excel Version: 2010

Re: Multi-threaded data save

Post by jstrygner »

lotsaram wrote:We have had multi-threaded server loading for some time now and can now also multi-thread data loads.
Multi-threaded server loading - yes, we have it, but AFAIK only for cubes (mentioned MaximumCubeLoadThreads) parameter. But what do you mean by multi-thread data loads? TI loading data to a cube (which I think always uses only one core), or you are still talking about data in cubes being load to RAM on server startup?
lotsaram wrote:I don't see why this shouldn't be possible ...
Agree with your enhancement and that it should be possible.

But maybe somebody could answer, why multi-threading (using more than one core) is not available by:
1. loading dimensions to RAM on server startup,
2. running TI processes,
3. building views,
4. other situations where you could make use of all cores you have, not just one being fully engaged?

Is it historical cause and a change is fundamental (not easy to implement) or maybe there is some kind of best practice in such solutions with advantages I am just not aware of?
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Multi-threaded data save

Post by lotsaram »

Yes multi-threaded server loading is for data only. Metadata is single threaded.

But no I was also talking about data loading no just server loading. This is getting off topic of the enhancement to enable multi-threaded data saves but in 9.4 and 9.5 at least it is very easy to do multi-threaded data loads. Multiple TI processes can run at once each loading to a separate cube. It is also possible to load multiple (non-overlapping) data sets to the same cube at the same time via multiple TI processes provided the server is in batch update mode.
User avatar
mattgoff
MVP
Posts: 516
Joined: Fri May 16, 2008 1:37 pm
OLAP Product: TM1
Version: 10.2.2.6
Excel Version: O365
Location: Florida, USA

Re: Multi-threaded data save

Post by mattgoff »

lotsaram wrote:We have had multi-threaded server loading for some time now and can now also multi-thread data loads. Together with 64 bit servers this has facilitated a huge expansion with the volume of data TM1 can handle.

However this all comes unstuck when a data save is executed following batch updates. As the data save is single threaded it can take a considerable amount of time to save data in a very large model.

My suggestion is to allow saving of .cub files to be multi-threaded in much the same way that loading of cubes is multi-threaded. Possibly with an additional config file parameter or possibly using the same settings as MaximumCubeLoadThreads.

I don't see why this shouldn't be possible ...
The bottleneck in loading is computing rules and firing feeders, so enabling CPU multithreading increases capacity. The bottleneck in saving is the disk subsystem, at least on my system, so adding compute capacity will not improve performance.

Matt
Please read and follow the Request for Assistance Guidelines. It helps us answer your question and saves everyone a lot of time.
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Multi-threaded data save

Post by lotsaram »

mattgoff wrote:The bottleneck in saving is the disk subsystem, at least on my system, so adding compute capacity will not improve performance.
Duh - Oh, Houston we have a problem. Poor old hard disk head can only do so many things, should have thought of that. Hmm maybe another take on things would be to enable a background save data mode that would not impose a server lock! I don't see why that should not be outside the realms of possibility ...
ParisHilton
Posts: 73
Joined: Fri Apr 23, 2010 11:35 am
OLAP Product: Tm1
Version: 9.5
Excel Version: 2007 2010

Re: Multi-threaded data save

Post by ParisHilton »

Or buy one of these funky new solid state disk drives !

The disk system is normally going to be the bottleneck, but it must be possible to write to disks in parallel.
“The way I see it, you should live everyday like its your birthday”


TM1 9.5 Cognos BI 8.4 Excel 2007. 128GB Ram. E7450@2.40Ghz -24 core. Fluffy white dog.
Post Reply