Transfer of data between servers

Suggest and discuss enhancements for TM1
Post Reply
User avatar
Alan Kirk
Site Admin
Posts: 6190
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2 Classic (PAW-free zone)
Excel Version: 2010 and 2016
Location: Sydney, Australia
Contact:

Transfer of data between servers

Post by Alan Kirk » Tue Jun 02, 2020 10:41 pm

This isn't one of mine, but one that I just noticed in this week's incarnation of the RFE system:
TM1 does not a user friendly and easy mechanism to transfer data between different TM1 servers. Currently there are 2 options, but none of them can be properly used:

1 - Use Replication to transfer and synchronize data from different TM1 servers: This option is not useful if data is not manually entered in TM1 and recorded via transaction logs. If source data is calculated via rules and consolidations, then it does not work. If source data is transferred to TM1 cube via TI process without logging, then it does not work. Therefore this option is not useful in 90% of the cases.

2 - Using ODBO as data source in a TI process. This option is not user friendly at all because it works via MDX queries. It never worked for our requirements.
I used to use Replication during the last Ice Age. I don't know how it is for anyone else but I found it to be as "reliable" as any given system for picking horse race winners; which is to say, "not". It would randomly stop working for no obvious reason and you'd have to unpick the replication and try it again.... until the next time it randomly stopped working, lather, rinse and repeat until you finally said "You know what, stuff this" and wrote TIs to export the structure for reimport.
As a solution to this problem:

a new feature can be added in either of the following 2 options:

1 - Add an option to replication/synchronization definition in a way not use transaction log files but read the source cube itself when doing the synchronization every time. Moreover replication feature should also have the capability to read source cube views that are calculated via rules or dimension consolidations so that a high level summary of a source cube can be transferred to a target cube with less dimensions.

2 - In a TI process, when using IBM Cognos TM1 as data source, it should allow, not only using the cubes in the same TM1 server, but it should also allow connecting to a different TM1 server and using its cubes in the same way. In this option, it should allow using existing cubes views or subsets in source TM1 server.
I'm in agreement with option 2. Given the new REST API, this should not be an impossible challenge.

It has 8 votes at the moment and is still in the "needs review" stage. If you like the idea, I encourage voting early and voting often via the link in the opening paragraph.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

User avatar
mce
Community Contributor
Posts: 330
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: Transfer of data between servers

Post by mce » Tue Jun 02, 2020 11:43 pm

I am also in favor of option 2. I do not understand how come IBMers can avoid adding this type of a very basic and must have functionality.
Such a tool that is not able to integrate its own data from its different databases. TM1 server A cannot read data from TM1 server B in a very simple and straightforward mechanism. That is a shame.

User avatar
mce
Community Contributor
Posts: 330
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: Transfer of data between servers

Post by mce » Tue Jun 02, 2020 11:45 pm

mce wrote:
Tue Jun 02, 2020 11:43 pm
Such a tool that is not able to integrate its own data from its different databases. TM1 server A cannot read data from TM1 server B in a very simple and straightforward mechanism. That is a shame.
In between... :)

scrumthing
Posts: 51
Joined: Tue Jan 26, 2016 4:18 pm
OLAP Product: TM1
Version: 2.0.9
Excel Version: too many

Re: Transfer of data between servers

Post by scrumthing » Wed Jun 03, 2020 8:52 am

I don't think something like this will happen. They want to get rid of TI and not improve it...
You can do it relatively simple with the rest api and for example tm1py. That offers in any case much more flexibility and a better performance than TI would, I think.
There is no OLAP database besides TM1!

User avatar
Alan Kirk
Site Admin
Posts: 6190
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2 Classic (PAW-free zone)
Excel Version: 2010 and 2016
Location: Sydney, Australia
Contact:

Re: Transfer of data between servers

Post by Alan Kirk » Wed Jun 03, 2020 10:37 pm

scrumthing wrote:
Wed Jun 03, 2020 8:52 am
I don't think something like this will happen. They want to get rid of TI and not improve it...
In the interview that I did with Hubert he stated straight out that that won't be happening in the next version; it's something further down the road. In any case, this isn't really about TI as such, it's about the ease of creating a data source which will just happen to be used in TI at present. In an ideal world it should be no more difficult to use an object on another server as a data source than it is to use an object on the current server, without the need to write screeds of code or query text.

Much as I have advocated strongly for users to learn MDX and the REST API, and continue to do so, I recognise that there will be quite a few people who simply don't want to do that through either lack of time or lack of inclination.
scrumthing wrote:
Wed Jun 03, 2020 8:52 am
You can do it relatively simple with the rest api and for example tm1py. That offers in any case much more flexibility and a better performance than TI would, I think.
That would require something external to the server, which is in my view something that should be a last resort. Second, if the TI data source were to use the REST API to pull from the other server then by definition there shouldn't really be any difference in performance.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

Wim Gielis
MVP
Posts: 2527
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Transfer of data between servers

Post by Wim Gielis » Wed Jun 03, 2020 11:04 pm

Alan Kirk wrote:
Tue Jun 02, 2020 10:41 pm
It has 8 votes at the moment
11 now, after having added mine.

Just to add that in tools such as Jedox, this is all standard. You would have different Integrator projects, all of which can interact with multiple Modeler projects.
Last edited by Wim Gielis on Fri Jun 12, 2020 9:56 pm, edited 1 time in total.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 112 TM1 articles and a lot of custom code
Newest blog article: Updating process line numbers with AutoHotKey

User avatar
Ajay
Regular Participant
Posts: 156
Joined: Wed May 14, 2008 8:27 am
OLAP Product: TM1
Version: 10.2.0, PA 2.0.9
Excel Version: 2016
Location: London

Re: Transfer of data between servers

Post by Ajay » Thu Jun 11, 2020 6:55 am

Totally agree, and have added my vote to this too.

I'm accomplishing this at the moment, with some TI code and batch files, as I am restricted to output what I need from server A, in *.txt files, then move it via batch code to another location, on server B and process the files there using TI.

It is a bit of a "faff", but I agree that a datasource would be the way to go with this.

Ajay

User avatar
Alan Kirk
Site Admin
Posts: 6190
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2 Classic (PAW-free zone)
Excel Version: 2010 and 2016
Location: Sydney, Australia
Contact:

Re: Transfer of data between servers

Post by Alan Kirk » Wed Sep 16, 2020 7:03 pm

At least this one has also now moved to the "Future Consideration" pile.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

User avatar
gtonkin
MVP
Posts: 812
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.3
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: Transfer of data between servers

Post by gtonkin » Wed Sep 16, 2020 7:12 pm

Just saw that too - no doubt it will be via the RestAPI but fine as long as it is native, easily configurable and lightning fast with error-handling etc. etc.

Post Reply