DimensionDeleteAllElements failover

Post Reply
vladino
Posts: 61
Joined: Sat Nov 06, 2010 10:10 am
OLAP Product: Cognos TM1
Version: 10.2.2
Excel Version: Excel 2013

DimensionDeleteAllElements failover

Post by vladino » Fri Jun 16, 2017 9:23 am

Hi guys,
I'm using DimensionDeleteAllElements in Prolog and then re-create some of elements back in Metadata to preserve the data.

But what if there's something wrong in Metadata and the script fails? Is my entire dimension (and the data) gone forever?

If yes - is there any way how to avoid this?

BR
Vladino

User avatar
Alan Kirk
Site Admin
Posts: 5657
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: 9.5.2 64 bit moving to 10.2.2
Excel Version: 2010
Location: Sydney, Australia
Contact:

Re: DimensionDeleteAllElements failover

Post by Alan Kirk » Fri Jun 16, 2017 9:41 am

vladino wrote:Hi guys,
I'm using DimensionDeleteAllElements in Prolog and then re-create some of elements back in Metadata to preserve the data.

But what if there's something wrong in Metadata and the script fails? Is my entire dimension (and the data) gone forever?

If yes - is there any way how to avoid this?
The way that a classic update of a dimension (that is, prior to the new REST API and the "Direct" TI functions) works is this:
- A copy of the dimension is created in memory;
- Updates are made to that copy in the Prolog tab and the Metadata tab;
- At the finish of the Metadata tab the dimension is validated for consistency and if that validation passes, then the copy of the dimension is registered to replace the real dimension.

Here's the problem; in my experience with the classic API a dimension with no elements will pass validation just fine.

So yes, if you delete all of the elements in the Prolog and your data source fails such that the elements are not added back in the metadata, it is conceivable that you will end up with a validated, empty dimension.

As you have envisaged, this will usually be A Very Bad Thing.

The solution? Never use the DimensionDeleteAllElements function for a data -bearing dimension. Ever. Not once. Never. Just don't. No, get your finger off it and step away from the console. Now. Usually, there is no need for it. You may need to strip down the consolidation tree but you don't need to strip out the N elements. And for mine, DimensionDeleteAllElements is just too damn dangerous.

The only time I ever use it is when the dimension is NOT used in a cube which stores data but is used as a pseudo array (since I'll win Powerball before we ever see real arrays in the TI language) and I'm just using it to store a collection of elements.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

Wim Gielis
MVP
Posts: 1419
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0
Excel Version: 2016
Location: Brussels, Belgium
Contact:

Re: DimensionDeleteAllElements failover

Post by Wim Gielis » Fri Jun 16, 2017 10:43 am

vladino wrote:But what if there's something wrong in Metadata and the script fails? Is my entire dimension (and the data) gone forever?
I wrote a generic TI process that can go through all processes' PRO files, and listing (in a control cube) many properties, line counts, parameters, ...
Also, it detects the presence of statements like DimensionDeleteAllElements, ExecuteProcess, and so on.
My clients can get a list of processes where this function exists. I use the function exactly like Alan mentions,
but if the client is using the TI wizard or writes his own scripts, I can signal this.

Best regards,

Wim
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 103 TM1 articles and a lot of custom code
Newest blog article: TM1 message log analysis with Power Query

User avatar
Alan Kirk
Site Admin
Posts: 5657
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: 9.5.2 64 bit moving to 10.2.2
Excel Version: 2010
Location: Sydney, Australia
Contact:

Re: DimensionDeleteAllElements failover

Post by Alan Kirk » Fri Jun 16, 2017 10:48 am

Wim Gielis wrote:...but if the client is using the TI wizard...
... then his eternal soul is condemned to purgatory, and may the coding gods show mercy.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

User avatar
mattgoff
MVP
Posts: 502
Joined: Fri May 16, 2008 1:37 pm
OLAP Product: TM1
Version: 10.2.2.6
Excel Version: 2016
Location: Florida, USA
Contact:

Re: DimensionDeleteAllElements failover

Post by mattgoff » Fri Jun 16, 2017 2:08 pm

Alan Kirk wrote:You may need to strip down the consolidation tree but you don't need to strip out the N elements..
Sounds like a good enhancement request. It would be handy to have a DimensionDeleteAllHierarchy as a easier alternative than coding it up yourself.

Matt
Please read and follow the Request for Assistance Guidelines. It helps us answer your question and saves everyone a lot of time.

Wim Gielis
MVP
Posts: 1419
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0
Excel Version: 2016
Location: Brussels, Belgium
Contact:

Re: DimensionDeleteAllElements failover

Post by Wim Gielis » Fri Jun 16, 2017 3:14 pm

mattgoff wrote:
Alan Kirk wrote:You may need to strip down the consolidation tree but you don't need to strip out the N elements..
Sounds like a good enhancement request. It would be handy to have a DimensionDeleteAllHierarchy as a easier alternative than coding it up yourself.

Matt
+1
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 103 TM1 articles and a lot of custom code
Newest blog article: TM1 message log analysis with Power Query

User avatar
mattgoff
MVP
Posts: 502
Joined: Fri May 16, 2008 1:37 pm
OLAP Product: TM1
Version: 10.2.2.6
Excel Version: 2016
Location: Florida, USA
Contact:

Re: DimensionDeleteAllElements failover

Post by mattgoff » Fri Jun 16, 2017 3:24 pm

Please read and follow the Request for Assistance Guidelines. It helps us answer your question and saves everyone a lot of time.

Mark RMBC
Posts: 67
Joined: Tue Sep 06, 2016 7:55 am
OLAP Product: TM1
Version: 10.1.1
Excel Version: Excel 2010

Re: DimensionDeleteAllElements failover

Post by Mark RMBC » Fri Jun 16, 2017 3:57 pm

It would be handy to have a DimensionDeleteAllHierarchy as a easier alternative than coding it up yourself.
Yes but having the option of excluding the very top consolidation level would be nice, if somewhat wiping our backsides!
I wrote a generic TI process that can go through all processes' PRO files
Would I be rude in asking to see that?!

User avatar
Alan Kirk
Site Admin
Posts: 5657
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: 9.5.2 64 bit moving to 10.2.2
Excel Version: 2010
Location: Sydney, Australia
Contact:

Re: DimensionDeleteAllElements failover

Post by Alan Kirk » Fri Jun 16, 2017 7:42 pm

Mark RMBC wrote:
It would be handy to have a DimensionDeleteAllHierarchy as a easier alternative than coding it up yourself.
Yes but having the option of excluding the very top consolidation level would be nice, if somewhat wiping our backsides!
Which gets you what, exactly? All you're left with is a C type element which has no children. If all of the other consolidation elements have been destroyed and need to be recreated, one extra consolidation node makes no difference. There is nothing special about a top consolidation node compared to any other one, especially as in any dimension with multiple trees there may be multiple "very top level" consolidations. And just to put the cherry on top of that cake, an element which is the "very top level" consolidation in one hierarchy may not be in another one.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

Wim Gielis
MVP
Posts: 1419
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0
Excel Version: 2016
Location: Brussels, Belgium
Contact:

Re: DimensionDeleteAllElements failover

Post by Wim Gielis » Fri Jun 16, 2017 8:01 pm

Mark RMBC wrote:
I wrote a generic TI process that can go through all processes' PRO files
Would I be rude in asking to see that?!
Voilà, there you go. 1,900 lines of coding ;-)

Unedited these processes will need at least TM1 10.2.2 FP4.

The MAIN process executes the 2 other processes.

Any questions / remarks / errors / improvements: let me know.
Certainly the process to analyse chores is new, all input is welcome.

Among (many) other things, these functions can list which processes execute other processes, how many times, in what tab of the process, etc.

EDIT June 18, 2017;

Attachments were removed and now I created a new topic here:
http://www.tm1forum.com/viewtopic.php?f=21&t=13373
The files in that topic are the latest versions.
Last edited by Wim Gielis on Sun Jun 18, 2017 3:11 pm, edited 10 times in total.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 103 TM1 articles and a lot of custom code
Newest blog article: TM1 message log analysis with Power Query

User avatar
Alan Kirk
Site Admin
Posts: 5657
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: 9.5.2 64 bit moving to 10.2.2
Excel Version: 2010
Location: Sydney, Australia
Contact:

Re: DimensionDeleteAllElements failover

Post by Alan Kirk » Fri Jun 16, 2017 8:02 pm

Happy 500th. I've given you another vote AND a watchlist addition, which apparently matters.

Of course, I was only able to do that after I followed the link above, hit the obligatory page which says:
Oops, that's not right!
Sorry, the page you're looking for cannot be displayed

Please try again later.
which both lies to you (yes, it can be displayed) and misleads you ("trying again later" is not the issue), logged in despite not being prompted to, was returned not to the RFE but to the cutsie little "Oops, that's not right" page, returned to this thread, clicked on the link again, and this time, because I was already logged in, went to the RFE.

All these years, and they still haven't fixed that.

"That's not right" indeed. In a perfect world anyone who was capable of such absolutely cr@p website coding would not be let anywhere near a keyboard, much less a production environment.

At present I just use a generic process for this. I would have written my own but since I already have the Bedrock processes installed Bedrock.Dim.Hierarchy.Unwind.Consolidation and Bedrock.Dim.Hierarchy.Unwind.All work well enough, so I just use those.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

User avatar
mattgoff
MVP
Posts: 502
Joined: Fri May 16, 2008 1:37 pm
OLAP Product: TM1
Version: 10.2.2.6
Excel Version: 2016
Location: Florida, USA
Contact:

Re: DimensionDeleteAllElements failover

Post by mattgoff » Fri Jun 16, 2017 9:24 pm

Alan Kirk wrote:At present I just use a generic process for this. I would have written my own but since I already have the Bedrock processes installed Bedrock.Dim.Hierarchy.Unwind.Consolidation and Bedrock.Dim.Hierarchy.Unwind.All work well enough, so I just use those.
Bedrock is awesome (with the caveat that a huge portion of what makes it awesome is that it plugs huge holes in the TI language), but it sooo cluttered up my process list, and I didn't really use much of it, so I got rid of it. I wish there were a couple levels of Display Control Objects where I could prefix something with a symbol, say {, or set a flag a la Enable Security Access to hide it in the GUI. Or maybe even if they allowed folders in places other than Applications.

I wish they would fix all this stuff before they invested in refactoring TM1 for containers and TM1 web for Tomcat. I DO appreciate some of the recent advancements, like auto-destructing subsets and views, MTQ, and persistent feeders.

Matt
Please read and follow the Request for Assistance Guidelines. It helps us answer your question and saves everyone a lot of time.

User avatar
Alan Kirk
Site Admin
Posts: 5657
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: 9.5.2 64 bit moving to 10.2.2
Excel Version: 2010
Location: Sydney, Australia
Contact:

Re: DimensionDeleteAllElements failover

Post by Alan Kirk » Fri Jun 16, 2017 10:20 pm

mattgoff wrote:
Alan Kirk wrote:At present I just use a generic process for this. I would have written my own but since I already have the Bedrock processes installed Bedrock.Dim.Hierarchy.Unwind.Consolidation and Bedrock.Dim.Hierarchy.Unwind.All work well enough, so I just use those.
Bedrock is awesome (with the caveat that a huge portion of what makes it awesome is that it plugs huge holes in the TI language), but it sooo cluttered up my process list, and I didn't really use much of it, so I got rid of it. I wish there were a couple levels of Display Control Objects where I could prefix something with a symbol, say {, or set a flag a la Enable Security Access to hide it in the GUI. Or maybe even if they allowed folders in places other than Applications.
Ayup. I think this thread about grouping objects into folders (from 6 years ago now) pre-dated the RFE system but it was hardly the first time it was brought up. Of course it also pre-dates the "we ain't doin' nothin' with Perspectives ever again" policy, so there isn't much point bringing it up as a formal RFE now.

At least with Bedrock the processes all sit together so I've learnt to just skim over them. The main reason that I've stuck with them is:
(a) I don't need to reinvent the wheel;
(b) Once I get used to them at least I know how they work, and they're already more or less documented (though some of the documentation is flat out wrong and was clearly written for an earlier iteration of the process). When I write my own stuff then something that seemed like a good approach one month may not seem so the following month. So either I sit down and write my own complete and full set of processes which behave consistently (which would seem to be a bit redundant when Bedrock is already there), or I end up with a bunch of generic processes that were written over time and are subtly different in how they act.

That said, I do indeed supplement it with my own generic processes and have "rewritten" one or two using the name BedrockX. Off the top of my head (I'm not connected to TM1 at the moment) I think one of those was where the process didn't have a parameter to disable logging when doing a large data movement, which resulted in multi-gigabyte log files. I therefore needed an incarnation which had the relevant parameter. Now that Bedrock has been moved to Github I may look at doing a branch for such things when I have the time.
mattgoff wrote:I wish they would fix all this stuff before they invested in refactoring TM1 for containers and TM1 web for Tomcat.
Who needs basic usability when you can point to Cutting! Edge! and (in the case of Docker) Flavour! Of! The! Month! development technology?

Though I'll admit that I've come around a little more to the Tomcat thing; I think in the longer term it's going to be a better idea than hitching the wagon to IIS if for no other reason than improved platform flexibility. And certainly the display of websheets seems to me to be a damn sight better than the "guess what I'll look like today" crayon drawings that you sometimes get in ye olde 1001 Excel Sessions method that pre-dated Tomcat.
mattgoff wrote:I DO appreciate some of the recent advancements, like auto-destructing subsets and views, MTQ, and persistent feeders.
Agreed.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

Mark RMBC
Posts: 67
Joined: Tue Sep 06, 2016 7:55 am
OLAP Product: TM1
Version: 10.1.1
Excel Version: Excel 2010

Re: DimensionDeleteAllElements failover

Post by Mark RMBC » Tue Jun 20, 2017 11:50 am

Wim – thanks for this, much appreciated.

Alan – Well I was thinking (probably not too clearly) that a deleteallhierarchy function would mean that any of the hierarchy that I create additionally would also be deleted when in fact I would want to keep those in place no matter what. I was thinking about when I link to an ODBC source, use SQL to build the dimension but then in the TI add in a few levels, usually high up to complete the dimensional hierarchy. So for example on a chart of accounts cost centre dimension I added in at the very top a level for the structure being used and underneath this top level I added in a level to capture Total Expenditure and Total Income.

So I wondered if this proposed function deleteallhierarchy could in some way have a parameter to exclude level(s) but then I narrowed it down to the top level because I thought adding in multiple levels to a parameter would be too complex.

Though I then thought, what the hell just go and recreate the additional levels in the TI because how much work would that be, but still thought I would add in the proposal anyway. What I was really after was a link to Wim’s code!

BrianL
MVP
Posts: 257
Joined: Mon Nov 03, 2014 8:23 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2 PA2
Excel Version: 2016

Re: DimensionDeleteAllElements failover

Post by BrianL » Thu Jul 13, 2017 6:56 pm

I know I'm a bit late to the party, but I think it's also worth noting that as of PA 2.0 you can explicitly ask TI to rollback and discard all work.

https://www.ibm.com/support/knowledgece ... lback.html

You'd still have to detect the metadata error in your TI code and then ask for a rollback to prevent the dimension destruction, but at least it's possible.

Post Reply