Bedrock 4 for IBM Planning Analytics is available

Post Reply
lotsaram
MVP
Posts: 3318
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Bedrock 4 for IBM Planning Analytics is available

Post by lotsaram » Tue Apr 23, 2019 11:25 am

Hi all TM1 & Planning Analytics fans,

A new version of the bedrock TurboIntegrator library has just been released. This is version 4 of bedrock and is a ground up rebuild of the library specifically for IBM Planning Analytics.

Bedrock is a modular TurboIntegrator library with the intentions of
  • significantly reducing the volume of custom TI code needed to perform most tasks
  • increasing TM1 code quality and robustness
  • improving system maintainability
  • flattening the learning curve for new developers
  • accelerating project development & implementation cycle time
Basically it is about making developers more productive and writing less code bringing a modular functional library approach to TM1 / Planning Analytics which is standard in other toolsets and languages. Don't re-invent the wheel, use bedrock!

Best of all bedrock is free and open source. All members of the TM1 / IBM Planning Analytics community are welcome and encouraged to contribute to the project and make it better!

To read about what's new in bedrock 4 vs. the last version please refer to the "what's new" post in the project wiki over at GitHub.

Note: that bedrock 4 requires a minimum TM1 server version of 11.3 to function with 11.4 being the suggested minimum version (that's PAL 2.0.5 & 2.0.6 respectively).
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
macsir
Community Contributor
Posts: 651
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1, SSAS
Version: 9.5 - 10.2
Excel Version: 97 - 2013
Contact:

Re: Bedrock 4 for IBM Planning Analytics is available

Post by macsir » Tue Apr 23, 2019 8:43 pm

Thanks for great work! Will check that out later. :D
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/

Wim Gielis
MVP
Posts: 2260
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Bedrock 4 for IBM Planning Analytics is available

Post by Wim Gielis » Tue Apr 23, 2019 10:20 pm

Many thanks Lotsaram and coworkers. Good job ! :P

Just a couple of ideas for new additions to the toolkit.

What I use a lot in my own TI library is creating permanent subsets in an automated way - like a filter by level 0 subset, Default subset, orphans subset, etc. You can do it for 1 dimension, for all dimensions, for all dimensions of a given cube, ... wildcards allowed. Subsets will be dynamic but static isan option too. The TI supports the languages that the TM1 interface also supports in Architect. The process is not yet adapted for hierarchies.

Another one is a TI to duplicate a dimension. Options are there to choose what is duplicated (structures, attributes, subsets, everything, ...), clean out the dimension if it exists, etc.

Elaborate TI’s to spot (possible) double counting issues.

TI to fill the }ElementAttributes cube for aliases in case the cell is empty (alias = principal name)

TI to fill a cube with information on views and subsets and their usage.

A process to maintain the “current / past / next” elements in dimensions like Year, Month, Scenario, Version.

A generic process to test the data source of a process (text files, ODBC, view, subset). Questions like: does the text file exist ? Does it exist and not opened in Excel ? Does it contain the right number of columns ? Isn’t it empty ?

Processes to delete objects satisfying certain name patterns. Like delete all private views with prefix “temp”.

Processes for emailing (PowerShell) or FTP of files or convert .xlsx to .csv.

And so on.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

lotsaram
MVP
Posts: 3318
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Bedrock 4 for IBM Planning Analytics is available

Post by lotsaram » Wed Apr 24, 2019 5:39 am

Wim Gielis wrote:
Tue Apr 23, 2019 10:20 pm
Just a couple of ideas for new additions to the toolkit....
Hi Wim,

Bedrock is open to contributions by all. You would be welcome to contribute.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

Bakkone
Posts: 102
Joined: Mon Oct 27, 2014 10:50 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2013

Re: Bedrock 4 for IBM Planning Analytics is available

Post by Bakkone » Wed Apr 24, 2019 9:47 am

Thx for your great work!

Looking forward to Wims additions. ;)

Wim Gielis
MVP
Posts: 2260
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Bedrock 4 for IBM Planning Analytics is available

Post by Wim Gielis » Wed Apr 24, 2019 10:17 am

Bedrock coding follows a certain format, which is good.
It means that any processes that are to be added, should also follow the format (unique view names, pLogOutput, etc.)
Therefore it means that processes need to be rewritten to adhere to the conventions in the bedrock library.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

lotsaram
MVP
Posts: 3318
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Bedrock 4 for IBM Planning Analytics is available

Post by lotsaram » Wed Apr 24, 2019 3:15 pm

To answer Wim's questions (but also for anybody else who might have the same or similar questions)

Yes to contribute any new processes there are certain conventions to be respected so that the library is coherent. For example:
  • pLogOutput as 1st parameter and consistent handling of informational and error level messages to server message log and return code finalization on epilog
  • consistent naming of parameters throughout library
  • internal views & subsets (withing single process) always temporary, otherwise give option to create permanent (temp by default)
  • consistent constants declaration on prolog
  • validation of input parameters
There are also a few principals to adhere to:
  • bedrock is TurboIntegrator only. No dependencies on any other objects, be that cubes, dimensions, batch files, powershell scripts, etc. The library should be as portable and lightweight as possible.
  • Don't break backward compatibility.
    • Add but don't remove
    • Within a major version the library should be upgradeable with no impact to any custom processes relying on bedrock.
    • Within a major version parameters should bot be deleted or renamed. Where new parameters are introduced running with the default value of new parameter should produce the same behaviour as running the same process before without the parameter.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

Wim Gielis
MVP
Posts: 2260
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: Bedrock 4 for IBM Planning Analytics is available

Post by Wim Gielis » Wed Apr 24, 2019 3:52 pm

lotsaram wrote:
Wed Apr 24, 2019 3:15 pm
There are also a few principals to adhere to:
  • bedrock is TurboIntegrator only. No dependencies on any other objects, be that cubes, dimensions, batch files, powershell scripts, etc. The library should be as portable and lightweight as possible.
No batch files but bedrock will execute commands as in:

Code: Select all

          sCmd = 'cmd /c "copy """' | cCubeRuleFileName | '"""  """' | cBackupDirFile |'""" "';
          ExecuteCommand(sCmd,1);
Also, I see that bedrock processes write their own batch files to execute them. This is allowed as a practice ?

Code: Select all

sFileName           = 'Bedrock.MkDir.bat' ;
ASCIIOUTPUT( sFileName, 'md "' | sBackupDir |'"' );
ExecuteCommand ( 'Bedrock.MkDir.bat ', 1 );
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 model with AutoHotKey as a service or application

lotsaram
MVP
Posts: 3318
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Bedrock 4 for IBM Planning Analytics is available

Post by lotsaram » Wed Apr 24, 2019 8:30 pm

There's no external dependencies as any batch file that bedrock writes it also removes in the same process. However, this is a little weird that the backup process writes a single line batch file versus calling the console directly. It does seem unnecessary.

Of course bedrock has come from a Windows background. The goal is that everything should run on Linux as well as Windows so any ExecuteCommand calls should work on either OS (or there should be a parameter switch in any process needing the command line to indicate Windows vs Linux). It isn't there yet but with enough smart people it will be.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

EvgenyT
Community Contributor
Posts: 275
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: Cognos TM 1
Version: 10.2.2
Excel Version: 2013
Location: Sydney, Australia

Re: Bedrock 4 for IBM Planning Analytics is available

Post by EvgenyT » Thu Apr 25, 2019 3:32 am

I like the new Parallelization option. Still up to a developer to make sure a process does not exceed maximum available threads or just simply use RushTI?

Cheers

Evgeny

lotsaram
MVP
Posts: 3318
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Bedrock 4 for IBM Planning Analytics is available

Post by lotsaram » Thu Apr 25, 2019 5:47 am

EvgenyT wrote:
Thu Apr 25, 2019 3:32 am
I like the new Parallelization option. Still up to a developer to make sure a process does not exceed maximum available threads or just simply use RushTI?

Cheers

Evgeny
At this point there's no mechanism when using RunProcess to query the status of the thread. So the calling process has no way to know which threads have finished and which are still running. We put in a mechanism to set the maximum number of threads to be called, that's all that can be done at this stage. May change in the future when RunProcess acquires status reporting.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

lotsaram
MVP
Posts: 3318
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Bedrock 4 for IBM Planning Analytics is available

Post by lotsaram » Tue Apr 30, 2019 4:14 pm

As of today PA 2.0.7 (TM1 11.5) is available on Fix Central. In this version the new RunProcess function (which has been present but incognito) is officially supported.
The new bedrock library already optionally makes use of RunProcess in processes which deal with reading and writing cube data.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
PavoGa
Community Contributor
Posts: 381
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.05
Excel Version: 2013 PAW
Location: Cleveland, Tennessee

Re: Bedrock 4 for IBM Planning Analytics is available

Post by PavoGa » Tue Apr 30, 2019 8:40 pm

Fantastic, thanks lotsaram.
Ty
Cleveland, TN

lotsaram
MVP
Posts: 3318
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Bedrock 4 for IBM Planning Analytics is available

Post by lotsaram » Thu May 02, 2019 3:53 pm

@PavoGa you also got your wish expressed in https://www.tm1forum.com/viewtopic.php? ... 023#p65002
for new method of exporting cube data.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
Alan Kirk
Site Admin
Posts: 5997
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2 Classic (PAW-free zone)
Excel Version: 2010 and 2016
Location: Sydney, Australia
Contact:

Re: Bedrock 4 for IBM Planning Analytics is available

Post by Alan Kirk » Sun Jul 14, 2019 1:23 am

Personally, I don't think I'll be jumping into the Bedrock 4 pool all that soon. I have it on all of my servers, but this morning was my first encounter with using it.

It was not auspicious.

It was a call to }bedrock.cube.viewandsubsets.create, with pLogOutput set to 1.

Yeah, all good, hunky dory, no issues here, nosiree!

Well, aside from the fact that no view was created. And no subsets were created. And no errors were recorded.

Aside from that, everything worked great, just as reported.

(And yes, I DID have the pTemp argument set to 0 because regardless of other opinions on the matter I do NOT, at this point, want views and subsets to be temporary by default.)

So I looked into the actual code for the process.

Oh. My. Fricking. God.

It's not that I can't understand it, it's that I can spend a couple of hours figuring out what it's doing OR I can try to get this last minute work that was thrown at me on a Sunday morning done and out of the way.

But it's not the enhanced complexity of the code that I have an issue with. It's this:
"No more pDebug. When bedrock was first released some 10 years ago the built in logging modes (0 no logging, 1 log actions, 2 log actions which would have been performed but don't do them) was one of the great features of the library. There was no alternative in TurboIntegrator other than to write text files for debugging purposes. But times have changed, with support in TM1's Rest API for debugging and TM1 IDEs like Arc for TM1 supporting step-through debugging there is simply no need for this anymore in the library, hence pDebug is no more."
Mmm. Times have changed. Quite.

You can faff around with the API to do debugging, unless you have some kind of commercial IDE to do it in.

Of course I could write my own REST-based IDE which could do this as well, and have already started dabbling in this, but finishing it by lunch time would seem a tad optimistic.

So the relatively easy way... is log files. Log files which allow you to capture the state of as many or as few variables as you need and see what the values are as the process runs. Log files which will work for non-programmer types who can write TIs but get brain freeze on things like API calls. Log files which will still work into 9.5.2 and 10.2.2.

Which Bedrock 4 does not have. If what it sees to be an error occurs, it may record it in the server log but when it fails to see an error, for whatever reason, then you have no idea what happened, or why. You just know that you didn't get the outcome that you were after.

My solution? I called the equivalent Bedrock 3 process instead.

Hey lookie, I got a view. With correctly defined subsets.

Obviously I may need to revisit this if we start doing alternative hierarchies, which obviously aren't supported in Bedrock 3. However if Bedrock 4 fails straight out of the gate with doing something as basic as creating a view, with me needing to dig into the code line by line to figure out why, I'm in no rush to adopt it.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

Post Reply