difference between prolog, metadata, data, epilog

Post Reply
plkkw
Posts: 23
Joined: Wed Sep 21, 2011 6:18 am
OLAP Product: TM1
Version: 9.4.1
Excel Version: 2003

difference between prolog, metadata, data, epilog

Post by plkkw » Fri Sep 30, 2011 5:52 am

Hi,

I get the definition of prolog, metadata, data, epilog:
==============================
Prolog
A series of actions to be executed before the data source is processed

Metadata
A series of actions that update or create cube, dimensions, and other metadata structures during processing.

Data
A series of data actions to be executed for each record in the data source.

Epilog
A series of actions to be executed after the data source is processed.
==================

My question is:
1) what is "data source"?
2) Is prolog unable to create cube, dimension (and metadata can do it)?
3) Can i take "prolog, epilog" as feature similar to (database table) before-trigger, after-trigger?
4) what is the processing order of "prolog, metadata, data, epilog"?
5) do you have practical example?

thanks.

winsonlee
Regular Participant
Posts: 180
Joined: Thu Jul 01, 2010 3:06 am
OLAP Product: Cognos Express
Version: 9.5
Excel Version: 2007
Location: Melbourne, Australia

Re: difference between prolog, metadata, data, epilog

Post by winsonlee » Fri Sep 30, 2011 6:16 am

1) data source definition is quite self explanatory. It is a data that you obtain from somewhere eg CSV file, or using SQL statement to get the data from sql server.

2) prolog (before) - you can use prolog to create cube and dimension eg CubeCreate or DimensionCreate.
metadata (during) - mainly populating the data to the dimension. It will actually goes through every single record from the datasource and write the data to the dimension.
data(during) - mainly writing data onto the cube.
epilog(after) - anything that you want to do after the process is completed, eg run a batch file, save the data on to the hard disk.

3) prolog, metadata and data, epilog are in sequence.

plkkw
Posts: 23
Joined: Wed Sep 21, 2011 6:18 am
OLAP Product: TM1
Version: 9.4.1
Excel Version: 2003

Re: difference between prolog, metadata, data, epilog

Post by plkkw » Fri Sep 30, 2011 8:52 am

then, metadata and data is quite similar. do you mean that the process will run coding in metadata first, and coding in data?

lotsaram
MVP
Posts: 3251
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: difference between prolog, metadata, data, epilog

Post by lotsaram » Fri Sep 30, 2011 9:25 am

plkkw wrote:then, metadata and data is quite similar. do you mean that the process will run coding in metadata first, and coding in data?
Metadata and data tabs are only similar in that all code on these tabs is an implicit loop and is repeated for each record in the data source. The metadata tab is (as the name implies) for updating dimensions. At the conclusion of metadata processing, that is after the last record of the source file is read and the connection to the source is closed then and only then are changes to the dimension(s) that have been updated saved.

If there are data updates only then there should be no code on the metadata tab, and vice versa if you only want to update dimensions. Otherwise you process the data source an additional time unnecessarily.

ParisHilton
Posts: 73
Joined: Fri Apr 23, 2010 11:35 am
OLAP Product: Tm1
Version: 9.5
Excel Version: 2007 2010

Re: difference between prolog, metadata, data, epilog

Post by ParisHilton » Thu Dec 29, 2011 3:25 pm

In my head, I'd add these 'explanations' to help newbies like me understand how it works.

The prolog tab hasn't seen the data yet. So it's good for operations like 'Create Dimension' , 'Delete all elements in a Dimension' , add in a hardcoded element to the dimension , such as 'ALL NL codes'.

The metadata tab, will run through the data source and it's here you can add elements to a dimension based on the data. For example if your data source has an NL code which is not already in the cube you can add it here . I guess (clarification from an expert would be nice), that you need to add it here, so that data can be loaded against it.

The data tab again loops over all the data. Primarily you are loading 'fact' data here in BI terms.

The epilog is for tidying up afterwards, I haven't used it for much other than writing out logs, but I am sure someone can add in a decent example!

Hopefully I've got this right , and if not it's a useful post to be picked apart to share the knowledge!

P
“The way I see it, you should live everyday like its your birthday”


TM1 9.5 Cognos BI 8.4 Excel 2007. 128GB Ram. E7450@2.40Ghz -24 core. Fluffy white dog.

MarenC
Posts: 9
Joined: Sat Jun 08, 2019 9:55 am
OLAP Product: Planning Analytics
Version: Planning Analytics 2.0
Excel Version: Excel 2016

Re: difference between prolog, metadata, data, epilog

Post by MarenC » Wed Jul 03, 2019 9:50 am

Hi,

I was told that in some situations processes needed to be split up because certain things created in one process would not be recognised until that process had fully run and therefore a second process would be required to fully complete the tasks. For example dimension builds.

Is there any definitive list which details what is and isn't recognised on the tabs? And when processes need to be split up?

regards, Maren

User avatar
orlando
Posts: 76
Joined: Fri Aug 04, 2017 8:27 am
OLAP Product: TM1
Version: PAL 2.0
Excel Version: Office 365

Re: difference between prolog, metadata, data, epilog

Post by orlando » Wed Jul 03, 2019 11:25 am

MarenC wrote:
Wed Jul 03, 2019 9:50 am

Is there any definitive list which details what is and isn't recognised on the tabs? And when processes need to be split up?

hi maren,

maybe you should read the documentation - this will give you a good overview of what TM1 can do and how it works.

https://www.ibm.com/support/knowledgece ... -gen2.html
https://www.ibm.com/support/knowledgece ... -gen2.html
https://www.ibm.com/support/knowledgece ... -gen5.html
https://www.ibm.com/support/knowledgece ... -gen6.html

Even if this links are for TM1 10.2.2 and not for PA 2.0 - it'll help.

Best regards
orlando

MarenC
Posts: 9
Joined: Sat Jun 08, 2019 9:55 am
OLAP Product: Planning Analytics
Version: Planning Analytics 2.0
Excel Version: Excel 2016

Re: difference between prolog, metadata, data, epilog

Post by MarenC » Wed Jul 03, 2019 11:58 am

Hi Orlando,

How do you know that I haven't read the documentation? Maybe I missed the bit about when you needed to split up processes!

But I take it you don't know and hopefully someone else can provide something useful!

Emixam
Posts: 16
Joined: Tue May 21, 2019 3:33 pm
OLAP Product: TM1
Version: TM1 10.2
Excel Version: 2016

Re: difference between prolog, metadata, data, epilog

Post by Emixam » Wed Jul 03, 2019 1:16 pm

Hi,

Splitting processes is more a "design technique" instead of a necessity.

I suggest you this white paper on the best practice to develop TM1 Turbo Integrator processes

https://static1.squarespace.com/static/ ... grator.pdf

Also, a good example of splitting process is the Bedrock library (https://github.com/cubewise-code/bedrock). For example, instead of creating a dimension directly in your TI, you can easily use an existing process from Bedrock (Bedrock.Dim.Create.pro).

Enjoy !
Last edited by Emixam on Wed Jul 03, 2019 1:22 pm, edited 2 times in total.

tomok
MVP
Posts: 2561
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: difference between prolog, metadata, data, epilog

Post by tomok » Wed Jul 03, 2019 1:17 pm

MarenC wrote:
Wed Jul 03, 2019 11:58 am
Maybe I missed the bit about when you needed to split up processes!
Maybe because there isn't anything in the documentation about splitting up processes and maybe because I've never heard of any such requirement. You can most certainly do dimension builds in one process. There is no technical requirement to break it up. Yes, there may be times, because of certain conditions, where you may split a dimension build into different processes. An example would be a process to unwind the existing hierarchy, another to add new leaf nodes and then another to re-build the hierarchy. However, this would be at the developers discretion, maybe because of code sharing or source data limitations, and not because TM1 forces you to. Maybe you should go back to the person who told you processes need to be split and ask them what they are talking about.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/

MarenC
Posts: 9
Joined: Sat Jun 08, 2019 9:55 am
OLAP Product: Planning Analytics
Version: Planning Analytics 2.0
Excel Version: Excel 2016

Re: difference between prolog, metadata, data, epilog

Post by MarenC » Wed Jul 03, 2019 1:28 pm

Hi,

ok thanks for the advice, something must have got lost in translation somewhere along the line!

Incidentally I didn't bring up the guides, Orlando did!

regards, Maren

Wim Gielis
MVP
Posts: 2087
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1
Version: PAL 2.0.7
Excel Version: Office 365 - latest
Location: Brussels, Belgium
Contact:

Re: difference between prolog, metadata, data, epilog

Post by Wim Gielis » Wed Jul 03, 2019 6:19 pm

For example in the prolog tab you can add an element for allocations. In the Data tab you can process data on the element and in the Epilog tab you cannot delete the element. Therefore you need to have the Prolog tab of a second process to do it.
Best regards,

Wim Gielis

Excel Most Valuable Professional, 2011-2014
http://www.wimgielis.com ==> 109 TM1 articles and a lot of custom code
Newest blog article: Set up a TM1 service with AutoHotKey

lotsaram
MVP
Posts: 3251
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: difference between prolog, metadata, data, epilog

Post by lotsaram » Wed Jul 03, 2019 7:55 pm

Love a bit of thread necromancy.
Wim Gielis wrote:
Wed Jul 03, 2019 6:19 pm
For example in the prolog tab you can add an element for allocations. In the Data tab you can process data on the element and in the Epilog tab you cannot delete the element. Therefore you need to have the Prolog tab of a second process to do it.
Well you should be able to and in most cases you can but there have been cases of versions with bugs which would crash the server when the source of a process was deleted on the epilog of the same process. (In the case of deleting a cube. In Wim's example you certainly can delete an element on the epilog with the DeleteDirect function, but admittedly you haven't always been able to do this.)
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

Catherine
Posts: 104
Joined: Wed May 20, 2009 7:30 am
OLAP Product: TM1
Version: 10.2.2 - PA
Excel Version: 2010
Location: Rennes, France

Re: difference between prolog, metadata, data, epilog

Post by Catherine » Fri Jul 12, 2019 1:22 pm

I had to split a process in 2 steps several times. But I have just one exemple in my mind at the moment.

This concerns security management.
In order to be sure not to lose private objects, here is the way I manage groups/clients in two processes:
Process 1
1. Prolog
-I delete the links between clients and groups
-I delete all groups (except admin)
-I define the source where I can read all goups and clients relevant for my instance
2. Metadata
-I add groups
-I add clients
3. Data
-I assign clients to groups
4. Epilog
Nothing because the last step is to delete clients which are not related to any group but I CAN NOT use DeleteClient function in an epilog
Process 2
I delete all clients which are not belonging to any group (using DeleteClient function)

Post Reply