More info on TM1Project.json file

Post Reply
luizg2019
Posts: 24
Joined: Thu Sep 12, 2019 11:02 pm
OLAP Product: TM1 - PAW & Architect
Version: PA 2.0.7 - PAW 2.0.44
Excel Version: office 2016

More info on TM1Project.json file

Post by luizg2019 » Thu Dec 05, 2019 6:09 pm

Hi!

I'm trying to use GIT on our servers but IBM's documentation is not very detailed on this subject, they say you "SHOULD" create a tm1project.json file for each model but don't say where to save it, I tried creating it on my data folder but it made no difference when I sent a GitPush command.

Does anyone know where should I create this file and how should I proceed to use this?

Thank you.

User avatar
Mike Cowie
Site Admin
Posts: 456
Joined: Sun May 11, 2008 7:07 pm
OLAP Product: TM1, MSAS
Version: Anything thru 11.x
Excel Version: 2003 - 2016
Location: Alabama, USA
Contact:

Re: More info on TM1Project.json file

Post by Mike Cowie » Thu Dec 05, 2019 11:00 pm

Hello:

The only way to work with the project file that I'm aware of is via the REST API-- I don't believe it can be managed directly with a file in the data directory structure.

To create/update that project file create json which defines your tm1project spec, for instance:

Code: Select all

{
  "Version": "1.0",
  "Name": "My Model",
  "Settings": {
    "Performance": {
      "MTQ": {
        "MTFeedersAtStartup": false
      }
    }
  },
  "Tasks": {
		"Refresh Security":
    {
      "Process": "Processes('Refresh Security')",
      "Parameters": []
    }
	},
  "Objects": {},
  "Files": [
		"\\admin\\*\\*"
	],
  "Ignore": [
		"Processes('Ignore Me')"
  ],
  "Deployment": {},
  "PrePull": [],
  "PostPull": [],
  "PrePush": [],
  "PostPush": []
}
And POST or PUT that json to this special TM1 REST API url used to access the tm1project definition:
.../api/v1/!tm1project

You can review what's currently in there with a GET-- by default I believe it's blank until you POST/PUT something there.

Hope that helps! (and, yes, it's not that clearly documented right now)

Edward Stuart
Community Contributor
Posts: 236
Joined: Tue Nov 01, 2011 10:31 am
OLAP Product: TM1
Version: All
Excel Version: All
Location: Manchester
Contact:

Re: More info on TM1Project.json file

Post by Edward Stuart » Fri Dec 06, 2019 8:54 am

Excellent work Mike, I'm looking forward to testing this further.

For those also looking into this Yury has posted (yet) another great blog on Git Integration

https://ykud.com/blog/cognos/tm1-cognos ... tm1-part-1

luizg2019
Posts: 24
Joined: Thu Sep 12, 2019 11:02 pm
OLAP Product: TM1 - PAW & Architect
Version: PA 2.0.7 - PAW 2.0.44
Excel Version: office 2016

Re: More info on TM1Project.json file

Post by luizg2019 » Fri Dec 06, 2019 12:25 pm

Mike Cowie wrote:
Thu Dec 05, 2019 11:00 pm
And POST or PUT that json to this special TM1 REST API url used to access the tm1project definition:
.../api/v1/!tm1project
Wow. I would've never guessed based on the documentation :lol: :lol: :lol:
Thank you very much for your help Mike!
Edward Stuart wrote:
Fri Dec 06, 2019 8:54 am
For those also looking into this Yury has posted (yet) another great blog on Git Integration
https://ykud.com/blog/cognos/tm1-cognos ... tm1-part-1
Thank you for sharing this link Edward, it's definitely more informative than the official documentation. :D

luizg2019
Posts: 24
Joined: Thu Sep 12, 2019 11:02 pm
OLAP Product: TM1 - PAW & Architect
Version: PA 2.0.7 - PAW 2.0.44
Excel Version: office 2016

Re: More info on TM1Project.json file

Post by luizg2019 » Mon Dec 09, 2019 2:07 pm

Mike Cowie wrote:
Thu Dec 05, 2019 11:00 pm

To create/update that project file create json which defines your tm1project spec, for instance:

Code: Select all

{
  "Version": "1.0",
  "Name": "My Model",
  "Settings": {
    "Performance": {
      "MTQ": {
        "MTFeedersAtStartup": false
      }
    }
  },
  "Tasks": {
		"Refresh Security":
    {
      "Process": "Processes('Refresh Security')",
      "Parameters": []
    }
	},
  "Objects": {},
  "Files": [
		"\\admin\\*\\*"
	],
  "Ignore": [
		"Processes('Ignore Me')"
  ],
  "Deployment": {},
  "PrePull": [],
  "PostPull": [],
  "PrePush": [],
  "PostPush": []
}
And POST or PUT that json to this special TM1 REST API url used to access the tm1project definition:
.../api/v1/!tm1project

You can review what's currently in there with a GET-- by default I believe it's blank until you POST/PUT something there.

Hope that helps! (and, yes, it's not that clearly documented right now)
Hi mike! Do you know how can I specify for control cubes to be included in the git context?
I tried using the "(not) ignore" section in tm1project but the control cubes were not added to git, and it made the gitpush took a lot more of time to complete.

Code: Select all

{
  "Version": "1.0",
  "Name": "Test",
  "Ignore": [
		"!Processes('}*')",
		"!Dimensions('}*')",
		"!Cubes('}*')"
  ]
}
Thank you.


EDIT:
Nevermind.

Found the answer. Turns out the wildcard is NOT supported for control objects :oops:
So you have to manually list one by one if you want them included in the git as well, which is very annoying because when you're moving new dimensions between servers, its attributes will not be transferred unless you specify each "}ElementAttributes_Dimension" cubes to be transferred as well.
Last edited by luizg2019 on Mon Dec 09, 2019 7:28 pm, edited 1 time in total.

User avatar
ykud
Posts: 109
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: More info on TM1Project.json file

Post by ykud » Mon Jan 06, 2020 5:01 am

luizg2019 wrote:
Mon Dec 09, 2019 2:07 pm
So you have to manually list one by one if you want them included in the git as well, which is very annoying because when you're moving new dimensions between servers, its attributes will not be transferred unless you specify each "}ElementAttributes_Dimension" cubes to be transferred as well.
Hi there,

How did you manage to include the control cubes in the end?
Via !Ignore, like this?

Code: Select all

  "Ignore": ["!Cubes('}ElementAttributes_DimensionA')"]
Or you're 'including' them explicitly in {objects}? Can you please share your tm1project.json?
I'm trying to add them in something like all_cubes.json file to be able to use Link: all_cubes.json, but failing so far. Quite annoying :(

Also, maybe you or Mike have played with it and know the answer:
Say I want to promote only 3 changed objects from one server (A) to another (B), but I have 10 more objects changed on A that I don't want to promote yet. I thought this would be something like listing the 3 objects in the "deployment" as objects so that it overrides the primary definition, but so far I am struggling to get any Pull request to ignore the other 10 objects. Any pointers?

Thanks,
Yuri

luizg2019
Posts: 24
Joined: Thu Sep 12, 2019 11:02 pm
OLAP Product: TM1 - PAW & Architect
Version: PA 2.0.7 - PAW 2.0.44
Excel Version: office 2016

Re: More info on TM1Project.json file

Post by luizg2019 » Tue Jan 07, 2020 1:04 pm

ykud wrote:
Mon Jan 06, 2020 5:01 am
Hi there,

How did you manage to include the control cubes in the end?
Via !Ignore, like this?

Code: Select all

  "Ignore": ["!Cubes('}ElementAttributes_DimensionA')"]
Hello!
Yes, using the (not) !ignore feature. But you have to save the tm1project settings before doing a git pull.
I didn't get to play a lot with the tm1project setting because I didn't have time and the bad documentation also discourages me.

But my tm1project.json was something like:

Code: Select all

{
  "Version": "1.0",
  "Name": "Test",
  "Ignore": [
		"!Dimensions('}Clients')",
		"!Dimensions('}ClientProperties')",
		"!Cubes('}ClientProperties')"
  ]
}
ykud wrote:
Mon Jan 06, 2020 5:01 am
Say I want to promote only 3 changed objects from one server (A) to another (B), but I have 10 more objects changed on A that I don't want to promote yet. I thought this would be something like listing the 3 objects in the "deployment" as objects so that it overrides the primary definition, but so far I am struggling to get any Pull request to ignore the other 10 objects. Any pointers?
I also ran into that issue. What I did was manually ignoring the objects on the tm1project.json from server A before doing a commit/gitpush so when you do a gitpull on server B it won't overwrite the objects that you ignored.

User avatar
ykud
Posts: 109
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: More info on TM1Project.json file

Post by ykud » Tue Jan 07, 2020 9:33 pm

luizg2019 wrote:
Tue Jan 07, 2020 1:04 pm
Hello!
Yes, using the (not) !ignore feature. But you have to save the tm1project settings before doing a git pull.
I didn't get to play a lot with the tm1project setting because I didn't have time and the bad documentation also discourages me.
Thanks heaps! That document is indeed very confusing and vague, I'm quite annoyed about having to 'guess & try' so much around it.

luizg2019 wrote:
Tue Jan 07, 2020 1:04 pm
ykud wrote:
Mon Jan 06, 2020 5:01 am
Say I want to promote only 3 changed objects from one server (A) to another (B), but I have 10 more objects changed on A that I don't want to promote yet. I thought this would be something like listing the 3 objects in the "deployment" as objects so that it overrides the primary definition, but so far I am struggling to get any Pull request to ignore the other 10 objects. Any pointers?
I also ran into that issue. What I did was manually ignoring the objects on the tm1project.json from server A before doing a commit/gitpush so when you do a gitpull on server B it won't overwrite the objects that you ignored.
Yep, I know that if you don't have any other objects in the repo (don't push or delete from git), they will no be pulled. Just hoping this is not the only way as that's kinda the end of trying to have a single git repo for your tm1 server. I'll try and see if I can get some pointers from IBM re this, things can't be that bad.

Thanks again,
Yuri

User avatar
macsir
Community Contributor
Posts: 710
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365
Contact:

Re: More info on TM1Project.json file

Post by macsir » Thu May 14, 2020 7:41 am

Hi, is there a way to add control objects into Git now?
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/

scrumthing
Posts: 43
Joined: Tue Jan 26, 2016 4:18 pm
OLAP Product: TM1
Version: 2.0.9
Excel Version: too many

Re: More info on TM1Project.json file

Post by scrumthing » Thu May 14, 2020 9:20 am

macsir wrote:
Thu May 14, 2020 7:41 am
Hi, is there a way to add control objects into Git now?
Still not as far as I know but I think IBM is by now aware of the issue.
It does not stop there. Even if you add all control objects manually you still do not have the data within those cubes (aka attribute values) which I find kind of really annoying especially if it comes down to stuff like picklists. I don't want to write a rule or process for every single picklist in my model.
ykud wrote:
Tue Jan 07, 2020 9:33 pm
Yep, I know that if you don't have any other objects in the repo (don't push or delete from git), they will no be pulled. Just hoping this is not the only way as that's kinda the end of trying to have a single git repo for your tm1 server. I'll try and see if I can get some pointers from IBM re this, things can't be that bad.
I tried it with only one repo as well and so far failed to do it properly. I think their idea is to have a separate branch for every merge you want to do and constantly experience "merge hell" between development and testing and production. :-(
I would love to set up a centralized repo with all standard processes for multiple instances and deploy them from there. But I am not sure that that is possible without merging and forking within GIT.
There is no OLAP database besides TM1!

lotsaram
MVP
Posts: 3417
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: More info on TM1Project.json file

Post by lotsaram » Thu May 14, 2020 12:20 pm

scrumthing wrote:
Thu May 14, 2020 9:20 am
I tried it with only one repo as well and so far failed to do it properly. I think their idea is to have a separate branch for every merge you want to do and constantly experience "merge hell" between development and testing and production. :-(
I would love to set up a centralized repo with all standard processes for multiple instances and deploy them from there. But I am not sure that that is possible without merging and forking within GIT.
This is why I think ultimately this is doomed to fail, not becasue of missing or misleading documentation or unfinished development and feature support, but becasue fundamentally this is not how TM1 development works.

The way IBM have conceptualized Git integration working is that each developer works on a separate branch, and each branch is essentially an individual copy of the entire TM1 server. The thing is when you are developing in TM1 it's not just code ("code" in TM1 sense being .pro and .rux files) but a model consisting of hierarchies and cube data. You can't develop the code in isolation of the model. Certainly not for more complex models, and definately not when it comes to most unit testing, UAT and troubleshooting. And well TM1 models can get pretty big, and having multiple developers each working on their own separate model can add up to a lot of resources and a lot of interdependencies.

Most of the time having a shared model being worked on by multiple developers makes a lot more sense. With developers reserving & checking-in / checking-out discrete packages of objects. And simple mechanisms to select a collection of objects to migrate between instances. But this is very difficult to manage (or is currently unmannegable) with how IBM have done the Git integration.

I'm still hopeful this might evolve into a decent tool, but I do admit to not being overly optimistic about it.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.

User avatar
macsir
Community Contributor
Posts: 710
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365
Contact:

Re: More info on TM1Project.json file

Post by macsir » Thu May 14, 2020 11:09 pm

The way IBM have conceptualized Git integration working is that each developer works on a separate branch, and each branch is essentially an individual copy of the entire TM1 server.
Actually, I think the working mode should be working. Each branch doesn't have to be the copy of entire TM1 server. They could have several objects you working on and later want to deploy to a target TM1 server. The central repo could have its own history and each branch could have history too. At any time, they could be merged back to central history. This is how Git works and I have tested. They are working fine. In Git, you can track history of code files, update and hot promote.
The only block for me is to figure out how to write proper tm1project file (I have learnt Ignore section but need more other sections) and how to deal with control objects (not individually). Hopefully IBM can provide more later.
For the data files, I am not concerning too much as it is not relevant to Git anyway. Git shouldn't maintain any history of any data file.
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/

User avatar
macsir
Community Contributor
Posts: 710
Joined: Wed May 30, 2012 6:50 am
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: Office 365
Contact:

Re: More info on TM1Project.json file

Post by macsir » Fri May 15, 2020 2:19 am

Further on control objects for tm1project files, it is not a big deal anymore. I can easily build the json file with extractions for all control objects from }Cubes, }Processes and }Dimensions dimensions. And they can be initialized into Git completely.
In TM1,the answer is always yes though sometimes with a but....
http://tm1sir.blogspot.com.au/

Post Reply