Automated TM1 Regression Testing Framework like nUnit?

Post Reply
bobgill
Posts: 4
Joined: Wed Aug 07, 2013 4:13 pm
OLAP Product: tm1
Version: 10.2
Excel Version: 2010

Automated TM1 Regression Testing Framework like nUnit?

Post by bobgill » Wed Aug 07, 2013 10:08 pm

Hi folks,

Is there something for TM1 following the nUnit style of automated test tools?

In theory, I would expect it to be something like the following
* A TI that runs all processes with a '.Test' suffix
* Each process with a .Test suffix runs unit tests on an underlying TI returning PASS or FAIL. i.e. The process p.Dim.Subset.Copy.Test runs unit tests of p.Dim.Subset.Copy.
* Notification is sent whenever any processes return FAIL

I can create this on my own, but I thought I would ask if there exists a common framework that everyone already collaborates on. Is there a sourceforge project already started? Is anyone interested in contributing to one if I were to start it?

Thanks for your help,

Bob

Darkhorse
Posts: 141
Joined: Wed Mar 09, 2011 1:25 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2003 2007 2010 2013

Re: Automated TM1 Regression Testing Framework like nUnit?

Post by Darkhorse » Wed Aug 07, 2013 11:12 pm

Hi bob

Neverheard of this process before but would this not work

While I <= dim size (}processes)
cElename = dimix(}processes, I)
If( scan("test", celename)<>o)
Executeprocess(celename);
Endif
I =I +1
End

You could add the code to the process if fail then add to dimension }processes an attribute which catches flags like fail

Then in epilogue create a subset based on that attribute for fails etc then ASCII output each fail or create an email to auto email yourself the answers

bobgill
Posts: 4
Joined: Wed Aug 07, 2013 4:13 pm
OLAP Product: tm1
Version: 10.2
Excel Version: 2010

Re: Automated TM1 Regression Testing Framework like nUnit?

Post by bobgill » Wed Aug 07, 2013 11:18 pm

You're completely right. That would work well.

I was just wondering if I'm reinventing the wheel or not. An Agile project should have a continuous integration component that automatically does regression testing and provides immediate visibility to the team if an error comes up. Is there not something already out there to provide this kind of testing framework?

In the TM1 world, is there something like sourceforge where developers can work together on foundational pieces to be shared in the community?

lotsaram
MVP
Posts: 3326
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Automated TM1 Regression Testing Framework like nUnit?

Post by lotsaram » Thu Aug 08, 2013 5:59 am

The closest or maybe the only thing resembling an open source library of unit functionality in TM1 is the bedrock codebase.

I'm suspicious as to the worth of a fully automated process for unit testing TI processes. Many processes will have parameters that need to be passed, without passing appropriate parameters a simple pass/fail based on process completion status doesn't give you much useful information, you could well argue it doesn't give you ANY useful information. For a valid test you need and expected outcome and compare the result to the expected outcome. Code that runs without producing an error is not enough.

User avatar
Harvey
Community Contributor
Posts: 218
Joined: Mon Aug 04, 2008 4:43 am
OLAP Product: PA, TM1, CX, Palo
Version: TM1 8.3 onwards
Excel Version: 2003 onwards
Contact:

Re: Automated TM1 Regression Testing Framework like nUnit?

Post by Harvey » Thu Aug 08, 2013 6:27 am

I'm pretty sure there is no bona-fide unit testing framework for TM1 on the market. It's something we've thought about frequently at Flow, but have yet to find the time to prioritize the product. We'd be interested to hear your thoughts and requirements.
Flow OLAP - thinking outside the cube.

Darkhorse
Posts: 141
Joined: Wed Mar 09, 2011 1:25 pm
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2003 2007 2010 2013

Re: Automated TM1 Regression Testing Framework like nUnit?

Post by Darkhorse » Sun Aug 11, 2013 12:18 am

I have constant chores on my server that run through all models giving me automatic emails from processes ran nightly to server statistics cube sizes dimensions garbage memory to users dormant users admins etc

These things are easily built but I believe a integrated system wouldn't work as most servers configs are different, in my workplace the servers are all slightly unique and all models although similar in nature work differently to the developers requirements and expertise,

User avatar
paulsimon
MVP
Posts: 718
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: Automated TM1 Regression Testing Framework like nUnit?

Post by paulsimon » Mon Aug 12, 2013 11:43 pm

Hi

I use something that I call a Master Process cube. One dimension is the Master Process dimension. Against each Master Process element you can set up a number of lines of processes to be run, and give parameters.

The master process execution process takes a parameter of an element in the Master Process dimension to be run. It then runs each process in turn, passing in the parameters. You can set each process to continue on error or to stop running further processes if there is an error. For an error testing suite you would probably want it to continue. At the end it will tell you whether each process succeeded or failed and how long it took to run.

However, the hard part is writing the processes to do the checks.

In practice I tend to incorporate the tests into the system rather than having a separate test suite.

For example, at the end of a dim build process there will be checks to ensure that all elements below a given top level element only have one parent.

For general day to day testing, I usually build an Excel framework where I perform the same calculations in Excel and compare the results to TM1. I also use it to report on Orphans (elements that have no parents in the hierarchy and which are linked to an Orphans consolidation that then consolidates into All elements.

The other part is setting up a reliable set of test data. In the past it took quite some time to set up a relational database with a set of test data for TM1 ( and also relational Cognos BI) so that we have a reliable set of data for test loads.

However, with a TI process based test suite you will only really be testing loads and dim builds or simulations of data entry, but you still won't be testing the data entry sheets, security rights, etc.

Regards

Paul Simon

CathyBoots
Posts: 19
Joined: Wed Aug 10, 2016 1:20 pm
OLAP Product: Tm1
Version: 10.2.2
Excel Version: Prof Plus 2013

Re: Automated TM1 Regression Testing Framework like nUnit?

Post by CathyBoots » Fri Jun 21, 2019 2:55 pm

hi all

I'd be very interested if there's a way to automate testing in TM1. We have a large model including around 500 excel/perspectives reports using VBA and 500 processes. If we could get at least some of that automated it would make upgrades and improvements easier and less risky!

Thanks
Cathy

User avatar
paulsimon
MVP
Posts: 718
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: Automated TM1 Regression Testing Framework like nUnit?

Post by paulsimon » Sun Jun 23, 2019 10:22 pm

Hi Cathy

It looks like you are more interested in comparing Excel workbook reports? If so, then we use a utility to do this. If you are looking at doing regression testing after a version upgrade, then we use the following approach:

Start the old and new versions on two different servers with identical data.

Run the same set of reports from each server
We also have a utility that slices and paste special values the default view on each cube in the server to Excel so that can be compared.

We then have a tool that compares the Excel reports from the two servers and reports any differences.

We then run key TI processes on the two servers, eg the month end rollover

We then run the reports and the comparison again.

Its not totally automated but for the frequency with which we upgrade it has been good enough for us.

Regards

Paul Simon

Post Reply