4 Things Learned from Automating Cognos BI Report Testing
Large companies have hundreds, if not thousands, of business-critical reports running from a business intelligence (BI) system such as Cognos or Power BI. Every time the BI system gets upgraded, testing must take place to ensure the reports can run without any change in data or formatting.
In a recent SPR project, our client was upgrading from Cognos 10 to Cognos 11, with about 700 reports and more being developed regularly. In previous test efforts, we prioritized and manually ran reports in the current QA environment and the upgraded QA environment. Manually, it was time-consuming and error-prone work. How can we test all reports in a timely manner and take out the element of human error?
Fortunately, tools are coming to the market such as MotioCI to set up and run automated test cases. Here is what we’ve encountered building experience with this style of automation.
Test case creation is a breeze
Setting up test cases proved to be very fast. It was straightforward, and they were set up in a bulk manner. Creating automated tests for all 700 reports took a small fraction of the time it would take to identify the corresponding manual test scenarios.
Test results are highly detailed, accurate and valuable
MotioCI identifies not only changes in report data, but also provides a detailed comparison as to the output formatting. If something in the Cognos 11 report output shifted visually, MotioCI picked it up and highlighted the differences. Some of these differences can be customized, to highlight or ignore things like coversheet formatting or a date/time stamp in the report footer (which would correctly be different every execution).
The tool ran well and confirmed Output Stability and Execution Time Stability assertions on the Cognos 10 test cases. While checking the report content, MotioCI also runs incremental performance testing between releases. If a report takes longer to run in the upgraded version than expected (a percentage value that is also customizable), an error is created. This is a powerful feature and will provide great value during each upgrade moving forward.
Tests can identify unused reports for cleanup
The MotioCI tool can set up test cases for all reports on a QA server or for certain folders. Having the test cases automatically tied to reports made it easier to find the obsolete reports and remove them from the test bed. Also, when reports were removed from Cognos, the corresponding test cases were automatically removed too. When reports changed in Cognos, MotioCI marked the test cases as Stale to prompt the tester to rerun them.
Proper environment setup is key
A tool like MotioCI is sensitive to the test environment configuration and setup. In our project, the testing environment was a challenge. Some project setup overhead to think about during initial planning includes coordinating with IT resources to adjust network configuration, server memory and RAM – in some cases multiple times before the automation would run.
Many tests fail shortly after initial installation and setup. Some reports are sensitive to time consideration, order in which the data is presented, etc. The more you try to use the full capability of the automation, the more likelihood there is of false negatives.
The test cases default to create results output in HTML, PDF and SQL formats. The HTML outputs were quite large, generating significant time overhead when running hundreds of tests. When those were removed, the test cases ran faster. If you don’t need output reports in HTML format, make sure this option is turned off prior to test execution.
Getting past the initial setup and parameter configuration to remove false negatives can be a challenge. But as the project moves on, and in future upgrades, MotioCI showed its strength in automating testing for Cognos reports. It is simple for a manual tester to use and maintain and reduces overall testing time for upgrade projects.
