The above picture conveys how you can feed test results from all your different tests into a database and then display it over a dashboard.
As your automation matures and grows in size/coverage, everyone expects more from it, and expect it to be faster and simpler, and do all kinds of things.
If you have 1000s of tests, running on a regular basis, keeping track of all those and then conveying this to all the stakeholders can become a big task in itself. And with 1000s of tests, its difficult to condense and provide an overall picture and status of the testing.
This is where having a Dashboard helps, as it radiates the status and progress of each test/feature in real-time.
Not just automated tests results, it can also pull data from Jira for Manual Tests, and keep all this info current, and easy to understand, and all your Directors and MDs would love to see a browser based real-time dashboard!
Also, the dashboard should not cause too much overhead in terms of implementation and actual use because otherwise it increases development and testing time.
Hence, ideally, the Dashboard calls should be wrapped in the Reporting framework itself, so that its use becomes seamless
Installing a dashboard solution on a Linux box seems to be the only way to get our own custom dashboard for reports.
Since, installing stuff on linux in enterprise setups is very time consuming I wanted the shortest/fastest possible solution with minimal dependencies.
Also, I need a more real-time solution which takes the data via an API/JSON rather than from a log/file because setting that up on CI/Agent would also be difficult.
Below is my experience of evaluating different solutions to implement a real-time Dashboard for Automated Test Results, where I have compared different options and tried to chose what best suites our need.
Have evaluated the following
- ELK
- ReportPortal
- Allure
- Klov
- Grafana + InfluxDB
ReportPortal -
- ReportPortal seems to be open source but has certain pre-requisites that need to be installed before we can use it - RabbitMQ, PostGreSQL and Elasticsearch; after this we still need to install ReportPortal and all the plugins that we need.
- If we have to end up installing ELK then I dont think there is any need to bother with ReportPortal, we can get the dashboard done with just ELK - no need for one more layer.
- Also, since this could only be installed on Linux, and I did not have a linux box of my own to play with, it was difficult to evaluate and use.
Allure -
- The problem that I faced with Allure was that even though it generates report files for each run, those report files can only be read after loading and processing in Allure, not directly like the way some other libraries that generate a easy to use HTML Result.
- This defeats the purpose of having a real-time report and a dashboard. This feels a veiled attempt to cause stickiness with the product, and I do not want to be ham-strung and be covertly dependent on any product.
- So clearly, Allure was not the product to use.
Klov -
- This comes from the makers of ExtentReports, and I really like ExtentReports, as its open source, and is really built from a tester's point of view, and is easy to implement and use - not to mention how the beautiful reports look.
- But I did not have an option to pay for Klov as its no open-source, and at some point I felt that I was getting too dependent on just one solution, which could cause difficulties in the future if there are some changes in the product which couldn't be handled.
- So thought of using other products, but I still continue to use ExtentReports.
- Also, paying for a Dashboard solution (eg., Klov) may not be an option for many teams, specially only just for reporting purposes, when your entire tool-set is open source.
In the end, I was left with just ELK and Grafana.
ELK -
- This is a great solution, and is being increasingly adopted by many teams. It has high scalability and has a ton of features and implementation options and is a becoming more of a standard now.
- Though ELK might be now set up in many organizations already, if its not, getting it set up could be difficult, at least it was difficult for me.
- And getting to install and configure all the different components could also prove time consuming which many QA teams cannot afford because you do need to configure different listners and appenders to capture and transmit all the info.
- It like installing many different pieces each with their own config and then trying to use all of them.
- Needless to say this has a long learning curve too.
Grafana -
- Grafana is a multi-platform open source analytics dashboard and InfluxDB is an open-source time series database.
- Grafana + InfluxDB does not need any middleware messaging hub/node as they communicate directly via API, so this is a simpler solution.
- Grafana fires up fast and has many options to customize the way you want to present your data on the dashboard. You install InfluxDB and then forget about it, there is no overhead. Its super simple to use and read.
- Also, the installation for both Grafana and InfluxDB is pretty simple without any major dependencies.
- And it can also be installed directly on Windows too, which really helped in evaluating all the different options that we were planning to build.
A Jira based dashboard is also possible and could be great too as it has its own well defined API, but I did not evaluate it this time, may be later.
So, at this point InfluxDB + Grafana seems to be the really simple solution but far better suited to our needs, though not easy in any way as it does have some learning curve.
Refer below links to get started with Grafana and InfluxDB
