The benefits of deploying Continuous Integration (CI), as part of any Software Development process, have been well documented. One of the best articles, in our opinion, is by Joe Green from smrtr.co.uk, published on the Codeship Blog on 4-11-2013.

A snippet of Joe’s article follows.

“In business terms, the value of Continuous Integration is:

• Reducing risk.
• Reducing overheads across the development & deployment process
• Enhancing the reputation of the company by providing Quality Assurance

…Integrating code more frequently leads to reduced risk levels on any project. So long as there are metrics in place to measure the health of the application, code defects can be detected sooner and fixed quicker.”

“Having a good CI system is akin to having a fire alarm in your house”

We couldn’t agree more with Joe as we see these benefits in our business every day. We’ve also learned that “having a good CI” is predicated on the system’s ability to provide meaningful and actionable results; results that are free from the noise of intermittent and existing known failures.

Test results that sound the alarm with the same failures for days, weeks at a time quickly diminish the value of any CI system. Having to remember if failures exist, and how many, is an unscalable process.

Not only is moving or disabling failing cases impractical, but it can also lead to forgotten issues making their way into production environments.

Continuing to report failing health can lead to new failures being ignored or overlooked. How many false alarms before we start ignoring the noise? Therein lies the challenge to frequent integration – enforcing a 100% pass rate on every commit – as there are simply times when dependencies or other priorities delay the resolution of failures.

The challenge can be especially true for larger organizations:

• Issues of higher severity.
• Dependencies on hardware updates.
• Defects blocked on parallel efforts.
• Software bugs too insufficient to hold up an important release

Addressing the challenge with process alone

The process approach requires individuals to triage and verify results on every commit – to ensure it’s a false alarm (known failures) and not an actual fire – primarily reducing risk by increasing overheads.

Addressing the challenge with thresholds.

Some CI reporting systems try to solve the problem by defining a failure threshold — an acceptable number of failures for a job to pass — independent of which cases are failing. The solution attempts to reduce overhead by increasing risk, essentially demoting a binary indicator to an educated guess. Additionally, using a pass/fail ratio as a quality gate assumes that all tests have equal value, rarely the case.

Managing Test Failures with Exemptions

In contrast to the above, Testspace allows you to manage failures by exempting tests from the calculation of software health. Just as we manage Static Analysis issues with priorities and deferments, we can manage individual test failures with short term exemptions.

Only non-exempted cases can fail the application’s health.

Exemptions persisted on the server, are applied to every subsequent commit. Users can subscribe to health failure notifications knowing that only new, unexpected failures trigger alarms

When integrated with GitHub, GitLab or Bitbucket, Testspace manages the branching and merging of exemptions in support of a seamless, branched workflow.

A Lightweight Process to Minimize Overheads

The process is light and straightforward. The view presents the results hierarchy with individual exemption control for every test case in every suite.

Just click the control icons on the right side of the Failures column and enter a reason the case is failing in the Notes box. Use markdown to link to your defect and issue reports. To exempt the failure from contributing to health of space select the Exempt box.

Clear the reason to remove the exemption, once the failure’s resolved. Exemptions are removed automatically after the 5th consecutive passing result.

The consecutive pass requirement provides a level of hysteresis to prevent unwanted health toggling by intermittent failures.

Tracking Test Failure Exemptions

To help manage failures, Testspace tracks all exemption activity. Project Notes and comments are created automatically for each exemption added. Subscribers can be notified when exemptions are removed, whether manually or automatically. Exemption totals are rolled up to each parent suite and folder, right up to the root when viewing and traversing the result set tree.

Testspace recalculates the application’s health as failing cases get exempted. You’ve managed all failures when the resulting health is passing again.

…when the red symbol of failed health

turns healthy and green again.

Exempt cases still have their pass/fail state shown in the report but are removed from case count and calculations.

A 100% pass rate with 0 exemptions as a gate for merging to main and master branches can still apply.

Conclusions

With minimal overhead and a lightweight process, managing failures with exemptions can help software teams and organizations achieve the most benefit from their CI system.

You can read more about Managing Health and how to get started with this practice in our guide.

You can find Joe Green at smrtr.co.uk or on Twitter @joegreen88. Joe Green’s full article, “The Benefits of Continuous Integration 04-11-2013 is published on the Codeship Blog here.

Get setup in minutes!

Try Testspace risk-free. No credit card is required.

Have questions? Contact us.