NCrunch v3.13 marks the first release of 2018, and a big step up in feature-set.  Most of the development effort of 2017 was spent on maintaining compatibility with recent platform level changes, particularly around the releases of VS2017 and .NET Core.  I'm happy to say that things are now looking very stable, with minimal problems reported with the v3.12 release.  This has given us the chance to renew our focus on building new features and overall improvements to the product.  With the pace of platform-level changes now slowing considerably, 2018 is looking like a good year for NCrunch feature development.

The Problem of Intermittently Failing Tests

As our software architectures keep getting bigger and more complex, and we pile more and more tests over them, we start discovering new problems and limitations in our automated testing systems.  Intermittently failing tests tend to rank near the top of the 'most annoying problems' list, and it seems that this is a problem that is encountered to a greater or lesser degree with almost any project of reasonable size where heavy test automation is employed.

We've had a growing problem with intermittent test failures when testing the NCrunch solution for a while now.  When I popped the question to Twitter on how people deal with intermittently failing tests, the responses ranged everywhere from 'swear at them' to 'delete them'.  Some people mentioned they employ some quite methodical solutions to solving intermittent behaviour, such as putting time aside on a schedule or having dedicated tasks intended to tackle fixing intermittent failures.  One thing, however, is universally clear: we hate these things.  They waste our time, interrupt us at the worst times, degrade our trust in our own code, and generally make us angry and frustrated.

So we decided to try and do something about this.

So can we just re-run them?

One of the more frequently requested solutions to this problem has been for NCrunch to have a feature that would automatically re-run failing tests to make them pass.  There could be a range of ways that this could be configured or controlled, perhaps with global settings or attributes to target tests that are known to have intermittent behaviour.  This approach was rejected for the following reasons:

1. It causes intermittent behaviour to accumulate without being addressed
2. Some intermittent failures are actually real problems with production code that should be investigated
3. Automatically re-executing tests whenever they fail to ascertain whether a failure is 'real' can become very expensive resource-wise.  Since most intermittently failing tests tend to be larger integration tests, this could have a significant effect on the responsiveness of NCrunch.
   

Introducing Churn Mode

NCrunch v3.13 introduces a new feature targeted towards solving the intermittently failing test problem.  The intention of Churn Mode is to take the intermittency out of the failure, bringing it up in a way that the failure can be more easily investigated and resolved.  Churn Mode does this by employing the brute-force approach of running tests endlessly until they fail, while putting the full capabilities of the NCrunch engine behind this.

The effectiveness of Churn Mode is determined very much by the resources that are available to the NCrunch engine.  If you have a high-end desktop machine with 8-10 execution threads, there's a good chance that you can cause an intermittent failure to appear within a fairly short space of time.  If you have a grid with 50-100 execution threads, you probably won't need to wait much longer than the normal execution time for the test.

Churn Mode has two modes of operation:

"Endless" Churn Mode

Endless Churn Mode is a global state of operation for the NCrunch engine.  When in this mode, NCrunch will continuously cycle all passing tests through the Processing Queue in a random sequence using all available execution threads.  When tests fail, they are removed from the queue and their failures will remain visible in the Tests Window.  Passing tests with failing 'siblings' (other tests under the same fixture) will also not be re-run, as this has the potential to clear data that might be useful when troubleshooting the failing sibling.

The intention is that this is an overnight mode of operation.  You turn this on when you're done for the day, so that NCrunch can churn out the failures overnight and you can troubleshoot them in the morning.

Targeted Churn Mode

In addition to churning tests in a global manner, it's also possible to tell NCrunch to perform a 'targeted' churn where only a specific range of tests will be churned until failure.  This is easily activated using a context menu option.  You just select the tests you want to churn, then choose the option and let NCrunch grind them down. 

We've found targeted churning useful for testing fixes/changes made to intermittently failing tests raised using Endless Churn Mode, to be sure that they won't just fail again intermittently some time in future. 

Effect of Churn Mode

While intermittent behaviour will never be easy to resolve, Churn Mode now gives us a way to manage this problem in a more predictable way.  Instead of having failing tests jumping out randomly throughout a dev session and needing to take time out to fix them, it's now possible to take a more methodical approach by setting time aside on a project to deal with them more efficiently.

On the NCrunch project itself, we've found that Churn Mode has made a massive difference to our ability to both discover and resolve intermittently failing tests.  Something that also surprised us was just how much intermittent behaviour was actually caused by problems in production code rather than by the tests themselves.  In some cases, this was resulting in defective behaviour that was often too rare for users to report, or otherwise too elusive for us to actually pin down and fix.  As a result. NCrunch v3.13 includes a range of fixes to intermittent problems discovered by our own churning.  Many of these problems have been around for a very long time.

Improvements to the NCrunch UI (ToolTips)

NCrunch v3.13 also introduces some badly needed improvements to the tooltips used by NCrunch for the coverage marker and corner status indicator.

The coverage marker tooltip is now much more visually appealing and contains a wealth of extra information.

The new results bar consists of a horizontal pie chart showing the portion of tests covering the line that are failing, in addition to using faded colours for tests that are out of date.  Execution times are now also much more visible and are graphed as a portion of the value of the 'Marker performance display sensitivity' configuration setting.

The exception tooltip has also been improved.

The main headline exception is shown in red with its message tidily below it.  Execution times are also reported on exception lines with a time greater than 0ms.

The corner status indicator tooltip has also been heavily upgraded.

This tooltip uses the same system for reporting test results as the coverage tooltip, with the dataset consisting of all tests in the solution tracked by NCrunch.

The Processing Tasks bar summarises the range of tasks being run by the engine, such as build, analysis and execution tasks.  It will also report on the portion of tasks being executed by remote clients on shared grid nodes, so you'll know if you're experiencing reduced capacity due to grid sharing.

The Core Load graph shows the overall load on the core of the NCrunch engine.  As opposed to a metric reporting task capacity, this is instead reporting the ability of NCrunch to orchestrate the tasks being executed and surface relevant data into the UI.  When Core Load reaches 100%, you'll notice the engine becomes less responsive as it becomes overloaded with the demands of coordinating and merging data from background tasks.  A high core load is likely to be quite rare unless the level of concurrency is extremely high, which is currently only really possible when heavily using distributed processing.  Presently, the 'Risk/Progress Bar' also has some performance issues that significantly increase core load.

Other Stuff

By popular demand, another new feature is the ability to reset the status of a test using a context menu option.  When you select this option, a test will be reverted to its 'Not run' state and all execution results for the test are discarded.

v3.13 also contains a range of fixes to recently reported problems, along with general internal improvements and enhancements to the console tool's reporting of test results into TeamCity.  I highly recommend upgrading as soon as possible.  See here for the full list of changes.

------------------------- Edit 6th of March 2018:

It's been reported that v3.13 includes a regression where the use of collapsed code regions in VS can cause the coverage markers to become desynchronised with the source code.  To address this issue, v3.14 has been released today.  v3.13 also contains a few improvements to the NCrunch Console Tool's integration with Team City.