Yes, it's finally here!  I had originally planned to have the NCrunch V2 Beta ready to ship back in September.  Of course, nothing ever goes to plan, but I do think this is a release that has been well worth waiting for.  As with any beta release, this one is far from perfect, but it has seen a good amount of real world testing and general thrashing around ... So it should be good enough to put to use and/or have a great play with.  I'm really excited about the contents of this release, and I hope that you will be too.

As with the NCrunch V1 Beta, usage of V2 while it's in a beta state is completely free.  So there's no reason not to take it for a spin!  Though I do expect that the V2 beta will be much shorter than it was for V1, which lasted for nearly 18 months.

There's a whole lot in V2.  Because the engine has seen so many internal changes and upgrades, it's hard to itemise them all.  Instead, I'm going to run through the major new feature areas and improvements and try to paint a picture of the bigger goals of this release, then hope that the rest will make sense as you put it to use.

Anyway, as many people already know, the biggest feature in V2 is ....

 

Distributed Processing!

NCrunch V2 brings the ability to offload all that hefty processing onto other computers, giving huge engine capacity improvements and opening the door to a whole range of new opportunities.

After all, why settle for just one desktop machine running your tests, when you can have a whole grid churning away?

When a server (or 'grid node') is hooked up, NCrunch will automatically replicate the solution open in VS over to the node and instruct it to start running builds and tests.  Because the node is responsible for its own builds, it's possible to configure your desktop machine ('grid client') to avoid running any builds or tests locally.  This means you can offload all the I/O heavy churning over to remote machines, leaving your development machine resources free for other tasks.

When NCrunch is connected to grid nodes, it will integrate their capacity into the processing queue.  You'll then see it start to use the nodes for executing tasks.

There are few restrictions on the nature of the machines that can be used as grid nodes.  As long as the machines can run .NET 4.0, accept connections over TCP, and have installed any components necessary to build your solution and execute its code, you should be fine.  Existing team build servers should work very well for this, as they usually have the various build targets needed for their solutions.

Grid nodes can be shared between clients, so you could have several team servers processing on behalf of multiple NCrunchers simultaneously.  The nodes each have a repository of solutions they commonly perform processing on (up to a configurable size limit), so once your solution has been transferred, it usually won't need to be copied over to the node again.  All data exchanged across the grid is compressed and encrypted.  The protocol has been designed to be lightweight enough that it can be used with limited bandwidth, which makes it ideal for scaling your grid onto nodes hosted in the cloud.

It's also possible to control (with much granularity) which projects can be built, and which tests can be executed on specific grid nodes.

There's a huge range of advantages that come with distributed processing, such as:

• Additional processing capacity:  The example above illustrates the engine offloading work onto two high capacity Amazon EC2 servers along with a local network machine.  In this situation, NCrunch was able to handle running 91 sets of test tasks concurrently with good speed and no issues.
• Less interference with the development (client) machine:  Because it's possible to operate in 'grid only mode', you can free up local resources to better deal with the tools you use in day to day development.  This means less sluggishness in the VS UI and less local disk space consumption.
• Easier parallel execution:  Many solutions used with NCrunch still don't support their tests being run in parallel.  Big heavy integration tests that work with local databases don't always function very well with NCrunch's parallel execution, and distributed processing provides a new way to scale these out.  The concurrency constraints used by NCrunch's engine only apply at machine/node level, which means it's possible to make constrained tests run in parallel by having them run on different servers.  For situations like this, you may be better off with a large grid consisting of many small nodes, as opposed to a small grid of high capacity nodes.
• Real-time testing using platforms not supported on the local development machine:  Sometimes teams need to perform testing using limited resources.  For example, you may have only one machine in your office capable of executing a particular test.  Setting this machine up as a grid node makes it possible to automate the testing over this machine, making life much easier for everyone.

So how much execution can the NCrunch engine handle?  This is a really good question and I'd like to hear more about your own experiences here.  Since the engine is no longer constrained by local resources, it's now limited only by the ability of the client to handle and merge the data being returned from the grid nodes.  My own testing suggests that it is possible to build grids that can scale well into the dozens (and potentially the hundreds) of concurrent tasks, although I'm sure that this depends entirely on the nature of the solution under test and the hardware being used.  It's probably worth mentioning that once you get up to 91 test tasks running concurrently, it may be worth looking at how to speed these tests up rather than adding more hardware .. although it's always good to have options :)

You can learn more about distributed processing in the V2 documentation.

 

UI Rebuild

V2 also includes a long promised rebuild of the NCrunch UI.

The new UI includes a new custom-made control to replace the old tree list used throughout V1.  The new control is much faster and cleaner, with improved keyboard navigation guided by feedback from V1 (many people commented that the old tree list was unintuitive with its keyboard options).  If you start typing while focused on the Tests Window tree list, you'll automatically be moved to the search box.  It's now also much easier to change the columns visible in the tree simply by right-clicking on the column headers.

Something else you'll also have noticed from the screenshots further above is that NCrunch now fully supports the Visual Studio 'Dark' theme.  In fact, the entire NCrunch UI is now completely aligned with Visual Studio and it will automatically adapt to changes in the VS theme without any configuration necessary.  Many people mentioned that the V1 UI stuck out like a sore thumb in the new versions of Visual Studio ... by comparison, V2 fits right in.

The Tests Window also has a new feature - you can now structure the tree based on several different grouping options.  This means you can now group the tree by category, or namespace, or a number of variations that I'm sure many people will find useful.

The Processing Queue Window has been upgraded with new filtering options, making it easier to see which tasks are being executed at any point in time.  You can also filter the queue by grid node, which is an added bonus when using NCrunch with distributed processing.

The recent revolution of high DPI tablet computers also puts new demands on the NCrunch UI, and I'm happy to say that high DPI is now fully supported by NCrunch.

Something else you'll notice is that there's something new and different with the coverage markers.

The markers are now situated in their own special gutter, completely separating them from the source code.  NCrunch V1 had problems with markers constantly overlapping source code, which was causing trouble for people using languages like F#.  The new gutter keeps the markers out of the way, while also keeping them visible when you scroll the code window horizontally.

 

Usability Tweaks

There's a range of usability tweaks also included in V2.  Many of these are very minor and probably won't be noticed by everyone, but here's a few you may be interested in:

• The 'Show Coverage For Selected Test Only' context menu option has been moved out of the 'Advanced' submenu to make it more accessible, as this option is very frequently used.
• The Metrics Window now considers the above code coverage filter, making it possible for you to view code coverage percentages based on a single test
• The code coverage markers will now automatically adjust their colour based on the background behind them.  This means that if your marker colour settings make the markers invisible, NCrunch will automatically find an alternative colour that makes them visible.  This is really useful when flicking between VS colour themes.
• The coverage marker colour page has been removed from the configuration wizard.  Many people mentioned that they find the wizard too long, so I've been looking at ways to reduce its size while still keeping it useful.  The marker colour page is now surplus to requirements and was the first to get the chop.

 

Windows Store App Support

As promised, NCrunch V2 introduces support for the testing of Windows Store Apps via MSTest.  When working with Windows Store App test projects, NCrunch will now register test project apps inside its workspaces, then sandbox them for test execution.  It all sounds really simple when you put it in one sentence, but this was a very difficult feature to implement.  Windows Store Apps have their activities very heavily constrained by the hosting O/S, so there was much to work around and much head scratching needed to make this happen.

One unfortunate constraint that does affect Windows Store App test projects is that they do not, at the moment, support parallel execution using NCrunch.  Windows itself limits the number of instances that can be created for a registered App, making parallel execution problematic.  Tests can still be run in parallel, but only as long as they exist in separate projects - as this then involves the creation of several Apps with different identities.

 

MS Fakes Support

Also promised for NCrunch V2 was support for Microsoft Fakes.  I'm happy to say that NCrunch now supports Fakes entirely - including shims.  There's nothing special you should need to do to make use of this... Just switch on NCrunch, and it should work nicely.

 

SpecFlow Integration Improvements

Throughout V1, many SpecFlow users have been asking for improved support around the handling of SpecFlow features.  While NCrunch has always supported SpecFlow through its NUnit/MSTest/xUnit integration, this support has always been limited to the files generated from SpecFlow's .feature files.

V2 introduces new improvements in SpecFlow support.  NCrunch will now show code coverage over SpecFlow's .feature files, with full test navigation.  NCrunch will now also automatically generate the feature code-behind files when it builds, so synchronisation issues around SpecFlow features are now a thing of the past.

 

Improved Task Handling

Something that was always lacking in V1 was the ability to stop an executing test dead in its tracks.  While you could always clear tests out of the processing queue, they would keep running until the test was done.  The only way to truly stop them was to find and kill the task runner process.

V2 fixes this, with the ability to kill test tasks that are busy executing.  On the Processing Queue Window toolbar, you'll notice a dangerous looking skull.

This option will become active whenever you select an executing test task in the queue.  As you'd expect, clicking the skull will terminate the underlying test task.

The Tests Window's stop button has also seen an upgrade in this area.  The red square is probably already familiar to you.

The first time you click this button, it will clear out the processing queue of all pending tests.  If there are tests still running, you'll notice the button changes.

Clicking the button a second time will immediately kill all test tasks currently running.

You'll also notice a new context menu option that allows you to kill an individual test when it's executing.

Tests that are terminated in this way won't be considered as failing.  NCrunch will simply discard the test run and leave the test status alone.

 

Stack Overflow Handling

Stack overflows have always been a pain in the neck for test runners.  As many people know, a stack overflow exception thrown under the CLR is an irrecoverable error.  The CLR will promptly terminate the process and throw up a great big error dialog asking if you'd like to debug it.  For a product like NCrunch that executes incomplete code, a stack overflow has usually needed to be handled by switching off the engine until the code has been fixed.

V2 introduces new change in this area.  When a stack overflow occurs, NCrunch will now detect it, handle it, and give you useful information to help you solve it.

 

Implicit Project Dependencies Settings

V2 introduces a new project-level configuration setting allowing you to define which projects a project depends on implicitly.  This offers a useful workaround for situations where projects depend on each other but do not specify this with a standard ProjectReference in their build XML.  More details about this setting are available in the documentation.

 

Performance Improvements

Performance has been a central focus right through the development of V2.  Because of the new stresses of needing to keep up with potential grid nodes, the engine has seen huge upgrades in its overall capacity.  There's also been many improvements right through from the UI (less stuttering of VS) down to the prioritisation of tasks in the processing queue, which is done much more intelligently.  Overall, V2 should be faster and more responsive than V1 ever was.

 

Other Stuff

V2 also includes a few other goodies:

• A new project-level configuration option has been added to disable code analysis build steps.  Code analysis build steps are now turned off by default, as they massively increase build times on even small projects.
• A new engine mode filter condition has been added allowing you to filter tests based on their execution time.  This means you can now have fast executing tests run continuously, and slower tests run manually.  This filter can also be applied in NCrunch's grid node configuration, allowing you to designate certain grid nodes as 'fast execution' or 'slow execution' only.
• NCrunch will now record and report MSBuild warnings in the Tests Window for any projects selected in the tree.
• The 'Rebuild' Tests Window project context menu option has been replaced with 'Reload and rebuild'.  This has been done to make it easier to solve problems with the engine if they appear, without a full reset being necessary.

 

A Note About Silverlight Support

The supported tools/frameworks page has for some time promised that V2 would include new compatibility extensions for Silverlight based testing frameworks.  At the time this page was first created, further Silverlight support seemed like a good idea, as Silverlight was still going strong with many people making use of it.

Unfortunately, with Microsoft's current direction and many changes in the market, the future for Silverlight is not as certain as it once was.  Google insights suggest that interest in Silverlight has halved in the last two years alone.

Support for a Silverlight based test environment in NCrunch is not without considerable complexity.  This is complexity that would need to be towed through the life of the product, with all runtime features needing special consideration for it.

It's with all this in mind that I've had to make the difficult decision not to extend NCrunch compability to Silverlight based test frameworks.  The long term cost is just too high, and the payoff too low.  I'm very sorry as I'm sure this will disappoint people.

Support for building Silverlight projects is something that has been progressively added throughout the increment V1 releases, and V2 will continue to have this going forward.  It is still possible to test Silverlight code by introducing references from non Silverlight test projects, and NCrunch is currently expected to always support this.

 

More Is Coming!

The list of features here is definitely not final for V2.  Although the major version is in its final stages of development, there are still other things planned to go in.  I hope to add these incrementally while the version is still in beta.

 

Getting Up and Running With NCrunch V2

NCrunch V2 uses a different naming notation for the .ncrunchproject and .ncrunchsolution files, which means it should be possible to use this version on teams with other people that are using NCrunch V1.  The separation of these files does mean that new configuration settings you add in V2 won't take effect for people using V1, so I wouldn't recommend staying on separate versions for too long.  My hope is that the V2 beta is stable enough that you'll never need to go back to V1.

Anyway, if you've read this far down the blog post without clicking on the download link, then let me encourage you!  GO GRAB NCRUNCH V2!