I'm happy to announce that NCrunch version 5 is now available for download.

This is the first major feature release we've had in a bit over 4 years, and in terms of both effort and feature-set, it's the largest release in the history of the product.

V5 contains two new feature areas.

JetBrains Rider Support

By enthusiastic user request, we now have a plugin that supports JetBrains Rider.

This wasn't easy to do, as the frontend for Rider is a JVM based application, while NCrunch is developed in .NET.  We host the NCrunch engine outside the IDE in our own .NET process, then pipe much of the UI into Rider through shared memory.

I recommend reading my previous blog post about our support for Rider, particularly if you're interested in how the plugin works and some of the history behind it.

Windows Only

Please note that even though we now have support for Rider, we are still limited to the Windows platform and cannot run on Linux or MacOS.  Our UI is still heavily dependent on Windows and we make heavy use of Win32 for internal optimisations, so porting over to another OS would be a serious project.

 

BETA State

I will also stress that as of now, our Rider plugin is most definitely in a BETA state.  It hasn't yet seen the sort of rigorous real-world testing that we've had with our VS plugin, and this is the first time it's been exposed to a wide user base.  Stuff will break.  There will likely be performance issues we haven't yet found or addressed.  Unfortunately, there isn't a way for us to ship a highly reliable plugin on the first try, and it will take us a while to work through the issues and solidify what we have.  Please take the time to report problems you encounter using the bug report option or by going through the support forum.  If you are able to share clear steps for how to reproduce a problem, this will help us to provide a speedy fix.

 

Limitations

The Rider plugin has two limitations that presently do not exist with our VS plugin:

No Support for Gallio/MbUnit: Because of the need to host our engine under .NET6 to work with the Rider backend, our integration with Gallio (and MbUnit) could not be easily ported over, as this framework is now very old and has been out of maintenance for quite a while.  We examined options for sandboxing it, but decided that with so few people still using it, the effort to maintain this integration point was simply not worth it.

No Support for Xunit v1: Our Xunit v1 support goes via Gallio, so this is being removed for the same reason.  Xunit v2 still works fine, so you have a clear migration path if you want to use NCrunch with Rider and Xunit.

It's expected that the next version of Visual Studio (2024?) will be running on .NET 8 or .NET 9, so we'll need to similarly host our engine in a way that includes the above constraints.  This means that our Rider support has feature parity with the next version of VS.

The UI performance of RDI under Rider is also something we are still working on.  Presently, the plugin has trouble rendering efficiently when dealing with large source files.  This is not an issue we expect to have to deal with long term as we have many ideas on how to optimise it.  RDI support for Rider was our last work item prior to release, so naturally it is our least 'finished' item on the board right now.

 

A Huge Thank You to JetBrains

We've been working with the team at JetBrains over the last 3 years to make this possible.  I'm not sure how we would have achieved it without their help.  They helped us greatly with the ability to 'wormhole' our UI through IPC and directly into Rider.  Without this, we would have needed to rebuild our UI from scratch, which would have made the project infeasible.

Over all, I've been extremely impressed with the helpfulness of everyone at JetBrains, throughout their technical teams, management and marketing.  It isn't easy to build an eco-system of plugins on a new IDE, but I think they are doing a stellar job of it.  I'm grateful for the chance to bring NCrunch to this platform.

I do also want to extend a huge thank-you to Michael Kroes for his work on integrating NCrunch with Rider.  Michael has been the architect of this work and was responsible for most of its development.  Without him, we would not have Rider support.

 

Runtime Data Inspection (RDI)

RDI is the second headline feature of the V5 release.  This is a feature that exists in both the VS and Rider plugins.

 

RDI is an analysis system that lets you examine the behaviour of your code as it was run by your tests, with no debugger necessary.

I've written a blog post that describes RDI in detail.  I recommend reading it as I think that RDI has the potential to change the way we develop software.

RDI is the result of 15 years of R&D.  It is reliant on the entirety of NCrunch's infrastructure and is fully integrated with all areas of the product (test runners, engine, UI, build system, distributed processing, IDEs, etc).

V5 includes a special new wizard showcasing RDI and giving you options to configure it the first time you open a solution.  I've been working toward shipping this feature for a significant portion of my life and I really hope you try it out.

 

State of RDI

We've been using RDI internally on the NCrunch solution for over a year now, and it's already been through several rounds of controlled testing.  I'm tempted to tag a BETA warning on it, simply because I expect that people will discover things we haven't thought of or tested for, but generally it's felt quite solid for us for a while now.  There are some feature gaps and issues that I still want to address, including:

Collection of complex types: Presently we just call ToString() on complex types and we make no effort to capture their fields or properties.  We have plans to do better in this area, as I think capturing full object trees (data limits permitting) would be very useful.

Collection of array/list contents: For arrays and lists, right now we're just capturing and reporting the number of items.  Future plans include capturing and reporting all items in the array/list up to a configurable limit.

Navigation: The navigation between frames isn't perfect.  Right now we don't have any handling for control flow diversions caused by the JIT when it loads assemblies or calls static constructors, so when this happens, navigating out of a frame can sometimes land you in an odd place.  Forward navigation also doesn't yet properly handle situations where multiple navigation points exist, which is quite noticeable when enumerating using lambdas.

Image collection: At the moment we only support collection of WPF UIElements and images with a render size set.  I'd like to expand this to other UI platforms too, and maybe even include options for customised image collection from objects.

Exception reporting: We don't yet report exceptions in RDI.  You'll simply see the control flow hit a point in the method, then stop.  I have some cool ideas on how we can visualise exceptions and I'm looking forward to having a play with this.

Data limit reporting: We do have some limited reporting for when data limits are reached when showing frames and values, but I think we can do better in this area.  Ideally, it should be extremely visible if RDI has capped collection due to a data limit.

Debugger access: Sometimes it's still useful to be able to run a debugger into an open frame to examine the state of memory.  There are some limitations here on deterministic behaviour around the process (for example, if a method is called a variable number of times), but I think it would be nice to have this.

Extra navigation options: It would be nice to have a top-down view on a whole navigation tree, something like a virtual call stack.

Optimisation: I'm not yet happy with RDI's engine memory consumption or its impact on engine startup/shutdown times when working with very large solutions.  There is work still to be done in this area.

 

A Bit of History

The concept of RDI is something that actually predates the work I did on the first NCrunch prototype back in early 2009.  My original idea was to have a tool that could visually show the execution path alongside the code, giving useful metrics that were collected passively.  It occurred to me that for this to be working at its best, it would need to run the tests continually and report this data back while the code was being written.  Some experimentation quickly showed that the concept was almost hilariously impractical.  My poor dual-core laptop quickly collapsed under even the most trivial sets of code, and no amount of optimisation or creativity was going to save it.  I eventually resigned to simply aggregating the coverage results instead, and NCrunch's inline coverage was born.

Years went by and hardware improved.  As the concept of NCrunch proved to be widely successful (inspiring a number of similar tools) and the engine's capabilities expanded, RDI started to seem like more than just an insane dream.  Unfortunately, to fully enable it, the NCrunch engine needed to be able to operate at a very high capacity.  It needed a high performance instrumentation system.  Meanwhile, Microsoft was busy rebuilding .NET from scratch at platform level, keeping us extremely busy just maintaining the existing features of NCrunch.  I find it hard to express how frustrating these years were.  RDI seemed achievable.  I had a plan for how to reach it .. I just didn't have the time to actually implement it.

Then things changed.  Michael Kroes joined me on the project.  Microsoft solidified the new platform.  We got on top of the workload.  COVID-19 happened, and everything slowed down.  There was finally a gap for more development work.

It came just in time.  For several years, we hadn't been able to visibly break much new ground with NCrunch.  Most of the work we'd done revolved around internal improvements and compatibility for more toolsets.  We've always had a very committed user base with NCrunch, but people weren't talking about the product anymore, making it harder for us to reach new users.  A decision had to be made.  Either we needed to implement something big (like RDI), or let things wind down and fade into obscurity.

I didn't want NCrunch to fade away.  But there was also a problem.  The emergence of JetBrains Rider meant that the .NET developer base was becoming split between two different IDEs, and it was getting really hard to justify not having support for Rider.  It wasn't enough to just implement support for Rider, and not have another big feature with it.  We also couldn't ship just RDI without making a lot of Rider users very upset.  V5 had to include both Rider support and RDI.

In hindsight, this was a logical decision.  But also, an extremely risky one.  There was no timeframe on how long it would take to complete either of these two features.  They might have even proved to be completely impractical.  We couldn't deliver either of them incrementally, so there would be no way to get user feedback or keep people excited in the project.

It was like stepping off a cliff with a blindfold on.  We didn't know how far we'd fall, what we'd land on, or even if anyone would still be around to witness the possibly giant mess at the end.

Meanwhile, the next version of Visual Studio was always just around the corner, built on a whole different variant of .NET and including unforeseen changes we would need to deal with.

I had to adopt a similar mindset to the one I used when working on NCrunch version 1.  Every day I told myself that it didn't matter if anyone would ever use what I was building, as long as I got to build something cool.  I accepted the very real possibility of spending several years building something for no reason other than being unable to forgive myself if I hadn't at least tried to build it.

And now, 3 years later, we have something to ship.  Is it perfect?  No.  But so far, it's exceeded my best expectations.  We're in Rider!  And now we can do things never done on such a scale before.

As I write this and look back on my life over the last 15 years, I feel myself getting quite emotional.  When I started on this project, I never imagined we would be here right now.

 

Onward

Over the next few weeks, our primary focus is the stability of RDI and our Rider integration.  With these features only just out of the oven, there are going to new problems discovered as more people start using them.  I expect we'll likely need to make a number of small releases shortly after v5.0 to address these issues.

When we're confident everything is working well with these features, we'll be focusing more on performance improvements and addressing feature gaps.

I fully expect that we'll receive a new barrage of feature requests during this time.  If there's anything you want to request, please do make the time to submit it through uservoice.  I know that I don't always respond to every request, but I do read them all.  We are also a team of just two people, so please understand that we need to prioritise our work carefully.

I want to stress that with both RDI and the Rider plugin being very new, these areas of the product haven't had the same level of optimisation as we've had for other areas of the product.  Please be understanding of this.  Performance optimisation and improving resource utilisation are an ongoing exercise on this project, and you can expect this to improve in the coming months.

You can find V5 available now on the Download page.  Happy crunching!