The Future of NCrunch

by Remco 18. September 2011 21:18

I often get asked about the future of NCrunch.  Many people cite the sentence at the bottom of the front page ('completely free during its beta test period') as implying that NCrunch will one day be a commercial product.  I want to point out that this is far from a certainty and is only one of the many potential paths that NCrunch may one day go down.

I originally started working on NCrunch out of frustration with the existing tools operating in the same space, and as a bit of side-project that I hoped would go into something new and exciting.  So far, it's more than exceeded my expectations on the excitement front, with a nice constant buzz of activity and feedback from the people that enjoy using it day to day.

But much more, it's been a fascinating learning experience for me.  It's not every day you can find a project even half as challenging as NCrunch.  The concept may be simple, but the implementation is far from it.  Most projects take their challenges around their integration with other tools, or their algorithmic complexity, or their level of concurrency, or their performance constraints.  In NCrunch, each of these aspects is equally very important.  Many people seem to think that writing an effective continuous test runner is as simple as just wiring up a test runner and kicking off the tests in a background thread - in reality, that's not even 5% of the problem domain.  Just even getting the engine in a working state where it could help on simple projects actually took me several years.

Above all else, the real challenge in NCrunch has been in getting it to work for as many solutions and environments as possible.  Anyone who has written a desktop application knows how many environmental considerations there are ... Is the user running Windows 7? Is it a 32 bit processor? Do they have a dodgy anti-virus installed? What about their version of Visual Studio? What versions of .NET do they have loaded?

When you add these environmental considerations to the sorts of complex build and test situations NCrunch is designed to handle, you get situations like: What if this project runs an ILMerge on a post build event? What if it uses an aliased MSBuild reference to a dependent assembly that may or may not be loaded in the GAC? What if the project contains a test that gets surfaced only using an attribute that inherits from a test attribute hosted within another assembly and uses an MTA thread when everything else should be STA?

When many people download NCrunch, they just skip straight past any documentation then go and boot it up against some huge, complex, multi-man-year solution with a legacy of build customisations and frameworks that do crazy stuff, then expect that NCrunch will simply deal with it.  It's easy to blame these people for not taking more of a careful approach, but it needs to be expected.  If someone picks up a tool they've never heard of before, they usually can't expect to spend 2 hours of their life evaluating whether or not it will work for them.  This is why I try so hard to encourage people to report any solution-specific problems as they find them, and is actually the whole purpose of the entire beta program.  No software developer on the planet can warrant that such a complex and integrated tool can work in every situation that's used, but I'm damn well going to try and see if I can get close.

One option I have been looking at is to eventually open-source the tool - as this may help many people to solve issues with having it work in their customised environments.

As a concept, NCrunch can move in many directions and solve many different problems.  I want to take the time to explore those options before I make any commitments about the future of the tool.  For now, I just hope everyone can enjoy coming along for the ride.


Comments (8) -

Hannes K.
Hannes K. Germany
9/20/2011 3:00:23 AM #

I really appreciate your hard work. I think the easiest way is to crowdsource some of the more "specialized" parts, e.g. TestRunner support, to the community, via Plugin System, ideally per something like MEF. This ensures that everybody can adjust the configuration part, like Nunit TestCase support etc., it would increase development speed in the long run, and you would still hold the IP. It would benefit all, no matter if you want to make money out of it (increasing the value) or opensourcing it (a first manageable step). But I realize that switching to some plugin system can be a big initial time hit.

Also, I wanted to thank you just at least one time for this awesome mod. Really flashed me the first time. Smile
Growl support! *cough*

Remco United States
9/20/2011 8:07:04 AM #

I do have high hopes for the current integration with Gallio.  Gallio is basically exactly what you just described - a pluggable architecture for test frameworks.  I think there is definitely more work that needs to be done here though, as the interfaces haven't been properly formalised (within the scope of NCrunch at least).

Growl support is on the list Smile  My only problem is that the list is really, really long!

Jeremy Canada
9/21/2011 6:16:16 AM #

The leap from working for enough people to working for enough people to qualify as a sellable, supportable product is a daunting one, that is for certain. For the moment I am just very happy that you've chosen to let us folks play with this amazing product. In the short time that I have been using it NCrunch has entirely changed the way I do my .NET work and I would never want to go back.

Understanding that you can't go commercial with NCrunch just yet, would it be possible to, for example, set up a donations system so some of us regular users can thank you for your work so far and contribute a bit to it going forward? And/or a list of some contributions you are looking for, along the lines of what Hannes mentioned, so that those who cannot contribute to NCrunch financially can do so with code? Or with hosting, software licenses, or hardware?

Remco United States
9/23/2011 7:55:29 AM #

A contribution system is a good idea, and I'll definitely give this one some more thought.  Meanwhile, the best kind of donation anyone can give to NCrunch is to keep submitting bug reports and feature requests.  I'm in a position where I've used it for quite a while, so features that are clumsy to new users may not be obvious to me - I really need as much feedback as I can get Smile

The step from having a cool little beta tool to having a refined product is always a big one, though something that few people are aware of is just how far NCrunch has come within a few short months.  Back in February, NCrunch's UI consisted of a couple of horrific colour-coded DataGrids showing lists of tests and queue tasks.. there were no wizards, filters or any extensive configuration.  It wasn't even possible to disable the engine.  The latest version still has a few niggles and it doesn't work on every solution under the sun, but it is now a useful tool that many people really enjoy using... watching that progress has been a reward in itself Smile

Steven United Kingdom
11/24/2011 11:38:32 PM #

I would just like to chime in here and say: there are very few .NET tools I would happily pay for, and this is one of them! I've only been using it a day but already I don't want to go back to the VS test runner.

James World
James World United Kingdom
4/30/2012 7:58:26 AM #

As I've mentioned before in the forums, this is an awesome tool that has had a significant effect on my productivity - thanks so much! I hadn't realized it's been "years" that you've worked on this. You must have a real sense of pride - and ownership - in NCrunch. I feel we are on the cusp of a sea change in regard to the demand for and uptake in continuous testing tools - much like the sudden explosion that happened around the awareness and uptake of unit testing tool around half a decade ago. There are a number of CT tools in active development now and it's only a matter of time before this scene changes up a gear. IMHO, NCrunch is the leading .NET tool in this space by some margin at the moment, however I fear it won't stay that way unless more developer resource is brought to bear. While I can appreciate that maybe you have a desire to keep a firm grasp on the reins of this tool into which you have poured so much time and effort,  I urge you to consider seeking commercial partnership or perhaps take the open source route to ensure NCrunch has the future it deserves - and to up the pace before competitors overtake you! Whatever you decide, keep up the great work. I would happily pay for this tool to further it's development.

gemini People's Republic of China
5/22/2012 1:23:56 AM #

thanks your work.

tc Poland
5/22/2012 8:50:56 PM #

I'd be happy to pay for this tool as its becoming invaluable in my daily work - I'm already hooked, it saves me time and makes my job easier and more pleasant.
I'm an independent developer so for me the price should be 200 USD tops, if it's more expensive I couldn't by it.
Also I'd rather see NCrunch go commercial and have solid long term support and be constantly developeded than go open source because it seems to me that this kind of advanced code base wouldn't find too many developers that would be commited to work for free and provide the required level of expertise - in my opinion the project would most probably go stale and die which would be a huge loss for us (currently there aren't many big and advanced OSS projects out there in the .NET space that are successfull and developed at a steady pace - i.e. NHibernate is more of an exception in this area).
Thanks for your hard work and keep it going.

Pingbacks and trackbacks (1)+

Comments are closed

Month List

Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download