NCrunch Pricing

by Remco 1. October 2012 06:19

I've been very busy over the last month working on the new NCrunch website and v1.42, reading everyone's comments about the commercialisation announcement, and planning for how things will progress over the next couple of months.  I'll try to share these plans here so that everyone has a good understanding of what will be happening and what to expect.

First of all, I have to say that I'm blown away by positive the response I've had from everyone around the commercialisation plans.  I wrote that last blog post with a considerable amount of dread .. expecting the worst.  I'm glad that most people understand the reasoning behind why NCrunch is going commercial and that I'm trying to do this in the best interests of the products future.  Unfortunately the wide circulation of the post also made it a target for spammers, so I apologise if I haven't been diligent enough with moderating/approving everyone's comments in a timely fashion.

I can now confirm 100% that v1.42 will be the first non-beta release of NCrunch.  While no software product is ever perfect, I'm convinced that NCrunch is stable enough that I'm prepared to drop the 'b' and move towards larger longer term release plans.  As such, v1.42 won't be a big feature release and instead will be focused only on fixing high priority issues, along with introducing the new licensing features.  Further revisions to v1 will be limited to fixes and minor compatibility adjustments with the bigger changes and features being rolled into the future NCrunch v2.  Many of the future features planned for NCrunch carry a degree of risk that I feel uncomfortable introducing without first conducting wider testing, so v2 will likely have its own beta programme that will kick off later in its development cycle.

After much careful consideration, I can now also share the pricing details for the new commercial model.

As mentioned earlier, NCrunch will have two different pricing tiers.  One for named users (aka 'Personal' license) and one for company seats (aka 'Corporate' license).

The Named User License is for individuals who purchase a license for themselves or as a gift for another named developer.  There aren't any kind of restrictions on the type of work that can be done with this license (i.e. personal vs professional), although the license is owned solely by an individual and cannot be purchased by a company.  These licenses will be on sale for $159 USD each.

The Company Seat License is available for purchase by companies, and license keys will be issued in the company name.  This means that if a developer leaves your organisation, the key stays with the company.  These licenses will be on sale for $289 USD per seat.

I've done my best to keep the prices in line with industry norms while taking into account my own overheads and processing fees, etc.  I can only hope that these figures won't be surprising to anyone and that there is an understanding that NCrunch will of course pay for itself very quickly in the productivity gains realised.

There will, of course, be a 30 day evaluation period for anyone that wants to try the product before paying for it.  30 days is quite generous and I hope this will be long enough for the product to prove its value and for people to be certain that it will work correctly in their development environment.

 

About two weeks from now, I'll be rolling out a major upgrade to this website to bring it up to a commercial standard.  The upgrade will coincide with the release of NCrunch v1.42, the first release to contain the new licensing features.  When the new website comes online, it will have all the facilities necessary for people to purchase license keys in order to make the move to v1.42.  The v1.41b release build will expire on the 1st of November (about a month from now), so this should give a good couple of weeks for everyone to upgrade and make sure the new version works correctly for them.

As v1.42 will be packaged with a 30 day evaluation license, it is theoretically possible (with a late upgrade) to continue using NCrunch without a commercial license up until the 1st of December.  If at all possible I strongly recommend not leaving your purchasing of a license until the last minute to ensure that you're not forced to run your tests manually without NCrunch.

Thanks everyone for your understanding!  I hope to make this process as smooth and painless for everyone as I possibly can.

 

--------- Update 3rd of October ---------

I've turned off further comments on this post as much of the discussion isn't bringing new information into light and is starting to disintegrate into arguments.  If anyone has any further questions about the pricing, please submit them via the contact form and I'll share any relevant answers here.  Thanks everyone for your feedback and understanding.

Tags:

Comments (29) -

Yang Lee
Yang Lee People's Republic of China
10/1/2012 9:43:33 AM #

I'm surprised that personal license is only $159 USD. I will buy one right after the new version is released.

What payment methods will be accepted?

Remco
Remco New Zealand
10/1/2012 11:35:07 AM #

All the standard payment methods will be supported - i.e. Credit Card, Check, Wire/Purchase Order.  The processing itself will be handled by a payment processor that specialises in just this sort of transaction, so I'm certain this won't be a problem for you.

Dave Leaver
Dave Leaver New Zealand
10/1/2012 11:26:53 AM #

Looking forward to getting the bosses check book out for this one Smile
Also, hello from some NCrunch users in Hamilton, NZ.

Remco
Remco New Zealand
10/1/2012 11:37:18 AM #

Great to hear!  New Zealand has been doing quite well with NCrunch adoption lately... It's making me proud to be a Kiwi Smile

Hannes Kochniß
Hannes Kochniß Germany
10/1/2012 5:18:18 PM #

Wow.. I Don't want to listen too negative, as I was holding back on the last post. I have, seeing other successfull products, my own opinions about the R# price model. I AM very surprised.. in a negative way. I told myself "you can still convince your company later on to build NCrunch", as I try to gently introduce it right now in my company.  I told the people "well, don't worry, it won't be as expensive as R#, because a) he's a  single dev, not a team like the jetbrains guys and b) NCrucnh just isn't as professional and all-around-usefull as Resharper.". Boy was I wrong.

I'm a bit sad, because I can understand the commercialisation (I still think that there should be a feature-cut nonpro version, _really_. You are cutting your customer base without a freemium model, like Joe Spolsky once said: "the biggest hamper to cusomer base growth is price"), but I think the price is too high. Really.
Just as a maybe single voice, but I know for a fact that I can't get my company to pay the same price for this as for R#. Especially because we just start to write unit tests. Sorry, big fan, but that's a bit unreasonable.

APC
APC United States
10/1/2012 6:18:50 PM #

I think the real question is, does it provide enough value compared to ReSharper? Because a R# personal license is $150 for the C# version. And R# is a veteran product that does a million things, is supported by a team of testers, developers, has lots of nice videos showing off its use does by professionals and stuff. Most importantly, ReSharper is indispensable, you literally cannot use Visual Studio without it unless all you do is F# or C++ programming. I just don't see NCrunch being that way.

James Murphy
James Murphy United Kingdom
10/2/2012 1:26:55 AM #

Of course you can use Visual Studio without R# (and, before anyone gets cute, you can also without Coderush - which is my current choice - or the similar toy from Telerik or any other refactoring and code assistance add-in).

Does R# add considerably to the experience? Of course it does, do people want to give it up once they've got it? Of course no but is it necessary? Hardly.

But the point is actually not whether it offers sufficient value compared to - say - resharper but whether it offers sufficient value in and of itself to whoever is signing the cheques (figuratively speaking) i.e. are you prepared to pay to retain the value it gives you. This is as true of choosing a level of visual studio as it is of choosing add-ins and extensions

Yang Lee
Yang Lee People's Republic of China
10/2/2012 12:56:51 AM #

NCrunch's price is much more reasonable than some other unit testing tools like Typemock and Justmock. A few weeks ago, I thought NCrunch could cost over 400 USD, because Typemock is 799 USD and Justmock is 399 USD. So I was quite surprised that NCrunch's personal license is only 159 USD.

Hannes Kochniß
Hannes Kochniß Germany
10/2/2012 7:14:32 PM #

Sorry, that's not comparable.
JustMock/TypeMock use Runtime Proxys (as well as the new MS Fakes framework), which allow a big company to test stuff that is untestable without refactorings. That is a _completely_ different ballgame, and has almost nothing to do with NCrunch. I wish people would know more about the "big picture" of Mocking tools.
So in short: these tools cost so much, because companies can start testing legacy codebases with the _guarantee_ to not break anything, and introducing 0 bugs. That is, esp. to bigger companies, well worth 400+ bucks. But that has _nothing_ to do with what Ncrunch does (beside the fact that NCrunch supports those Runtime proxies, but that's all). Your comparison is (unfortunately) unfitting.

Jim
Jim United Kingdom
10/3/2012 5:42:05 AM #

To take that to its logical conclusion, NCrunch is a bargain because a Ferrari costs $200,000.

Sorry NCrunch, I'm uninstalling your beta. $159 is just too expensive for an open-source hobbyist like me. I'll stick with TestDriven.net for the much more reasonable price of $0.

I know you don't want to be free but not having an open source / private license at a much lower price seems like cutting out a lot of your audience.

Hannes Kochniß
Hannes Kochniß Germany
10/1/2012 5:24:16 PM #

I meant "buy", not "build".
Again, I don't want to sound too harsh. You have to realize, that a bunch of "oh that's great" comment posts attracts further "oh that's great" comments posts. The same goes for negative posts. They keep people with other opinions away. I f you want the real opinion, make a poll.

I followed the project, I think it's great, I just think that the (almost) same price as R# is really over the top. I have no problem with this for my own, I just see people shy away to the two alternatives. Please reconsider freemium, or dropping the pricer a bit. It's not too late. Smile

Remco
Remco New Zealand
10/1/2012 6:43:47 PM #

Hi Hannes -

Thanks for taking the time to share your views on this.  As nice as it is to go to sleep at night with only positive comments on this blog, I'd be naive to think that the expected prices would suit every company and every individual, and I appreciate hearing from someone that doesn't agree with the pricing, in part because it does give me a bit more of an opportunity to explain why the prices are where they are and hopefully help to open a broader perspective on the subject.

As I think your main point describes, software is usually priced around people's perception of 'price points', namely, the prices of other products that sit within a similar space or could in some shape or form be considered comparable.  Because we don't have a per-unit manufacturing cost, and the value of any software product is often subjective, this is the reference many people use in order to decide whether or not the product they are buying is good value, or a rip-off.  For the sake of argument, here is a list of price points that someone might consider relevant to a tool like NCrunch.  Since I think you're most interested in the company/corporate licensing, I'll limit the list to just those values:

Jetbrains' Resharper - $349
Jetbrains' DotCover - $199
NCover Desktop - $658
TestDriven.NET - £105 (about $170)
TypeMock Isolator - $799 ($1598 for the full deal)
Telerik Test Studio - $2499
Telerik JustMock - $399

NCrunch - $289

Resharper is an amazing product, and it deserves all the hype and credit it gets (IMHO, perhaps more).  Because it's so ubiquitous, whenever people think of a developer productivity tool, they usually first think of Resharper.  NCrunch is a productivity tool too, so the association seems logical.  However, in reality, the tools could not be more different from each other and they actually complement one another very well.  They are designed to cater for different situations and actually target entirely different markets.  Resharper can be found useful by just about every C# and VB.NET developer on the planet, where NCrunch caters only to a small subset of .NET developers that have the discipline and quality-centred focus needed to write automated tests.

I really wish I could say that most developers write tests, but if I did, it would be a flat outright and dirty lie.  Any consultant, contractor or employee with a few years experience knows just how bad this industry really is.  NCrunch has helped many people into taking a more disciplined approach to quality software, but I would be surprised if we're even 15% of the way there towards truly widespread adoption of TDD.  The uptake of NCrunch shows this.  I've been really happy at the response to NCrunch, but this website isn't taking thousands of hits every day from hungry developers eager for testing.  The economic scale of a product like NCrunch is just totally different to Resharper.  Someone from Jetbrains may jump out and correct me on this, but I would say that as a developer you get far less for what you put in.

In reality, many of the other price points listed above are probably far more applicable to a tool like NCrunch.  They all target the test driven developer and all aim to reduce the cost and complexity of software within the .NET field.  If you line up many of the prices shown in this list with the price offer for NCrunch, $289 may look like a much better deal.

I don't assume to know the effort that other developers have put into the products listed above.  All I can do is to look at what they offer and ask myself whether I feel they represent good value.  Does paying $399 or $799 for a cutting edge mocking framework make sense?  It must to some developers, otherwise it wouldn't be worth the while of the companies that develop this software.  I've worked on projects where tools like this would have paid for themselves very quickly, and I genuinely believe that NCrunch does the same.

I'm not going to be drawn into an argument over whether a company deserves to charge more money simply because they are a team v.s. an individual developer trying to stay above the breadline.  People are entitled to their own opinions on whether or not the appearance of a company has any real impact on their ability to produce valuable software.  I chose to play an open and personal hand with the users of NCrunch because this is how I prefer to work.  I could have just as easily started out as a faceless company and hid behind flash web design and ticketing systems in an effort to present a more professional image, but I chose instead to invest that time in making a better product.  Every business starts somewhere.  Microsoft didn't get to 90,000 employees on their first day.

Anyway, as you've already said, it's not too late to change things.  If the whole user base suddenly evaporates because of a $289 price tag, I could still just drop the product and forget the whole thing entirely.  The sad reality is that there just isn't enough demand for good TDD to sell $50 licenses, and even if there was, just trying to support the product would cost far in excess of any money that would come from it.  I'll see what the response is like over the next few days... as I'm writing this reply, more comments are already appearing here.

Thanks!

Remco

Hannes Kochniß
Hannes Kochniß Germany
10/2/2012 8:08:45 PM #

I agree with some of the points, especially you don't appeal to as much devs as R# does. Ok, fair.
The problem I have is really: I hoped for NCrunch, when it goes commercial, to really have a landslise effect on the .NET landscape. Like you often see in the current Webspace, where new sites with freemium totally change the landscape and attract a huge folllowership, and then the people find a way to make big money with freemium or similar models. What really sucks to me is: NCrunch won't be the big "wakeup" call to companies like JetBrains at this pricepoint.

So again, most points are fair, but where it gets a bit unfair is this sentence: "In reality, many of the other price points listed above are probably far more applicable to a tool like NCrunch".
*buzzer* No. Wink Unfortunately I'm a former user of all of these, as my position is the one of an all-around testlead in my company. And as it the n-th comparison of TypeMock/JustMock with something else, I guess I'll have to write something down.

I'm sure you know as well as I do, that JustMock/TypeMock just don't belong into the list, as well as TestStudio.
I'll reiterate from my other answer above: JustMock/TypeMock/Fakes framework from MS allow me (primary usecase) to use Runtime Proxies to inject any TestCode, it's for old legacy codebases, and guarantee my company to introduce 0 bugs to get inmemory, unittest-like tests running, without refactorings. Again: key is _guarantee_ to not introduce bugs into an old codebase. NO refactorings need. That.. is.. HUGE. Totally different usecase, for often different codebases in sometimnes different companies, compared to NCrunch usecases/users.
Short: TypeMock/JustMock: targeted at brownfield projects, big legacy codebases, customer base, normally a lot of money involved in that codebase. NCrunch: mostly TDD greenfield projects, often opensource, small private projects, which tend to not make millions.. But of course that will change. I'm just talking about the usecases right now.

When NCrunch gets a runtime proxy feature that can do the same as TypeMock/JustMock (I doubt it ever will, given the scope of the product), let me know, your product all of a sudden increased its worth by at least 400$ per seat.

TestStudio: as a former user, that is a BIG suite positioned against MS TestStudio, so the functionality (Testers recording tests, videos, test reporting) is totally different ballgame. Not even near. Statistics, bringing teams together, make manual testing actually bearable with a recorder that works, lots of stuff in there. TestRecorder, Test Database. Such stuff. Totally different to NCrunch. Therefor pricepoint > 2000$.

NCover Desktop: Coverage statistics, most complete feature set about everything that is coverageanalysis. You also took the full pro version: Website that shows that stuff, full support team, etc. A lot of tooling around it. But having said that, I think the price point lives from the fact that until now, there is no real competitor, and NCrunch is still an underdog. But still: different usecases, no real comparison.

Fairest comparsion:
Jetbrains' DotCover - $199
Your product is like DotCover, +the MUCH needed Live update. +metrics, -export feature (dotcovers HTML export is actually an unknown and the ONLY way to get per line coverage markers in JSON, which I deperately need for an internal, but that's a minor point). -support (right now, guess that changes; I don't care too much about that). -R# integration, -a bit of polish on the UI side. -Attaching to external process. But I don't want to be mean, I know with the pace you have, that minus points (not R# integration I guess) will be fixed.

So is it more worth than 199$ DotCover? For the standard usecase in TDD: of course! Compared, the $289 seem reasonable.

Just don't write it's "cheap", as it is rather "normal placed" in the market. The problem: the .NET coverage tools market is not a young one anymore, it suffers from no real market dynamics (price points changing), and you're really sitting yourself right into it.. That's all I'm saying. Don't want to sound aggressive, again. After all, still big fan and yes, this tool is needed. As the two alternatives maybe are, sadly saying that, in the future.

So in closing: great product, just _please_ don't slam the door for people who want to continue using it (meaning 1.41 should stay free, like the NCover situation). Otherwise I sadly can't introduce it into my company. And I'm still a firm beliver of the freemium model. Just sayin'.

Hannes Kochniß
Hannes Kochniß Germany
10/2/2012 8:16:45 PM #

After all, to reiterate my point, a "or I'll drop the product" would of yourse be a kneejerk reaction. Noone wants that. Just I think it could be 10-20% cheaper, OR introduce 2 products (Normal and Pro) version) which would probably already fix the most problems. That's what I meant by "reconsider". A bit more choice.

After all: I'm following you (even twitter and HerdinCode podcast) since some time.. I really like the vision you have and that you pulled through so long. I will also buy the product anyway. I just see adoption rate of the product (out of some weird "make the world a better place" view) as really important, and I think you can still get the same money out of it if you had freemium or a second product version at a lower point.

As this is my last post for a while, I want to make sure I get that across: .NEt landscape needs it badly. That's why I care about adoption rate in the future. But after all: great product, please keep going, in whatever way.

Remco
Remco New Zealand
10/3/2012 6:59:23 AM #

I think that we're more or less on the same page here.  I won't go through the motions of comparing product features between NCrunch and other tools - By mentioning at all, I was just trying to make a point.  Around the time that your original comment was posted, there was a big push of many people comparing the price with R#, and I felt many people were missing the big picture.

Anyway, I agree with you that $289 isn't a bargain bin price, and most likely it will make NCrunch inaccessible for many people lacking access to this kind of money or lacking the justification to spend it.  But factoring in the market size, effort involved and value brought, I feel it is a fair price.  From my own personal tally of positive reactions vs negative reactions, a wide majority of people agree that it is fair price.  Most of those that disagree seem to have expectations more in the $50-100 mark, which would make a tool like NCrunch economically unviable.

I do feel the pain regarding the need to make the world a better place.  In this industry, we've all been tangled with rubbish quality code for far too long, and nothing made me happier than the fact that I'd managed to design a tool that could make a positive impact and really make people think about the quality of their code - and be more productive for doing it.  By putting up price barriers I do realise that I'm hindering this adoption.  I'm hindering the change that we all so badly need to make.  I'm making it harder for new developers to get on board with practices that will make the world a better place to live in.  Believe me, this isn't something I'm happy about at all.

But the unfortunate reality is that this is a very small market.  There are others that have built tools with very similar objectives to NCrunch and have failed to stay afloat for the simple reason that there just aren't many developers that are interested in continuous testing.  A freemium model requires a certain size of market in order for paying users to carry the weight of  free ones, and my own observations are that the current market for NCrunch is not of this size.

To reiterate from my last blog post, the motive behind commercialising NCrunch is so that the project can continue to survive.  I've had to face an unfortunate reality of:
1. Find a way to get enough money to finance NCrunch V2
2. Get a haircut, and get a real job

Consider that the current version of NCrunch barely does even 30% of what I'd currently like it to in the long term.  The remaining 70% is sitting in a list of objectives half a mile long, and they're all possible and all within reach if enough people are prepared to buy licenses.  The 30% of this concept that is implemented has been enough to significantly change the way that many people develop software.  Just imagine what the 70% will bring.  This is the big deal.  This is why I'm prepared to risk lower adoption of the product.  Because if I don't get the chance to see that 70% happen, then I'm convinced that everyone loses big time.

Anyway, thanks for sharing your thoughts.  I appreciate both your honesty and your intentions.  And thanks for supporting NCrunch by offering to buy a license Smile

sils
sils United Kingdom
10/1/2012 5:56:00 PM #

Hi Remco,

Pricing seems reasonable, and about what I was expecting.  Think R# has a bigger potential market hence JB can keep the price lower.

Two questions:
- does the price include 12-months of upgrades, plus a discount after that?
- is the personal licence applicable to a one-person company?

Thanks

Remco
Remco New Zealand
10/1/2012 6:48:51 PM #

Hi Sils -

The upgrade policy will include a free upgrade to any major revision released within 6 months of the license purchase.  There'll also be an 'upgrade' license available that will allow you to move up to a later version for a lower price (I'm thinking probably a 30-50% discount).

The personal license is suited to individuals for both personal and professional purposes, so if you like, you can purchase a personal license under your own name and then use it for your work within the company.  There aren't any constraints on what kind of work you do with the license, only on the name of the purchaser (i.e. if it's a company name, it needs to be a company seat).

Thanks for your post!

Remco

sils
sils United Kingdom
10/1/2012 9:24:48 PM #

Thanks for the response.   Keep up the great work!

David Thompson
David Thompson United Kingdom
10/1/2012 6:34:08 PM #

@Hannes, i'm totally in agreement with you, sadly I showed the pricing model to my line manager and he laughed when I told him the commercial price, he made the same point about Resharper with its features, sad really as I do think NCrunch is a fantastic tool, and I totally understand the need to charge (we all need to make a living) but at those prices its going to be a hard sell to my bosses.

Remco
Remco New Zealand
10/1/2012 6:50:37 PM #

Hi David -

I totally understand.  I've done my best to explain my reasoning in the reply to Hannes' post.  I wish you all the best in your dealings with your line manager, and I hope you'll watch this space closely.

eviltobz
eviltobz United Kingdom
10/1/2012 8:27:56 PM #

Is there any plan to offer reductions on the commercial licence based on the number of seats bought? Comparisons have already been made to Resharper's pricing, and they offer bulk discounts when buying 5 or more licences, and based on the "get quote" link I presume that the discount grows with order size rather than just being a completely fixed affair.

Remco
Remco New Zealand
10/1/2012 8:34:18 PM #

A number of people are asking after this one, and I am listening Smile  For NCrunch V1 I'm going to try to keep things simple with just the flat scheme - more because it keeps things much easier from a sales perspective than for any financial reason.  I'll review this later down the line as the support patterns for the commercial version become clearer.  I recognise that support costs are likely lower when licenses are purchased in volume, so be assured that I'll be doing a full assessment of whether this makes sense for future versions.

Stephen
Stephen United States
10/2/2012 3:11:10 PM #

Fact Check:  
ReSharper is $249 for either c# or VB.
The price you quoted is for the 10% of developers who code in both so much that they need the full version.

Stephen
Stephen United States
10/2/2012 3:15:55 PM #

I am not saying your price is bad. ReSharper operates on a demand level you will not see (at least right now).

I just wanted to make sure you are quoting the "real" price.

Remco
Remco New Zealand
10/2/2012 3:30:38 PM #

I think I understand why you posted this, although I hope I managed to make my point about why comparisons between NCrunch and Resharper are very much like comparing apples and oranges.  I'm aware that Jetbrains have different pricing options for their products.  To start picking apart price options would always boil down to comparing features between the tools, which I feel would be a useless and destructive discussion.  NCrunch isn't priced like Resharper.  Indeed, neither are any of the other products listed above.

Stephen
Stephen United States
10/2/2012 11:39:55 PM #

As a side note. I have been avoiding NCrunch because I knew it would go commercial. Now that it has, and the price is reasonable, I plan to give it a look.

Jim Sowers
Jim Sowers United States
10/3/2012 3:48:13 AM #

Yeah, I am disappointed with the price. I love NCrunch and have used it all along. The commercial pricing will probably take it out of my department. Even if my manager would pay for it, there would be a month gap. It will expire at the end of November and for a good sized department, we would have to move the request into the fiscal plan for 2013. We will not be able to swing several thousand dollars at the end of the budget year.
Remco, I am very impressed with the tool you have written. I think it is amazing, but I think you have priced us out. I do not think your price is unreasonable but I had a difficult enough time getting resharper in house.
Thanks,
Jim

Azzlack
Azzlack Norway
10/3/2012 4:39:36 AM #

Awesome work @Remco!

Ncrunch is already better than the competition, and worth every penny IMHO.
It has already saved me countless development hours.

Will buy it the instant it's released. Smile


Just curious about 1.42 - is there a changelog somewhere?
I have some tests that run fine in the R# test runner, but fails in the NCrunch runner and was wondering if there are any fixes related to that.

Remco
Remco New Zealand
10/3/2012 6:14:41 AM #

Hi Azzlack,

Thanks for this comment! You've made my morning Smile

I usually publish changelogs with every release, although as 1.42 is still a work in progress I can't yet 100% confirm what will be in it.  I can, however, say that there aren't any fixes planned for this release that involve changes in test behaviour.

There are some subtle differences with the way that NCrunch executes tests that can sometimes cause them to behave unexpectedly.  If you haven't already, I really recommend having a read of the test troubleshooting guide in the Wiki.  If you can post some information about the failing tests in the support forum, I'll help you with getting them up and running Smile

Cheers,

Remco

Pingbacks and trackbacks (1)+

Comments are closed