Potentially moot with private clusters? Do you think this is worth the effort today?

Oh, I can skip it if you aren't too concerned about it, it just came up and I didn't want to forget about it. I think it's like 10 minutes of work, but I'm perfectly happy to not do that work too. I don't think having this data would change any decisions I'd make, at least, even if the data showed extreme results:

• I'd interpret 100% normal growth to mean that we're doing great, and the pricing change might not be super valuable but saved us some support workload.
• I'd interpret mostly large instance growth to mean that the pricing change was good and identified a real need.
• I'd only be concerned if there was zero normal growth, but from peeking at the data I know the number is somewhere in the middle of these extremes anyway anyway, which is what I'd expect. And we can detect zero normal growth from "Active Standard Instances" and actual revenue anyway.

Yeah I think these stats are super interesting. Probably prefer nux stuff.

My only new thoughts on free tier are concerns over it being a "leaky faucet" money wise, ie, that we're paying out hard cash for daemons, etc, for instances that aren't being used. I'm not as worried about support since we're holding the stack. Which is why we offer "test instances" which we GC now.

Essentially, I feel cool with spending $x/month for a real small startup to use Phacility, but not $10x/month for someone to import 5 linux kernels and leave us holding the bag indefinitely.

We can limit the number of instances, but I think the big question we want to answer is how many free instnaces convert to paying instances over some long period of time.

I'd expect pretty low conversions (in the realm of a few percent?) so the limit probably has to be pretty high. If we only let in 100 free instances, a couple of data points could make 1% conversions look like 3% conversions, or 2% conversions look like 0% conversions. So I think the minimum sample size we can get any kind of statistical effect from is probably more like a minimum of 1K, not 100.

My biggest concern with this is that it means I will more often need to look up information about who is sending us mail to support@phacility.com before I can respond, since we can't reasonably give free tier users the same level of support that we give paid users in the long run, and it might bias the experiment by providing a much better support product than a permanent tier would get (though I'm not sure which way that bias would go).

The actual lookup is kind of a pain right now, especially since users sometimes don't include their instance name in their email. In this case, I have to go fish around for their username or email address (often in the database) and connect it to their instance, then go from there.

I feel like I'm already pretty overloaded (in general, not with support specifically) so I'm hesitant to add more stuff to my workload, especially high-urgency interrupt stuff where I may do a lookup only to figure out my time is being wasted which we're committing to for a long period of time (e.g., many of these instances will probably still be around in 12+ months). Routing support through Nuance is a clean technical solution to this problem (we can prevent free instances from entering the high priority queue, have Nuance look up instance data, and display it on the Item page immediately).

I'd be more open to trying this sooner if:

1. we can decrease my overall workload so I feel like I can take on more high-urgency support stuff like this without burning out (e.g., by hiring, although that will probably take 6+ months after hiring to start paying dividends); or
2. we can find some way to prevent free instance support from reaching the paid instance queue so I don't have to worry about seeing a Phacility support email that's from a free instance; or
3. we have some reason to believe that running this experiment now is a lot more valuable than running it in the future.

On (1), I don't think there are any quick fixes. The gears are in motion on hiring but that won't happen overnight, and even if we hire now that will likely increase my workload for a long time. A lot of my recent total resource committment is outside of the professional realm anyway (e.g., family stuff, litigious neighbor stuff) and probably has no technical solution.

On (2), I think we need Nuance or something similar. The critical part is doing user-to-instance lookup and instance-based routing, so even if we found some third-party software that we could plug in to fill the gap today, it probably couldn't solve that problem (at least, very cleanly). We could forward all support mail to some pre-queue and hire someone to forward it from there to us, but that would impact paid users negatively (by adding a queue delay) and require us to build tooling for lookup and potentially expose a lot of fairly sensitive information to a third party.

We could try to build a hacky version of this which receives the mail, makes an effort at doing an automated lookup, and immediately forwards it on. However, this isn't trivial and would pretty much be completely thrown out when we built Nuance.

On (3), my gut is that the number of ~1-4 person companies who are seriously interested in paid hosting for Phabricator but turned off by the $60/month price tag for 3 users is extremely small. We already provide a very long free evaluation period, and I can't really recall any direct feedback from users that I'd characterize in this way, outside of "I want everything for free" users.

Just based on experience, I think most 1-4 person companies don't want/need code review or even real bug tracking. I'll pull some stats on actualy instance size and send them privately, they may be informative. But I don't have an idea in mind of who this is targeting: a 4-person startup that is totally convinced of the value of code review, but not willing to pay $80/month for the best tool for it? I get that budgets are tight for startups, but even using absurdly conservaitve numbers this startup is spending at least $10K/month on salaries, and adding Phacility reduces their runway by something like a few hours. If this is who we're targeting (4-person startups who want code review and are bad at math?) we could just have a special startup program for these users. Or maybe just write a "how to do runway math, you clowns" page.

My guess is that conversions from these instances to paid instances (in cases where having the extended free period was a deciding factor) will be very rare, and that these instances will, on average, generate more support issues than paid instances. We also probably can't literally ignore these instances, support-wise -- we still need to do stuff like data exports, I think?

Thinking on it, I think this experiment also won't necessarily tell us much of anything, because we won't be able to create A/B buckets effectively. Users will have no real reason to start with a paid instance, so every instance which would have been a paid instance and is small enough will start as a free instance instead. We can maybe exclude instances which spend less than 1 month in the free tier and compare the effect of the free tier to the periods before and after we open it up and catch some of them if they have to upgrade to get real support, but we'll still be comparing very dissimilar samples and probably be unable to conclusively get any real data about what effect this tier has. Our month-to-month variance is also likely to significantly overwhelm any measurable effect.

An alternate experiment we could run today without some of these risks is "your 2nd, 3rd, 4th and 5th users are free", i.e. $20/month for 1-5 users (then at least everyone hitting the support queue is paying us something). However, from the $10 experiment it seems like our customers are generally not particularly price sensitive in practice. And we also can't really draw any conclusions from this one. I would expect this to pretty much just be a repeat of the $10 experiment.

My mental timeline for this was more like 'after private clusters'. Sorry, didn't mean to add thinking about this to your immediate plate. I'm pretty chill right now waiting for CORGI data to come in (2-3 months) and NUX.