Here's what we currently have:

I'm going to kick this out of the ops space since none of this should be sensitive.

The current ruleset implements all those rules correctly, except that it 200's directly for /status/.

redirect to Javelin

On secure, two not-so-great fixes are:

These are now up to date.

In T12296#216132, @epriestley wrote:

While the git ls-remote change isn't really motivated as a performance improvement, it does seem to have reduced CPU usage a measurable amount (deployed on the morning of 3/18), maybe 15%:

While the git ls-remote change isn't really motivated as a performance improvement, it does seem to have reduced CPU usage a measurable amount (deployed on the morning of 3/18), maybe 15%:

The setsid() appears to have resolved the issue, in the sense that daemons are now starting cleanly every time without receiving any inexplicable signals.

The MySQL stuff seems to be in the old daemons (them trying to do logging while MySQL restarts), not the new daemons, and not the cause of the issue.

Even with a very conservative policy, we're going to drop 80% of the instances it sounds like.

Well, I think it's reasonable to possibly not log in for a while like I keep dormant projects on GH. I don't want them to disappear, but I prefer to keep them in the cloud.

We can also do other stuff like "never archive an instance with 2+ users", "after an instance emails us to unsuspend, have a flag to mark it as exempt from cleanup in the future", "don't destroy data for 365 days, instead of 90, if an instance had hosted repositories", or other sorts of things like that, but I suspect measures like that will be largely unnecessary for years to come and that "no one loaded a page for 45 days" is already a nearly perfect signal for "no one cares about this instance".

I'd say 180 day destruction window, and mostly that's just for us to feel good about deleting stuff.

Fair points.

Picking a random shard, 80% of the active instances have zero sessions in the last 30 days -- so no user has loaded a single page in at least a month. Most (maybe all?) of these have only one account. These include (anonymized) free instances like "alsdknfasf", "test2", "test2017", "testinstance9", "testforme", "mytest", "myinstance", "bbbbbbb", "ddeemmoo". Of the 20% of active instances which do have activity in the last 30 days, many are recently-launched free instances with only one user session, and I'd guess many of those users are never coming back and these instances will join the boneyard shortly.

My thinking is that v1 would work like this:

I suppose we could get pretty far changing the language from "Disabled" to "Inactive". That is, send a warning after n days, then after 3 warnings, set it to "Inactive" but have a means they can contact us or click a button to re-activate it?

So I've read this a few times and I'm still a little unclear what happens if "activity requirement" isn't met. It sounds like we'll just disable them like we do non-paying instances - which I don't think users expect and feels like a crumby experience. Would it just be possible to "hibernate/sleep" the instance (no daemons run, user has to wait and spin them up to log back in if 90 days later they get back to using the instance)?

I haven't been able to reproduce this locally by artificially making services unavailable or connections fail. Debugging this in production will require momentary service interruptions on admin until I can narrow things down so I'm going to wait until off-peak to poke at it.

In terms of application/code stability, it would be good to handle this properly.

Specifically, I suspect it won't affect repo because db restarts separately, so the database won't be unavailable when daemons on repo restart.

It looks like this might be an issue with MySQL not being available yet when the daemons start. The deployment script restarts MySQL, then immediately restarts the daemons, and a bunch of this stuff is ending up in the log in the 1-2 seconds after the restart:

Earlier, I predicted:

I've deployed everything so I believe this is now resolved. I'll keep an eye on it going forward.

The pool appears to be scaling up and down properly now. I'm going to deploy the fix to the secure and repo tiers since it may affect all instances.

I'm picking D17433 now and sending it to admin.

I'm picking those to stable and sending them to production now.

For wikimedia's "scap" tool we built a feature called "checks" which are configurable commands that run on each target machine and confirm if the desired commands were successful.

I did the last deploy (T12316) with Phage again and things went much more smoothly after the various bugfixes and improvements.

Cool, now I don't feel so bad since I know you only recently added the option 😄

(That said, my expectation is that behaviors around unreachable commits have not changed -- D17279 added the "Unreachable" option to solve this problem.)

Likely somewhere in T9430 / T10978, which rewrote a big chunk of the audit code.

That works, Im curious when this changed, as I didn't need to filter since the last few weeks.

If you don't want to see unreachable commits, use a query which hides them:

The reason I think something isn't working correctly is in my dashboard, I list recent commits, and since I updated my install a few weeks ago my commit list looks like this a lot:

It is expected that some deleted commits will import and others will not.

@epriestley all other commits that have the message "This commit has been deleted in the repository: it is no longer reachable from any branch, tag, or ref." have imported, so I don't think the reachable code is working correctly, as until within the last few weeks I wasn't seeing these "This commit has been deleted in the repository: it is no longer reachable from any branch, tag, or ref." commits showing up, but now I see them all the time.

Total cost is about 1.6s. Of this:

This is still a little bit of a pain but far more reasonable now, here's an actual profile:

I cycled the rest of the tier and things look better now.

After manually deploying that to repo001, I'm now seeing a consistent Taskmaster/PullLocal count. I'm redeploying the rest of the repo hosts.

I think the issue is that the --reserve flag for reserving system memory now applies to even the first daemon in a pool, because all daemons are now part of pools. Previously, it only applied only to the 2nd..Nth daemon.

Some taskmasters failed to start on repo001 and perhaps elsewhere.

I suspended and destroyed the largest instance on the shard, a free instance with one user and no logins for >30 days (the user appears to have imported a huge repository and then never logged back in). We still have backups, but this instance was way over reasonable resource use anyway. The volume is now 78% full.

I pruned some very old backups (backup pruning is currently very conservative in several cases). The volume is now 72% full. This could easily be reduced much further by allowing bin/host prune to remove backups older than 90 days for suspended/terminated instances that have been out of service for 90 days, but backup utilization is generally relatively stable so I'm not concerned about this.

I synchronized services for this shard and destroyed old instances. The volume is now 68% full.

I synchronized services for this shard and destroyed old instances. The volume is now 44% full.

This was a bunch of repository file data in /tmp. Each file was unique, so this doesn't look like any sort of buggy loop or anything. There's some actual bug here but since there's a long list of other issues here I just "fixed" this by clearing /tmp.

Some reserved instances ended on Feb 19th; I should make sure reserved instances match production instances.

Shards 017 through 020 are now in production.

The re-launched hosts seem OK, I'm waiting for DNS to update and then redeploying them.

The hosts launched into an "Auto-Assign Public IPv4" subnet but didn't get IPv4 addresses. Restarting them didn't help, and there's no apparent way to assign public addresses after the fact. I'm going to throw them away and try again.

I'm going to put more hosts into production immediately before pursuing other issues.

• Some reserved instances ended on Feb 19th; I should make sure reserved instances match production instances.

• The tier as a whole is 88% full.

The commit cache should be a fairly large performance win.

Hibernating daemons currently show as "Waiting" in the Daemon console, but I'm not going to worry about that for now.

Ah, okay. This is the relevant part:

@epriestley
https://test-3uoifmrptebb.phacility.com/R2:80233a27d7b72bb03710d358e12a531ce46808a9