Yeah I'm seeing SSL errors as well. Not sure if it helps but there are some items in yellow on https://www.ssllabs.com/ssltest/analyze.html?d=secure.phabricator.com

Sorry about that -- that patch should fix this one. Let me know if you hit anything else.

It's a single host and I didn't touch anything in the last couple hours.

Can you be more specific? I can't reproduce this.

I think I'm going to bring .org onto this host for simplicity/consistency, too. Putting it on corp is messy because of SSL, but putting it here for now should be straightforward.

Oh, I need to hook up diviner / symbols, too. I'm probably just going to cron those. So still-remaining stuff I'm aware of is:

I manually installed xhprof on the host since it needs some special setup steps that aren't trivial to integrate into the deployment tools, and isn't particularly important to have available.

Backups work but won't be run automatically by cluster administration services, although they are reported. I'm just going to cron them for now.

Stuff I'm still tracking:

• I configured HTTP -> HTTPS in a standard way (on this host, it involves some legacy oddness).
• I added host SSH keys and switched the tier to use them.
• I verified external connectivity and reasonable-seeming repo connectivity.
• I created secure001.phacility.net and sbak001.phacility.net devices on Admin, so generating routine backups will depend on the cluster being available. This seems acceptable as a dependency.
• I added run.repo to the host attributes to get repo backups working and verified repo backups get written.
• I dumped all the data and moved it over.
• I renamespace'd the database to secure and loaded it, then made config adjustments to use this as the storage namespace.
• I unpacked the repos into /core/repo/secure, then made a config adjustment for this.
• I allocated a new elastic IP and swapped DNS over (we still have javelinjs on the old box, plus we can't reuse a non-VPC elastic IP in a VPC).

I've completed the cluster deployment and am resuming this now.

Thanks for the report. That should fix the issue, let us know if you run into anything else.

Please don't pentest on a production install.

"Audit" is basically "Query commits", just a little un-modern / funky.

I think this is kind-of working now. I'm going to take a break for today and pick it back up tomorrow since I'm pretty close to bringing the data over, which might take a while, and I'm sort of committed to finishing once I start it.

• I created a new instance.secure security group. I removed port 25 (SMTP), since we now use Mailgun for inbound mail. I removed port 843 (Flash sockets) since we no longer use them. I changed administrative SSH from 222 to 2222 for consistency with the vault tier. This leaves us with 22, 80, 443, 2222, and 22280.
• I launched a secure001 host and added a secure001 DNS entry for it in Route 53.
• I created and attached sbak001, srepo001 and sdata001 volumes. This is a little overkill but should work well with the tooling.
• I created a new "secure" host role in rCORE44a038f, and deployed the bastion to get it to recognize the role.
• I used bin/remote deploy secure001 --port 22 to do an initial deploy.
• I added secure.conf and the missing sshd library in rCORE95a18f3.
• I added a bunch of config files and stuff in rCORE78c6c9a. This got SSH moved to 2222 correctly, so --port 22 is no longer required (like vault, this tier does need it for initial deploys).
• Kept fiddling with config (see diffs), thing is sort of deploying properly now, at least.

I'm starting this now.

My experience is inline comments usually aren't chat-like and would not make sense if synchronized in real time. It's intentional that they don't publish for other users until the author is ready to share them. For example, I often make an inline asking about something I don't understand, but later understand it after looking at more of the diff. I also often make an inline pointing out an instance of a larger theme, but the comment on its own doesn't make sense without the larger explanatory text. In both of these cases, publishing them before I'm ready to share the comments would be confusing or unclear.

Stuff that went poorly:

• I set instances.open to 0 on the 001 services and to 1 on the 002 services.
• I created a new instance and verified it came up on the 002 services.
• I verified that the test-on-002 instance still worked. Verified that a known 001-instance (meta) still works.

• I connected to the test instance, enabled Darkconsole, verified it was hitting the right DB, etc.
• I created, cloned, and pushed a repository.
• I ran host backup explicitly on both machines. This wanted device entries for the volumes for logging. I created them; this completed smoothly afterward.
• I verified that the backups appeared in the web UI log, that the repository was on the correct host, etc.

AppArmor uses file modification time heuristics to decide whether it can read a cache or not. This heuristic guesses the wrong result in the deployment case. I added -T to stop it from using the cache in rCORE8371c05b. Instance upgrade/sync operations now work correctly.

• I created Almanac service entries for repox002 and dbx002, following the 001 templates. Notably, repo bindings need a protocol specified. I didn't specify "instances.open" on these yet.
• I created a new instance and verified it came up on repox001 and dbx001 still, since the new services aren't open.
• I manually edited the instance to move it to the 002 services, then tried to upgrade the database. This failed.

• I created Almanac device entries for these hosts, following db001 and repo001. Both deploys went cleanly once the device entries were created.
• I used remote ssh <host> to connect to the devices and verified connectivity, volume attachment, deployed libraries, branches, that mysql works on db002, iptables rules, and that nothing appeared otherwise funky or broken. I didn't catch anything suspicious, except that remote deploy doesn't initialize or mount the backup volume: we do that when we actually perform backups. It probably should, just to catch misconfiguration/errors earlier. For now, I'll do this manually later on. I filed T8688 to follow up.

When creating device entries, I discovered AlmanacServiceQuery has a bug where name is not returned in the paging map. This prevents paging when an install has more than 100 service entries.

When a new host comes up with no instances, we don't handle it cleanly at the tail end of deployment.
This cleared MySQL deployment and just left us with the "no instances" issue on both hosts.

We don't actually install MySQL onto hosts in the db role.

• I launched repo002 and db002 into the cluster.
• I created .phacility.net DNS entries for the hosts.
• I created ddata002, dbak002, rdata002 and rbak002 volumes and attached them according to Adding Cluster Hardware.
• I used bin/remote deploy <host> to deploy each host. Both hosts deployed mostly cleanly.

I'm starting this now. This should ideally be routine, but I expect there will be a few hiccups.

For communication, my plan today would just be to use IRC/email/phonecalls since we have a small team, run operations mostly in the open, and are all regularly available on these other channels, and the likely course of action is "wait for AWS to come back up". I think that's generally reasonable until we have a larger operations team and more operational responses available, so I'm not particularly concerned about Conpherence/Maniphest on this instance being down.

I can't reproduce this, and haven't seen other reports of it. If this is a draft issue, it will likely be resolved by T5031.

I think this is fixed now, and additional failures should now be more obvious.

T8682 describes another recipe, if [body][does not contain]['magic word'], [block commit] to let you create warnings/prompts when committing.

This is a very niche use case. It also tends to conflate the pusher (who we may not know, for imported repositories), the committer, and the author. This makes me hesitant about implementing it.

Ahhh, some of the files in /var/repo/D/objects/ are owner by root, possibly because I ran sudo ./bin/repository import at one stage.

T8238#116697

Spaces are now available in master. They'll promote to stable in this week's release, and are likely to leave prototype next week.

What's worse is that I don't know what to do with this revision... it seems that I can't attach a diff to it by creating a new diff via arcanist (arc diff --preview) and then attaching it via the dropdown. I could remove it with ./bin/remove destroy, but that seems a bit aggressive.

I think the root cause of this issue is non-utf8 rawDiffs failing to encode when being passed to the publisher.

I've I'd be to work on a patch to add a config for this, if there any chances that this is going to be accepted ?

I'm tentatively going to do something in the vein of (2) to stop this from bleeding and pursue a more complete fix after cutting stable. We have a general issue with this code not making sense:

• D13435 should probably fix the root issue.
• D13436 should make these kinds of issues louder/more obvious.

This will return a list of tasks removed for a project within a certain time period, correct? If this can be done like this, this is exactly what is needed.

Several possible approaches spring to mind here:

I think this is a race between the HarbormasterBuildWorker queued by harbormaster.sendmessage, and the DifferentialTransactionEditor attaching the diff to the revision.

Er, not that one. T8650.

See T8670.

If there were a reflexive write of task transaction metadata to some project specific storage, (even some form of non-db log), this would be very useful, and much more elegant and smaller than having to query millions of Facts for the same information.

Maybe I'm still just not understanding what you need. You can get all the data you want today like this, right?