Hit a couple of bumps when deploying master ahead of the release cut. Here's what happened.

First, 20180124.herald.01.repetition.sql failed to apply:

[secure002] COMMAND
[secure002] /core/lib/phabricator/bin/storage upgrade --force --no-adjust
[secure002] 
[secure002] STDOUT
[secure002] Applying patch "phabricator:20180124.herald.01.repetition.sql" to host "secure001.phacility.net:3306"...
[secure002] 
[secure002] 
[secure002] STDERR
[secure002] [2018-01-26 21:13:10] EXCEPTION: (AphrontQueryException) #1265: Data truncated for column 'repetitionPolicy' at row 1 at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:355]
...

The issue here is that we had two NULL values in our herald_rule table, in the two very oldest rules from 2011 (or earlier). This is normally not possible and I didn't have any in my test data, which is how I missed this.

When we convert a NULL int to a non-nullable string, MySQL complains in strict mode, apparently. I thought it would just silently discard the NULL but either misremembered or MySQL got a little more strict or something.

I fixed this with rPfd49acd0 which just sets the column to 1 for values where it is NULL before running the migration. That got things to work. Later, I spot-checked the table and it looks like that worked fine.

Then, I hit this:

[secure001] # Synchronization of "secure001.phacility.net" from leader "secure002.phacility.net" failed: Command failed with error #128!
[secure001] COMMAND
[secure001] '/usr/bin/sudo' -E -n -u 'ubuntu' -- git fetch --prune -- 'ssh://172.30.1.91:22/source/phabricator/' '+refs/*:refs/*'
[secure001] 
[secure001] STDOUT
[secure001] (empty)
[secure001] 
[secure001] STDERR
[secure001] fatal: ssh variant 'simple' does not support setting port

This is adjacent to https://discourse.phabricator-community.org/t/issues-with-git-2-16-0/1004/2. I fixed it with rP7e772080, by setting GIT_SSH_VARIANT explicitly.

My understanding of this is that git tries to automatically detect what GIT_SSH is. Since we set it to bin/ssh-exec, Git isn't sure if that's SSH or not and defaults to simple.

This feature doesn't seem to be new, so it's not clear what part of this behavior changed:

• Did it previously detect it as ssh?
• Did it previously not care about ports for simple?
• Did something else happen?

The server picked up Git 2.16.1 (we're on an accelerated release channel because we needed a security bugfix in the past) so this can probably be figured out by examining the 2.16.0 release notes.

In any case, telling it explicitly with GIT_SSH_VARIANT seems correct.