Since I typed up most of this anyway:

We're reading this from $_SERVER['REMOTE_ADDR'], which I believe is never populated with a list of addresses.

I would guess that you're loading X-Forwarded-For into REMOTE_ADDR directly, probably in support/preamble.php. You probably have a line of code like this:

$_SERVER['REMOTE_ADDR'] = $_SERVER['HTTP_X_FORWARDED_FOR'];

(We currently suggest this in Configuring a Preamble Script, but should probably be more surgical about it. This is normally fine, but not if you send requests through several load balancers; this can occur if you put a CDN in front of your web load balancer.)

If this is present, you can use a block like this to load only the pre-LB IP:

if (isset($_SERVER['HTTP_X_FORWARDED_FOR'])) {
  $forwarded_for = $_SERVER['HTTP_X_FORWARDED_FOR'];
  $forwarded_for = explode(',', $forwarded_for);
  $forwarded_for = end($forwarded_for);
  $forwarded_for = trim($forwarded_for);
  $_SERVER['REMOTE_ADDR'] = $forwarded_for;
}

This will trust exactly one level of forwarding. If multiple levels of the header are trustworthy, you may need to add additional logic to strip off trusted addresses.

We may also move to trusting this header automatically if the request came from a recognized address in cluster list -- we currently trust cluster requests slightly more on these counts, but do not currently trust the headers per se. (The documentation is actually slightly incorrect here, and suggests we trust them strictly; we currently do not, and only relax requirements for HTTPS when receiving a request from a cluster device.)

Thanks! That is exactly what we had, and I was not aware of it. And yes, we have this on purpose because of there being a load balancing (trusted) proxy in front of Phabricator. For whatever reason, I assumed Phabricator had it's own X-Forwarded-For handling. Mainly my mistake here was jumping to the assumption that Phabricator was looking at X-Forwarded-For based on the value - even though the error message never indicated that it was.