See PHI497. An instance ran into an issue where the remote was complaining about too many unreachable objects:
remote: warning: There are too many unreachable loose objects; run 'git prune' to remove them.
We don't currently run git prune or git gc automatically since it has never caused issues before, but should probably start running git gc on working copies every so often.
One possible issue with this is that git gc can take a very long time to complete on large working copies. Previously, in PHI386, an unusually large repository for the same instance took about 5 hours to git gc. However, it took less than a minute to git prune, so maybe git prune is more safe to run regularly.
A possible workaround is to run these operations as part of bin/remote optimize, or some similar workflow, in the cluster, only.