I think the root of this is largely building a commit mapping from "draft" commits to their equivalent "published" commits. I believe all of the approaches available to us are:

This install now exposes two separate actions in a new submenu, although the backend is only sort of halfway converted. My plan is:

In T3025#181309, @jayvdb wrote:

I saved "Ignore Conflict" here and on phabricator.wikimedia.org , but a few hours later the warning message overlay has reappeared. Im going to ignore it here, so that the devs can see when I last saved "Ignore Conflict".

In T11085#181202, @scode wrote:

@eadler Did you have enough insight/recollection as to whether all our reported cases were due to users editing in the web UI?

I'd like to have a clearer sense of how we're planning to resolve T10034 before moving forward here, even if we don't actually build any of it yet. Briefly, I think some of the problems here are:

I saved "Ignore Conflict" here and on phabricator.wikimedia.org , but a few hours later the warning message overlay has reappeared. Im going to ignore it here, so that the devs can see when I last saved "Ignore Conflict".

Hit by this again on a trip to a nearby country. Can the "Ignore Conflict" option be moved out of the list so the user doesnt have to click "Save Timezone" to in fact register they want no change. I instinctively want to hit the Cancel button on this popup/overlay, as I dont want to change my timezone. I just want the message to stop.

In T4631#181162, @epriestley wrote:

you'll need to figure out how to get users to upgrade arc on your own until T5055.

I'll comment on the remainder later.

We discussed this briefly out-of-band, but the draft part of this is probably only sensible as "soft" prioritization:

• Does that description sound approximately like what you're hitting?
• Are both smarter arc land and arc cascade interesting, or are you only running into pain in one of the cases?
• Do you anticipate cases where git reflog and local notes could fail as sources (lots of pushing/pulling to different working copies, I guess)?
• Anyone know offhand how to git rebase -i master; delete the first few lines; save and exit; non-interactively?

I think the immediate issue here tends to manifest as arc land requiring users to manually rebase before it will work.

@eadler Did you have enough insight/recollection as to whether all our reported cases were due to users editing in the web UI?

Here are the interactions I can come up with which we'll potentially get wrong by implementing this naively:

From a technical point of view I think the ideal solution to this is to make the commit message format unambiguously parseable, but I'm not immediately sure how to do that without making other things worse. It's valuable for messages to be human-readable and human-editable, and I think we'd be worse overall if we tried to escape or encode messages in Git itself (or represent them as XML or whatever).

I'm not sure I have a good answer to that. :/

What's the behavior you'd expect or prefer when a user writes this commit message?

My understanding is that the root issue here is that you're adding reviewers via Herald/Owners (per T10939) so the "no reviewers" warning is inaccurate or misleading in many cases because owners will be added automatically after the revision is created.

This spawned a healthy set of followups, but I believe the core work described here is now resolved and accounted for and all future work is covered in narrower followup tasks.

Appears resolved and accounted for.

I believe this is resolved and accounted for, let us know if anyone runs into issues.

This appears to be resolved and accounted for. There's one followup in T11120 (8bit vs base64 mail encoding) which also might be fixed now. If anyone runs into issues, let us know.

I believe this is now resolved and accounted for. See the remarkup documentation for some details.

I believe this is resolved and accounted for, let us know if anyone is hitting issues.

I believe this is now fixed and all the business stuff is wrapped up on our side. If anyone runs into issues, feel free to file a followup describing what you're hitting.

FWIW, examining every ref like that is a complete no-go for our situation

OK, so I just manually removed the branch from info/refs and packed-refs and I think that maybe that worked?

I think D16136 will "fix" the problem and I can't come up with any problems it creates, although it would be nice to figure out where the mystery ref is coming from.

okay so basically ghosts?

In T9028#180932, @joshuaspence wrote:

Ah interesting. From our GitLab host, I see that git branch --all shows a whole bunch of local branches and only a single branch of the form remotes/origin/url_shortener.

Ah interesting. From our GitLab host, I see that git branch --all shows a whole bunch of local branches and only a single branch of the form remotes/origin/url_shortener.

I think checking if git ls-remote origin in the working copy lists a refs/remotes/origin/XYZ ref in the authoritative remote is the easiest way to check, although I'm not 100% sure that refs/remotes/ isn't magic with respect to ls-remote.

I wonder if the actual authoritative repository has a refs/remotes/origin/XYZ ref which we're mirroring?

We fetch bare working copies with git fetch origin +refs/*:refs/* --prune.

In T9028#180926, @epriestley wrote:

Hrrm -- is the repository a non-bare working copy (e.g., has files and folders in it instead of just config/, hooks/, info/, etc)?

Is there a lack of a git remote prune origin or similar?

Hrrm -- is the repository a non-bare working copy (e.g., has files and folders in it instead of just config/, hooks/, info/, etc)?

OK, running ./bin/repository discover --verbose rX gives me a bunch of output including this:

We have Diffusion configured to track only the master branch, but now Diffusion is importing commits from all branches, it seems.

Updated our Phabricator install just now. One issue I discovered is that we are having a bunch of audits triggered for commits that are newly discovered. We have Diffusion configured to track only the master branch, but now Diffusion is importing commits from all branches, it seems.

Yeah. I just wanted to make sure whether mark-reachable was safe to run without fear of accidentally marking something unreachable that isn't (i.e., if the command were to imply "mark everything in the import queue as unreachable no matter what"). Sorry it was probably a bit of a lazy question since a more careful re-reading of the rest of the ticket would probably have answered it for me.

It basically runs git log --all, then checks every commit in the database against that list and marks the missing ones as unreachable.

If the mark-reachable command is used, is it subject to races in the sense that in-flight legit newly importing commits may be marked unreachable, or does it just trigger the process of going back and re-evaluating true reachability of all commits?

I've pushed this here, and enabled dangerous changes on rGITTEST if anyone wants to try anything.

I'm vaguely suspicious that I may have missed some weird stuff here since this pipeline is complicated, but I'm going to land this stuff now and just plan to hold the release a bit if it feels shaky tomorrow.

I've deployed this and configured this server to use AES-256-CBC, although I haven't retroactively migrated existing data.

We can't reproduce this and haven't seen other reports of similar issues (including from other installs of greater size and complexity), so we have no way to move forward on this.

To revive deleted commits when they are re-pushed, I think it is sufficient to ignore deleted commits when filling the discovery cache (PhabricatorRepositoryDiscoveryEngine->fillCommitCache()) and allow recordCommit() to detect existing commits and revive them.

Current state of the world after D16108 is:

Thanks for testing, that's quite helpful.

In T8510#179818, @epriestley wrote:

D16094 should improve things, let me know if you identify specific problems after deploying that. It's deployed on this server now, although we have fewer similarly-named projects so it's harder to see the effects.

I'm planning to do this:

Confirmed {rP2e450212503e80e07c55821d5ba68d6370e98d04} fix the bug. Thanks!