Thanks for looking into this further.

Looks like that fix affects Differential, but not Diffusion? After reading through that bug, I can totally reproduce this on demand in Audit
I have a test instance here that reproduces this problem (both actively able to reproduce and has commits in the database that are exhibiting the behavior already): https://test-kdauzhhtarbi.phacility.com/rLIBdde2f74f2793f0216f0d76618ed335a9c802cfec, but I reproduced it reliably on 3 separate instances now.

Sounds similar to T11092, which was recently resolved.

I'm unable to reproduce on a test install right now, but then again, I stopped reproducing on my install after rebooting the server. It looks like something happens eventually to cause this to break and I start getting these broken comments.
Looks like every time I get a broken one it doesn't have a transaction PHID associated with it and all comments submitted together have the same wrong associated commitPHID.

oh neat!

You can use a free instance at http://phacility.com/ if you need a clean, up to date place to test privately.

This is, by the way, on a brand new install that's not even a month old.
I tried to reproduce it on another install that is older and I'm not having any luck.

This actually is a lot worse than I originally thought.
All inline comments that are made on an audit are actually applied to the audit I was previously commenting on (like an off by 1 error). I've isolated the following things:

• This happens across repositories (i.e. if I have multiple repositories, commenting on one commit and then switching to another repository's commit and commenting on it will produce a comment on the first commit)
• This happens for all users
• This happens regardless of audit status - i.e. commits with audit requested exhibit this as much as commits that don't require audit.
• Submitting an empty comment first thing before adding any content to a commit works around this problem - subsequently added commits appear as expected.
• The broken state can be easily detected by unavailability of the "hover state" of the overall commit field - pressing "z" does nothing

This appears to be stable, and is now accounted for.

Bug reports MUST include reproduction steps that allow us to reproduce the issue locally. Without this information, we can not move forward.

(I don't know if this is the exact same bug, but we'd fix them both in the same pass)

Nice! Thanks.

I don't know offhand, See T10978, we have a lot of cruft in Audit that just needs tossing out.

Never noticed that before? Is this specific for Audit?

FWIW, examining every ref like that is a complete no-go for our situation

OK, so I just manually removed the branch from info/refs and packed-refs and I think that maybe that worked?

I think D16136 will "fix" the problem and I can't come up with any problems it creates, although it would be nice to figure out where the mystery ref is coming from.

okay so basically ghosts?

In T9028#180932, @joshuaspence wrote:

Ah interesting. From our GitLab host, I see that git branch --all shows a whole bunch of local branches and only a single branch of the form remotes/origin/url_shortener.

Ah interesting. From our GitLab host, I see that git branch --all shows a whole bunch of local branches and only a single branch of the form remotes/origin/url_shortener.

I think checking if git ls-remote origin in the working copy lists a refs/remotes/origin/XYZ ref in the authoritative remote is the easiest way to check, although I'm not 100% sure that refs/remotes/ isn't magic with respect to ls-remote.

I wonder if the actual authoritative repository has a refs/remotes/origin/XYZ ref which we're mirroring?

We fetch bare working copies with git fetch origin +refs/*:refs/* --prune.

In T9028#180926, @epriestley wrote:

Hrrm -- is the repository a non-bare working copy (e.g., has files and folders in it instead of just config/, hooks/, info/, etc)?

Is there a lack of a git remote prune origin or similar?

Hrrm -- is the repository a non-bare working copy (e.g., has files and folders in it instead of just config/, hooks/, info/, etc)?

OK, running ./bin/repository discover --verbose rX gives me a bunch of output including this:

We have Diffusion configured to track only the master branch, but now Diffusion is importing commits from all branches, it seems.

Updated our Phabricator install just now. One issue I discovered is that we are having a bunch of audits triggered for commits that are newly discovered. We have Diffusion configured to track only the master branch, but now Diffusion is importing commits from all branches, it seems.

Yeah. I just wanted to make sure whether mark-reachable was safe to run without fear of accidentally marking something unreachable that isn't (i.e., if the command were to imply "mark everything in the import queue as unreachable no matter what"). Sorry it was probably a bit of a lazy question since a more careful re-reading of the rest of the ticket would probably have answered it for me.

It basically runs git log --all, then checks every commit in the database against that list and marks the missing ones as unreachable.

If the mark-reachable command is used, is it subject to races in the sense that in-flight legit newly importing commits may be marked unreachable, or does it just trigger the process of going back and re-evaluating true reachability of all commits?

I've pushed this here, and enabled dangerous changes on rGITTEST if anyone wants to try anything.

I'm vaguely suspicious that I may have missed some weird stuff here since this pipeline is complicated, but I'm going to land this stuff now and just plan to hold the release a bit if it feels shaky tomorrow.

To revive deleted commits when they are re-pushed, I think it is sufficient to ignore deleted commits when filling the discovery cache (PhabricatorRepositoryDiscoveryEngine->fillCommitCache()) and allow recordCommit() to detect existing commits and revive them.

On second thoughts, the flexibility of a Herald rule may already be better for me, as I want to filter out Merge Commits from auditing (at least when they are merged by an Owner), since the individual commits that are merged already have their own audit.

I know you have focussed on the missing background, but I'm not overly fussed about that. But I am surprised that the commit subject shown twice in a row, once truncated inside a table and once not, is anything else but a bug.

I know I didn't retrieve all the info you asked for, so let me know if you want to see anything else or want me to decode the base64 email body.

I've learned some interesting things.

Can you provide:

Yes, that would also solve the problem for me, and would probably be an easier option for new users.

If the "Audit" setting for packages had three settings instead, would that solve your use case?

https://secure.phabricator.com/book/phabricator/article/support/

I created the task, then changed the tags manually (selecting only the tags I felt necessary - also removing the BUG tag). Seems I may have found a bug with the bug tracker. ;)

That is, if you go to the Diffusion workboard, we give you a generic create form instead of a choice between the various normal create forms, currently.

You can file stuff without those tags from workboards, and maybe "Support Request" just got caught in the crossfire of typing every possible tag, so that's a legitimate-albeit-unusual pathway here. But I wouldn't expect the short-lived "New Support Request" form to be accessible any longer.

This tracker is for bug reports and feature requests, and we're surprised you were able to file something without either of those tags.

How did you file this task?

If a package is defined with multiple owners (or a group ownership), it is a reasonable expectation that even when an owner commits a change, other owners will review that change. This is how owners has worked up until the T10939 changes recently.

A sub-problem here is that if daemons discover commits but those commits are deleted before they can be processed, the tasks fail forever. This is normally unusual (git won't GC the commits for a few weeks, which often gives the daemons enough time to process them) but cases like fiddling with tracking, re-cloning a repository, everyone going on vacation for a couple weeks, or clustering an observed repository can exacerbate them.

Can you list out a complete set of reproducible steps to follow? Is this a Dashboard you're building and installing on other people homepages, for example?