Page MenuHomePhabricator
Create Task
Maniphest T11180

Daemons fail to discover when a repository contains tags which tag other tags
Closed, ResolvedPublic
Actions

Description

Instance is at stable HEAD running a series of Git repositories self-hosted by Phabricator. Upon upgrade to week 25 stable release, daemon for repository reports:

Error updating working copy: No such commit 'XXX' in repository!

Upon performing git show XXX locally the commit does in fact have an existence in history but the remote within Phabricator cannot find it. Upon performing bin/repository reparse --all REPO --message a series of daemon tasks are now stuck in an unprocessed state for other missing commits as well. All signs point to branches that were eventually deleted.

Additional chat activity is at:

https://secure.phabricator.com/conpherence/1336/#27549

Revisions and Commits

rP Phabricator
D16149rP83121f78980b (stable) Handle tag tags properly in discovery
D16149rP9a2c2505a013 Handle tag tags properly in discovery

Related Objects

Event Timeline

mbishopim3 created this task.Jun 20 2016, 4:32 PM
mbishopim3 mentioned this in Z1336: General Chat.
epriestley added a subscriber: epriestley.Jun 20 2016, 4:58 PM

Can you show me the output of these commands?

phabricator/ $ ./bin/repository discover --verbose

And in the working copy for the repository (/var/repo/REPO) or similar:

/var/repo/REPO/ $ git for-each-ref

You can censor branch names if they're sensitive, but please leave ref prefixes (e.g., refs/heads/CENSOR-THIS-PART).

mbishopim3 added a comment.Jun 20 2016, 5:05 PM

The repo uses callsign P and refers to Platform.

First command:

Discovering "rP"...
Discovering commits in repository "rP".
Examining ref "develop", at "ce99dead0bd8642e55972452bd08955635b5a33f".
Looking for new commits.
Examining ref "master", at "326ba09c22ab7c7d2e5af849ce86320e5f7b2211".
Skipping, HEAD is known.
Examining ref "v3.3", at "313902a33fc0a97f2e06c666491e04dbace05974".
Skipping, HEAD is known.
Examining ref "v3.2", at "f5742fcceed7d8cd1357f9a8e5c789f4e192cd0d".
Skipping, HEAD is known.
Examining ref "v3.1.1", at "a591360919b77c9598357f4559d4140a842a6df2".
Skipping, HEAD is known.
Examining ref "v3.1.0", at "d830c5a93bb84d0f305180ab59067481446568af".
Skipping, HEAD is known.
Examining ref "v3.0.0", at "16ac928d07ed2272b55b67c3764aacd270750592".
Skipping, HEAD is known.
Examining ref "v2.13.0", at "ef385ae6e1816060ffed7f89fa1528897b762d56".
Skipping, HEAD is known.
Examining ref "v2.12.2", at "331f4ca12a6918a9b88243c66f036d482699fdf1".
Skipping, HEAD is known.
Examining ref "v2.12.1", at "2d50255047c8517561809d21d99e74f791685130".
Skipping, HEAD is known.
Examining ref "v2.12.0", at "8a7b5cd4273a8248f87161850dc57ea154349765".
Skipping, HEAD is known.
Examining ref "v2.11.3", at "f27f98a190bcea375689c8cc62221e8c28b70267".
Skipping, HEAD is known.
Examining ref "v2.11.2", at "4a2907603ee0a609c640b37bb21818f45e15ea41".
Skipping, HEAD is known.
Examining ref "v2.11.1", at "78daf8de9fe15dce0b4d0e1945a57c778e0536d7".
Looking for new commits.
[2016-06-20 17:02:25] EXCEPTION: (Exception) No such commit '78daf8de9fe15dce0b4d0e1945a57c778e0536d7' in repository! at [<phabricator>/src/applications/repository/daemon/PhabricatorGitGraphStream.php:85]
arcanist(head=stable, ref.master=b87138356a9c, ref.stable=18b27b03fa3d), phabricator(head=stable, ref.stable=59bc6adc0e22), phutil(head=stable, ref.master=c3fd3a8bb2c3, ref.stable=8aa8612a094b)

#0 PhabricatorGitGraphStream::parseUntil(string) called at [<phabricator>/src/applications/repository/daemon/PhabricatorGitGraphStream.php:36]
#1 PhabricatorGitGraphStream::getParents(string) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryDiscoveryEngine.php:374]
#2 PhabricatorRepositoryDiscoveryEngine::discoverStreamAncestry(PhabricatorGitGraphStream, string, boolean) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryDiscoveryEngine.php:177]
#3 PhabricatorRepositoryDiscoveryEngine::discoverGitCommits() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryDiscoveryEngine.php:77]
#4 PhabricatorRepositoryDiscoveryEngine::discoverCommitsWithLock() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryDiscoveryEngine.php:53]
#5 PhabricatorRepositoryDiscoveryEngine::discoverCommits() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementDiscoverWorkflow.php:49]
#6 PhabricatorRepositoryManagementDiscoverWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:410]
#7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:303]
#8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22]

And the second:

ce99dead0bd8642e55972452bd08955635b5a33f commit refs/heads/develop
326ba09c22ab7c7d2e5af849ce86320e5f7b2211 commit refs/heads/master
7ea0ad343687fd78d05b76e2468bdb56fc5315d9 tag refs/tags/v1.2
ad91a77e8f6e7cb3523ed2d55b69c5d68497f43c tag refs/tags/v1.3
01fe4f2f119092ff6a6c92d3c7518509abbb6159 tag refs/tags/v1.4
03c0c6061365159ac2cba7695ebf9ef365465e59 tag refs/tags/v1.5
4a0492efbd749dc5506f8ee20e5f2112d139a6f3 tag refs/tags/v2.0
603d7ebf12589ddeeb1e38eab64c056125a07594 tag refs/tags/v2.1
51519369aca1f61aeefe55521dfae62762d8eb90 tag refs/tags/v2.10
271a9758863a668e9a6515e576a33c3be0229046 tag refs/tags/v2.10.1
2f210f8cb2f67e9e74d3624b4e951f2de85f9277 tag refs/tags/v2.10.2
6d6ee6c4fc0f42a4d666098c1c358add62f67030 tag refs/tags/v2.11
a79b7864bf4da8b1fb06500dcd056fcea574edf2 tag refs/tags/v2.11.1
ca328eac8baf7d8eff026ef9f40aca9b34e82db4 tag refs/tags/v2.11.2
0e55e227f2d0b5da1c26473b8e4be9e7690f3237 tag refs/tags/v2.11.3
3c52380bc4b6a4eadfde8bf9022c863660fb9000 tag refs/tags/v2.12.0
f16d28f5f495d27810465dbcdf7e15ebcb729a5e tag refs/tags/v2.12.1
04e6220780788ef5a25098b5863b95d8ab3f06bb tag refs/tags/v2.12.2
65aac362f135434b00035ad729732a3b6d65ec4a tag refs/tags/v2.13.0
e741715b382beb40a14c702741ceda44491493ed tag refs/tags/v2.2
b8c972b47b97687b950853e7c1428e912364a140 tag refs/tags/v2.3
15365d00b006aceae8e873bdd06e7909f3bc279d tag refs/tags/v2.4
7cbe4889ba3dac37ada683f0c28c1c16af4f7261 tag refs/tags/v2.5
540a3fe06bc0870a18d9729d8fe11d368f6e1fda tag refs/tags/v2.6
328801397d77c99046088fa5ab921699a7367a3b tag refs/tags/v2.7
de712a532da58cef2b781122bf5d9e735b19ca8b tag refs/tags/v2.7.1
dc1296e6f82d2a21d4413856aff91b8aa76f331a tag refs/tags/v2.7.2
f9ade410c32ab31a4b5dcf02e13771359bbbc0ea tag refs/tags/v2.8
c41e2429c0cb7167e425b0ce0d2d9185c48f79d1 tag refs/tags/v2.8.2
cf73e9bf2cf9db32a5eb35f7b41fb289b620261d tag refs/tags/v2.8.3
be04c7eb30f06b23000e445f509ce40821f2573d tag refs/tags/v2.9
a12da7ca86c0cdadc1409dac9fc487040c6004d6 tag refs/tags/v3.0.0
15f5d30fa8d4c1cdb089a8f1949cae7ac54341a4 tag refs/tags/v3.1.0
14853263cc0fc8310a7293855cc5077f7f50f863 tag refs/tags/v3.1.1
811d6fde5ff729b38b4ba1fd201b84c8a6bd4c1f tag refs/tags/v3.2
f18a314433d691c7a32759880987d0281380ef21 tag refs/tags/v3.3

epriestley added a comment.Jun 20 2016, 5:25 PM

Hrrm, I'm very confused by that behavior.

bin/repository discover should pretty much be running git for-each-ref, then examining each ref. However, it looks like it's getting a completely different result for refs/tags/v2.11.1:

Examining ref "v2.11.1", at "78daf8de9fe15dce0b4d0e1945a57c778e0536d7".
a79b7864bf4da8b1fb06500dcd056fcea574edf2 tag refs/tags/v2.11.1

Can you try this? Run:

phabricator/ $ ./bin/repository discover --trace --verbose rP

Find the git for-each-ref command it issues, which should look something like this and appear above all the "Examining ref..." stuff:

...
<<< [17] <query> 122 us
>>> [18] <exec> $ git for-each-ref --sort='-creatordate' --format='%(objectname)%01%(objecttype)%01%(refname)%01%(*objectname)%01%(*objecttype)%01%(subject)%01%(creator)' -- 'refs/'
<<< [18] <exec> 48,298 us
...

Then run that command in /var/repo/P/ (or wherever) and show me the output?

epriestley added a comment.Jun 20 2016, 5:26 PM

(I suspect v2.11.1 is a tag-object and the tag-hash and object-hash are getting mixed up somehow, but hopefully the full command will shed some light on things.)

mbishopim3 added a comment.Jun 20 2016, 5:31 PM

Had to redact a bit:

ce99dead0bd8642e55972452bd08955635b5a33fcommitrefs/heads/developIsolate production cache, simplify key namesTest User <test@test.com> 1466440502 -0400
326ba09c22ab7c7d2e5af849ce86320e5f7b2211commitrefs/heads/mastermerge developTest User <test@test.com> 1465220341 -0400
f18a314433d691c7a32759880987d0281380ef21tagrefs/tags/v3.3313902a33fc0a97f2e06c666491e04dbace05974committagging v3.3Test User <test@test.com> 1465219400 -0400
811d6fde5ff729b38b4ba1fd201b84c8a6bd4c1ftagrefs/tags/v3.2f5742fcceed7d8cd1357f9a8e5c789f4e192cd0dcommitTagging v3.2Test User <test@test.com> 1447940045 -0500
14853263cc0fc8310a7293855cc5077f7f50f863tagrefs/tags/v3.1.1a591360919b77c9598357f4559d4140a842a6df2committagging v3.1.1Test User <test@test.com> 1444219883 -0400
15f5d30fa8d4c1cdb089a8f1949cae7ac54341a4tagrefs/tags/v3.1.0d830c5a93bb84d0f305180ab59067481446568afcommittagging v3.1.0Test User <test@test.com> 1438866825 -0400
a12da7ca86c0cdadc1409dac9fc487040c6004d6tagrefs/tags/v3.0.016ac928d07ed2272b55b67c3764aacd270750592committagging v3.0.0Test User <test@test.com> 1436439081 -0400
65aac362f135434b00035ad729732a3b6d65ec4atagrefs/tags/v2.13.0ef385ae6e1816060ffed7f89fa1528897b762d56committagging v2.13.0Test User <test@test.com> 1434022899 -0400
04e6220780788ef5a25098b5863b95d8ab3f06bbtagrefs/tags/v2.12.2331f4ca12a6918a9b88243c66f036d482699fdf1committagging v2.12.2Test User <test@test.com> 1428403958 -0400
f16d28f5f495d27810465dbcdf7e15ebcb729a5etagrefs/tags/v2.12.12d50255047c8517561809d21d99e74f791685130committagging v2.12.1Test User <test@test.com> 1426593614 -0400
3c52380bc4b6a4eadfde8bf9022c863660fb9000tagrefs/tags/v2.12.08a7b5cd4273a8248f87161850dc57ea154349765committagging v2.12.0Test User <test@test.com> 1424438124 -0500
0e55e227f2d0b5da1c26473b8e4be9e7690f3237tagrefs/tags/v2.11.3f27f98a190bcea375689c8cc62221e8c28b70267committagging v2.11.3Test User <test@test.com> 1418323305 -0500
ca328eac8baf7d8eff026ef9f40aca9b34e82db4tagrefs/tags/v2.11.24a2907603ee0a609c640b37bb21818f45e15ea41committagging v2.11.2Test User <test@test.com> 1415798839 -0500
a79b7864bf4da8b1fb06500dcd056fcea574edf2tagrefs/tags/v2.11.178daf8de9fe15dce0b4d0e1945a57c778e0536d7tagtagging v2.11.1Test User <test@test.com> 1413916998 -0400
6d6ee6c4fc0f42a4d666098c1c358add62f67030tagrefs/tags/v2.1120f7cbd7e12595269f0db97977e55c4d1b00ff06committagging v2.11Test User <test@test.com> 1413387450 -0400
2f210f8cb2f67e9e74d3624b4e951f2de85f9277tagrefs/tags/v2.10.25cb7994a7b724ddbf219bd358137ba074fe2d3e1committagging v2.10.2Test User <test@test.com> 1409839386 -0400
271a9758863a668e9a6515e576a33c3be0229046tagrefs/tags/v2.10.1814be0cc8db8a1341f50c070d005483aec062311committagging v2.10.1Test User <test@test.com> 1406832927 -0400
51519369aca1f61aeefe55521dfae62762d8eb90tagrefs/tags/v2.106fd9559d1a6704c89d619501d5bfb21d05dc3383committagging v2.10Test User <test@test.com> 1406127182 -0400
be04c7eb30f06b23000e445f509ce40821f2573dtagrefs/tags/v2.9121a1d01e22ac71977ce22b4556da894cd151680committagging v2.9Test User <test@test.com> 1404913995 -0400
cf73e9bf2cf9db32a5eb35f7b41fb289b620261dtagrefs/tags/v2.8.357c11946c28389f9e7667c1254f0d745d58c312ecommittagging v2.8.3Test User <test@test.com> 1401368975 -0400
c41e2429c0cb7167e425b0ce0d2d9185c48f79d1tagrefs/tags/v2.8.25c930948440dcafd9b6fd59c6322e4ee23e2698dtagtagging v2.8.2Test User <test@test.com> 1400075291 -0400
f9ade410c32ab31a4b5dcf02e13771359bbbc0eatagrefs/tags/v2.89c11e2cb9ad83720aec690776a143abbe62ec0dccommittagging v2.8Test User <test@test.com> 1399318216 -0400
dc1296e6f82d2a21d4413856aff91b8aa76f331atagrefs/tags/v2.7.2e010dab75fcbf509e48f07f1d04eef35170ca56ccommittagging v2.7.2Test User <test@test.com> 1396382197 -0400
de712a532da58cef2b781122bf5d9e735b19ca8btagrefs/tags/v2.7.144bb2dd0842143222ff1f2d3c42cb2908d88fbbccommittagging v2.7.1Test User <test@test.com> 1394033658 -0500
328801397d77c99046088fa5ab921699a7367a3btagrefs/tags/v2.73c25c2a1e35bde37ec961b58e0682515fb162a74tagtagging v2.7Test User <test@test.com> 1393256004 -0500
540a3fe06bc0870a18d9729d8fe11d368f6e1fdatagrefs/tags/v2.6ce05dd9e0864409d88a3d374a82d79e10e4c5e44committagging v2.6Test User <test@test.com> 1392039113 -0500
7cbe4889ba3dac37ada683f0c28c1c16af4f7261tagrefs/tags/v2.57763ba22f0ca32372065df7ae76a1db408c852e8committagging v2.5Test User <test@test.com> 1389206896 -0500
15365d00b006aceae8e873bdd06e7909f3bc279dtagrefs/tags/v2.411919b6008a8498e40b890aaa21c72e63974425ccommittagging v2.4Test User <test@test.com> 1386967704 -0500
b8c972b47b97687b950853e7c1428e912364a140tagrefs/tags/v2.3bfe9c2ed086e33ae2cda6e65864607c06736ef4acommittagging v2.3Test User <test@test.com> 1385046814 -0500
7ea0ad343687fd78d05b76e2468bdb56fc5315d9tagrefs/tags/v1.2d2ef04697eef216abcd3496b3edd02e6d0858dfdcommittagging v1.2Test User <test@test.com> 1381926316 -0400
e741715b382beb40a14c702741ceda44491493edtagrefs/tags/v2.29ad588a6e2998f243800c772b3ffd93f83f26059committagging v2.2Test User <test@test.com> 1381926130 -0400
603d7ebf12589ddeeb1e38eab64c056125a07594tagrefs/tags/v2.1c9511e46bbb3298934bb049405d7c2fe4f2c4da7committagging v2.1Test User <test@test.com> 1381422582 -0400
4a0492efbd749dc5506f8ee20e5f2112d139a6f3tagrefs/tags/v2.04ed98b7ad3b1c008d479cf66afd120ae1b0716e2committagging v2.0Test User <test@test.com> 1373982255 -0400
03c0c6061365159ac2cba7695ebf9ef365465e59tagrefs/tags/v1.5b3dfb0aa4c093c69f9214dc71a3e5911e6c5528acommittagging v1.5Test User <test@test.com> 1373982206 -0400
01fe4f2f119092ff6a6c92d3c7518509abbb6159tagrefs/tags/v1.4f1b6a4086bd6fef6636da1ed0e323409115687e6committagging v1.4Test User <test@test.com> 1373982154 -0400
ad91a77e8f6e7cb3523ed2d55b69c5d68497f43ctagrefs/tags/v1.3e8ce63e74f9fb92d292d6c8e1293d1236605a3dbcommittagging v1.3Test User <test@test.com> 1373982102 -0400

mbishopim3 added a comment.Jun 20 2016, 5:33 PM

If it requires us to delete the v2.11.1 tag then so be it, we understand that maybe we did something wild with history in the past but that tag doesn't really give us anything anymore so it can go.

epriestley added a comment.Jun 20 2016, 5:37 PM

We should be able to parse any repository in any state, so don't delete the tag!

Things do look less crazy now, since this at least shows where both the hashes (a79b7... and 78daf...) came from:

a79b7864bf4da8b1fb06500dcd056fcea574edf2tagrefs/tags/v2.11.178daf8de9fe15dce0b4d0e1945a57c778e0536d7tagtagging v2.11.1Test User <test@test.com> 1413916998 -0400

Can you show me these? You can redact most of them if there's anything sensitive:

git cat-file -p a79b7864bf4da8b1fb06500dcd056fcea574edf2
git cat-file -p 78daf8de9fe15dce0b4d0e1945a57c778e0536d7
git log --format=%H 78daf8de9fe15dce0b4d0e1945a57c778e0536d7
    ^^^ only need the first few lines of that one if it outputs a ton of stuff
mbishopim3 added a comment.Jun 20 2016, 5:41 PM

First:

object 78daf8de9fe15dce0b4d0e1945a57c778e0536d7
type tag
tag v2.11.1
tagger Test User <test@test.com> 1413916998 -0400

tagging v2.11.1

Second:

object ad3da3a15b7db907f62a9dec2ffbd339c8ff083a
type commit
tag v2.11.2
tagger Test User <test@test.com> 1413916438 -0400

tagging v2.11.2

Third:

ad3da3a15b7db907f62a9dec2ffbd339c8ff083a
f60e71a190c038ec47969adf3c17c6d712d81d7c
4266d24b468189757d263e04637670330b351536
69654b60777509b8b88a79de6a639ccabd8c1d9c
76abff6d8d6fc47a1e773e2c94f4dcb000813fad
1a1b1c424fbaa3a12c9e32bfc995cd833f9f3eaf
ae211160e5ee7a9076c7c7cb7595fdc7a3cc4dea
856f0dd64ec2e003d49e67ff51e7392d3f9df87c

epriestley added a comment.Jun 20 2016, 5:43 PM

Thanks! Looks like we have things flipped for some reason. Let me see if if I can track that down. I think I should have a patch in an hour or so, or I'll let you know if I need anything else.

epriestley added a subscriber: 20after4.Jun 20 2016, 6:00 PM

Okay, I figured this out:

... tag tagging v2.11.1 ...

You've tagged a tag. That is, you have Tag v2.11.1 → Tag v2.11.2 → Commit ad3da3a....

I'll see if I can make Phabricator ignore these.

@20after4, this is likely what you were seeing too.

I had zero clue you could do this and didn't anticipate it when writing the logic.

chad added a subscriber: chad.Jun 20 2016, 6:05 PM

O.o

mbishopim3 added a comment.Jun 20 2016, 6:06 PM

This is in all likelihood a mistake made forever ago but I'll wait for a potential patch.

epriestley added a revision: D16149: Handle tag tags properly in discovery.Jun 20 2016, 6:08 PM
epriestley renamed this task from Daemons cannot recover from missing commits to Daemons fail to discover when a repository contains tags which tag other tags.Jun 20 2016, 6:09 PM
epriestley claimed this task.
epriestley triaged this task as Normal priority.
epriestley added a project: Diffusion.

Thanks for all the help digging into this, there is pretty much zero chance I would have guessed that on my own. I think D16149 should fix this properly, I was able to reproduce a similar error once I knew what I was after.

epriestley closed this task as Resolved by committing rP9a2c2505a013: Handle tag tags properly in discovery.Jun 20 2016, 6:10 PM
epriestley added a commit: rP9a2c2505a013: Handle tag tags properly in discovery.
epriestley added a comment.Jun 20 2016, 6:13 PM

Let me know if that doesn't fix things. Expected behavior is:

  • bin/repository discover --verbose now works, emitting a relevant message ("skipping tag of tag") instead of failing.
  • Daemons work normally.
epriestley awarded a token.Jun 20 2016, 6:13 PM
mbishopim3 added a comment.Jun 20 2016, 6:25 PM

Thanks, this is working now.

mbishopim3 mentioned this in T11181: Old missing commits from deleted branches stuck as queued tasks.Jun 20 2016, 6:35 PM
epriestley added a commit: rP83121f78980b: (stable) Handle tag tags properly in discovery.Jun 20 2016, 6:36 PM
avivey mentioned this in T11301: git tags can tag anything, break daemons.Jul 8 2016, 8:47 PM
Phabricator · Built by Phacility