Page MenuHomePhabricator
Create Task
Maniphest T6922

Diffusion git repository import stuck at 99.99%
Closed, ResolvedPublic
Actions

Description

We are noticing that on two separate repositories after all tasks have been processed during the initial import, the repository is stuck at 99.99% importing.

After doing some debugging, we found that on both repositories, all stuck commits are reverts of previous commits. And attempting to reparse them results in an exception (see log below)

The smaller of the two repositories is ~1000 commits, and there appears to be a single stuck commit. Relevant debugging from that repository below...

P1700 Commands from stuck repository import
1(names and emails have been redacted out)
2
3Ok, what's stuck?
4
5$ ./bin/repository importing MYREPO
6rMYREPOf628ef8ca552a0c9ca6d1736576b45e7bbc635f5 Herald
7
8Ok, let's reprocess that thing then
9
10$ ./bin/repository reparse rMYREPOf628ef8ca552a0c9ca6d1736576b45e7bbc635f5 --herald --trace
11>>> [2] <connect> phabricator_repository
12<<< [2] <connect> 1,106 us
13>>> [3] <query> SELECT * FROM `repository` WHERE callsign = 'MYREPO'
14<<< [3] <query> 383 us
15>>> [4] <query> SELECT * FROM `repository_commit` WHERE repositoryID = 5 AND commitIdentifier = 'f628ef8ca552a0c9ca6d1736576b45e7bbc635f5'
16<<< [4] <query> 334 us
17>>> [5] <query> SELECT * FROM `repository_commit` WHERE `id` = 64098
18<<< [5] <query> 234 us
19>>> [6] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
20<<< [6] <query> 235 us
21>>> [7] <query> SELECT commit.* FROM `repository_commit` commit WHERE (commit.id IN (64098)) ORDER BY commit.id DESC
22<<< [7] <query> 227 us
23>>> [8] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
24<<< [8] <query> 213 us
25>>> [9] <query> SELECT * FROM `repository_commitdata` WHERE commitID in (64098)
26<<< [9] <query> 212 us
27>>> [10] <query> SELECT * FROM `repository_auditrequest` WHERE commitPHID IN ('PHID-CMIT-njbauecpnouwc3rwesnr')
28<<< [10] <query> 201 us
29>>> [11] <query> SELECT commit.* FROM `repository_commit` commit WHERE (((commit.commitIdentifier LIKE '4edd28f6008c14051421559a00aa61a41972e65e%' AND LENGTH(commit.commitIdentifier) = 40))) ORDER BY commit.id DESC
30<<< [11] <query> 306 us
31>>> [12] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
32<<< [12] <query> 217 us
33>>> [13] <exec> $ git diff -M -C --no-ext-diff --no-color --src-prefix=a/ --dst-prefix=b/ -U3 'f628ef8ca552a0c9ca6d1736576b45e7bbc635f5^' 'f628ef8ca552a0c9ca6d1736576b45e7bbc635f5' -- '.'
34<<< [13] <exec> 18,819 us
35>>> [14] <query> SELECT edge.* FROM `edge` edge WHERE (edge.src IN ('PHID-CMIT-njbauecpnouwc3rwesnr')) AND (edge.type IN ('21')) ORDER BY edge.dateCreated DESC, edge.seq DESC
36<<< [14] <query> 324 us
37>>> [15] <query> SELECT commit.* FROM `repository_commit` commit WHERE (((commit.commitIdentifier LIKE '4edd28f6008c14051421559a00aa61a41972e65e%' AND LENGTH(commit.commitIdentifier) = 40))) ORDER BY commit.id DESC
38<<< [15] <query> 353 us
39>>> [16] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
40<<< [16] <query> 223 us
41>>> [17] <query> SELECT commit.* FROM `repository_commit` commit WHERE (commit.phid IN ('PHID-CMIT-c7kzeg4za63fwd5ql7vl')) ORDER BY commit.id DESC
42<<< [17] <query> 256 us
43>>> [18] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
44<<< [18] <query> 224 us
45>>> [19] <query> SELECT commit.* FROM `repository_commit` commit WHERE (((commit.commitIdentifier LIKE '4edd28f6008c14051421559a00aa61a41972e65e%' AND LENGTH(commit.commitIdentifier) = 40))) ORDER BY commit.id DESC
46<<< [19] <query> 328 us
47>>> [20] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
48<<< [20] <query> 222 us
49>>> [21] <query> SELECT commit.* FROM `repository_commit` commit WHERE (commit.phid IN ('PHID-CMIT-c7kzeg4za63fwd5ql7vl')) ORDER BY commit.id DESC
50<<< [21] <query> 228 us
51>>> [22] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
52<<< [22] <query> 218 us
53>>> [23] <query> SELECT commit.* FROM `repository_commit` commit WHERE (commit.phid IN ('PHID-CMIT-c7kzeg4za63fwd5ql7vl')) ORDER BY commit.id DESC
54<<< [23] <query> 221 us
55>>> [24] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
56<<< [24] <query> 229 us
57>>> [25] <query> SELECT commit.* FROM `repository_commit` commit WHERE (((commit.commitIdentifier LIKE '4edd28f6008c14051421559a00aa61a41972e65e%' AND LENGTH(commit.commitIdentifier) = 40))) ORDER BY commit.id DESC
58<<< [25] <query> 303 us
59>>> [26] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
60<<< [26] <query> 217 us
61>>> [27] <query> SELECT commit.* FROM `repository_commit` commit WHERE (commit.phid IN ('PHID-CMIT-c7kzeg4za63fwd5ql7vl')) ORDER BY commit.id DESC
62<<< [27] <query> 231 us
63>>> [28] <query> SELECT * FROM `repository` r WHERE (r.id IN (5)) ORDER BY r.id DESC
64<<< [28] <query> 234 us
65>>> [29] <query> SELECT edge.* FROM `edge` edge WHERE (edge.src IN ('PHID-CMIT-njbauecpnouwc3rwesnr')) AND (edge.type IN ('55')) ORDER BY edge.dateCreated DESC, edge.seq DESC
66<<< [29] <query> 224 us
67>>> [30] <query> SELECT edge.* FROM `edge` edge WHERE (edge.src IN ('PHID-CMIT-njbauecpnouwc3rwesnr')) AND (edge.type IN ('52')) ORDER BY edge.dateCreated DESC, edge.seq DESC
68<<< [30] <query> 234 us
69[2015-01-09 08:58:27] EXCEPTION: (PhabricatorApplicationTransactionNoEffectException) Transactions have no effect:
70 - Edges already exist; transaction has no effect. at [<phabricator>/src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php:1635]
71 #0 PhabricatorApplicationTransactionEditor::filterTransactions(PhabricatorRepositoryCommit, array) called at [<phabricator>/src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php:649]
72 #1 PhabricatorApplicationTransactionEditor::applyTransactions(PhabricatorRepositoryCommit, array) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitHeraldWorker.php:91]
73 #2 PhabricatorRepositoryCommitHeraldWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:44]
74 #3 PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:91]
75 #4 PhabricatorWorker::executeTask() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementReparseWorkflow.php:297]
76 #5 PhabricatorRepositoryManagementReparseWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:396]
77 #6 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:292]
78 #7 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22]
79
80
81
82
83The offending commit:
84
85$ git show f628ef8ca552a0c9ca6d1736576b45e7bbc635f5 --format=raw
86commit f628ef8ca552a0c9ca6d1736576b45e7bbc635f5
87tree c1f99a7a3c34f452fac3d25cecf05b842de60282
88parent 4edd28f6008c14051421559a00aa61a41972e65e
89author Some User <example@example.org> 1385425306 -0800
90committer Some User <example@example.org> 1385425306 -0800
91
92 Revert "$THE_COMMIT_MESSAGE"
93
94 This reverts commit 4edd28f6008c14051421559a00aa61a41972e65e.
95
96
97
98The reverted commit
99
100$ git show 4edd28f6008c14051421559a00aa61a41972e65e --format=raw
101commit 4edd28f6008c14051421559a00aa61a41972e65e
102tree 182ea14203caf1147ae604d3d229b56241467815
103parent b5f5d51e0bb3ecad58b298b3e19caa08b0f5a998
104author Some User <example@example.org> 1385424584 -0800
105committer Some User <example@example.org> 1385424586 -0800
106
107 $THE_COMMIT_MESSAGE

Revisions and Commits

rP Phabricator
D11304rP9b21274138cf Allow `PhabricatorRepositoryCommitHeraldWorker` to continue on certain types…

Related Objects

Event Timeline

frenchs created this task.Jan 9 2015, 5:28 PM
frenchs raised the priority of this task from to Needs Triage.
frenchs updated the task description. (Show Details)
frenchs added a subscriber: frenchs.
chad added a project: Diffusion.Jan 9 2015, 5:30 PM
epriestley added subscribers: joshuaspence, epriestley.Jan 9 2015, 6:01 PM

The PhabricatorAuditEditor in PhabricatorRepositoryCommitHeraldWorker should probably have both:

->setContinueOnMissingFields(true)
->setContinueOnNoEffect(true)
joshuaspence claimed this task.Jan 9 2015, 9:31 PM
joshuaspence added a revision: D11304: Allow `PhabricatorRepositoryCommitHeraldWorker` to continue on certain types of exceptions.Jan 9 2015, 10:41 PM
joshuaspence closed this task as Resolved by committing rP9b21274138cf: Allow `PhabricatorRepositoryCommitHeraldWorker` to continue on certain types….Jan 9 2015, 10:43 PM
joshuaspence added a commit: rP9b21274138cf: Allow `PhabricatorRepositoryCommitHeraldWorker` to continue on certain types….
gabe awarded a token.Jan 13 2015, 8:56 PM
gabe added a subscriber: gabe.

Was stuck with this same problem, just pulled the latest changes and can now report that it's working fine. Thanks!

frenchs added a comment.Jan 13 2015, 9:00 PM

Many thanks for knocking this one out. Pulled as well, and it fixed both our impacted repos.

staticshock added a subscriber: staticshock.Jan 20 2015, 11:51 PM
Phabricator · Built by Phacility