HomePhabricator

Add controllers/search/edit engine functionality to RepositoryIdentity

Description

Add controllers/search/edit engine functionality to RepositoryIdentity

Summary: Depends on D19423. Ref T12164. Adds controllers capable of listing and editing PhabricatorRepositoryIdentity objects. Starts creating those objects when commits are parsed.

Test Plan: Reparsed some revisions, observed objects getting created in the database. Altered some Identity objects using the controllers and observed effects in the database. No attempts made to validate behavior under "challenging" author/committer strings.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T12164

Differential Revision: https://secure.phabricator.com/D19429

Event Timeline

This seems to have caused exceptions for us when parsing commit messages like:

Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] [2018-06-04 15:30:46] EXCEPTION: (PhutilProxyException) Error while executing Task ID 498956. {>} (AphrontQueryEx
ception) #1048: Column 'identityNameRaw' cannot be null at [<phutil>/src/aphront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:355]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] arcanist(head=stable, ref.master=cdaad096fa8d, ref.stable=733ac8050162), phabricator(head=stable, ref.stable=2362
53df8cd6), phutil(head=stable, ref.master=a8143ced646d, ref.stable=f3e10579f640)
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #0 <#2> AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [<phutil>/src/ap
hront/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:289]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #1 <#2> AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [<phutil>/src/aphront/storage
/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:185]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #2 <#2> AphrontBaseMySQLDatabaseConnection::executeRawQuery(string) called at [<phutil>/src/xsprintf/queryfx.ph
p:8]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #3 <#2> queryfx(AphrontMySQLiDatabaseConnection, string, string, string, array, string)
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #4 <#2> call_user_func_array(string, array) called at [<phutil>/src/aphront/storage/connection/AphrontDatabaseConnection.php:58]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #5 <#2> AphrontDatabaseConnection::query(string, string, string, array, string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1262]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #6 <#2> LiskDAO::insertRecordIntoDatabase(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1106]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #7 <#2> LiskDAO::insert() called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:1075]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #8 <#2> LiskDAO::save() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepositoryIdentity.php:73]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #9 <#2> PhabricatorRepositoryIdentity::save() called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:94]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #10 <#2> PhabricatorRepositoryCommitMessageParserWorker::updateCommitData(DiffusionCommitRef) called at [<phabricator>/src/applications/repository/worker/commitmessageparser/PhabricatorRepositoryCommitMessageParserWorker.php:42]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #11 <#2> PhabricatorRepositoryCommitMessageParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phabricator>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:51]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #12 <#2> PhabricatorRepositoryCommitParserWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:123]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #13 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #14 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #15 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
Daemon 31221 STDE [Mon, 04 Jun 2018 15:30:46 +0000] #16 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]
Daemon 31221 FAIL [Mon, 04 Jun 2018 15:30:46 +0000] Process exited with error 255.
Daemon 31221 WAIT [Mon, 04 Jun 2018 15:30:46 +0000] Waiting 5 second(s) to restart process.

Fixed with: ./bin/repository rebuild-identities <repo>

Fixed with: ./bin/repository rebuild-identities <repo>

Thanks for the report. The underlying issue is fixed in D19466.

Shouldn't there be something in the Changelog about this change?

Shouldn't there be something in the Changelog about this change?

This round of changes just starts populating the new table and adds a workflow to populate them by hand. There will be a large migration soon that starts to actually depend on this table, and that week’s change log will have extensive guidance.

Got the same issue / callstack as @nochum after today upgrade, which was fine (especially storage upgrade)

Symptoms :

  • new pulled commits are visible but in "Still Importing..." state
  • high cpu
  • daemons exception -> callstack visible by doing things like ./bin/phd log --id 1529788

Got the same issue / callstack as @nochum after today upgrade, which was fine (especially storage upgrade)

Symptoms :

  • new pulled commits are visible but in "Still Importing..." state
  • high cpu
  • daemons exception -> callstack visible by doing things like ./bin/phd log --id 1529788

What version of the code are you running? If you haven't pulled since D19466 and have an SVN repository, you're likely to hit the same bug.

I'm at 236253df8cd69a3c7d3da984bd2f1837e92330cc

I did the rebuild-identities, which was slow (big repo.), but solved the issue