Page MenuHomePhabricator

Repository-sync kills my server!
Closed, InvalidPublic

Description

Hello,

due to a misconfiguration of a repository remote access, the xen image on which my phabricator runs has been stopped to be accessable, and it stopped even the hosting-xen server. I had to hard-reset the complete xen-server :(

PLEASE, don't advise me to configure it right - I did that successfully in the meanwhile of course.

The problem is that the daemon retries to access endlessly, although the system already recognized that the configuration is wrong!

I researched the logs and found in the combined sync-logs the following entry several HUNDRED times:

[2014-05-13 15:52:42] EXCEPTION: (PhutilProxyException) Error while updating the "rOIM" repository. {>} (CommandException) Command failed with error #255!
COMMAND
 '/var/www/phabricator/phabricator/bin/repository' update -- 'OIM' 

STDOUT
 (empty)

STDERR
[2014-05-13 15:52:42] EXCEPTION: (Exception) Repository "OIM" does not have a correctly configured remote URI. The remote URI for a Subversion repository MUST point at the repository root. The root for this repository is "https://XXX.de/intsvn/MF4", but the configured URI is "https://XXX.de/intsvn/MF4/OdourInspectionManager". To resolve this error, set the remote URI to point at the repository root. If you want to import only part of a Subversion repository, use the "Import Only" option. at [/var/www/phabricator/phabricator/src/applications/repository/engine/PhabricatorRepositoryDiscoveryEngine.php:332] 
 #0 PhabricatorRepositoryDiscoveryEngine::verifySubversionRoot(Object PhabricatorRepository) called at [/var/www/phabricator/phabricator/src/applications/repository/engine/PhabricatorRepositoryDiscoveryEngine.php:241] 
 #1 PhabricatorRepositoryDiscoveryEngine::discoverSubversionCommits() called at [/var/www/phabricator/phabricator/src/applications/repository/engine/Ph... (1,206 more bytes) ... at [/var/www/phabricator/libphutil/src/future/exec/ExecFuture.php:398] 
 #0 phlog(Object PhutilProxyException) called at [/var/www/phabricator/phabricator/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:335] 
 #1 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(Object PhabricatorRepository, Object ExecFuture, 15) called at [/var/www/phabricator/phabricator/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:198] 
 #2 PhabricatorRepositoryPullLocalDaemon::run() called at [/var/www/phabricator/libphutil/src/daemon/PhutilDaemon.php:85] 
 #3 PhutilDaemon::execute() called at [/var/www/phabricator/libphutil/scripts/daemon/exec/exec_daemon.php:112]

Event Timeline

cgaeking raised the priority of this task from to Needs Triage.
cgaeking updated the task description. (Show Details)
cgaeking added a project: Repositories.
cgaeking added a subscriber: cgaeking.

It's expected that it will retry, but should not retry more than once every 15 seconds. Running one command ever 15 seconds should not kill a server. Are the errors in the log separated by less than 15 seconds?

Thank you for the response. Ok, I rechecked the logs and it was every 15 seconds - you are right. (I see 4 entries for each wrong location). There were 2 locations entered which were wrong at the same time. The repository is generally quite large.
Hm.
I can not completely exclude that the server's failure was something different.
But always when the synchronization started it didnt take long until the server said good bye.
Btw. the svn-log shows really tons of accesses during that time.

Why are you not deactivating the obviously wrong respository-entry after some tries automatically?

We can't tell that the failure is because of configuration, versus the server being briefly offline for maintenance, versus a network problem, etc. In some of these cases the failure is permanent, but in other cases it is temporary and the correct behavior is to continue retrying until the server comes back up, the network issue is cleared up, etc.

btrahan claimed this task.