Okay, thanks for the info.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
All Stories
Nov 19 2015
Since we don't have access to whatever GitLab is measuring, we can't understand this issue in greater detail. Particularly, we can not reproduce the underlying heavy load in a way we can examine. We have not seen similar reports from other users, except @jcarrillo7, who has not filed any details. We have many installs with hundreds of repositories and my expectation is that the operation is cheap, so I suspect this may be an issue unique to GitLab.
Sounds like you could easily fork and change the polling frequency; The calculation is at https://secure.phabricator.com/diffusion/P/browse/master/src/applications/repository/storage/PhabricatorRepository.php$1527, and you can drop it to an hour just by changing that line.
I am not sure of the underlying implementation of GitLab, but our phabricator instance has been flagged as the #1 user of GitLab resources. Our GitLab instance hosts thousands of repositories, so being the single largest user of GitLab is concerning since we only have ~50 repositories.
Based on my memory, I think it's possible that exception is the cause of the issue; when it occurs, the exception ends the current worker task with a permanent failure, and the task won't be revisited. Thus the build remains pending because the worker that would have performed the next update failed to do so.
My next step would be to put more crumbs in the UI, but we're missing a little data right now. Basically, I'd do this:
We can leave this at "reduced expected incidence rate to < 1/3k" and I'll come back here if it happens again if you don't have any more obvious ideas.
huge amount of traffic
At that scale, is there any reason to not use Phabricator's repository hosting instead of GitLab? You can mirror repositories from Phabricator to GitLab if you still need some GitLab functionality (see the Mirrors option under Edit Repository).
Oh I lied, checked the old logs, this error was present the last two times it occurred, but not present when it happened this evening.
This perfectly recreates the symptoms, except it generates an error in the daemon log:
Oops! Thats embarrassing. These numbers were gathered from multiple places/multiple people, and mass confusion ensued. The 2Hz number came from the total requests to GitLab instead of requests per repository. Ill update the title/description accordingly.
I don't think that is going to work here are some things I pulled from the extension migration page (unless we ifdef large blocks of code):
zval
- PHPNG doesn't require any involvement of pointers to pointers to zval. Most occurrences of zval** variables and parameters have to be changed into zval*. The corresponding Z_*_PP() macros that work with such variables should be changed into Z_*_P().
- In many places PHPNG work with zval directly (eliminating need for allocation and deallocation). In these cases corresponding zval* variable should be converted into plain zval, macros that use this variable from Z_*P() into Z_*() and corresponding creation macros from ZVAL_*(var, …) into ZVAL_*(&var, …). Be always careful about passing addresses of zval and & operator. PHPNG almost never require passing address of zval*. In some places & operator should be removed.
- zval allocation macros ALLOC_ZVAL, ALLOC_INIT_ZVAL, MAKE_STD_ZVAL are removed. In most cases their usage indicate that zval* need to be changed into plain zval. Macro INIT_PZVAL is removed as well and its usages in most cases should be just removed.
Specifically, if you put like a sleep(10) directly after applyPlan() I would expect that to trigger the race ~100% of the time as long as the queue isn't too full. You could see if the behavior is similar to the behavior you've observed.
Yeah, I think that can race. Here's an example -- open two connections to MySQL, mysql1> and mysql2>. In the first window do this:
I'm not sure I understand your math -- here's what I get:
DAO's prefixed CISubmitQueue are in their own database. There has never been any problems with referential integrity / never seen anything in the database that would indicate we'd written records incorrectly. But you may be right about the tasks being written before the plan is written. I suspect that would cause errors though, I can try to reproduce that first in order to rule out our specifics being the problem.
Let me post the whole block, it didn't seem relevant:
Oh, you removed openTransaction(), so that's probably not the culprit.
It's vaguely possible that openTransaction() is the culprit. applyPlan() queues a daemon task, and I could imagine the race might be:
@epriestley I know you mentioned that a diffusion.looksoon would trigger this, but we have no known users of this command.
Only due to the fact that it failed 250 times and got removed by Phabricator.... I suspect it will happen again the next time that file changes.
It sounds like this no longer reproduces for you.
You can do this with Phurl today.
Nov 18 2015
Is there anything I could do to facilitate discussion or acception of this proposal?
- If the changes are relatively minor, add a bunch more #ifdefs? We already have some of this to deal with prior API changes, and I believe this approach is historically a standard one across other extensions.
- If the changes are substantial, maybe we need to get more creative.
After a few months of using P1094, we have found it invaluable in getting an overview of big and deep tasks. It is sufficient for our needs, we're happy with it continuing to work locally.
@epriestley, due to the fact that the Zend APIs for PHP7+ are different than those in PHP5* how should changes to XHProf be submitted so we don't break stuff?
@epriestley I have a reasonable implementation of this working on the new Drydock architecture. I want to pursue this before EC2 so we can ensure that all of the locking mechanisms are working correctly before we use the new Drydock on something that potentially costs money (and in the event that Drydock goes haywire, may cost thousands of dollars). KVM offers a safe alternative, because if Drydock goes haywire, the worst that will happen is that it won't be able to allocate new VMs on the KVM host due to out of memory errors.
In T9807#144874, @epriestley wrote:If you want to test that something in this vein works, I'm happy to bring it upstream.
Closing here, without a way to reproduce the issue, not sure anything else can be done.
We don't have any current plans to pursue this and I can't guess what or how Phabricator may work years from now. Thanks for the feedback.
Ah - just saw that the diffs I was looking at were outdated and that link is a thing now. Saw the guide explaining all of the custom fields :)
Nov 17 2015
Wow thank you - I was planning to look at this probably sometime this weekend (with pretty much the change you pasted). I'll edit the task description to provide a workaround if anyone else runs into this.
If you want to test that something in this vein works, I'm happy to bring it upstream.
We should be able to just omit the final argument if remote is not specified. We can refine that later, but our usage seems like it's in the wrong to me.
sadly I can't run this
bin/worker execute --id <id>
since the task has been archived
https://wiki.php.net/phpng-upgrading should contain a lot of relevant information.
If a Phurl doesn't have a name, should we still show the empty name property on the object view?
No, that's not currently possible, but it might be worth filing a Maniphest task for that.
Definitely want to weigh in, and say that this is a vastly important feature for myself. Having the tree view in other bug trackers is helpful for planning, triaging, and almost every aspect of software development.
I have metamta.differential.inline-patches set to 100
metamta.email-body-limit is set to the Global Default 524288