You are going to have to post more information. I am able to correctly see the error you posted following these steps.

Can you post your arclint file

See Planning for timelines on how we work. If this issue is urgent and blocking your company, see Support Resources for prioritization options.

@chad could u take a look again

I put up a diff that fixes this however I'm not sure if it's the correct solution.

Do you have any other repositories to try this on? Smaller ones, or some public repo from BitBucket?

Thanks!

I'll retarget this to add a setup warning about micro instances, this isn't the first time they've done surprising things (T2727, T8204, T8210).

I did, and it certainly seems to match up. I'm surprised it's so dramatic. I guess since it was working fine before, the sensible thing is probably to just wait for the credits to recover and go back to things on the t2 micro.

If you fiddle with the dropdown in the upper right you should be able to get historic data and maybe see if the graph went not-so-good around then. Doesn't seem implausible.

My graph certainly seems to imply that this is the issue.

Here's the actual credit graph in the AWS web console, under the Monitoring tab (lower right):

One thing we could possibly do is try to hit the AWS meta-service during startup:

It's hard to say because workloads differ, but I'd steer away from the CPU credited "burstable" t2.* instances based on generally negative experiences with micros in the past.

Hm, that's interesting. I know basically nothing about AWS, but I'll try adjusting phd.taskmasters, and if that doesn't work, I'll try using a bigger instance. Is the next size up sufficient, in your experience?

In AWS, my understanding is that micro instances have CPU credits and their CPU is stopped or heavily throttled if you exceed the credit hours. Has the instance exhausted its CPU credits? I think there's a graph on the console, maybe in the "CloudWatch" tab or something like that? I don't actually have any micro instances up right now to check.

(Is your server an underclocked Raspberry Pi emulated on an Amiga?)

EC2 t2.micro

(Is your server an underclocked Raspberry Pi emulated on an Amiga?)

Not that was printed to the console.

That looks pretty normal to me.

P2005. This was all written to stderr, not stdout. I let it run for a little while and then interrupted.

Yeah sorry a " " snuck in there while I was copy/pasting it.

@jonah214 I vote for see stuff that goes by. You can use the paste application https://secure.phabricator.com/paste/ so it's easier for us to look at.

>>> [536] <exec> $ hg log --template='{node}' --rev ''\''caac1231de800ccaa18c96
98ae72f6cb7dfca637'\'''

I don't know a lot about hg internals, but 1,000 branches and 10,000 commits doesn't sound like too much to process.

@michaeljs1990:

do you have any more information from the logs?

Should I post the output from bin/phd log? I can't even tell where the actual logfiles are.

Does /usr/bin/env php -v also print 5.6, versus 7?

I'm running 16.04 with PHP5.6 as well, haven't seen this issue. Here's my installed PHP info if that helps. Might want to verify what PHP your webserver is running, not just CLI.

do you have any more information from the logs? This also seems like more a question than a bug report since daemons not working right is rather hard to debug.

/var/log/httpd/error_log shows nothing with proc_open()

Well I typed in one of those...

If you try to run /usr/bin/python /usr/bin/hg log --template={node} --rev 'some rev hash' manually in /projects/phabricator/2, does it break or do you get anything useful back?

Well scratch that last comment. It looks like it has pulled in commits from the last 3 days. It looks like under recent commits when I'm viewing the repo doesn't have the most recent commits from 3 days ago, but I just installed a panel on the dashboard showing the most recent commits, and all my commits area there. The panel runs off Audit, and the repo Diffusion.....maybe there's a disconnect? idk.

D16460 is marked as fixing this, and we always send data via Files afterwards. This is different from the approach I'd outlined above (no mode where we continue to return text directly), but two additional considerations have arisen:

Yeah I just double checked, it is a full clone of the original repo at the time it was at, but looking at it now, it hasn't picked up a new commit for 3 days.

Than probably not a reasonable case of "too many open files".

we'll discuss internally next week and let you know. thanks for now

You're work-around is fine.
What does cat /proc/sys/fs/file-max say? Is it possible you just have too many open files?

I'm happy to rewrite your script so it works, but we charge $1,500/hr for custom development (see Consulting) because time spent helping individual users with unique problems takes away from developing and improving Phabricator. Are you interested in this?

this is fine since the way python serializes over HTTP is certainly peculiar; but what about JSON ?

This is not a bug.

This doesn't look like a bug report nor a feature request. I don't know what you want us to do with this.

I've seen this happening here too (B13428)

So, yes, the bundle2 thing is definitely orthogonal to this. For now, I'd suggest that if phabricator breaks bundle2, filter the response to the capabilities command and omit bundle2=.* from the response.

The action is indistinguishable from an update from Herald's perspective. Herald runs on the state of the object, not changes to it, so "Changed file content" will just match whatever the diff changes.

Commandeer is a pretty infrequent action so this doesn't matter much, but out of curiosity could I avoid this with a better Herald rule? I guess I can try Changed file content matches regexp '.'.

T7385 is probably also approximately this.

Commandeer fires Herald rules so stuff like "Require Legalpad Signature" or "Add Reviewers: that intern's wrangler" work.

T7074 may not be this, exactly, but they're close enough for government work.

Ah, thanks! I think that should be relatively easy to fix.

This happens for everyone here in the company on windows chrome with the HRV (Croatian) layout, where the @ symbol is achieved with the right side alt (ALTGR) + "V". Think that registers ALT + CTRL + V.

Yes, I have tried to do storage upgrade but it ends up with an error.

Have you tried following the instructions given in the error message?

Forgot to mention we're running stable week 34.

Let me see if I can reproduce this, but this is probably straightforward to fix -- I suspect some code just didn't get updated when Owners packages became mailable.