I've updated both hosts and manually fixed D16464 to display the correct changes.

Yeah, I just neglected to push secure002 when I pushed secure001. I'll fix this shortly.

Oh, this looks like a smoking gun:

There's no changeset in the database, so this isn't just a weird permissions issue -- legitimately, no changeset was extracted.

bin/differential extract rPHU491ebc74d816dbe2fc8bbbbc992e8a14f8c613be also extracts the change correctly:

This might actually something more narrowly scoped since, e.g., D16468, D16469, and D16470 worked fine.

Giving full access to all users for the host repo folder solved the issue. As this is a vulnerability, I shouldn't left it as 'chmod a+w'.

I have the same problem.

This has happened about 2 months ago, so a lot of the damage has already happened. We had a listing somewhere about what to do, but I can't find it now, so I'll re-write it here for posterity:

This can cause a storm of commit mail when upgrading. If there are old tags which weren't imported at the time, when Phabricator is upgraded to a version containing D16129 it can fire off thousands of commit emails, flooding inboxes and making some folks grumpy at their Phabricator administrators.

D16460 is marked as fixing this, and we always send data via Files afterwards. This is different from the approach I'd outlined above (no mode where we continue to return text directly), but two additional considerations have arisen:

Introducing an epoch64 type ends up being pretty messy:

Perhaps a better solution is to introduce an epoch64 datatype and store these timestamps faithfully, since they're at least sort of possible to write and do have a relatively unambiguous meaning. Introducing a new type instead of changing the meaning of the existing epoch would let us accommodate this without requiring tons and tons of migrations or exposing us to a lot of weird fallout.

After some effort, I'm unable to figure out how to generate these commits using vanilla git. I can't figure out how get them with --date or GIT_COMMITTER_DATE to git commit: dates after 2100 are treated like the current date.

The particular issue is that that the change removed a file containing the bytes ... 0xE5 0x76 .... The 0xE5 is probably an å in Latin 1.

I extracted the response body prior to encoding as a serialized()'d object so I can narrow down what's causing the encoding to fail and it looks reasonable, looking like it's almost certainly a UTF8 issue now.

Not too surprisingly, the response is empty. AphrontResponse->encodeJSONForHTTPResponse() uses json_encode() and does not detect UTF8-related encoding failures, so the chance that this is a UTF8 issue is pretty high. We should also probably fix that so fail earlier and more explicitly (and don't emit HTTP 200).

The next step I want to take is figuring out what the actual problematic response body looks like, but that's slightly tricky right now because there's no way to make Conduit calls from the CLI.

In the read/write case, it's too late once we hit a hook -- we need to be able to decide which cluster node to send the traffic to, and we need to know if it's a read or write request before we make that decision, since there may be a nearby read-only node and a distant read/write node. If we can examine the stream and make a determination that the request is read-only, we can freely pick a nearby read-only node.

It's not entirely phabricator specific, as others probably have similar use cases. I'm literally the only hg dev that knows this is an issue (someone flagged me on twitter a month ago, and it's only a strange inability to sleep tonight that got me deep enough in my stack taht I got to this), but if you go to our ML with a decent problem statement ("we need a hook to let us identify read vs write transactions and do our own ACL checking" sounds like a start?) you'll get more help than just me when I'm unable to sleep. :)

Aha

I'm not a phabricator developer so I'm not sure if supporting mercurial hooks is something they're willing to take upstream. I can join the hg mailing list, but wouldn't this conversation be specific to phabricator and beneficial to be carried out here?

Today, we technically do not need to decode the protocol, but we will need to be able to distinguish between read and write requests to implement cluster read-only nodes (T10883) in the future and need to be able to rewrite the protocol to keep parity with fork-like/virtual-ref features that we plan to support in Git (T10691, T8092).

Ah, okay. We can probably do a lot better with some sort of pre-transaction hook then (that was my gut feeling to begin with). Once you've got an idea what the use case is, maybe we can chat on the hg mailing list?

I am totally on-board with finding something more elegant/maintainable, and would be able to help with some testing and possibly development. I'll try to look over the code in the near future to recall what it was doing, but I believe it's trying to determine whether the mercurial operations are read/write in order to determine whether the actor has necessary permissions to do so. I *believe* that's what it's doing, but it's been close to a year since I was looking through this.

We've got a volunteer (slowly) working on documenting formats, but right now bundle2 is not documented extensively outside of the code and the wiki page already mentioned in this thread.

@durin42 - that would probably be better than parsing the bundle stream but, is that documented anywhere? I tried looking fairly extensively for details on the bundle2 protocol and never came across that. Also, the bundle stream itself isn't what's being parsed/modified, it's the capabilities/outer protocol which I found at least some documentation, which is what the parsing/filtering is based on.

I'm not sure why phabricator is trying to parse the bundle stream (I don't know anything about phabricator), but you should be able to invoke hg with bundle2 disabled by setting experimental.bundle2-advertise to false in an hgrc (or using --config if you're shelling out to hg). Given that it's an experimental flag, it's not something I can promise will exist forever, but it might be a start.

We haven't seen other requests in this vein, exactly, in about two years, and we generally don't expect users to rewrite published commits as a routine workflow.

We haven't seen other users run into difficulty with this in the last couple of years, so I don't currently plan to change it.

We have an existing repository_badcommit table which is used to mark commits as "bad", where "bad" approximately means "the repository as a whole works, but the VCS tools can not read or should not read this particular commit for whatever reason". This is rare, but can happen when, e.g., third-party tools are used to write into repositories.

This is like two years old now, and I'm pretty sure it works now since no one else has reported it and it would likely have come up when clustering was written a few months ago if it was still an issue. If anyone is still seeing issues, please feel free to file a new report with reproduction steps.

I believe this was resolved by T6878 or related work. In the modern codebase, we fetch with --prune and explicit refspecs, which is sufficient according to git help fetch:

Yes, this seems to be exactly what the other issue was referring to. We've just recently upgraded (previous codebase was 2+ years old) and massive amounts of commit email are getting sent out.

Yeah, this is likely the same as T11188. See that task for some additional discussion.

Yes, this is intended behavior. See T10262 for discussion. See T10262#158198 for comparison to other systems, many of which use the same approach to access control for the same reasons.

This is intentional (and not a security issue), the URI contains a security token valid for 30 days (or until the policy is changed).

This should be fixed in HEAD of master, and in the documentation on this host whenever it regenerates (every other rainy Tuesday during a full moon).

I'm just going to merge this into T5028 since I think that ultimately is essentially the same issue.

thanks Evan, that did the trick!

(T8850 is vaguely related.)

we are using Apache webserver, and yes we are mangling the REMOTE_ADDRESS in preamble as recommended in the configure_preamble article section "adjusting client ips" because we are behind a load balancer.

cluster.instance is only used when serving multiple separate instances from a single cluster (for example, when hosting Phabricator for multiple clients), and should normally be left alone. This setting is used in the Phacility cluster to make x.phacility.com, y.phacility.com, etc., work correctly. This is sort of confusing because we use "cluster" to refer both to instancing and redundancy/scale; both are "clusters" in some sense but we don't generally expect users to care about instancing.

I just tested the "Replace %s%n%n%b with %B" and it fixed it, Auditors were parsed without any problems.

Yeah, I think this is because of --format=%s. Some discussion in T5028, which is probably most of this.

The windows style was from netbeans, I did the previous ones from command line (if you want I can run netbeans as well to prove it). The title was from when I was strictly using netbeans.

When I run git cat-file -p 69d24cb35c46a4a2fed6a851aa17962d7dc0bd26 | hd, I get only \ns, no \rs at all... same with git log and git log ---format='%B' (git version 2.8.1.210.gb8b4d93 on linux).

Sorry about that, I didn't know the test git repository existed otherwise I would have done this right after.

READMEs are only rendered in the browse view:

Was rendering disabled altogether? https://secure.phabricator.com/diffusion/P/browse/master/README.md shows up as a mere textarea.

I've used a mac for ~6 years now and still get annoyed that Finder won't list folders above files sorted by name. For me the folders work as a navigation control instead of content which is why I usually like them separate. My preference said, it doesn't really affect me in Diffusion.

See additional discussion in D9343.

I added you to Community, granting you sweeping janitorial powers on this install.

This is resolved on my end.

You could also do this with no custom code by using Herald to trigger a Harbormaster build and having that make an HTTP request to something which runs the logic, or theoretically by using feed.http-hooks or Doorkeeper (although these are probably better after T10363).

D16385 increases the storage length of the underlying column. Summaries will now be cut to 255 bytes or 80 display glyphs, whichever is shorter.