We may fopen() an additional stdout and/or stderr handle, but do not fclose() it?

I am suspicious that D21794 may have broken something subtle with unix magic, since I'm seeing some hangs out of deployment scripts wrapping daemon management scripts. I think the issue is probably one of:

What options exist for inter-process communication?

Here's how I'm thinking about overengineering this:

How do modern servers written in C/C++ handle parallelizing requests?

Just for my own notes:

FWIW I've found by far the easiest way to work with microcontrollers is using micropython / circuitpython on any of these chips: ESP32, ATSAMD21/ATSAMD51 and RP2040. The esp32 is in many ways the easiest and most practical because it's extremely cheap and includes a wifi radio.

What are the practical limits of multi-system or multi-architecture binaries?

I generated D21794 with a native binary that has no dependency on system PHP (but does depend on system cURL).

1. PHP doesn't link cURL statically and doesn't appear (?) to provide any easy way to link it statically.
   1. • Possible solution: build static cURL.
      • Possible solution: link cURL dynamically.
      • Possible solution: replace cURL with mbedTLS + a first-party HTTP client, since we don't care about 99% of what cURL does. Or implement first-party TLS hahaha except I'm half-serious? If HTTP/HTTPS is going to happen from an embedded context on my coffee maker I need a healthy embeddable TLS + HTTP stack anyway.
   2. The STDERR and STDOUT constants are defined by the CLI SAPI, and not present in the embed SAPI.
      • Possible solution: define them in the C wrapper.
      • Possible solution: polyfill them at startup in PHP (I'm not entirely sure this is possible).
      • Possible solution: abstract around them and use php://stderr and php://stdout instead. See also T13556. This is likely desirable anyway.
   3. When arc tries to load PHP code, it needs to read data out of the executable binary in some set of conditions (e.g., "if we miss on disk").
      • Possible solution: hook zend_compile_file(), which seems to be the expected way to approach this. The flow in phar_compile_file() seems similar.
   4. To run unit tests, arc depends on the presence of a php binary on the system.
      • Possible solution: accept that you must also have PHP to run arc's tests (and that system PHP may differ from arc php).
      • Possible solution: also include the PHP CLI wrapper and invoke it when arc is executed as php, providing what is essentially a fallback toolset.
      • Possible solution: provide a "unit test helper" fallback toolset.
      • Possible solution: rewrite the 5 trivial cases where we need this (support/unit/*) in shell script (but: the reason to do these in PHP in the first place was so that they're portable to Windows).

Can we build a single binary with a bunch of data in it (e.g., a picture of a cat) without breaking anything?

Is the purpose of implementing introspection of libraries to allow for building native components that would get picked up and auto-loaded by arcanist?

I find this very intriguing.

Phabricator would benefit from having access certain services (full-text search, full-codebase search, repository graph storage) that very likely aren't practical to write in PHP

Is the purpose of implementing introspection of libraries to allow for building native components that would get picked up and auto-loaded by arcanist?

Browsing Diffusion (namely, loading a file while the published changeset is head, and trying to Skip past commit):

I couldn't immediately reproduce this; it's also probably a bit of a messy fix.

I'll plan to investigate further if possible. I believe in this case the repository was running in observed mode rather than hosted in case that might be a factor.

How can PHP be built statically? How hard is this?

Browsing Revision

I caught one cosmetic issue in D21711, but this works properly for me locally now. Thanks!

(After picking up D21697, the approach works properly in my environment where I originally ran into this, but there are a couple of rebase calls still present in ArcanistMercurialLandEngine that don't use the new API yet.)

Marking this as resolved by D21700. If there are further details or input we can re-open to further address.

I'm also surprised that rebase extension isn't enabled by default. I guess I have been turning it on in my default setup.

I like that idea. I'll take a look.

...perhaps we should consider a way to do this automatically or at least less-manually.

Interestingly this is the output I get when using hg log to print out the commit message containing smart quotes with different encodings -- this behavior is the the same for both PowerShell (chcp 65001) and cmd.exe (chcp 437) and no errors occur.

$ hg log -r tip --template "{desc}" # "default" encoding
A test

I think the global flag is reasonable.

In Mercurial, with certain character sets, some arc workflows may fail in arc-hg while trying to perform UTF8 operations.

I'm doing some testing to find an appropriate update to arcanist for this. I ran into another issue, when patching down a revision which has unicode characters in the summary on a windows machine. I could try to whack-a-mole these different issues but since Mercurial's --encoding argument is a global option do you think it would be wise to update ArcanistMercurialAPI::buildLocalFuture() with this?

diff --git a/src/repository/api/ArcanistMercurialAPI.php b/src/repository/api/ArcanistMercurialAPI.php
index e5c2078b..ac72c356 100644
--- a/src/repository/api/ArcanistMercurialAPI.php
+++ b/src/repository/api/ArcanistMercurialAPI.php
@@ -15,7 +15,7 @@ final class ArcanistMercurialAPI extends ArcanistRepositoryAPI {
   protected function buildLocalFuture(array $argv) {
     $env = $this->getMercurialEnvironmentVariables();

...this would need some kind of smarter scope guard...

I also can't get O_NONBLOCK to survive process exit on macOS. This is possibly because macOS is now zsh, and this RedHat bug suggests that zsh clears O_NONBLOCK:

arc may leave stdout/stderr nonblocking.

D21643 "fixes" this by using copy(). However:

We might be able to detect that the --strategy-option theirs merge has done something suspicious by trying to merge C^ into the final state (or vice versa). If it merges cleanly, I think we can be fairly confident that the changes are actually independent. However, I'm not sure offhand what we could conclude if it doesn't merge cleanly.

A particular example is:

In the general case, the problem is that some invocations of arc, including arc land --pick A, may select an arbitrary range of commits to land which have some ancestors that we do NOT want to land.

git diff ...

Broadly, I believe these "certain changes" are approximately changes which include a merge commit.

See also PHI1987 for another case of this.

In T3277#254315, @epriestley wrote:

grep -v master | grep -v '^[0-9a-f.]*$'

This part may prove slightly trickier to implement correctly in the general case.

grep -v master | grep -v '^[0-9a-f.]*$'

+1 for this command. We have use a web-based UI for landing so our branches stick around locally. To fix this I'm currently using:

arc branches | grep ' Closed ' | sed 's/[^ ]* //' | sed 's/ .*//' | grep -v master | grep -v '^[0-9a-f.]*$' | xargs git br -D

Just want to add a note that with git lfs managed files, arc patch won't work unless one fetch the ref tag and then the lfs blobs from the staging server first. See https://discourse.phabricator-community.org/t/arc-patch-fails-with-git-lfs-files/2447/3

Same issue here.

• D21457 fixes approximately 20 additional errors identified by the linter in phabricator/.

• D21454 fixes the specific issue in HTTPSFuture.

Check if arc lint catches this.