Just thinking aloud to myself

Off-hand I don't know of a solution that could provide that same guarantee for cherry-picking. Our current process also requires that the engineer who authored the fix is the one responsible for merging it, rather than having release engineers or an unlucky engineer responsible for performing all merges into the subsequent versions.

This could probably be handled by having the server-side hook check for commits with the same task number, as we require all commits to start with a task number. It's not as "comfortable" as ensuring the same changeset exists in the branch but since the changesets require merge commits which can potentially introduce incorrect merges that comfort might be just as misleading.

The commits created as a side effect of arc land --hold aren't actually deleted explicitly, but they're effectively gone in Git as soon as you update your working copy to any other state. I'm not sure how closely this holds (or can hold) in Mercurial.

Ah yea this is one difference with Mercurial and Git's treatment of branches. In Mercurial there is no automatic cleanup, so things must be stripped/pruned manually before they will go away -- regardless of whether they're tagged with a bookmark or not. I'm still piecing together Git's handling of branches in my mind as I have used Mercurial for so long. Conceptually I know how I want to manipulate the repository but then things happen which reveal that how I'm thinking the repository is laid out isn't what Git's doing (Arcanist has been very useful in this respect).

Remove TODO from ArcanistMercurialWorkingCopyRevisionHardpointQuery. Add some comments to functions in ArcanistLandEngine.

Oh just noticed your comment on this, marking for review so emails go out (I don't plan to make further changes here).

Made updates from comments. Re-ran the "Landing a non-tip revision" test and everything still looks good.

I'll make these last few minor updates and do a quick few tests to make sure everything is still in working order

One thought here is that updating variables_order is fine on the server, but I really don't want to force arc users to muck with php.ini.

This seems to be resolved already but just wanted to add a note - I believe Windows users currently need to modify php.ini to enable the curl extension for arcanist to work. This is from our internal documentation for installing PHP for arcanist:

Browse to C:\tools\php[some version number]. Edit php.ini as listed below.

• Note: The following modifications remove the semi-colon at the start of the line. Semi-colons are comments in .ini files and these changes are uncommenting and updating the lines.
  • Find the line ;extension=curl and remove the semicolon.
  • Locate the line ;date.timezone = and replace it with date.timezone = UTC to prevent PHP from yelling at you.

Sorry for the churn on this, it sorta became a whackamole for an assortment of issues I ran into. I think most of the test cases I've tried out now indicate that the behavior is pretty solid at this point. For now I don't have any remaining test cases to cover and I think this is in a state for review.

Uncovered an issue when using the evolve extension, in which landing a non-tip revision would leave behind a set of commits in a hideous state.

Don't actually unset the found rebased active commit. Not sure why I thought that was a good idea.

Just did another pass-through of all these changes to do some cleanup.

• Removed an old line of code I was testing out trying to handle multiple onto-markers when restoring the working state
• Updated some comments and formatting of marker ref queries
• Found some logic which didn't quite carry over from refactoring of newPushCommands() in handling of cleaning up markers to be deleted

Whoops - that last change results in the same original behavior. It turns out confirmCommits() reverses the order that selectCommits() creates.

To confirm the min/max change I reviewed the revset x..y documentation along with the expected ordering of commits in the ArcanistLandCommitSet, then renamed + added some comments to clarify this behavior.

I normally use arc:bookmark however I recently switched it to arc:this so it's also compatible with git. All of my tests I've run currently were using arc:this.

Went back through https://secure.phabricator.com/book/phabricator/article/arcanist_commit_ranges/ and realized I can specify a ruleset of arc:bookmark, arc:this to get the behavior I want in both mercurial and git. Huzzah!

I'm guessing the --base argument would affect what commits get snagged up during landing (or is that captured during diff phase only?)

Ah it looks like the --base flag isn't part of diff workflow

$ arc land test --base arc:bookmark
Usage Exception: Argument "--base" is unrecognized. Use "--" to indicate the end of flags.

I'm going to do a few more tests -- I'm guessing the --base argument would affect what commits get snagged up during landing (or is that captured during diff phase only?). I normally use arc:bookmark however I recently switched it to arc:this so it's also compatible with git. All of my tests I've run currently were using arc:this.

Add back in the rebase --abort for now

When landing from a working directory that isn't in the set of changes being landed, restore the working directory to that commit after the land.

Not-working state test

• I created a diff with an inactive bookmark test but working directory on an older commit not related to the one being landed
• I landed that diff
• I verified the ending resulting working state was on the older commit I was on prior to landing

Doing a basic test this behavior looks good

$ hg wip
@   19:823c7457b7ef cspeck (30 seconds ago) tip test             <- draft commit to be landed, working state when running `arc land`
|     test commit
| o   17:33c3ae4d1277 cspeck (8 minutes ago) master
|/      test commit
o   14:82892d507c28 cspeck (24 minutes ago)
|     test commit
~

Determine whether the working directory was rebased, and if so update to it during reconcileLocalState()

To sum up what I think the issues I ran into last night were

1. I was not purging caches when switching versions of mercurial or when making changes to DiffusionMercurialBlameQuery
2. I had improperly inverted strpos(':', $line) instead of strpos($line, ':')
3. When using --template I think I was not properly escaping quotes for how they are eventually invoked when launching the process

There's one minor bug with annotations using mercurial 4.2 for file lines that contain : not being annotated but it works on 5.8.

This was because I had made modifications to the resolveBlameFuture() in my testing. I undid these changes and things are working again. I tested both mercurial 5.8 and 4.4, purging all caches in between and verifying that annotations and history appear correctly, including annotations of files with code lines including ':'.

Update the handling of template arguments to avoid using quotes around the template being used

I've been incorporating ./bin/cache purge --all into my test process and things are making a lot more sense now. I'll hopefully have some changes up here shortly. There's one minor bug with annotations using mercurial 4.2 for file lines that contain : not being annotated but it works on 5.8.

Is it possible the annotate results are cached somewhere and might be causing some of the inconsistency I’m seeing?

I'll take another look tomorrow. Somehow I managed to get the code to emit output and found the issue with strpos() but then after fixing that it now seems busted again for both versions of mercurial. No exceptions are happening but it's not actually building up the annotation data properly (which is just the ordered list of the commit hashes I think?)

Fix strpos()

Best guess is that it's maaaaaaybe because we pass HGPLAIN=1 in the environment (DiffusionMercurialCommandEngine) and your version of Mercurial has annotate set up as an extension? That seems like a big stretch, but wouldn't show up.

I don't believe annotate is setup as an extension. This is our global .hgrc

# Ansible managed

I added some logs and then removed them but functionality nothing else should have changed (except the --template quoting I mention) and now when running hg 4.4 it no longer errors, but it the blame doesn't list changesets (but does put color next to the lines which I think is just a default if nothing was found?)

Buh there must be something going on in the environment that's causing issues. I logged out the command it's running and tried running it directly in the repo

[cspeck@repo-testing1 85]$ hg --version
Mercurial Distributed SCM (version 4.4.2)
(see https://mercurial-scm.org for more information)

Oh that error happens when resolveBlameFuture() returns null because there was a problem invoking the command... looking a little more into what issue is happening

Hmm this is the error I'm getting when using mercurial 4.4

2021/07/08 22:35:31 [error] 9693#9693: *1 FastCGI sent in stderr: "

I setup mercurial to run version 4.4, created a new repository, added some commits, and verified all the above behavior still works properly.

This isn't quite true -- annotations don't appear to be showing up. I'll take a quick look

More updates based on comments. Tested out history view and annotation view and things look good.

Fix a regex typo and add back in the trim() on the hash

Found one issue, which seems to fix the improper graph structure but now it's rendering disjointly

Something isn't quite right here. Looking at the history of a repo, the graph rendering doesn't match what it was previously doing

Run arc liberate for the new test file

forgot to submit my comments~

Do capability-check for using hg annotate --template vs --debug, set the required version to 2.4, address other comments

Adding a capability check for hg annotate --template

One other case where "Phabricator" appears is in HTTP and Email headers, e.g. X-Phabricator-XYZ. For a separate project wanting to update these I think a slow migration approach is needed, to allow recipients currently expecting the existing fields. I'm guessing there isn't a reasonable change here for the upstream.

Depending on how many of these we end up with, I think an email-header-prefix sort of config option might be reasonable.

Oh and then for existing installs this would just default to X-Phabricator-XYZ to continue with existing behavior?

In D21679#275638, @epriestley wrote:

This specific issue is probably the relevant one, the other thing linked from there looks like it's just a --debug problem, which this patch should fix.

https://discourse.phabricator-community.org/t/error-cloning-mercurial-repository-http-after-first-install/1200/2

Thank you for the tips.

1. Bring the 2.4 version bump upstream, without directly addressing annotate --debug.
2. Deal with annotate --debug by either:
   1. bumping the minimum version to 4.6 in your fork and using --template; and/or
   2. defining another Mercurial version capability test for annotate --template and selecting that in versions of Mercurial which support it (this can come upstream).

Your call on whether you want to do the legwork on (2B).

I'll plan to do (2B) now that I'm more familiar/comfortable with the version-checking approach. Bumping the required version to something so much newer isn't really justified for this, especially considering the versions that are available on official distribution packages (centos7 currently has mercurial-2.6). Using --template for newer versions will be less error-prone though so I'll do the legwork. I'll try to get something up later tonight.

This is still in early stage but marking for review because I wanted to get some early feedback if I'm on the right track. With these changes when I land a diff with other arbitrary local commits the pull -B master that it ends up running does pull down and update the local master bookmark to the right revision, on a hosted mercurial repository. The changes to splitting the $tail command array into two separate arrays feels a bit off to me. As for updating the working directory I think that change is going to be a bit more involved but I'm not sure if that's something appropriate to be done in pushChange() or that should happen higher up in ArcanistLandEngine::exeute() by adding some other functions to be overridden by the different engines or similar.

"fix" "typos"

Okay I have these changes in and verified that things seem to be working fine with mercurial 5.8, though I'm not sure your thoughts on requiring mercurial 4.6 as the minimum version. If we restore the hg annotate --debug then the minimum version required would only jump to 2.4.

According to their changelog the {p1node} style was added in 2.4 (2012) and was deprecated in 4.9 (2019) in favor of the new {p1.node} format
https://www.mercurial-scm.org/wiki/WhatsNew/Archive#Mercurial_2.4_.282012-11-01.29

I started a draft change in D21679. I also have changes for rARC but haven't yet created a diff for that (or yet figured out the exact versions to define)

Ah thank you for those details for the better change. I was planning to play around with what would be needed to remove the --debug flag(s) but I hadn't yet gone into the details. I'll take a look at these changes. From quick testing on recent mercurial versions (5.8) it looks like the output does not include the local revision number, e.g instead of
693:8b39f63eb209dd2bdfd4bd3d0721a9e38d75a6d3 -1:0000000000000000000000000000000000000000
it prints out
8b39f63eb209dd2bdfd4bd3d0721a9e38d75a6d3 0000000000000000000000000000000000000000

For the invalid branch cache message remove the (served): bit which might differ on hosted vs. observed repositories.

I removed the "install RHEL" and "install Ubuntu" scripts outright since I don't have any reasonable way to test them and don't plan to maintain them.

Ah I did come across these as well and forgot to note.

In T13658#255994, @epriestley wrote:

Create Platform::getProjectRepo() to return "phabricator", or instead detect the repository name.

I suspect having this method do anything nontrivial may lead to some weird boostrapping issues, where it gets called very early during startup or with a partially built environment or during exception handling or whatever and not everything you expect to be available is actually available, so I suspect you'll have the best luck by just hard-coding this.

(For example, Phabricator reads PHABRICATOR_INSTANCE from the environment very early.)

Ah I did have a suspicion that would be tricky but was thinking to give it a shot. Hardcoding is probably just going to be easiest.

As another test I tried creating a new commit on Windows and using smart-quotes and em-dash in the commit message and mercurial bombed with

$ hg ci # nvim opened an editor in UTF-8 encoding mode and let me write in a commit message: Test commit with “smart quotes” and "em—dash"
transaction abort!
rollback completed
note: commit message saved in .hg\last-message.txt
note: use 'hg commit --logfile .hg/last-message.txt --edit' to reuse it
abort: decoding near 't quotesΓÇ¥ and "emΓ': 'charmap' codec can't decode byte 0x9d in position 34: character maps to <undefined>!

I ran through another test just in case and things do seem to function fine. Another interesting point is when using nvim on Windows it will properly render the smart-quotes and em-dash, but mercruial's output uses the weird characters. Just in case I made sure to test having smart quotes and em-dash both in the Title and Summary fields as well as the content of the file change. Patching and landing the change from Windows, then pulling the commit onto the Mac and the end result on the Mac looks correct for all places.

Interestingly this is the output I get when using hg log to print out the commit message containing smart quotes with different encodings -- this behavior is the the same for both PowerShell (chcp 65001) and cmd.exe (chcp 437) and no errors occur.

$ hg log -r tip --template "{desc}" # "default" encoding
A test

In Mercurial, with certain character sets, some arc workflows may fail in arc-hg while trying to perform UTF8 operations.

I'm doing some testing to find an appropriate update to arcanist for this. I ran into another issue, when patching down a revision which has unicode characters in the summary on a windows machine. I could try to whack-a-mole these different issues but since Mercurial's --encoding argument is a global option do you think it would be wise to update ArcanistMercurialAPI::buildLocalFuture() with this?

diff --git a/src/repository/api/ArcanistMercurialAPI.php b/src/repository/api/ArcanistMercurialAPI.php
index e5c2078b..ac72c356 100644
--- a/src/repository/api/ArcanistMercurialAPI.php
+++ b/src/repository/api/ArcanistMercurialAPI.php
@@ -15,7 +15,7 @@ final class ArcanistMercurialAPI extends ArcanistRepositoryAPI {
   protected function buildLocalFuture(array $argv) {
     $env = $this->getMercurialEnvironmentVariables();

For landing the revision can that only be done through the "Land Revision" button? It's greyed out for me.

Re-balance the text line formatting with the new phrasing