See PHI1378, which is interested in "at least N", for N = 2.

There was one issue with this: bin/host stop --instance X (which is rarely used, and has no prior automated callers) used --force, which stops all daemons on the host. The recent changes to the --force flag in D20601 also had an indirect impact here. I fixed this and redeployed the repo tier.

See downstream https://phabricator.wikimedia.org/T230001 for a similar issue on project pages.

The repro case is something like:

In D20221#262095, @epriestley wrote:

Oh, good catch. Yeah, that would be an improvement.

use cases seem very weak.

Oh, good catch. Yeah, that would be an improvement.

This thread suggests that some version of ssh-keygen is sensitive to trailing whitespace in private keys:

you can add libreoffice/soffice to your list. last time we had to do sth like this we used soffice --headless --convert-to pdf
not really better than running a browser though... in our use case we needed to use merchant uploaded doc/docx templates in combination with our data. so even less options.
but yeah most options around pdf generation are clunky.

Yeah, I'm really hoping to not require us to run an entire browser or depend on an external service to generate PDFs. The approaches I found in my research were:

Just my two cents here, but chrome has an headless mode that allows to "print to pdf" : https://developers.google.com/web/updates/2017/04/headless-chrome

When I apt-get install wkhtmltopdf (or something like that) on aux001, which is Ubuntu 14, I get this:

I've worked with wkhtmltopdf several times, and I don't remember it actually requiring any special setup - in all cases, I basically dropped the binary in the machine (Although it's possible all my machines had X11 built in for some reason? I've never intentionally installed X11).

Just some observations after popping open PDF invoices which have shown up in my inbox recently:

See PHI1367, which featured a specific case where git repack unambiguously did something good instead of being magic fairy dust that we sprinkle around to ward off demons.

This is also more "while I'm here", but there's no web UI way to void an invoice right now. It would be nice to have a "Void Invoice" button for staff instead of requiring database fiddling, since this comes up occasionally.

Per T13366, I'm just going to get rid of this behavior for now, with the expectation that we'll soon begin generating side-channel emails for "Billing Department Email Address" under the umbrella of T8389 and could put "Merchant Emails" on a similar channel if we want.