This broadly looks reasonable, some discussion inline.

I'd guess the signature algorithm needs to look something like:

• Build up some object with the parameters and also all the metadata.
• Encode the object as a value, with these rules:
  • If the value is an object/dictionary, sort the keys. Write "O", then the number of keys, then a colon, then encode each key and each value in order.
  • If the value is an array, write "A", then the number of elements, then a colon, then encode each value in order.
  • If the value is a string, write "S", then the number of bytes, then a colon, then the bytes of the string.
  • If the value is an integer, write "I", then the number of digits in the integer, then a colon, then the digits of the integer.
  • If the value is null, write "N", then a colon.
  • If the value is false, write B0:.
  • If the value is true, write B1:.
• Sign the resulting string.

I think this should be pretty easy to implement in another language, and easy to understand/debug (no ambiguity around which characters get urlencoded, for example, and theoretically fairly easy to implement in-place in memory if you're into that kind of thing), and binary safe, while giving every request a unique encoding.

(Or we could look if there's a standard way that everyone else does it, but the need to be binary-safe and support signatures across encodings seems likely to make things complicated and prevent reuse of any standard signing technique.)

bencode is similar and unambiguous, but doesn't have null or bool types.

We might be able to use protobuf. That has implementations in most languages and I think it should support all the types we need.

(Then we just need to document the ordering semantics since fields are by numeric ID)

We need at least one binary encoding as an alternative to JSON anyway, although the id-encoding of protobuf might be less convenient than BSON or messagepack.

I think messagepack is probably very close, although some of the encodings aren't human-readable (minor) and array ordering isn't part of the spec (easy to compensate for by sorting before or during encoding).

I guess messagepack technically has ambiguous encodings, like the integer 1 can be encoded as either 0x01 or 0xcc 0x01, but presumably no encoders do this and there's an unambiguous minimal encoding.

Actually, if you have the private key, you should be able to generate the correctly formatted public key with:

openssl rsa -in <private-keyfile> -pubout

...without needing to use some other intermediate host to run a good ssh-keygen. But that requires the private key, so we can't use it on the host (which only has knowledge of the public key).

Is there no newer binary for ssh-keygen on OSX / version through homebrew or something like that? Presumably it's possible to grab the OpenSSH code and build a more modern version that supports pcks8 (and then no caching is needed for a single platform unlikely to be used in production).

If you wanted to host a cluster on OSX you'd have to do that, yeah. For development, it's probably easier to just prime the cache.

I don't think it's a version problem, I think it's an issue with Apple's build. Mavericks (where I'm hitting the issue) has 6.2p2, which is recent.

I also found a couple of references to it possibly working in 6.1 and being broken in 6.2, but this server (secure.phabricator.com) also has 6.2p2 and everything works fine, which leads me toward believing this is an Apple thing.

(We want the cache no matter what, I don't want to shell out to ssh-keygen on every Conduit request, so the only special code would be priming it artificially.)

Oh right, because the pcks8 version isn't being stored when the Almanac device is registered (like in the previous diffs), so it has to be calculated + cached at runtime anyway.

Yeah. That cache could technically go in the SSH key table (and maybe I'll just put it there) but we don't have the cache anywhere else now since I unified SSH key storage across users and devices.

Oops, forgot to get you in on this action.

I know 3ish Mac OSX developers... :D I think it would be neat if we had some command like

bin/almanac gen-and-prime-ssh-key <public-key>

this command should fatal horribly on mac osx, letting you know it won't work, and then otherwise it should kind of collapse steps 2 and 3 from above.

Just to respond to this specifically, since I'm about to shoot a diff out for it:

• If your ssh-keygen works, we're just doing the whole thing inline, so you don't have to do any work -- everything works automatically, and you don't have to run any scripts. This is everywhere except OSX.
• If you're on OSX, ssh-keygen is busted, so writing a CLI script that says "ssh-keygen is busted" wouldn't be very helpful. The automatic thing is going to fail with a message "ssh-keygen is busted, go run xxxxx" instead, and then you'll have to go run that. I think there's no real way around this.