what, if any other providers are there than Twilio for this sort of thing?

Twilio is the only one I'm actively aware of / recognize the brand name of. We might want a "drop in a hole" adapter for testing, but I don't think we need to support anything else out of the gate.

Looks like there are a ton of them in general, though. I don't recognize any of these names other than Twilio (and the major carriers) offhand:

http://blog.mashape.com/post/56272188360/list-of-50-sms-apis

should this be a full blown app? (as opposed to some code in the infrastructure folder)

I don't think so -- I'd guess that (like MetaMTA) all the info ends up being too sensitive to put on the web. I'd probably shove the code in infrastructure/ or metamta/ and then make a CLI bin/sms to provide a list, a send-test, etc.

any big changes needed here based on what you see?

Nope, this looks great to me.

would be good to know what other providers do before making this work (e.g. what can I assume the message ID will look like?)

I'd probably just guess / go with your gut, it should be pretty easy to update this table later. We can GC it after 90 days or something like we do with mail, so it won't be too big.

As such, re-jiggered this a bit to have a poll inside the "doWork" method, and before checkin I think I need to retry if and only if the external provider has said "we definitely failed." thoughts?

Ideally, we should distinguish between "this is failed forever" (e.g., bad "to" number) and "this is appropriate to retry" (e.g. rate limit).

Thinking about this, for v0, I kind of thing we should maybe just never retry. A lot of mail we really want to try hard to deliver, but the only SMSes we're going to send right now are multi-factor SMSes, which are no good if they take more than a minute or so to show up. Generally, SMSes are likely to be time sensitive, and delivering them much later is probably annoying at best (your phone blows up) and confusing at worst (you get a bunch of SMS codes which aren't valid anymore, or alerts for things that already happened).

So the logic would look like:

• If we get a good request, record the queued/sending status but just "return;" from the task and call it a success, assuming it will do the right thing. Presumably, it is very rare and/or nonexstent that Twilio queues a message and then fails to deliver it for reasons where retries would be appropriate.
• If we get a bad request, no matter what, just throw PermanentFailure so we don't retry.

Then based on our experience in the real world we can see if that's good enough or not? My guess is that it will be like 98% good, and then adding retries for specific errors (e.g., rate limit) will get us to like 99%+ good.

any pro-tips for debugging daemons and workers? finding it hard to surface phlog, echo, etc and will probably write to a temp file if there's no better way to surface info.

Stop all the daemons, then run bin/phd debug taskmaster to guarantee that process gets to process everything. Then all the log(), etc., will show up in your console.

in doWork, if everything went okay, you just return? what if things went badly but not "fail permanently" badly? is there a way to sleep for a bit then hop in the top of the doWork method without that counting as a try?

You can throw new PhabricatorWorkerYieldException(15); to mean (approximately): don't increase the failure count, and retry this task in about 15-ish seconds. (It may take longer than 15 seconds before the task gets retried, but that should be OK here I think).

Also, we use "multiplex" elsewhere but I think your usage is actually correct and they should all be "demultiplex". I think I just never looked it up, I'll maybe fix that at some point.