Let me send you something for streaming reads.

Just saw your update. Was typing this missive, which is obsoleted by handing me streaming stuff --

On the daemon bit, there should architecturally be two pieces to keep twitter happy

• reading as fast as possible, while reacting nicely to timeouts, keeping only a single connection per account open, generally not opening a bunch of connections, etc
• actually creating issues, running herald rules, whatever

The provided daemon was just going for reading...

Given this break down, I find it wildly intriguing to use something open source for the "reading" bit that would then just call into Conduit to write a lightweight version of what was just read. (https://dev.twitter.com/docs/open-source-examples ) PHP versions all seem broken.

Then some heavier, full-on Phabricator bot could run against this lightweight queue.

D7724 probably adds streaming.

For architecture, I think it can work something like this:

• One FirehoseDaemon or StreamDaemon or whatever, which holds all of the connections we need open.
• When it gets a message, it just inserts a task into the task queue with the data.
• A Taskmaster picks it up and does whatever processing needs to be done.

Seem reasonable?

(Where "all of the connections we need" is theoretically "tons and tons of different services", but in practice is probably just "twitter".)

For parsing the thing, I think you're going to need something like PhutilJSONProtocolChannel which holds a buffer and emits messages. Roughly like this:

$new_data = $future->read();

$messages = $protocol_buffer_object->write($new_data);

foreach ($messages as $message) {
  // queue a new task
}

The idea is the object returns some messages if $new_data completes any message frames, and otherwise it just keeps those bytes inside itself and returns nothing ("no complete messages yet").

Axing it seems fine to me. We don't let you, e.g., change a Git repository into a Subversion one either.

some inlines on stuff i need to fix and a Q or two

QUESTION - as far as I can tell, the existing infrastructure handles keeping one connection open, and each "future" is just a different request on that connection. Correct? (This makes things waaaay easy for me if so as I was thinking of each future as a connection...)

It should work like this, yes, assuming the connections are sequential, although we generally can't tell the difference so I'm not 100% certain this is actually what cURL will do in all cases.

Generally, this all looks great to me. It would be nice to figure out how to share that HTTP parsing code but I agree it's not very easy/obvious. We can sort that out later. (Yell if I missed anything.)