Intecture shouldn't block while waiting for output. Instead it should stream it back to the client.

[rushmorem]

[petehayes102]

👍 Totally agree. This will take a bit of finesse however!

[rushmorem]

This is very easy to do using Tokio. I believe with ZMQ we just need to use the right protocol (PUB/SUB perhaps?).

[rushmorem]

Using .spawn() instead of .output() is the key.

[petehayes102]

Yeah the mechanics are simple, but there's more to it than that - do we establish a new PUB/SUB socket pair for each user connection (which is a pain to firewall) or do we have a permanent PUB sock on the Agent? This has implications for agent.json and config in general (port numbers have to be shared among all parties). Or, is it better to reuse the existing socket and establish a different communication protocol to handle async data streams? Or, should we keep the API as is and pursue an out-of-band streaming option? Lots of broader considerations...

I don't want to spend loads of time on this at the moment because there's a major architectural shift on the horizon which will make this issue obsolete. Though it's probably 12 months away so an interim solution might be appropriate.

[petehayes102]

If you think of passive FTP as an example of the challenges of firewalling ephemeral ports. Also you have to consider authentication for any endpoints that are exposed. Lots of overheads.

[rushmorem]

I'm not familiar with ZMQ and their protocols but with Tokio you can do this without requiring any additional ports. This is typically what we do when writing database clients.

[petehayes102]

I'm assuming that this is achieved over raw sockets sent in the clear?

[rushmorem]

Not necessarily. The connections can still be encrypted.

[petehayes102]

We're basically duplicating what ZMQ+CurveZMQ already give us, except that we'd have to roll our own crypto. It'd be better to rewrite the comm protocol to allow for streaming data to the existing endpoint. The API will need some love too in order to...:

1. Convert the ZMQ socket stream into Box<BufRead>
2. Optionally return Box<BufRead> or appropriate existing type (for example, CommandResult)
3. Still have the ability to return CommandResult, which would contain the exit code, unlike the stream

However, I want to create a new project soon, called build (<-- catchy, no? 😛), which will be responsible for building and running code on demand as a service. Users can request builds as normal via the CLI, but it'd also allow Agents to request them too. This is a big missing feature at the moment where other tools allow hosts to periodically run their configuration scripts instead of relying on the user to do it manually. That project has implications for how we handle this issue, because the comm path between CLI <=> Build (API) <=> Agent is via sockets, so Box<Read> from the API would need to be re-converted into a socket anyway, which is a waste of time.

[petehayes102]

There's been some movement on this. I've been busy of late rewriting Intecture using Tokio and Futures. I've just merged a POC of the new code, though it doesn't use a streaming API. Over the next few days I hope to merge in a fully streaming version of the Command endpoint that will resolve this issue :)

If you're happy to, I'll CC you on the PR so you can review.

[rushmorem]

OK, please do.