A generic HTTP Pipes client

[v1gnesh]

Hi everyone,

Loving data platform week, and a special call-out to @slopp's 30 days of orchestration is so excellent. Your design team is so good that it's pleasing to even look at the product and the presentation material. Wishing y'all grander & grander successes.

A qq - what do y'all think about a generic http client as a pipes operator?
It'll help cover a lot of ground via Pipes without needing to create separate clients.

[danielgafni]

Hey @v1gnesh, thanks for the kind words!

By "operator" do you mean a PipesClient? What is your vision for it? Pipes invocations involve running something remotely, for example, an OS process, a Kubernetes pod, or a Spark job. What role would a "generic http client" play here, what exactly would this abstraction represent? ok you did not mean that :)

Or are you talking about creating an HTTP endpoint in Dagster which could be used to send Pipes events over the network from remote processes? If so, I see 2 ways how this could be made possible:

1. We could potentially make a PipesHTTPMessageReader and PipesHTTPMessageWriter Pipes components. PipesHTTPMessageReader could start a web server from the Dagster process and PipesHTTPMessageWriter would send events (invoked in the remote process) to this server via http requests. I can see this working, although making them available to each other over the network would be the user's responsibility (it should be more simple in some environments like Kubernetes and more complicated in others).

2. In the future, we could maybe have a similar endpoint available as part of the Dagster deployment. Perhaps, via GraphQL. This could simplify networking setups communication between these Pipes. I vaguely recall @schrockn mentioning something about such possibility, but I don't remember any details.

I wonder how big would be the demand for (1) in our community 🤔

[v1gnesh]

Hi @danielgafni, yes, sorry I mean a PipesClient.

On point 2, if at all possible, please consider prefering HTTP (or even a Pythonic way) over GraphQL... for the client-facing interface.

A generic HTTP listener (in Dagster's side) and a generic HTTP client helper(s) can help smoothen onboarding of various integrations. OS process, Kubernetes pod, etc. are prevalent components; so it's normal that either Dagster Labs or the community builds / has built explicit pipes client(s) for them.
But there are a lot more generic systems/services that can give info over HTTP.
Having a generic HTTP pipe will allow all of them to communicate with Dagster in one sweep, with little case-specific customizations.
Further, this will help with the orchestration federation vision too, as the HTTP client/server can potentially expand over time to allow comms with different sub-systems of Dagster (sending asset defs over HTTP).

[schrockn]

ewYes I have spoken about this in the past. There are a few levels of this.

• Standardizing on an OpenAPI client interface even in cases where the transport is not http. This would provide nice consitency and a famiilar standard across pipes implementations across all languages.
• Then there is the notion of using HTTP directly. This has desirable properties in a lot of cases, but would also end up with challenges. This is namely because with the way that Dagster works the orchestration process needs to be able to process the events coming in from the external process. There are paths forward to doing that I see:
  • Each run worker would need to host an HTTP server
  • The remote pipes process would need to be an http server
  • The events emitted by the remote process would have to be enqueued somewhere and then processed by the run worker.

All of those approaches have their challenges.

[v1gnesh]

Each run worker would need to host an HTTP server

https://github.com/aio-libs/aiohttp

The remote pipes process would need to be an http server

Why is this?

The events emitted by the remote process would have to be enqueued somewhere and then processed by the run worker.

https://github.com/aio-libs/aiojobs

I realize it's not as easy as plopping these in. Sharing these, hoping that, if you find time, you can tell us more about why these may/may not help that much.