Add ability to pass custom http client into node

[silesky]

Package: @segment/analytics-node.

Please see the types/default implementation here: packages/node/src/lib/http-client.ts

This is based on work / research from @MichaelGHSeg!

Use a custom fetch-like implementation with proxy (simple, recommended)

import { HTTPFetchFn } from '@segment/analytics-node'
import axios from 'axios'

const httpClient: HTTPFetchFn = async (url, options) => {
  return axios({
    url,
    proxy: {
        protocol: 'http',
        host: 'proxy.example.com',
        port: 8886,
        auth: {
          username: 'user',
          password: 'pass',
        },
      },
    ...options,
  })
}

const analytics = new Analytics({
  writeKey: '<YOUR_WRITE_KEY',
  httpClient,
})

Augment the default HTTP Client

import { FetchHTTPClient, HTTPClientRequest  } from '@segment/analytics-node' 
 
class MyClient extends FetchHTTPClient {
  async makeRequest(options: HTTPClientRequest) {
    return super.makeRequest({
         ...options, 
         headers: { ...options.headers, foo: 'bar'  }
      }})
  }
}

const analytics = new Analytics({ 
  writeKey: '<YOUR_WRITE_KEY>', 
  httpClient: new MyClient() 
})

Option 2: Completely override the full HTTPClient (Advanced, you probably don't need to do this)

import { HTTPClient, HTTPClientRequest } from '@segment/analytics-node'

class CustomClient implements HTTPClient {
  async makeRequest(options: HTTPClientRequest) {
    return someRequestLibrary(options.url, { 
      method: options.method,
      body: JSON.stringify(options.data) // serialize data
      headers: options.headers,
    })
  }
}
const analytics = new Analytics({ 
  writeKey: '<YOUR_WRITE_KEY>', 
  httpClient: new CustomClient() 
})

[changeset-bot]

🦋 Changeset detected

Latest commit: 84ac43b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@segment/analytics-node	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[danieljackins]

Was this added accidentally?

[silesky]

Good catch!

[MichaelGHSeg]

Did you have a specific fetch replacement library in mind that doesn't support 'signal' here? I have questions about simultaneously allowing access to tinker with our internals and passing around non-standard objects.

[silesky]

All fetch libraries support AbortSignal, but not all HTTP Libraries do (e.g Request).

That's why signal is present in the FetchHTTPClient class/interface but not the HTTPClient interface -- it's fetch specific.

Does this answer your concern? I am not entirely sure I am following you on you standard objects / monkeying with internals -- maybe you can elaborate on what part of the examples (see PR description) you feel is problematic. I wanted to allow a couple layers extensibility -- users can swap out the default fetch client, or, if they need, replace the entire http client with something that doesn't use fetch / have any relationship with the fetch interface at all.

[MichaelGHSeg]

I just wanted to know what the impetus was. Figured you had a reason for moving some of the fiddly bits into the user replaceable thing that they then have to repeat.

[chrisradek]

What's the reasoning for changing this from a string to an object? Seems like it saves us from having to do a JSON.parse() in our tests, but I'd also expect an HTTP body to be a string if I'm specifically listening for HTTP requests/want to know exactly what got sent to the service.

(Also this would technically be a breaking change going from a string to an object - someone already doing a JSON.parse() would now encounter an error, though I don't expect many users of this API yet)

[silesky]

The serialization part is the client's responsibility, and I didn't want the http client to have take an emitter -- so the emitting part now happens in the publisher code (before the request is passed to the http client). I saw no downsides to this change, and only upsides.

[chrisradek]

I've seen a lot of different HTTP-related timeouts (timeout to establish connection, socket idle timeout, time for response headers, time for full response body) - versus browsers that were way more limited with what XHR provided. Can we be more detailed on what this timeout covers?

[silesky]

Good point. We currently support httpRequestTimeout which is "timeout for full response body" -- I can rename it to match our config.

[chrisradek]

It feels a little weird for an HTTP client to have data instead of body. (headers and query params are data too!)

How come we accept an object rather than a string/byte array here? I would expect the HTTP Client to only be concerned with making the HTTP request, not transforming the body.

[oscb]

I can see the possibility of wanting to change the data before sending to the server and hence wanting to have the object and not parse from JSON.

But we have already a few places to do these operations, so I agree with this, let's keep the HTTP Client dumb to discourage having too much logic here

[silesky]

The reason I used data here is:

1. to distinguish the higher level representation of the body (JSON object) from the serialized body (string) -- fetch takes a string
2. To allows people to use a custom JSON serializer if they want.
3. To allow people to make modifications to "data/payload" without deserializing and re-serializing (expensive)
4. This is what axios uses, so it's a comment convention

The HTTPClient is meant to be generic in terms of what we support (for example, if we didn't support timeout, that wouldn't be an option).

PS: this pattern is also used by Stripe in their SDK's HTTPClient.

PPS:The shape will be { batch: [{ ..... }] }

[chrisradek]

Kind of wonder if we really need ok - it's just a boolean that is true if status if 200-299. We get status already so could check that ourselves (then if someone uses an HTTP client that doesn't return ok, they don't have to worry about it)

[chrisradek]

Ah I see, timeout is time to response headers (to answer my earlier question 😄)

[oscb]

Looks good. I +1 on some of the comments before, I think the most important one would be to remove the ok from the responses to prevent weird issues where it's missed to turn on (or the other way around.

[oscb]

I can see the possibility of wanting to change the data before sending to the server and hence wanting to have the object and not parse from JSON.

But we have already a few places to do these operations, so I agree with this, let's keep the HTTP Client dumb to discourage having too much logic here

[oscb]

👍