HTTP 1.1 Icecast source client

[virtualfunction]

Hi, not a big issue in the grand scheme of things, but just wondered if there is any chance it's easy for you make it so that the Icecast streaming can also use HTTP 1.1 (using chunked encoding) as an extra option on top of HTTP 1.0.

The reason I mention this, is that I'd like to stream to a Node JS server. I'm having all sorts of ball-aches relating to Node because it tries to force HTTP 1.1 for everything.

[toots]

Hi, if you mean sending SOURCE connections using HTTP/1.0, it shouldn't be too hard to hack on. Just edit this line:
https://github.com/savonet/ocaml-cry/blob/master/src/cry.ml#L348
You may also have to fuss with the headers, here for instance:
https://github.com/savonet/ocaml-cry/blob/master/src/cry.ml#L424

Please let us know what works for you and perhaps submit a PR with your changes if you wish. I'd be interested to hear of your project with nodeJS also!

[virtualfunction]

I wish it was just a case of that.

As I understand it, Liquidsoap send HTTP 1.0 headers and, waits for some sort of headers to come back from the Icecast server and then simply writes the data out directly on the socket, and with HTTP 1.0 provided the connection stays open this is seen to be acceptable.

HTTP 1.1 is much stricter when it comes to streaming, and thus it expects a Content-Length header. If this is not possible then it data must use chunked encoding. From what I can tell, Node is dropping the connection after the first network packet of data that comes after the headers. My educated guess is that Node discards the connection as soon as it realises the data is not using chunked encoding as precautionary measure as it doesn't know what to do.

So to summarize I think it's a case of using chunked encoding for the audio data, as well as sending HTTP 1.1 on the first header line and using a Transfer-Encoding: chunked header.

If you want me to stick up a dummy node instance for you to test/run against, I can do so.

[toots]

So chunked transfert doesn't seem too hard to implement, I'll look at it later. Another thing that we could think of is defining a source protocol fit for WebSockets. This way, we could start implementing a source client using node and the browser..

[virtualfunction]

Yes, WebSockets could be handy too :)

I worked out with Node that the reason it was dropping the connection was not just because it wanted HTTP 1.1 chunked encoding, but if chunked encoding isn't used the HTTP specs (RFC 1945, section 7.2 & RFC 2616 section 4.4.5) both state that Content-Length must be provided (except when 1.1 chunked transfer encoding is used). As a result sticking a Content-Length header with an obscurely huge number seems to do the job. Obviously supporting chunked encoding would eliminate the need for this hack.

The only other annoyance with node is that it's HTTP parser is strict over HTTP verbs, I think there are long term plans to change it when they get round to re-working http_parser.c so I've got some hacky node code that replaces SOURCE with POST or PUT at socket level. While this shouldn't affect you, it might be worth knowing if you are also playing with node.

[smimram]

Hi, I have added (untested) support for chunked transfer in savonet/ocaml-cry#1. You should be able to activating by adding ~chunked:true on https://github.com/savonet/liquidsoap/blob/master/src/outputs/icecast2.ml#L311. Please tell us if you get it working!

[virtualfunction]

Thanks, I'll try and get round to testing over the next few days and let you know when I do.

[bradisbell]

@virtualfunction It isn't clear whether or not you are developing a source or a client, but there is no need to modify the Node.js HTTP client, and from what I've heard, it doesn't sound like it will be rewritten anytime soon anyway. A super simple monkey patch is all you need. http://stackoverflow.com/a/12119765/362536

[smimram]

Now that savonet/ocaml-cry#1 was merged. Maybe could we use chunked transfer in Liquidsoap and close this bug report?

[toots]

Sure, that shouldn't be too hard to do. Two things that I think need to happen tho:

• We need to be able to use a different HTTP verb than SOURCE. A PUT or POST should be possible.
• We need to be able to receive those stream in liquidsoap. That'll provide a reference implementation on the receiving side..

I'll look at this ASAP.