Feat/car iter

[coryschwartz]

This is intended to fix #8323

It uses a new method in go-car to prevent the need for buffering the entire dag. Each node is pushed to the client, who can examine the error for each node and make a decision about what to do about it.

[rvagg]

@coryschwartz I have a local branch editing export.go to switch from using mdag to ipld-prime's selector walk function and uses SelectiveCar instead.

At this stage I think it would be unfortunate to further embed the use of mdag and its associated Walk functions into CAR use rather than be migrating to ipld-prime's traversal package. I got a bit distracted with some selector work in Lotus recently but perhaps I could get my branch pushed and you could see if that approach solves the problem rather than introducing further functionality to go-car? I'll try and get that done in the next day or so.

[aschmahmann]

@lidel @rvagg feel free to disagree, but my understanding is that this is making a breaking change to the ipfs dag export HTTP API and I'm not sure that this is worth it. By popular request we started exposing /api/v0/dag/export via the gateway endpoints in go-ipfs v0.9.0. This potentially means that there is a bunch of code out there using this API endpoint that could break.

I understand that browsers have problems with HTTP trailing headers, but do we have other options here? I guess we could introduce a new flag or command that has the different behavior, but I'm wondering if we've got more friendly alternatives.

[rvagg]

#8506 is the alternative I was working on, I don't know if it solves for the error this is dealing with though, I'm mainly concerned about moving further away from being able to use go-ipld-prime selectors for this API

[aschmahmann]

Closing based on not wanting to make breaking changes here that are inconvenient for users who want to just pipe car file outputs as described in #8323 (comment) and #8503 (comment)