IPIP-431: Opt-in Extensible CAR Metadata on Trustless Gateway

src/http-gateways/trustless-gateway.md

@@ -80,7 +80,8 @@ Below response types SHOULD be supported:
 - [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car)
   - Disables IPLD/IPFS deserialization, requests a verifiable CAR stream to be
     returned, implementations MAY support optional CAR content type parameters
-    (:cite[ipip-0412]) and the explicit [CAR format signaling in HTTP Request](#car-format-signaling-in-request).
+    (:cite[ipip-0412]), the explicit [CAR format signaling in HTTP Request](#car-format-signaling-in-request)
+    and the optional [metadata block](#meta-content-type-parameter).
 
 - [application/vnd.ipfs.ipns-record](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record)
   - A verifiable :cite[ipns-record] (multicodec `0x0300`).
@@ -301,6 +302,112 @@ of their presence in the DAG or the value assigned to the "dups" parameter, as
 the raw data is already present in the parent block that links to the identity
 CID.
 
+## `meta` (content type parameter)
+
+The `meta` parameter allows clients to request the server to include additional metadata about the CAR along with the response body.
+
+The value of this parameter includes both the location where the metadata is given (e.g. `eof`) as well as the type of data received (e.g. `json`) separated by a `+`, to give a value such as `meta=eof+json`
+
+When the location parameter is set to `eof`, which is currently the only supported value, the server SHOULD respond with the following response body:
+
+```
+<Response body as CARv1 stream> <0x00 byte> <Metadata>
+```
+
+The only supported value for the data type parameter is `json`. This signifies that the metadata MUST be a JSON object.
+
+This parameter MUST only be used with CAR `version=1`.
+
+When the parameter is not set or does not equal `eof+json`, the server SHOULD not add any extra blocks to the response, neither the 0x00 byte nor any metadata.
+
+When `meta=eof+json`, the JSON object SHOULD conform to the following [JSON schema](https://json-schema.org/).
+
+```json
+{
+  "type": "object",
+  "properties": {
+    "data": {
+      "type": "object",
+      "description": "Properties of the response"
+    },
+    "error": {
+      "type": "string",
+      "description": "Error message"
+    },
+    "sig": {
+      "type": "string",
+      "description": "A signature, using the server's Ed2559 identity, over the `data` object serialized as JSON."
+    },
+    "required": []
+  }
+}
+```
+
+The properties object can include any fields that the server would like to implement. The following JSON schema explicitly mentions certain properties fields in order to reach a convention on their definition as they have existing use cases.
+
+```json
+{
+  "type": "object",
+  "properties": {
+    "car_bytes": {
+      "description": "The total byte length of the CAR stream (excluding the 0x00 byte and the metadata block)",
+      "type": "integer"
+    },
+    "data_bytes": {
+      "description": "Total byte length of the flat file before it was encoded into a CAR file",
+      "type": "integer"
+    },
+    "block_count": {
+      "description": "Total number of blocks present in the CAR stream (excluding the 0x00 byte and the metadata block, but including duplicates when present)",
+      "type": "integer"
+    },
+    "car_cid": {
+      "description": "A hash of the CAR stream giving a CIDv1 with 0x0202 codec",
+      "type": "string"
+    },
+    "b3checksum": {
+      "description": "A Blake3 hash (checksum) of the CAR stream (excluding the 0x00 byte and the metadata block). The value should be serialized as a multihash with multibase prefix, preferably using Base58 encoding.",
+      "type": "string"
+    },
+    "content_path": {
+      "description": "The url path in the request as executed by the gateway, e.g. `/ipfs/bafy1234/cat.jpg`. The query string MUST BE stripped from the path.",
+      "type": "string"
+    },
+    "dag_params": {
+      "description": "A map with DAG params like dag-scope, entity-bytes from [IPIP-402](https://specs.ipfs.tech/ipips/ipip-0402/)",
+      "type": "object",
+      "properties": {
+        "dag-scope": {
+          "description": "See [IPIP-402](https://specs.ipfs.tech/ipips/ipip-0402/) for the definition",
+          "type": "string"
+        },
+        "entity-bytes": {
+          "description": "See [IPIP-402](https://specs.ipfs.tech/ipips/ipip-0402/) for the definition",
+          "type": "string"
+        }
+      },
+      "required": []
+    },
+    "car_params": {
+      "description": "A map with CAR content type params like order and dups from [IPIP-412](https://specs.ipfs.tech/ipips/ipip-0412/)",
+      "type": "object",
+      "properties": {
+        "order": {
+          "description": "See [IPIP-412](https://specs.ipfs.tech/ipips/ipip-0412/) for the definition.",
+          "type": "string"
+        },
+        "dups": {
+          "description": "See [IPIP-412](https://specs.ipfs.tech/ipips/ipip-0412/) for the definition.",
+          "type": "string"
+        }
+      },
+      "required": []
+    }
+  },
+  "required": []
+}
+```
+
 ## CAR format parameters and determinism
 
 The default header and block order in a CAR format is not specified by IPLD specifications.

src/ipips/ipip-0431.md

@@ -0,0 +1,197 @@
+---
+title: "IPIP-0431: Opt-in Extensible CAR Metadata on Trustless Gateway"
+date: 2023-08-08
+ipip: proposal
+editors:
+  - name: Miroslav Bajtoš
+    github: bajtos
+    affiliation:
+        name: Protocol Labs
+        url: https://protocol.ai/
+  - name: Patrick Woodhead
+    github: patrickwoodhead
+    affiliation:
+        name: Protocol Labs
+        url: https://protocol.ai/
+relatedIssues:
+  - https://github.com/filecoin-project/boost/issues/1597
+order: 431
+tags: ['ipips']
+---
+
+## Summary
+
+Define an optional enhancement of the CARv1 response that allows a Gateway server to provide
+additional metadata about the CARv1 stream. Introduce a new content type that allows the client
+and the server to signal or negotiate the inclusion of extra metadata.
+
+## Motivation
+
+SPARK is a Filecoin Station module that measures the reputation of Storage Providers by periodically
+retrieving a random CID. Since both SPs and SPARK nodes are permissionless, and Proof of Retrieval
+is an unsolved problem, we need a way to verify that a SPARK node retrieved the given CID from the
+given SP. To enable that, we want the Trustless Gateway serving the retrieval request to include a
+retrieval attestation after the entire response was sent to the client.
+
+Aside from this specific use case, the IPFS Ecosystem at large has no reliable
+mechanism to signal that a CAR file transmission over HTTP completed successfully.
+
+We need such signalling mechanism in order to be able to use CARs as a way of serving streaming
+responses for queries. One way of solving this problem is to append an extra block at the end of the
+CAR stream with information that clients can use to check whether all CAR blocks have been received.
+
+## Detailed design
+
+CAR content type
+([`application/vnd.ipld.car`](https://www.iana.org/assignments/media-types/application/vnd.ipld.car))
+already supports optional parameters like `version` and `order`, which allows
+HTTP client to opt-in via `Accept` header and Gateway to indicate via
+`Content-Type` header which CAR flavor is returned with the response.
+
+The proposed solution introduces a new parameter for the CAR content type in HTTP requests
+and responses: `meta`.
+
+The `meta` parameter allows clients to request the server to include additional metadata about the CAR along with the response body.
+
+The value of this parameter includes both the location where the metadata is given (e.g. `eof`) as well as the type of data received (e.g. `json`) separated by a `+`, to give a value such as `meta=eof+json`
+
+When the location parameter is set to `eof`, which is currently the only supported value, the server SHOULD respond with the following response body:
+
+```
+<Response body as CARv1 stream> <0x00 byte> <Metadata>
+```
+
+The only supported value for the data type parameter is `json`. This signifies that the metadata MUST be a JSON object.
+
+This parameter MUST only be used with CAR `version=1`.
+
+When the parameter is not set or does not equal `eof+json`, the server SHOULD not add any extra blocks to the response, neither the 0x00 byte nor any metadata.
+
+This results in a example content type of `application/vnd.ipld.car;version=1;meta=eof+json`
+
+See [CAR `meta` (content type parameter)](/http-gateways/trustless-gateway/#car-meta-content-type-parameter)
+in Trustless Gateway specification for more details.
+
+## Design rationale
+
+The proposal introduces a minimal change allowing Gateways and retrieval clients to explicitly opt
+into receiving additional metadata block at the end of the CAR response stream.
+
+The metadata block is designed to be very flexible and able to support new use-cases that may arise
+in the future.
+
+### User benefit
+
+- Clients of trustless gateways can use the fields from the metadata as an attestation that they
+performed the retrieval from the given server.
+
+- For example, the metadata block could include a `car_bytes` field, the byte length of the CAR stream (excluding the metadata block). This would allow clients to verify whether they received all CAR
+bytes, which provides a backward-compatible solution for the [CARv1 streaming problem](https://github.com/ipfs/specs/pull/332) until new CAR version is introduced.
+
+- As another example, the metadata object includes the `error` field, allowing the server to pass back additional information about why the response is an error, such as why the CAR stream was incomplete.
+
+- In the SPARK use case, retrieval clients would like to prove they have retrieved an entire file from a specific retrieval provider that has implemented the trustless gateway spec. The additional metadata block allows checksums and signatures to be passed along with the data, allowing the retrieval client to create a proof of correct retrieval.
+
+- The metadata `sig` field SHOULD also be populated, returning a signature, using the server's Ed2559 identity, over the metadata properties object. This allows gateway clients to submit the metadata block as an attestation of retrieval that 3rd parties can verify.
+
+### Compatibility
+
+The new feature requires clients to explicitly ask the server to include the extra block via `Accept` header,
+therefore the change is fully backwards-compatible for all existing gateway clients.
+
+Gateways receiving requests for the CAR content type can ignore the `meta` parameter they don't
+support and return back a response with one of the CAR content types they support. This makes the
+proposed change backwards-compatible for existing gateways too.
+
+All metadata fields are optional to allow different applications to experiment with different metadata. Future IPIPs may standardize metadata fields that are observed to be widely used.
+
+### Security
+
+#### Zero-length-block insertion attacks
+
+The idea of using the zero-length block (a single byte `0x00`) to signal the end of the CARv1 stream has been already considered in the past.
+
+> CARv1 is nicely sectioned, such that each section has a specific length, you know when it ends.  In the [ZeroLengthSectionAsEOF](https://pkg.go.dev/github.com/ipld/go-car/v3#ZeroLengthSectionAsEOF) mode, when it gets to a new section and reads a 0x00, i.e. zero length (sections are prefixed with a length varint), it treats that as the end of the CAR. So all it takes with this turned on is to attach a 0x00 to the end of a stream and you get your EOF.
+>
+> The background for this is the power-of-two padding that is needed for a Filecoin sector — stick a CAR into the sector and fill it out with zeros but have no way of saying that the CAR is x-bytes long; hence the need for an EOF signal, which is this.
+
+However, introducing a `0x00` into CARv1 spec would create a security vulnerability:
+- Tools and services not aware of this new semantics will happily accept a CARv1 payload containing zero-length blocks in the middle.
+- Tools and services treating `0x00` as EOF will discard the remaining blocks in such CARv1 file
+ after encountering the zero-length block.
+
+ Our proposal avoids this attack vector:
+ - It does not change the current semantics of CARv1. Zero-length blocks remain invalid.
+ - Instead, we treat the response body as a new container format combining the CARv1 file with additional data.
+ - Clients must explicitly request this new container format. Existing clients not aware of the new metadata will not receive responses in the new format.
+
+#### Denial of Service attacks
+
+Computing the signature for the metadata blcok has a non-negligible performance cost. To mittigate DoS attacks, we designed the metadata to be highly cacheable. When a gateway receives two requests for the same content, it can return the same metadata block in both responses, including the signature. This allows gateway operators to deploy a traditional caching layer operating at the HTTP protocol, the cache does not need to understand any specifics of IPFS and Trustless Gateway protocols.
+
+### Alternatives
+
+#### HTTP Trailers
+
+Instead of adding a new content type argument, we were considering sending the additional metadata
+in HTTP response trailers. Unfortunately, HTTP trailers are not widely supported by the ecosystem.
+Nginx proxy module discards them, [browser `Fetch API` does not allow JS clients to access trailer
+headers](https://github.com/mdn/browser-compat-data/issues/14703), neither does the Rust `reqwest` client.
+
+#### New Content-Type
+
+We could introduce a new content type that is not CARv3, but a thin envelope
+around CARv1 with purpose of streaming over HTTP (e.g. `Content-Type:
+application/vnd.ipld.car-stream`).
+
+It would have three fields:
+- `car-stream-header` (optional DAG-CBOR)
+- `car` (same as `application/vnd.ipld.car;version=1`)
+- `car-stream-end` (optional DAG-CBOR)
+
+This will be enough to append DAG-CBOR  manifest at the end of the stream. It
+would be effectively the same CAR byte stream, but with different
+`Content-Type`.
+
+Upside of this solution:
+
+- does not require registering new codec, or mixing data plane with control
+  plane, no sniffing the last DAG-CBOR block
+
+Downsides of this solution:
+
+- maintenance cost, requires duplicating of all CAR-related tests and features
+- ecosystem opportunity cost, in creating new content type, we increase
+  cognitive overhead for everyone working with IPFS over HTTP
+- no backward-compatible interop with existing tools and gateways that only
+  speak `application/vnd.ipld.car`
+- distracts us away from working on things like large blocks and CARv3
+
+#### Create CARv3
+
+We could admit we've clearly hit limitation of what we can do with HTTP and CARv1 and CARv2 and stop abusing existing CARv1 by mixing data plane with control plane.
+
+Spend energy on creating CARv3 that solves the problems from "Motivation" section and more:
+- optional index or key-value metadata before or after data
+- native truncation detection and standardized error handling and passing during streaming
+- support for things like [Large Blocks](https://discuss.ipfs.tech/t/supporting-large-ipld-blocks/15093/)
+
+TODO: link to some public artifact about CARv3
+
+#### Create a new multicodec for this metadata block
+
+Initially, we proposed to create a new multicodec for this metadata block called `car-metadata`. This was ruled out due to some concerns that you can find documented [here](https://github.com/multiformats/multicodec/pull/334#issuecomment-1668086641).
+
+#### Using CBOR instead of JSON for the metadata block
+
+We could use CBOR instead of JSON for the metadata block. However it was [decided](https://github.com/ipfs/specs/pull/431#issuecomment-1719634928) to opt for user readibility over number of bytes since CBOR doesn't greatly reduce the number of bytes in a key value map compared with JSON.
+
+## Test fixtures
+
+TBD
+
+Using one CID, request the CAR data using various combinations of content type parameters.
+
+### Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).