Implement the standard HTTP for BN #1434

paulhauner · 2020-08-03T01:59:28Z

Description

This issue tracks the implementation of the standard Eth2 API at https://github.com/ethereum/eth2.0-APIs.

Note: this issue is assuming that #1569 has already been merged into master.

Overall status

The standard is mostly implemented in #1569, however there are still some endpoints missing. Most notably, SSE endpoints have not been implemented.

Per-Endpoint

This section documents the progress of each API endpoint. An endpoint should only be marked as complete if it is 100% complete, with nothing else to do.

Beacon

Debug

/eth/v1/debug/beacon/states/{state_id}
/eth/v1/debug/beacon/heads

Node

/eth/v1/node/identity
- Partially implemented, missing fields.
- [Merged by Bors] - Shift metadata to the global network variables #1631 adds one of the required fields.
/eth/v1/node/peers
/eth/v1/node/peers/{peer_id}
/eth/v1/node/version
/eth/v1/node/syncing
/eth/v1/node/health

Config

/eth/v1/config/fork_schedule
/eth/v1/config/spec
/eth/v1/config/deposit_contract

Validator

/eth/v1/validator/duties/attester/{epoch}
/eth/v1/validator/duties/proposer/{epoch}
- Only returns duties for the current epoch to make caching easier. It would be nice to add a non-cached path for historical requests, too.
eth/v1/validator/blocks/{slot}
/eth/v1/validator/attestation_data
/eth/v1/validator/aggregate_attestation
/eth/v1/validator/aggregate_and_proofs
/eth/v1/validator/beacon_committee_subscriptions

Events

/eth/v1/events

Known Issues

The text was updated successfully, but these errors were encountered:

## Issue Addressed - Resolves #1550 - Resolves #824 - Resolves #825 - Resolves #1131 - Resolves #1411 - Resolves #1256 - Resolve #1177 ## Proposed Changes - Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes. - Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate. - Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate. - Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc. - Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client. - Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties. - Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`). - Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API. - Add functions to `BeaconChainHarness` to allow it to create slashings and exits. - Allow for `eth1::Eth1NetworkId` to go to/from a `String`. - Add functions to the `OperationPool` to allow getting all objects in the pool. - Add function to `BeaconState` to check if a committee cache is initialized. - Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`. - Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response. - Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality). - Impl `Display` and `FromStr` for several BLS fields. - Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do. ## Additional Info - See #1434 for a per-endpoint overview. - Seeking clarity here: ethereum/beacon-APIs#75 ## TODO - [x] Add docs for prom port to close #1256 - [x] Follow up on this #1177 - [x] ~~Follow up with #1424~~ Will fix in future PR. - [x] Follow up with #1411 - [x] ~~Follow up with #1260~~ Will fix in future PR. - [x] Add quotes to all integers. - [x] Remove `rest_types` - [x] Address missing beacon block error. (#1629) - [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix - [x] ~~Follow up with validator status proposal~~ Tracked in #1434 - [x] Unify graffiti structs - [x] ~~Start server when waiting for genesis?~~ Will fix in future PR. - [x] TODO in http_api tests - [x] Move lighthouse endpoints off /eth/v1 - [x] Update docs to link to standard ## Blocked On - ~~Blocked on #1586~~ Co-authored-by: Michael Sproul <[email protected]>

- Resolves #1550 - Resolves #824 - Resolves #825 - Resolves #1131 - Resolves #1411 - Resolves #1256 - Resolve #1177 - Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes. - Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate. - Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate. - Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc. - Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client. - Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties. - Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`). - Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API. - Add functions to `BeaconChainHarness` to allow it to create slashings and exits. - Allow for `eth1::Eth1NetworkId` to go to/from a `String`. - Add functions to the `OperationPool` to allow getting all objects in the pool. - Add function to `BeaconState` to check if a committee cache is initialized. - Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`. - Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response. - Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality). - Impl `Display` and `FromStr` for several BLS fields. - Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do. - See #1434 for a per-endpoint overview. - Seeking clarity here: ethereum/beacon-APIs#75 - [x] Add docs for prom port to close #1256 - [x] Follow up on this #1177 - [x] ~~Follow up with #1424~~ Will fix in future PR. - [x] Follow up with #1411 - [x] ~~Follow up with #1260~~ Will fix in future PR. - [x] Add quotes to all integers. - [x] Remove `rest_types` - [x] Address missing beacon block error. (#1629) - [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix - [x] ~~Follow up with validator status proposal~~ Tracked in #1434 - [x] Unify graffiti structs - [x] ~~Start server when waiting for genesis?~~ Will fix in future PR. - [x] TODO in http_api tests - [x] Move lighthouse endpoints off /eth/v1 - [x] Update docs to link to standard - ~~Blocked on #1586~~ Co-authored-by: Michael Sproul <[email protected]>

michaelsproul · 2020-10-13T00:17:06Z

It would be nice to have /eth/v1/node/peers soon, as it's used by eth2stats

realbigsean · 2020-10-13T01:20:19Z

It would be nice to have /eth/v1/node/peers soon, as it's used by eth2stats

Yes, I've started work on it

michaelsproul · 2020-10-15T01:23:54Z

Something else I noticed while using the API:

The /eth/v1/beacon/states/{state_id}/validators endpoint doesn't support filtering by validator ID or status, and the statuses returned are Lighthouse enum names, not the status strings from the spec. For example, the query /eth/v1/beacon/states/head/validators?status=active_slashed&status=exited_slashed returns:

{
  "data": [
    {
      "index": "0",
      "balance": "31760402067",
      "status": "Active",
      "validator": {
        "pubkey": "0x8fcf28896a85e5e76ee9e508438e23e7253da1a23a6501e3a7d56182520dbcf4cdb44af3267318188f1f4168342146da",
        "withdrawal_credentials": "0x0010361af430aa7ab4a9567eaaca50ec5e02315ca1513d9ee8d73bde96370091",
        "effective_balance": 31000000000,
        "slashed": false,
        "activation_eligibility_epoch": 0,
        "activation_epoch": 0,
        "exit_epoch": 18446744073709552000,
        "withdrawable_epoch": 18446744073709552000
      }
    },
    ...
  ]
}

The relevant bit of code is:

lighthouse/beacon_node/http_api/src/lib.rs

Line 405 in a0634cc

// GET beacon/states/{state_id}/validators

guybrush · 2020-10-15T07:15:55Z

We need /eth/v1/validator/duties/proposer/{epoch} to also work for historical epochs for the explorer.

paulhauner · 2020-10-15T19:44:35Z

statuses returned are Lighthouse enum names, not the status strings from the spec

I'm working to update the spec for this: ethereum/beacon-APIs#94 (comment) We will indeed need to switch to strings once that's finalized.

## Issue Addressed `node` endpoints in #1434 ## Proposed Changes Implement these: ``` /eth/v1/node/health /eth/v1/node/peers/{peer_id} /eth/v1/node/peers ``` - Add an `Option<Enr>` to `PeerInfo` - Finish implementation of `/eth/v1/node/identity` ## Additional Info - should update the `peers` endpoints when #1764 is resolved Co-authored-by: realbigsean <[email protected]>

mcdee · 2020-10-24T18:57:14Z

We need /eth/v1/validator/duties/proposer/{epoch} to also work for historical epochs for the explorer.

Wanted to add my support for this, as well as the same for /eth/v1/beacon/states/{stateid}/committees. An example of where this is useful beyond explorers: ethdo has a block info command that dumps information about a block. The committees are required to unpick the aggregation bits in the block's attestations and show who attested for what.

I'd expand this to suggest that all data under /eth/v1/beacon/states and /eth/v1/beacon/blocks should allow arbitrarily far back state/block IDs. If the node doesn't have the data it can spit back a 404, but for data that can be retrieved or reconstituted it is very handy to be able to do so.

## Issue Addressed Michael's comment here: #1434 (comment) Resolves #1808 ## Proposed Changes - Add query param `id` and `status` to the `validators` endpoint - Add string serialization and deserialization for `ValidatorStatus` - Drop `Epoch` from `ValidatorStatus` variants ## Additional Info Please provide any additional information. For example, future considerations or information useful for reviewers.

dankrad · 2020-11-03T00:21:37Z

We need /eth/v1/validator/duties/proposer/{epoch} to also work for historical epochs for the explorer.

Just want to throw my weight behind this -- apparently this endpoint is required for having a block explorer work with lighthouse, and I would very much like us to be able to have lighthouse based block explorers as a backup to prysm on launch.

realbigsean · 2020-11-03T02:41:31Z

I've just added historical lookups for /eth/v1/validator/duties/proposer/{epoch} to #1831 👍

roman-blox · 2020-11-06T14:02:43Z

@paulhauner this endpoint https://ethereum.github.io/eth2.0-APIs/#/Validator/getAttesterDuties should accept POST request, but your implementation doesn't do so. I use openapi generate tool to generate API client based on that openapi.

realbigsean · 2020-11-06T17:58:19Z

@roman-blox this will be updated with #1831

begmaroman · 2020-11-08T12:14:39Z

@realbigsean do you have any ETA of this PR?

begmaroman · 2020-11-08T12:16:01Z

@realbigsean one more issue. This generate tool generates multiple structures with the same name. This affects the block already redeclared error. Do you know how to fix that?

winlin · 2020-11-25T08:39:49Z

hello, we need /eth/v1/beacon/states/{state_id}/validator_balances to realize the pool function, it's not work when I pass the <slot>, <hex encoded stateRoot with 0x prefix>

realbigsean · 2020-11-25T13:31:58Z

@winlin if you're querying an old slot or state root, the queries can take a while to complete, but they should be working

paulhauner · 2020-12-04T00:09:36Z

Although we may not be exactly on spec, the spec is not yet stable and is still being modified. As such, I think we've achieved the goal of this issue and we can close it.

We can open new issues as we learn about spec changes or bugs.

Well done @realbigsean for getting this across the line!

## Issue Addressed Resolves #1434 (this is the last major feature in the standard spec. There are only a couple of places we may be off-spec due to recent spec changes or ongoing discussion) Partly addresses #1669 ## Proposed Changes - remove the websocket server - remove the `TeeEventHandler` and `NullEventHandler` - add server sent events according to the eth2 API spec ## Additional Info This is according to the currently unmerged PR here: ethereum/beacon-APIs#117 Co-authored-by: realbigsean <[email protected]>

paulhauner added the enhancement New feature or request label Aug 3, 2020

This was referenced Aug 3, 2020

Irritating error message of the validator_client before genesis start. #1424

Closed

Separate metrics server and API server (different ports) #1411

Closed

Tracking: October security review #1436

Closed

paulhauner added the sept-sec-review Required for the September 1st security review label Aug 3, 2020

ghost added A0 HTTP-API labels Sep 9, 2020

paulhauner removed enhancement New feature or request sept-sec-review Required for the September 1st security review labels Sep 21, 2020

This was referenced Sep 21, 2020

[Merged by Bors] - Implement standard eth2.0 API #1569

Closed

Cargo audit vuln ignored: RUSTSEC-2020-0043 #1669

Closed

paulhauner added A1 and removed A0 labels Sep 29, 2020

MysticRyuujin mentioned this issue Oct 9, 2020

Support for Lighthouse v0.3.0+ Alethio/eth2stats-client#60

Closed

guybrush mentioned this issue Oct 12, 2020

Feature/api v1 gobitfly/eth2-beaconchain-explorer#292

Closed

This was referenced Oct 15, 2020

[Merged by Bors] - Node endpoints #1778

Closed

[Merged by Bors] - Beacon state validator id filter #1803

Closed

paulhauner added A0 and removed A1 labels Nov 8, 2020

paulhauner assigned realbigsean Nov 16, 2020

realbigsean mentioned this issue Nov 17, 2020

[Merged by Bors] - Server sent events #1920

Closed

paulhauner closed this as completed Dec 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement the standard HTTP for BN #1434

Implement the standard HTTP for BN #1434

paulhauner commented Aug 3, 2020 •

edited

Loading

michaelsproul commented Oct 13, 2020

realbigsean commented Oct 13, 2020

michaelsproul commented Oct 15, 2020

guybrush commented Oct 15, 2020

paulhauner commented Oct 15, 2020

mcdee commented Oct 24, 2020

dankrad commented Nov 3, 2020

realbigsean commented Nov 3, 2020

roman-blox commented Nov 6, 2020

realbigsean commented Nov 6, 2020

begmaroman commented Nov 8, 2020

begmaroman commented Nov 8, 2020

winlin commented Nov 25, 2020

realbigsean commented Nov 25, 2020

paulhauner commented Dec 4, 2020

Implement the standard HTTP for BN #1434

Implement the standard HTTP for BN #1434

Comments

paulhauner commented Aug 3, 2020 • edited Loading

Description

Overall status

Per-Endpoint

Beacon

Debug

Node

Config

Validator

Events

Known Issues

michaelsproul commented Oct 13, 2020

realbigsean commented Oct 13, 2020

michaelsproul commented Oct 15, 2020

guybrush commented Oct 15, 2020

paulhauner commented Oct 15, 2020

mcdee commented Oct 24, 2020

dankrad commented Nov 3, 2020

realbigsean commented Nov 3, 2020

roman-blox commented Nov 6, 2020

realbigsean commented Nov 6, 2020

begmaroman commented Nov 8, 2020

begmaroman commented Nov 8, 2020

winlin commented Nov 25, 2020

realbigsean commented Nov 25, 2020

paulhauner commented Dec 4, 2020

paulhauner commented Aug 3, 2020 •

edited

Loading