Massa gRPC API design and challenges

[aoudiamoncef]

Context

Recently, a workshop took place to discuss the adoption of gRPC as the technical stack for the newer version of the Massa API. Feedback from the innovation team, builders, and the community was gathered, yet some blocking points remain.

See #3406 for more details.

Caching Model

As you know, the Massa API produces a large amount of ephemeral data including blocks, operations, endorsements and execution events. Presently, this data is accessed through a limited, short-term cache. While this method can retrieve the necessary information, it introduces an element of non-idempotency over time.

Questions:

• Should we continue relying on the short-term cache ?
• Should we create a secondary layer of caching that keeps state in disk storage ?
• Could delagating caching to the Massa Explorer work, streaming all without requiring a retention policy ?

UX vs Accuracy

Having considered the collected information, one proposed design would involve breaking down existing methods into smaller segments. However, this could introduce issues with integrity and atomicity.

Questions:

• Should we keep the current, bulkier methods (e.g. get_status)?
• Can we break down these methods and add additional context to the responses?
• Could a single bulk (batch) endpoint ensure atomicity of execution?

In my opinion, in a distributed system and message based architecture, every exchanged message should contain all relevant metadata. I look forward to hearing feedback before beginning implementation.

[damip]

Here are my thoughts:

• let's not use caching at the node level for now, it can be delegated upstream
• methods should be broken down (to not have bulky methods)
• methods for which it matters should return the metadata explaining at which slot they were computed
• extra "execution_batching", "consensus_batching", "network_batching" etc... endpoints should be made available and allow querying those modules with batches of requests that will be executed atomically. This will only support requests for which atomicity is relevant. Note that unifying those endpoints would yield misleading outputs because those modules are mutually asynchronous, which is why I split them. Batched requests should also return the slot at which they were executed.

[aoudiamoncef]

Here are my thoughts:

• let's not use caching at the node level for now, it can be delegated upstream

I certain scenarios, when we have a lot of throughput, the cache is cleared and the API call will return nothing. THe only way to handle this is to save the stream of all produced events.

• methods should be broken down (to not have bulky methods)
• methods for which it matters should return the metadata explaining at which slot they were computed

This could deep changes in certain modules. cc @AurelienFT

• extra "execution_batching", "consensus_batching", "network_batching" etc... endpoints should be made available and allow querying those modules with batches of requests that will be executed atomically. This will only support requests for which atomicity is relevant. Note that unifying those endpoints would yield misleading outputs because those modules are mutually asynchronous, which is why I split them. Batched requests should also return the slot at which they were executed.

This should be well designed and scoped, we'll try to be agile and make a POC about it ASAP.

[aoudiamoncef]

I have an example of wrong response in the current API due to the ephemeral data, this is the output of get_addresses, in the response "created_blocks": [] is empty, but we don't have any indicator or context. We quickly reach the limits of our actual approche. We don't have any warranty to get a correct response.

{
    "jsonrpc": "2.0",
    "result": [
        {
            "address": "A12hVZr4DfpE4AWwaP4tyjX8wQ78e7E3MmkeM4985uHBTkokFxMK",
            "thread": 27,
            "final_balance": "87.96078707",
            "final_roll_count": 65,
            "final_datastore_keys": [],
            "candidate_balance": "87.984316478",
            "candidate_roll_count": 65,
            "candidate_datastore_keys": [],
            "deferred_credits": [],
            "next_block_draws": [],
            "next_endorsement_draws": [
                {
                    "slot": {
                        "period": 59951,
                        "thread": 10
                    },
                    "index": 15
                },
                {
                    "slot": {
                        "period": 59951,
                        "thread": 17
                    },
                    "index": 2
                },
                {
                    "slot": {
                        "period": 59953,
                        "thread": 5
                    },
                    "index": 13
                },
                {
                    "slot": {
                        "period": 59954,
                        "thread": 22
                    },
                    "index": 3
                },
                {
                    "slot": {
                        "period": 59955,
                        "thread": 4
                    },
                    "index": 1
                },
                {
                    "slot": {
                        "period": 59955,
                        "thread": 11
                    },
                    "index": 0
                },
                {
                    "slot": {
                        "period": 59956,
                        "thread": 11
                    },
                    "index": 13
                },
                {
                    "slot": {
                        "period": 59956,
                        "thread": 26
                    },
                    "index": 9
                },
                {
                    "slot": {
                        "period": 59957,
                        "thread": 1
                    },
                    "index": 11
                },
                {
                    "slot": {
                        "period": 59957,
                        "thread": 5
                    },
                    "index": 12
                },
                {
                    "slot": {
                        "period": 59957,
                        "thread": 16
                    },
                    "index": 2
                },
                {
                    "slot": {
                        "period": 59957,
                        "thread": 22
                    },
                    "index": 1
                },
                {
                    "slot": {
                        "period": 59958,
                        "thread": 7
                    },
                    "index": 15
                },
                {
                    "slot": {
                        "period": 59959,
                        "thread": 11
                    },
                    "index": 6
                },
                {
                    "slot": {
                        "period": 59959,
                        "thread": 22
                    },
                    "index": 15
                },
                {
                    "slot": {
                        "period": 59959,
                        "thread": 29
                    },
                    "index": 8
                }
            ],
            "created_blocks": [],
            "created_operations": [],
            "created_endorsements": [
                "E13JHzNeyqgRCt6TrnH21npCZ8HvoLyMWk7jXeKyEv4Sg7EHaVZ",
                "E1BY8Kx2kawHrZ97WdcPPSfUJRV4VYNu8RsRkXfohhmL1SXVDdm",
                "E1KpKADECXAfpbPCZMtiQVQeqWXSadvJEr2pDMHzZX5Go3LyCP8",
                "E12oXd92EKedFeTSSDD3nXVqgRKDDaD2aeMURYe8XyLpMTreBdy2",
                "E12VxE1Uk6QCsaFA7wMuGs6gvu437dCenzWqxL1woVXpe8zGUvCz",
                "E12PxdrWTfzaiqLguZ8zWmdt74bbTfgJgmhS1YPjLXErddDVaSys",
                "E1bsMJpLdADr4HxTgJZKf5jGoGVUCWkxJBaqUCue8t4S9P17yhD",
                "E12NtKyEbYjvScSc94sRwouzt3kLBQHVnYGboCJhNavCLZM6uE2B",
                "E1vH8XkP7E6hHgXPEzQzA5Fe53QX8dpkg36W9WsPiCEuDZCumt5",
                "E12M6xU8nNom1opC3aSp9SMFLYqzYxBDEEGLUyAkAnJbbrnsagfX",
                "E12dmZ2GaEpCKTrVsvmzu62K7RLAwjXDYZSA9NH1wCyZTTPgFGXd",
                "E12YHcYrmBrKSnx4uSp8SK2UevJsT7bDA8Axewh1jymwJDo5kjyH",
                "E129RZQvZL2x9ixviRnNY3gkGZRY4dE9L1yj8eA4hWkA89GSWvyh",
                "E12Jg5XW5vcayePu6DguKH4Vhu1Y1nP1VXxNJVSM1trWp8LotLe2",
                "E1QqJ3XZdz82SmueWmR3CrmZsMDvVUV4J6Ybp3YwgL6SKkarBPT",
                "E128HHaZhWfUCavjU6XSxQNLJapS2TJx1djBZcWtoY6McCbQoaCC",
                "E12hX85hNZahAizh5oGevq6Z9LnooJ2zEVe4y2WN4FWT7ZrUutqS",
                "E1SyTEH4tas1EpKZoXUSTMKUnVAZ7C3iwEyUDAeZoqiR5xGC8rt",
                "E1MwgmeqPBGtM6cfaiJiJsy24HPyR5ko3eTEmeFDiKogphhQLT1",
                "E12RC4hG5goERxG1rJvjf1447Hm2YgeYuygGkw91pwy7Wu34M48H"
            ],
            "cycle_infos": [
                {
                    "cycle": 464,
                    "is_final": true,
                    "ok_count": 13,
                    "nok_count": 0,
                    "active_rolls": null
                },
                {
                    "cycle": 465,
                    "is_final": true,
                    "ok_count": 10,
                    "nok_count": 0,
                    "active_rolls": null
                },
                {
                    "cycle": 466,
                    "is_final": true,
                    "ok_count": 8,
                    "nok_count": 0,
                    "active_rolls": null
                },
                {
                    "cycle": 467,
                    "is_final": true,
                    "ok_count": 6,
                    "nok_count": 0,
                    "active_rolls": 65
                },
                {
                    "cycle": 468,
                    "is_final": false,
                    "ok_count": 3,
                    "nok_count": 0,
                    "active_rolls": 65
                }
            ]
        }
    ],
    "id": 1
}

[aoudiamoncef]

API service

get_status:

• GetBlockchainInfo
• GetCurrentBlockchainInfo
• GetNodeConfig
• GetConsensusStats
• GetPoolStats
• GetOperationPoolStats
• GetNetworkStats
• GetExecutionStats

get_addresses:

• GetBalance
• GetStakingActivity || GetStakingHistory

execute_read_only && execute_read_only_bytecode

• ScReadOnlyCall || ExecuteScReadOnlyCall

get_stakers:

• GetStakers

get_graph_interval:

• GetGraphByInterval || GetGraph || GetGraphBlocks || GetGraphSlice || GetGraphBySlots

get_filtered_sc_output_event:

• GetScExecutionEvent || GetSlotExecutionOutputs

get_blockclique_block_by_slot:

• GetBlockcliqueBySlot || GetBlockclique

get_staking_addresses:

• GetStakingAdresses

NODE service
add_staking_secret_keys:

• StartStaking || AddStakingSecretKey

node_peers_whitelist

• GetPeersWhitelist

node_add_to_peers_whitelist

• AddToPeersWhitelist

node_remove_from_peers_whitelist

• RemoveFromPeersWhitelist

node_bootstrap_blacklist

• GetBootstrapBlacklist

node_add_to_bootstrap_blacklist

• AddToBootstrapBlacklist

node_remove_from_bootstrap_blacklist

• RemoveFromBootstrapBlacklist

node_bootstrap_whitelist
-GetBootstrapWhitelist

node_bootstrap_whitelist_allow_all

• AllowAllInBootstrapWhitelist // BootstrapWhitelistAllowAll

node_add_to_bootstrap_whitelist

• AddToBootstrapWhitelist

node_remove_from_bootstrap_whitelist

• RemoveFromBootstrapWhitelist

node_ban_by_id

• BanById

node_unban_by_id

• UnbanById

node_ban_by_ip

• BanByIp

node_unban_by_ip

• UnbanByIp

node_sign_message

• SignMessage || Sign

stop_node

• ShutdownNode

May be for batched methods, we could keep the same name ? (GetAddresses, GetStatus)

[aoudiamoncef]

We'll split the gRPC API into two services:

Public

It will be available on port 33037 with the following methods:

• GetStatus: this will be a newer version with a limited set of information compared to the existing one in JsonRPC. Connected nodes will be removed

• QueryState: this a game changer in API side as it's designed to query all kind of information about the state of the blockchain in a batched way and it guaranty the atomicity. This will remove a lot of existing methods. See Redesign Execution queries #4075

• ScReadOnlyCall: we'll fuse the two existing JsonRPC methods into one.

Private

It will be available on port 33038.

We'll remove node prefix and keep the actual names used the private JsonRPC API.

The public API service will be priorities as it's heavily used.