[experimental] A types-only RPC implementation using JavaScript proxies #1190

steveluscher · 2023-03-09T01:30:23Z

This PR introduces an implementation of the client RPC that does as little work as possible. It essentially passes calls to the server RPC unmodified, and enforces correctness through nothing but Typescript types.

Problem

The Connection class today contains concrete implementations for every RPC method, past, present, and future. This implies that the bundle size of this library may grow unbounded as the number of RPC methods grows.

Solution

The RPC interface is simple and conforms to JSON RPC 2.0. The only things that are required to make an RPC call are a method name and a list of parameters.

The solution herein creates an rpc object on which you can call any method. That method name and the params get passed through, unmodified.

// Example: this call...
rpc.doSomething(transport, 1, 'foo', {bar: 'baz'});

// ...gets put on the wire as a JSON RPC call of the format:
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "doSomething",
  "params": [1, "foo", {"bar": "baz"}]
}

In this way, the set of RPC methods can grow, unbounded, without increasing the size of the client library at all.

Correctness, in this world, is enforced only by Typescript. The input params are completely specified on the JavaScript Proxy, and the return types change with the inputs given. See the complicated getAccountInfo commit on this PR for an example of how this is achieved using function overloads.

Crucially, this library will never concern itself with how to send RPC calls – only with how you should format your calls and reason about the return values. The sending is left up to the transport, which you can read about in #1187.

How to review this PR

Click the ‘commits’ tab and treat each commit as though it were its own PR.

…Blocks`

codecov-commenter · 2023-03-09T01:39:59Z

Codecov Report

Merging #1190 (c59286a) into master (ad23683) will not change coverage.
The diff coverage is n/a.

❗ Current head c59286a differs from pull request most recent head b87de1a. Consider uploading reports for the commit b87de1a to get more accurate results

@@           Coverage Diff           @@
##           master    #1190   +/-   ##
=======================================
  Coverage   76.35%   76.35%           
=======================================
  Files          56       56           
  Lines        3151     3151           
  Branches      475      475           
=======================================
  Hits         2406     2406           
  Misses        578      578           
  Partials      167      167

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

steveluscher · 2023-03-09T01:36:41Z

packages/rpc-core/src/types/rpc-methods/getBlockHeight.d.ts

+    getBlockHeight(
+        transport: IJsonRpcTransport,
+        config?: readonly {
+            // Defaults to `finalized`
+            commitment?: Finality;
+            // The minimum slot that the request can be evaluated at
+            minContextSlot?: Slot;
+        }
+    ): Promise<GetBlockHeightApiResponse>;


See https://docs.solana.com/api/http#getblockheight.

steveluscher · 2023-03-09T01:36:51Z

packages/rpc-core/src/types/rpc-methods/getBlocks.d.ts

+    getBlocks(
+        transport: IJsonRpcTransport,
+        startSlot: Slot,
+        endSlotInclusive?: Slot,
+        config?: readonly {
+            // Defaults to `finalized`
+            commitment?: Exclude<Finality, 'processed'>;
+        }
+    ): Promise<GetBlocksApiResponse>;


See https://docs.solana.com/api/http#getblocks.

Looks like the docs are wrong on this one :-\

I fixed them! I don't know why the change hasn't deployed yet. https://github.com/solana-labs/solana/pull/30351/files

Ah… https://discord.com/channels/428295358100013066/966782389839876096/1083467864650039347

ah yes. Anyway, thanks for fixing!

steveluscher · 2023-03-09T01:40:52Z

packages/rpc-core/src/types/rpc-methods/getAccountInfo.d.ts

+    getAccountInfo(
+        transport: IJsonRPCTransport,
+        address: Base58EncodedAddress,
+        config?: readonly {
+            encoding: 'base64';
+        } &
+            GetAccountInfoApiCommonConfig &
+            GetAccountInfoApiBase64EncodingCommonConfig
+    ): Promise<GetAccountInfoApiResponseBase & GetAccountInfoApiResponseWithEncodedData>;
+    getAccountInfo(
+        transport: IJsonRPCTransport,
+        address: Base58EncodedAddress,
+        config?: readonly {
+            encoding: 'base64+zstd';
+        } &
+            GetAccountInfoApiCommonConfig &
+            GetAccountInfoApiBase64EncodingCommonConfig
+    ): Promise<GetAccountInfoApiResponseBase & GetAccountInfoApiResponseWithEncodedZStdCompressedData>;
+    getAccountInfo(
+        transport: IJsonRPCTransport,
+        address: Base58EncodedAddress,
+        config?: readonly {
+            encoding: 'jsonParsed';
+        } &
+            GetAccountInfoApiCommonConfig
+    ): Promise<GetAccountInfoApiResponseBase & GetAccountInfoApiResponseWithJsonData>;


This is a much more complicated example, where you can see how modulating the inputs changes the output type (eg. specifying different encoding results in the output being different.

Things to pay attention to:

jsonParsed encoding results in data being a parsed data structure or [bytes, encoding] as a fallback if no parser is available.

only base64 encoding accepts dataSlice as an input

See https://docs.solana.com/api/http#getaccountinfo.

This is extremely cool, and not nearly as complicated to read as I feared when I first opened the file 😅

joncinque

Only one tiny substantial point -- looks really great overall!

joncinque · 2023-03-09T15:49:48Z

packages/rpc-core/src/types/rpc-methods/common.d.ts

+declare type Slot =
+    // TODO(solana-labs/solana/issues/30341) Represent as bigint
+    number;


I know we've talked about this before, so pardon asking it again here... with the BYO-Transport model, is it possible to require that transports properly handle u64s and make this a bigint?

So, fast forward to when everything is bigint (or rewind, and have represented all u64s as JavaScript strings in the first place) and all that the transport has to do is to serialize bigints as string over the wire.

We can't do this today, because the Solana RPC doesn't accept them.

# params is normally [5, 10] but here I'm representing them as ["5", "10"] curl https://api.mainnet-beta.solana.com -X POST -H "Content-Type: application/json" -d ' { "jsonrpc": "2.0", "id": 1, "method": "getBlocks", "params": ["5", "10"] } ' {"jsonrpc":"2.0","error":{"code":-32602,"message":"Invalid params: invalid type: string \"5\", expected u64."},"id":1}

We need to change the server to parse strings as u64 before we can do that on the client.

But I see what you're saying. You're saying: make it a bigint now at the outer part of the interface, and have the transport (temporarily) downcast that to a JavaScript number, maybe optionally warning when it overflows. Is that right?

I love this!

I was wondering it if it would be possible at the transport level to use bigint everywhere, but I forgot that the problem is at the RPC level, and not web3.js. But yeah, since all of these interfaces are new, it might be worth doing everything right and use bigints everywhere.

joncinque · 2023-03-09T15:51:19Z

packages/rpc-core/src/types/rpc-methods/getBlockHeight.d.ts

+        config?: readonly {
+            // Defaults to `finalized`
+            commitment?: Finality;
+            // The minimum slot that the request can be evaluated at
+            minContextSlot?: Slot;
+        }


All these levels of optionals... I can swee why graphql would make this so much neater

joncinque · 2023-03-09T16:14:14Z

packages/rpc-core/src/types/rpc-methods/getBlocks.d.ts

+    getBlocks(
+        transport: IJsonRpcTransport,
+        startSlot: Slot,
+        endSlotInclusive?: Slot,
+        config?: readonly {
+            // Defaults to `finalized`
+            commitment?: Exclude<Finality, 'processed'>;
+        }
+    ): Promise<GetBlocksApiResponse>;


Looks like the docs are wrong on this one :-\

joncinque · 2023-03-09T18:22:30Z

packages/rpc-core/src/rpc.ts

+import { IJsonRpcTransport } from '@solana/rpc-transport';
+import { JsonRpcApi } from './types/jsonRpcApi';
+
+export const rpc = /* #__PURE__ */ new Proxy<JsonRpcApi>({} as JsonRpcApi, {


TIL what a Proxy is, very cool!

joncinque · 2023-03-09T18:24:59Z

packages/rpc-core/src/rpc.ts

+                const normalizedParams = params.length ? params : undefined;
+                const result = await transport.send(method, normalizedParams);
+                return result;
+            } as unknown as JsonRpcApi[TMethodName];


Learning a lot today with ... as unknown as ... -- does this cause potential footguns for transport implementers?

At that point, it's our job to make sure that the implementation of rpc.* is perfect. Essentially what we're saying is that the dynamic get returns a method that will definitely conform to JsonRpcApi[TMethodName] but the implementation on lines 14-18 does not, from Typescript's perspective.

src/rpc.ts:14:13 - error TS2322: Type '(transport: IJsonRpcTransport, ...params: Parameters<JsonRpcApi[TMethodName]>) => Promise<unknown>' is not assignable to type 'JsonRpcApi[TMethodName]'. Type '(transport: IJsonRpcTransport, ...params: Parameters<JsonRpcApi[TMethodName]>) => Promise<unknown>' is not assignable to type '{ (transport: IJsonRPCTransport, address: Base58EncodedAddress, config?: ({ encoding: "base64"; } & GetAccountInfoApiCommonConfig & GetAccountInfoApiBase64EncodingCommonConfig) | undefined): Promise<...>; (transport: IJsonRPCTransport, address: Base58EncodedAddress, config?: ({ ...; } & ... 1 more ... & GetAccountIn...'. Type '(transport: IJsonRpcTransport, ...params: Parameters<JsonRpcApi[TMethodName]>) => Promise<unknown>' is not assignable to type '{ (transport: IJsonRPCTransport, address: Base58EncodedAddress, config?: ({ encoding: "base64"; } & GetAccountInfoApiCommonConfig & GetAccountInfoApiBase64EncodingCommonConfig) | undefined): Promise<...>; (transport: IJsonRPCTransport, address: Base58EncodedAddress, config?: ({ ...; } & ... 1 more ... & GetAccountIn...'. Types of parameters 'params' and 'address' are incompatible. Type '[address: Base58EncodedAddress, config?: ({ encoding: "base64"; } & GetAccountInfoApiCommonConfig & GetAccountInfoApiBase64EncodingCommonConfig) | undefined]' is not assignable to type 'Parameters<JsonRpcApi[TMethodName]>'. 14 target[p] = async function (transport: IJsonRpcTransport, ...params: Parameters<JsonRpcApi[TMethodName]>) { ~~~~~~~~~

It seems like these two should be compatible:

[address: Base58EncodedAddress, config?: ({ encoding: "base64"; } & GetAccountInfoApiCommonConfig & GetAccountInfoApiBase64EncodingCommonConfig) | undefined]

Parameters<JsonRpcApi[TMethodName]>

…but they aren't.

Ah I see, that's really weird, thanks for explaining. Is that a typescript bug? Or does Parameters need to have some extra declaration to make that clear?

I'm not sure, but I think I made a mistake including transport in all the method signatures. I wonder what would happen if I took that out…

joncinque · 2023-03-09T18:34:25Z

packages/rpc-core/src/types/rpc-methods/getAccountInfo.d.ts

+        lamports: number; // TODO(solana-labs/solana/issues/30341) Represent as bigint
+        owner: Base64EncodedAddress;
+        rentEpoch: number; // TODO(solana-labs/solana/issues/30341) Represent as bigint
+        space: number; // TODO(solana-labs/solana/issues/30341) Represent as bigint


Maybe I'm missing something, but in testing I didn't see space returned here. Here's a token account:

{ "jsonrpc": "2.0", "result": { "context": { "apiVersion": "1.13.6", "slot": 181671824 }, "value": { "data": [ "xvp6877brTo9ZfNqq8l0MbG75MLS9uDkfKYCA0UvXWEs07wIp7mh6Iz0YahgZDSe/Psf+oXDeVKNALXKNdFOFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", "base64" ], "executable": false, "lamports": 2039280, "owner": "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA", "rentEpoch": 0 } }, "id": 1 }

Haha. In the docs, this is called size, in the docs' example it's called space, in the current web3.js implementation it's absent, and in the server implementation it's space and it's an Option<u64>.

🤔 solana-labs/solana#28421

Looks like this is only in v1.15.

joncinque · 2023-03-09T18:35:30Z

packages/rpc-core/src/types/rpc-methods/getAccountInfo.d.ts

+    getAccountInfo(
+        transport: IJsonRPCTransport,
+        address: Base58EncodedAddress,
+        config?: readonly {
+            encoding: 'base64';
+        } &
+            GetAccountInfoApiCommonConfig &
+            GetAccountInfoApiBase64EncodingCommonConfig
+    ): Promise<GetAccountInfoApiResponseBase & GetAccountInfoApiResponseWithEncodedData>;
+    getAccountInfo(
+        transport: IJsonRPCTransport,
+        address: Base58EncodedAddress,
+        config?: readonly {
+            encoding: 'base64+zstd';
+        } &
+            GetAccountInfoApiCommonConfig &
+            GetAccountInfoApiBase64EncodingCommonConfig
+    ): Promise<GetAccountInfoApiResponseBase & GetAccountInfoApiResponseWithEncodedZStdCompressedData>;
+    getAccountInfo(
+        transport: IJsonRPCTransport,
+        address: Base58EncodedAddress,
+        config?: readonly {
+            encoding: 'jsonParsed';
+        } &
+            GetAccountInfoApiCommonConfig
+    ): Promise<GetAccountInfoApiResponseBase & GetAccountInfoApiResponseWithJsonData>;


This is extremely cool, and not nearly as complicated to read as I feared when I first opened the file 😅

joncinque · 2023-03-09T18:38:36Z

packages/rpc-core/src/types/rpc-methods/common.d.ts

+};
+
+// TODO: Eventually move this into whatever package implements transactions
+declare type Finality = 'confirmed' | 'finalized' | 'processed';


nit: Is Finality the new term? If so, we should probably update the docs and all that. I'd prefer to keep this Commitment to make mental mapping easier, but I won't put up a big fight about it.

Ooh. I thought these three were what we're calling finality, as opposed to the olde deprecated suite of commitments.

From web3.js today:

solana-web3.js/packages/library-legacy/src/connection.ts

Lines 476 to 501 in ad23683

/**

* The level of commitment desired when querying state

* <pre>

* 'processed': Query the most recent block which has reached 1 confirmation by the connected node

* 'confirmed': Query the most recent block which has reached 1 confirmation by the cluster

* 'finalized': Query the most recent block which has been finalized by the cluster

* </pre>

*/

export type Commitment =

| 'processed'

| 'confirmed'

| 'finalized'

| 'recent' // Deprecated as of v1.5.5

| 'single' // Deprecated as of v1.5.5

| 'singleGossip' // Deprecated as of v1.5.5

| 'root' // Deprecated as of v1.5.5

| 'max'; // Deprecated as of v1.5.5

/**

* A subset of Commitment levels, which are at least optimistically confirmed

* <pre>

* 'confirmed': Query the most recent block which has reached 1 confirmation by the cluster

* 'finalized': Query the most recent block which has been finalized by the cluster

* </pre>

*/

export type Finality = 'confirmed' | 'finalized';

So yeah, I have this wrong, somehow. I'll rename it.

@joncinque

…the RPC # Summary @joncinque pointed out in #1190 that the Typescript types of `@solana/rpc-core` relied on a cast through `unknown` to work. The more I thought about it the more I realized that the `transport` shouldn't form part of the typespec for the RPC. In this PR we remove it from the source types, and map it back in where it's needed in the implementation..

@joncinque

…the RPC # Summary @joncinque pointed out in #1190 that the Typescript types of `@solana/rpc-core` relied on a cast through `unknown` to work. The more I thought about it the more I realized that the `transport` shouldn't form part of the typespec for the RPC. In this PR we remove it from the source types, and map it back in where it's needed in the implementation..

@joncinque

…the RPC (#1196) # Summary @joncinque pointed out in #1190 that the Typescript types of `@solana/rpc-core` relied on a cast through `unknown` to work. The more I thought about it the more I realized that the `transport` shouldn't form part of the typespec for the RPC. In this PR we remove it from the source types, and map it back in where it's needed in the implementation..

github-actions · 2023-03-16T21:46:59Z

🎉 This PR is included in version 1.73.4 🎉

The release is available on:

Your semantic-release bot 📦🚀

github-actions · 2023-03-31T00:16:55Z

Because there has been no activity on this PR for 14 days since it was merged, it has been automatically locked. Please open a new issue if it requires a follow up.

steveluscher requested review from joncinque and jordaaash March 9, 2023 01:30

steveluscher force-pushed the proxy-rpc branch from 113bd69 to 7b821cf Compare March 9, 2023 01:35

steveluscher added 4 commits March 9, 2023 01:35

Boilerplate code for @solana/rpc-core

0bc8528

Type definition for two simple RPC methods: getBlockHeight and `get…

e3f892d

…Blocks`

Implementation of the RPC using JavaScript proxies

b2a8a82

Type definition of a more complicated RPC method, getAccountInfo

b87de1a

steveluscher force-pushed the proxy-rpc branch from 7b821cf to b87de1a Compare March 9, 2023 01:35

steveluscher commented Mar 9, 2023

View reviewed changes

joncinque reviewed Mar 9, 2023

View reviewed changes

Rename Finality to Commitment

1cd7604

steveluscher merged commit 9402bd4 into solana-labs:master Mar 9, 2023

steveluscher deleted the proxy-rpc branch March 9, 2023 21:34

steveluscher mentioned this pull request Mar 11, 2023

[experimental] Remove the concept of the transport from the types of the RPC #1196

Merged

github-actions bot added the released label Mar 16, 2023

steveluscher added this to the web3.js Roadmap – March 2023 milestone Mar 16, 2023

github-actions bot locked as resolved and limited conversation to collaborators Mar 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[experimental] A types-only RPC implementation using JavaScript proxies #1190

[experimental] A types-only RPC implementation using JavaScript proxies #1190

steveluscher commented Mar 9, 2023

codecov-commenter commented Mar 9, 2023 •

edited

Loading

steveluscher Mar 9, 2023

steveluscher Mar 9, 2023

joncinque Mar 9, 2023

steveluscher Mar 9, 2023 •

edited

Loading

steveluscher Mar 9, 2023

joncinque Mar 9, 2023

steveluscher Mar 9, 2023

joncinque Mar 9, 2023

joncinque left a comment

joncinque Mar 9, 2023

steveluscher Mar 9, 2023 •

edited

Loading

steveluscher Mar 9, 2023

steveluscher Mar 9, 2023

joncinque Mar 9, 2023

joncinque Mar 9, 2023

joncinque Mar 9, 2023

joncinque Mar 9, 2023

joncinque Mar 9, 2023

steveluscher Mar 9, 2023

steveluscher Mar 9, 2023

joncinque Mar 9, 2023

steveluscher Mar 11, 2023

steveluscher Mar 11, 2023

joncinque Mar 9, 2023

steveluscher Mar 9, 2023

steveluscher Mar 9, 2023

steveluscher Mar 9, 2023

joncinque Mar 9, 2023

joncinque Mar 9, 2023

joncinque Mar 9, 2023

steveluscher Mar 9, 2023

github-actions bot commented Mar 16, 2023

github-actions bot commented Mar 31, 2023

	/**
	* The level of commitment desired when querying state
	* <pre>
	* 'processed': Query the most recent block which has reached 1 confirmation by the connected node
	* 'confirmed': Query the most recent block which has reached 1 confirmation by the cluster
	* 'finalized': Query the most recent block which has been finalized by the cluster
	* </pre>
	*/
	export type Commitment =
	\| 'processed'
	\| 'confirmed'
	\| 'finalized'
	\| 'recent' // Deprecated as of v1.5.5
	\| 'single' // Deprecated as of v1.5.5
	\| 'singleGossip' // Deprecated as of v1.5.5
	\| 'root' // Deprecated as of v1.5.5
	\| 'max'; // Deprecated as of v1.5.5

	/**
	* A subset of Commitment levels, which are at least optimistically confirmed
	* <pre>
	* 'confirmed': Query the most recent block which has reached 1 confirmation by the cluster
	* 'finalized': Query the most recent block which has been finalized by the cluster
	* </pre>
	*/
	export type Finality = 'confirmed' \| 'finalized';

[experimental] A types-only RPC implementation using JavaScript proxies #1190

[experimental] A types-only RPC implementation using JavaScript proxies #1190

Conversation

steveluscher commented Mar 9, 2023

Problem

Solution

How to review this PR

codecov-commenter commented Mar 9, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveluscher Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joncinque left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveluscher Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 16, 2023

github-actions bot commented Mar 31, 2023

codecov-commenter commented Mar 9, 2023 •

edited

Loading

steveluscher Mar 9, 2023 •

edited

Loading

steveluscher Mar 9, 2023 •

edited

Loading