-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OpenRPC Support #4711
OpenRPC Support #4711
Conversation
This implement the basic functionality for the method over HTTP RPC. Signed-off-by: meows <[email protected]>
Signed-off-by: meows <[email protected]> Conflicts: go.mod go.sum
Signed-off-by: meows <[email protected]>
Signed-off-by: meows <[email protected]> Conflicts: go.mod go.sum
This is for development only. Versions need to be bumped when they're ready for use as canonical remotes. Signed-off-by: meows <[email protected]>
This eliminates code duplication. Signed-off-by: meows <[email protected]>
Signed-off-by: meows <[email protected]>
Also fixes casing stuff for the rest of Filecoin. methods. Signed-off-by: meows <[email protected]>
This reverts commit 116898e.
This fixes an issue caused with the latest reverting commit. Signed-off-by: meows <[email protected]>
…orted stuff Signed-off-by: meows <[email protected]> Makefile: fix docgen refactoring for makefile use of command Signed-off-by: meows <[email protected]>
There are quite of few of these already registered for the docgen command, so it makes sense to use those! Signed-off-by: meows <[email protected]>
Signed-off-by: meows <[email protected]>
…ect versions Signed-off-by: meows <[email protected]>
This function will handle the manual configurations for app-specific data types w/r/t their json schema representation. This is useful for cases where the reflect library is unable to provide a sufficient representation automatically. Provided in this commit is an initial implementation for the integerD type (assuming number are represented in the API as hexs), and a commonly used cid.Cid type. Signed-off-by: meows <[email protected]>
Conflicts: cmd/lotus/rpc.go
Oooo, this is interesting. At a first glance the integration seems pretty clean, will review further tomorrow |
Hey @whyrusleeping, just bumping on this. Feedback would be very welcome. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this idea, but there are a couple of issues
I'd strongly prefer not running any generation at runtime, to not include new / unaudited (by us) dependencies in the lotus/lotus-miner binaries
Instead of hooking into the RPC handler directly, which I don't think is needed, I'd:
- Generate the doc json blob with docgen (ok if it uses external deps, but please limit changes to deps we already include); store that blob in
build/openrpc/[full/miner].json
- Create go-rice loaders like the one for proof parameters json file (https://github.com/filecoin-project/lotus/blob/master/build/parameters.go); as long as that call is in the
build/
package, go-rice should figure out embedding automagically - Add a
Discover(ctx) (json.RawMessage, error)
method to CommonAPI interfaces/structs inapi/
,api/struct/
,node/impl/common
- Given that OpenRPC spec requires the method to be called
rpc.discover
, we can handle this in go-jsonrpc by adding a RPCServer.Alias method, and aliasing that method
- Given that OpenRPC spec requires the method to be called
- In the implementation just return that pre-generated blob
go.sum
Outdated
github.com/ethereum/go-ethereum v1.9.12 h1:EPtimwsp/KGDSiXcNunzsI4kefdsMHZGJntKx3fvbaI= | ||
github.com/ethereum/go-ethereum v1.9.12/go.mod h1:PvsVkQmhZFx92Y+h2ylythYlheEDt/uBgFbl61Js/jo= |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll need to tame the dep tree before merging this
go-ethereum is LGPL which isn't exactly compatible with MIT+Apache
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(go-ethereum being LGPL is the reason why we've implemented go-jsonrpc in the first place)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cmd/lotus/rpc.go
Outdated
|
||
doc.RegisterReceiverName("Filecoin", fullAPI) | ||
|
||
rpcServer.Register("rpc", &RPCDiscovery{doc}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like it will make all RPC methods be rpc.*
, which is a breaking change
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed this approach with a63eeef
…flect This removes a problematic dependency on github.com/ethereum/go-ethereum, which was imported as a dependency for a couple github.com/etclabscore/go-openrpc-reflect tests. etclabscore/go-openrpc-reflect v0.0.36 has removed this dependency, so this commit is the result of bumping that version and then running 'go mod tidy' This is in response to a review at filecoin-project#4711 (review) Date: 2020-11-21 06:52:48-06:00 Signed-off-by: meows <[email protected]>
This allows the command to EITHER generate the doc for Full or Miner APIs. See comment for usage. Date: 2020-11-21 07:48:05-06:00 Signed-off-by: meows <[email protected]>
Generating the Miner API OpenRPC doc (via 'go run ./api/openrpc/cmd miner') caused the example logic to panic because some types were missing. This commit adds those missing types, although I'm not an expert in the API so I can't suggest that the example values provided are ideal or well representative. Date: 2020-11-21 07:50:21-06:00 Signed-off-by: meows <[email protected]>
…full/miner].json docs These will be used as static documents provided by the rpc.discover method. Date: 2020-11-21 07:51:39-06:00 Signed-off-by: meows <[email protected]>
Date: 2020-11-21 08:23:06-06:00 Signed-off-by: meows <[email protected]>
Instead of generating the doc on the fly, we're going to serve a static asset. Rel filecoin-project#4711 (review) This removes the runtime implementation from the RPC server construction. Date: 2020-11-21 08:41:20-06:00 Signed-off-by: meows <[email protected]>
… and structs Date: 2020-11-21 08:41:56-06:00 Signed-off-by: meows <[email protected]>
This depends on a currently-forked change at filecoin-project/go-jsonrpc 8350f9463ee451b187d35c492e32f1b999e80210 which establishes this new method RPCServer.AliasMethod. This solves the problem that the OpenRPC spec says that the document should be served at the system extension-prefixed endpoing rpc.discover (not Filecoin.Discover). In fact, the document will be available at BOTH endpoints, but that duplicity is harmless. Date: 2020-11-21 09:18:26-06:00 Signed-off-by: meows <[email protected]>
…of string Instead of casting the JSON asset from bytes to string, unmarshal it to a map[string]interface{} so the server will provide it as a JSON object. Date: 2020-11-21 09:27:11-06:00 Signed-off-by: meows <[email protected]>
Conflicts: Makefile api/api_common.go api/apistruct/struct.go api/docgen/docgen.go go.mod go.sum
Date: 2020-11-22 07:19:36-06:00 Signed-off-by: meows <[email protected]>
This keeps the errcheck linter happy. Date: 2020-11-24 06:33:14-06:00 Signed-off-by: meows <[email protected]>
Date: 2020-11-24 06:36:07-06:00 Signed-off-by: meows <[email protected]>
Conflicts: go.mod
Tidying up after resolving the merge conflicts with master at go.mod Date: 2020-11-24 06:40:45-06:00 Signed-off-by: meows <[email protected]>
This is a repeat of 76e6fd2, since the latest merge to master seems to have reverted this. Date: 2020-11-24 06:42:30-06:00 Signed-off-by: meows <[email protected]>
…schema examples Removing method example pairings since they were redundant to schema examples and were not implemented well. Improved schema examples by using the ExampleValue method instead of the map lookup. Made a note in the comment here that this is not ideal, since we have to make a shortcut assumption /workaround by using 'unknown' as the method name and the typea as its own parent. Luckily these values aren't heavily used by the method logic. Date: 2020-11-27 12:57:36-06:00 Signed-off-by: meows <[email protected]>
Previously used an integer schema assuming hex encoding. It appears, based on review some of the examples, that this may not be the case. Obvioussly this schema could be more descriptive, but just shooting for mostly likely to be not wrong at this point. Date: 2020-12-15 14:44:37-06:00 Signed-off-by: meows <[email protected]>
Conflicts: api/apistruct/struct.go api/docgen/docgen.go cmd/lotus/rpc.go go.mod go.sum node/impl/storminer.go
Date: 2021-01-19 12:30:42-06:00 Signed-off-by: meows <[email protected]>
…worker.json.gz: run 'make docsgen' Date: 2021-01-19 12:33:55-06:00 Signed-off-by: meows <[email protected]>
Date: 2021-01-19 12:39:48-06:00 Signed-off-by: meows <[email protected]>
Date: 2021-01-19 12:52:04-06:00 Signed-off-by: meows <[email protected]>
…docsgen' Date: 2021-01-19 12:55:52-06:00 Signed-off-by: meows <[email protected]>
1d7f5de
to
7b7cd4f
Compare
This will returns empty comments/docs maps. This should fix issues like: https://app.circleci.com/pipelines/github/filecoin-project/lotus/12445/workflows/4ebadce9-a298-4ad1-939b-f19ef4c0a5bf/jobs/107218 where the environment makes file lookups hard or impossible. Date: 2021-01-19 13:04:58-06:00 Signed-off-by: meows <[email protected]>
It seems the failing test is not related to this PR... 🤞 |
* main: init implement rpc.Discover RPC method This implement the basic functionality for the method over HTTP RPC. Signed-off-by: meows <[email protected]> * main,go.mod,go.sum: init example with go-openrpc-reflect lib Signed-off-by: meows <[email protected]> Conflicts: go.mod go.sum * main: make variable name human-friendly Signed-off-by: meows <[email protected]> * main,go.mod,go.sum: init impl of go-openrp-reflect printing document Signed-off-by: meows <[email protected]> Conflicts: go.mod go.sum * go.mod,go.sum: use go-openrpc-reflect and open-rpc/meta-schema hackforks This is for development only. Versions need to be bumped when they're ready for use as canonical remotes. Signed-off-by: meows <[email protected]> * main,openrpc,main: refactor openrpc supporting code to own package This eliminates code duplication. Signed-off-by: meows <[email protected]> * main: add rpc.Discover to openrpc document Signed-off-by: meows <[email protected]> * openrpc: fix rpc.discover method name casing Also fixes casing stuff for the rest of Filecoin. methods. Signed-off-by: meows <[email protected]> * Revert "main: add rpc.Discover to openrpc document" This reverts commit 116898e. * main: fix document creation method name This fixes an issue caused with the latest reverting commit. Signed-off-by: meows <[email protected]> * main,docgen,openrpc: refactor to share api parsing, etc as docgen exported stuff Signed-off-by: meows <[email protected]> Makefile: fix docgen refactoring for makefile use of command Signed-off-by: meows <[email protected]> * openrpc: add schema.examples to app reflector There are quite of few of these already registered for the docgen command, so it makes sense to use those! Signed-off-by: meows <[email protected]> * openrpc: init method pairing examples Signed-off-by: meows <[email protected]> * go.mod,go.sum: bump go.mod to use latest meta-schema and openrpc-reflect versions Signed-off-by: meows <[email protected]> * openrpc: init SchemaType mapper function This function will handle the manual configurations for app-specific data types w/r/t their json schema representation. This is useful for cases where the reflect library is unable to provide a sufficient representation automatically. Provided in this commit is an initial implementation for the integerD type (assuming number are represented in the API as hexs), and a commonly used cid.Cid type. Signed-off-by: meows <[email protected]> * go.mod,go.sum: tame dependencies by bumping etclabscore/go-openrpc-reflect This removes a problematic dependency on github.com/ethereum/go-ethereum, which was imported as a dependency for a couple github.com/etclabscore/go-openrpc-reflect tests. etclabscore/go-openrpc-reflect v0.0.36 has removed this dependency, so this commit is the result of bumping that version and then running 'go mod tidy' This is in response to a review at #4711 (review) Date: 2020-11-21 06:52:48-06:00 Signed-off-by: meows <[email protected]> * main: add 'miner' arg to openrpc gen cmd This allows the command to EITHER generate the doc for Full or Miner APIs. See comment for usage. Date: 2020-11-21 07:48:05-06:00 Signed-off-by: meows <[email protected]> * docgen: add missing examples for Miner API Generating the Miner API OpenRPC doc (via 'go run ./api/openrpc/cmd miner') caused the example logic to panic because some types were missing. This commit adds those missing types, although I'm not an expert in the API so I can't suggest that the example values provided are ideal or well representative. Date: 2020-11-21 07:50:21-06:00 Signed-off-by: meows <[email protected]> * build/openrpc/full.json,build/openrpc/miner.json: add build/openrpc/[full/miner].json docs These will be used as static documents provided by the rpc.discover method. Date: 2020-11-21 07:51:39-06:00 Signed-off-by: meows <[email protected]> * build: init go-rice openrpc static assets Date: 2020-11-21 08:23:06-06:00 Signed-off-by: meows <[email protected]> * main: remove rpc.discover implementation from runtime plugin Instead of generating the doc on the fly, we're going to serve a static asset. Rel #4711 (review) This removes the runtime implementation from the RPC server construction. Date: 2020-11-21 08:41:20-06:00 Signed-off-by: meows <[email protected]> * api,apistruct,common: add Discover(ctx) method to CommonAPI interface and structs Date: 2020-11-21 08:41:56-06:00 Signed-off-by: meows <[email protected]> * main: use rpc server method aliasing for rpc.discover This depends on a currently-forked change at filecoin-project/go-jsonrpc 8350f9463ee451b187d35c492e32f1b999e80210 which establishes this new method RPCServer.AliasMethod. This solves the problem that the OpenRPC spec says that the document should be served at the system extension-prefixed endpoing rpc.discover (not Filecoin.Discover). In fact, the document will be available at BOTH endpoints, but that duplicity is harmless. Date: 2020-11-21 09:18:26-06:00 Signed-off-by: meows <[email protected]> * api,apistruct,build,common: rpc.discover: return json object instead of string Instead of casting the JSON asset from bytes to string, unmarshal it to a map[string]interface{} so the server will provide it as a JSON object. Date: 2020-11-21 09:27:11-06:00 Signed-off-by: meows <[email protected]> * Makefile: merge resolve: docsgen command path Date: 2020-11-22 07:19:36-06:00 Signed-off-by: meows <[email protected]> * apistruct,main,docgen,openrpc: merge resolve: fix func exporteds, signatures Date: 2020-11-22 07:31:03-06:00 Signed-off-by: meows <[email protected]> * go.mod,go.sum: 'get get' auto-bumps version Date: 2020-11-22 07:31:44-06:00 Signed-off-by: meows <[email protected]> * Makefile,docgen,main,build/openrpc: refactor openrpc documentation generation This creates Makefile command docsgen-openrpc-json, and refactors the docsgen command to generate both the markdown and openrpc json documents, redirecting the output of the openrpc json documentation to the build/openrpc/ directory, where those json files will be compiled as static assets via go-rice boxes. The api/openrpc/cmd now uses usage argumentation congruent to that of the docgen command (switching on API context). Date: 2020-11-22 08:01:18-06:00 Signed-off-by: meows <[email protected]> * main,docgen_openrpc: rename api/openrpc -> api/docgen-openrpc Renames the package as well. This is intended to parallel the existing docgen package and command namespacing. Date: 2020-11-22 10:34:46-06:00 Signed-off-by: meows <[email protected]> * api,apistruct,docgen,build,build/openrpc: use typed Discover response Instead of using a map[string]interface{}, use a typed response for the Discover method implementation. This avoids having to set a docgen Example for the generic map[string]interface{} (as an openrpc document) which both pollutes the generic type and lacks useful information for the Discover method example. Date: 2020-11-22 08:31:16-06:00 Signed-off-by: meows <[email protected]> * apistruct,build,main,impl: implement Discover method for Worker and StorageMiner APIs Methods return static compiled assets respective to the APIs. Date: 2020-11-22 08:57:18-06:00 Signed-off-by: meows <[email protected]> * docgen_openrpc,build/openrpc: remove timestamping from openrpc doc info This should allow openrpc docs generated at different times to be equal. This is important because the CI (Circle) runs the docgen command and tests that the output and the source are unchanged (via git diff). Date: 2020-11-22 10:47:07-06:00 Signed-off-by: meows <[email protected]> * main,docgen_openrpc,main,build: fix lint issues Fixes goimports, staticcheck, golint issues. Date: 2020-11-22 11:06:46-06:00 Signed-off-by: meows <[email protected]> * docgenopenrpc: fix: don't use an underscore in package name (golint) Date: 2020-11-22 11:07:53-06:00 Signed-off-by: meows <[email protected]> * go.sum: fix: mod-tidy-check (run 'go mod tidy') Date: 2020-11-22 11:09:48-06:00 Signed-off-by: meows <[email protected]> * go.mod,go.sum: bump filecoin-project/go-jsonrpc dep to latest This version includes the necessary RPCServer.AliasMethod method. Date: 2020-11-23 12:16:15-06:00 Signed-off-by: meows <[email protected]> * Makefile,main,build,build/openrpc: init gzipped openrpc static docs Date: 2020-11-24 06:15:06-06:00 Signed-off-by: meows <[email protected]> * build: refactor gzip reading Date: 2020-11-24 06:18:34-06:00 Signed-off-by: meows <[email protected]> * build: add basic test for openrpc doc from static assets Date: 2020-11-24 06:30:23-06:00 Signed-off-by: meows <[email protected]> * build: handle reader Close error This keeps the errcheck linter happy. Date: 2020-11-24 06:33:14-06:00 Signed-off-by: meows <[email protected]> * go.sum: run 'go mod tidy' Date: 2020-11-24 06:36:07-06:00 Signed-off-by: meows <[email protected]> * go.mod,go.sum: go mod tidy Tidying up after resolving the merge conflicts with master at go.mod Date: 2020-11-24 06:40:45-06:00 Signed-off-by: meows <[email protected]> * go.mod,go.sum: bump filecoin-project/go-jsonrpc to latest This is a repeat of 76e6fd2, since the latest merge to master seems to have reverted this. Date: 2020-11-24 06:42:30-06:00 Signed-off-by: meows <[email protected]> * docgenopenrpc,build/openrpc: remove method example pairings, improve schema examples Removing method example pairings since they were redundant to schema examples and were not implemented well. Improved schema examples by using the ExampleValue method instead of the map lookup. Made a note in the comment here that this is not ideal, since we have to make a shortcut assumption /workaround by using 'unknown' as the method name and the typea as its own parent. Luckily these values aren't heavily used by the method logic. Date: 2020-11-27 12:57:36-06:00 Signed-off-by: meows <[email protected]> * docgenopenrpc: use generic number jsonschema for number types Previously used an integer schema assuming hex encoding. It appears, based on review some of the examples, that this may not be the case. Obvioussly this schema could be more descriptive, but just shooting for mostly likely to be not wrong at this point. Date: 2020-12-15 14:44:37-06:00 Signed-off-by: meows <[email protected]> * cmd/lotus,go.mod,go.sum: maybe fix straggling merge resolution conflicts Date: 2021-01-19 12:30:42-06:00 Signed-off-by: meows <[email protected]> * build/openrpc/full.json.gz,build/openrpc/miner.json.gz,build/openrpc/worker.json.gz: run 'make docsgen' Date: 2021-01-19 12:33:55-06:00 Signed-off-by: meows <[email protected]> * api/apistruct,node/impl: (lint) gofmt Date: 2021-01-19 12:39:48-06:00 Signed-off-by: meows <[email protected]> * api/docgen: maybe fix parse error: open ./api: no such file or directory Date: 2021-01-19 12:52:04-06:00 Signed-off-by: meows <[email protected]> * api/docgen,build/openrpc: maybe fix no such file error and run 'make docsgen' Date: 2021-01-19 12:55:52-06:00 Signed-off-by: meows <[email protected]> * api/docgen: return if AST comment/groupdoc parsing encounters any error This will returns empty comments/docs maps. This should fix issues like: https://app.circleci.com/pipelines/github/filecoin-project/lotus/12445/workflows/4ebadce9-a298-4ad1-939b-f19ef4c0a5bf/jobs/107218 where the environment makes file lookups hard or impossible. Date: 2021-01-19 13:04:58-06:00 Signed-off-by: meows <[email protected]> * api: Don't depend on build/ * make: support parallel docsgen * openrpc gen: Use simple build version * methodgen * goimports Co-authored-by: meows <[email protected]>
Merged in #5843 |
Rel filecoin-project/devgrants#165
This is an implementation adding OpenRPC Document support for the Filecoin API(s).
Note that the
docsgen
program (easily accessed viamake docsgen
) also uses reflection to generate Markdown documentation, and that program has been refactored to share some logic with the OpenRPC documentation to avoid code duplication (eg. examples and comments).Would love to get some feedback on the general architecture (ie Is plugging directly in to the
rpc
package a best approach?), as well as a few other things, like Is re-using the given examples from the docsgen acceptable?, Is it desirable to add more detailed schemas for common data types (which has been exemplified by a tentativecid.Cid
)?We'd be happy and eager to schedule a meeting to talk through some of these ideas and questions if interest and time allows.
Here's a gist of what the rendered
rpc.discover
response can look like at the OpenRPC Playground: https://playground.open-rpc.org/?url=https://gist.githubusercontent.com/meowsbits/2f22b61ce46f6897d343f850ac86b540/raw/openrpc_doc.json