Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid p2p dependency #160

Merged
merged 6 commits into from
Aug 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions README.org
Original file line number Diff line number Diff line change
Expand Up @@ -78,24 +78,23 @@ configured user permissions for a user named ~joe~, the following would connect
to a local Postgres database at port 5432:

#+begin_example
chainweb-data <command> --service-host=<node> --p2p-host=<node> --dbuser=joe --dbname=chainweb-data
chainweb-data <command> --service-host=<node> --dbuser=joe --dbname=chainweb-data
#+end_example

*** via a Postgres Connection String

#+begin_example
chainweb-data <command> --service-host=<node> --p2p-host=<node> --dbstring="host=localhost port=5432..."
chainweb-data <command> --service-host=<node> --dbstring="host=localhost port=5432..."
#+end_example

** Connecting to a Node

~chainweb-data~ syncs its data from a running ~chainweb-node~. The node's
P2P address is specified with the ~--p2p-host~ command. The node's
Service address is specified with the ~--service-host~ command.
If custom ports are used, you can specify them with ~--service-port~ and ~--p2p-port~
If a custom service enpoint port is used, you can specify it with ~--service-port~

#+begin_example
chainweb-data <command> --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com ...
chainweb-data <command> --service-host=foo.chainweb.com ...
#+end_example

*** Configuring the Node
Expand Down Expand Up @@ -133,7 +132,7 @@ will automatically run fill once a day to populate the DB with missing blocks.
configuration value is ~true~.

#+begin_example
> chainweb-data listen --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
> chainweb-data listen --service-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
DB Tables Initialized
28911337084492566901513774
#+end_example
Expand All @@ -146,11 +145,11 @@ As a new block comes in, its chain number is printed as a single digit.
~server~ is just like ~listen~, but also runs an HTTP server that serves a
few endpoints for doing common queries.
Additionally, it can serve an OpenAPI v3 spec of the API when the hidden
`--serve-swagger-ui` option is enabled, offering a basic interface for interacting
~--serve-swagger-ui~ option is enabled, offering a basic interface for interacting
with the API. This feature, however, is kept unofficial for now due to
its rudimentary documentation.

By specifying the optional `--no-listen` argument, the server can be made read-only,
By specifying the optional ~--no-listen~ argument, the server can be made read-only,
allowing multiple servers to serve from the same database.

**** Endpoints
Expand Down Expand Up @@ -199,7 +198,7 @@ so it's possible to keep searching by making successive calls with subsequent ~C
been improved to encompass all block filling operations.

#+begin_example
> chainweb-data fill --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
> chainweb-data fill --service-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
#+end_example

*** backfill
Expand All @@ -215,7 +214,7 @@ chain first via ~listen~ before doing ~backfill~! If ~backfill~ detects any
empty chains, it won't proceed.

#+begin_example
> chainweb-data backfill --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
> chainweb-data backfill --service-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
DB Tables Initialized
Backfilling...
[INFO] Processed blocks: 1000. Progress sample: Chain 9, Height 361720
Expand Down Expand Up @@ -245,7 +244,7 @@ releases. Use the ~fill~ command instead.
~single~.

#+begin_example
> chainweb-data gaps --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
> chainweb-data gaps --service-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
DB Tables Initialized
[INFO] Processed blocks: 1000. Progress sample: Chain 9, Height 361624
[INFO] Processed blocks: 2000. Progress sample: Chain 9, Height 362938
Expand All @@ -257,7 +256,7 @@ releases. Use the ~fill~ command instead.
~single~ allows you to sync a block at any location in the blockchain.

#+begin_example
> chainweb-data single --chain=0 --height=200 --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
> chainweb-data single --chain=0 --height=200 --service-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
DB Tables Initialized
[INFO] Filled in 1 blocks.
#+end_example
Expand All @@ -271,5 +270,5 @@ when orphans/forks are present at that height.
~check-schema~ is used to perform a check of the ORM definitions against the DB schema.

#+begin_example
> chainweb-data check-schema --service-host=foo.chainweb.com --p2p-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
> chainweb-data check-schema --service-host=foo.chainweb.com --dbuser=joe --dbname=chainweb-data
#+end_example
4 changes: 2 additions & 2 deletions cabal.project
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ source-repository-package
source-repository-package
type: git
location: https://github.com/kadena-io/chainweb-api.git
tag: ef2bee367136e298dfb45ea118d122350e7b3bc7
--sha256: sha256-VQ/JhpZaINeTZHNt7TZ9WaVGIXJEf9bDQ51cSG+auVI=
tag: 7fb1e73d2f7a0f0dd5703873c7204287cd7e8d25
--sha256: sha256-KaXVlMBVy6hDrKL2z54+jH4CIVjoWbS4X2RwT/QUMNo=

source-repository-package
type: git
Expand Down
4 changes: 0 additions & 4 deletions haskell-src/exec/Chainweb/Gaps.hs
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ import Network.Connection (TLSSettings(TLSSettingsSimple))
import Network.HTTP.Client
import Network.HTTP.Client.TLS
import System.Logger hiding (logg)
import qualified System.Logger as S
import System.Exit (exitFailure)
import Text.Printf

Expand Down Expand Up @@ -123,7 +122,6 @@ _test_headersBetween_and_payloadBatch = do
manager <- newManager $ mkManagerSettings (TLSSettingsSimple True False False) Nothing
let urlHost_ = "localhost"
serviceUrlScheme = UrlScheme Http $ Url urlHost_ 1848
p2pUrl = Url urlHost_ 443
nodeInfo = NodeInfo
{
_nodeInfo_chainwebVer = "mainnet01"
Expand All @@ -137,7 +135,6 @@ _test_headersBetween_and_payloadBatch = do
_env_httpManager = manager
, _env_dbConnPool = undefined
, _env_serviceUrlScheme = serviceUrlScheme
, _env_p2pUrl = p2pUrl
, _env_nodeInfo = nodeInfo
, _env_chainsAtHeight = undefined
, _env_logger = undefined
Expand Down Expand Up @@ -185,7 +182,6 @@ _test_getBlockGaps dbHost dbPort dbUser password dbName = withHandleBackend defa
_env_httpManager = m
, _env_dbConnPool = pool
, _env_serviceUrlScheme = UrlScheme Http $ Url "localhost" 1848
, _env_p2pUrl = Url "localhost" 443
, _env_nodeInfo = ni
, _env_chainsAtHeight = fromMaybe (error "chainsAtMinHeight missing") $ map (second (map (ChainId . fst))) <$> (_nodeInfo_graphs ni)
, _env_logger = lr
Expand Down
8 changes: 4 additions & 4 deletions haskell-src/exec/Chainweb/Lookups.hs
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ payloadWithOutputs env (T2 cid0 hsh0) = do
pure res
where
v = _nodeInfo_chainwebVer $ _env_nodeInfo env
url = showUrlScheme (UrlScheme Https $ _env_p2pUrl env) <> T.unpack query
url = showUrlScheme (_env_serviceUrlScheme env) <> T.unpack query
query = "/chainweb/0.0/" <> v <> "/chain/" <> cid <> "/payload/" <> hsh <> "/outputs"
cid = T.pack $ show cid0
hsh = unDbHash hsh0
Expand All @@ -184,8 +184,8 @@ queryCut :: Env -> IO (Either ApiError ByteString)
queryCut e = do
let v = _nodeInfo_chainwebVer $ _env_nodeInfo e
m = _env_httpManager e
u = _env_p2pUrl e
url = printf "%s/chainweb/0.0/%s/cut" (showUrlScheme $ UrlScheme Https u) (T.unpack v)
u = _env_serviceUrlScheme e
url = printf "%s/chainweb/0.0/%s/cut" (showUrlScheme u) (T.unpack v)
req <- parseRequest url
res <- handleRequest req m
pure $ responseBody <$> res
Expand Down Expand Up @@ -350,7 +350,7 @@ mkTxEvents :: Int64 -> ChainId -> DbHash BlockHash -> (CW.Transaction,Transactio
mkTxEvents height cid blk (tx,txo) = zipWith (mkEvent cid height blk (Just rk)) (_toutEvents txo) [0..]
where
rk = DbHash $ hashB64U $ CW._transaction_hash tx


mkEvent :: ChainId -> Int64 -> DbHash BlockHash -> Maybe (DbHash TxHash) -> Value -> Int64 -> Event
mkEvent (ChainId chainid) height block requestkey ev idx = Event
Expand Down
7 changes: 3 additions & 4 deletions haskell-src/exec/Main.hs
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,9 @@ main = do
logg Info $ "Constructing rich list using given db-path: " <> fromString fp
return fp
richList logg fp version
Args c pgc us u _ ms mbMigFolder -> do
Args c pgc us _ ms mbMigFolder -> do
logg Info $ "Using database: " <> fromString (show pgc)
logg Info $ "Service API: " <> fromString (showUrlScheme us)
logg Info $ "P2P API: " <> fromString (showUrlScheme (UrlScheme Https u))
withCWDPool pgc $ \pool -> do
runMigrations pool logg ms mbMigFolder
let mgrSettings = mkManagerSettings (TLSSettingsSimple True False False) Nothing
Expand All @@ -82,7 +81,7 @@ main = do
case mcids of
Nothing -> logg Error "Node did not have graph information" >> exitFailure
Just cids -> do
let !env = Env m pool us u ni cids logg
let !env = Env m pool us ni cids logg
case c of
Listen etlenv ->
case etlenv of
Expand Down Expand Up @@ -112,7 +111,7 @@ main = do
& loggerConfigThreshold .~ level
backendConfig = defaultHandleBackendConfig
getLevel = \case
Args _ _ _ _ level _ _ -> level
Args _ _ _ level _ _ -> level
RichListArgs _ level _ -> level
MigrateOnly _ level _ -> level
CheckSchema _ level -> level
Expand Down
10 changes: 7 additions & 3 deletions haskell-src/lib/ChainwebData/Env.hs
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ import Text.Printf
type MigrationsFolder = FilePath

data Args
= Args Command Connect UrlScheme Url LogLevel MigrationAction (Maybe MigrationsFolder)
= Args Command Connect UrlScheme LogLevel MigrationAction (Maybe MigrationsFolder)
-- ^ arguments for all but the richlist command
| RichListArgs NodeDbPath LogLevel ChainwebVersion
-- ^ arguments for the Richlist command
Expand All @@ -81,7 +81,6 @@ data Env = Env
{ _env_httpManager :: Manager
, _env_dbConnPool :: Pool Connection
, _env_serviceUrlScheme :: UrlScheme
, _env_p2pUrl :: Url
, _env_nodeInfo :: NodeInfo
, _env_chainsAtHeight :: [(BlockHeight, [ChainId])]
, _env_logger :: LogFunctionIO Text
Expand Down Expand Up @@ -260,10 +259,15 @@ envP = Args
<$> commands
<*> connectP
<*> urlSchemeParser "service" 1848
<*> urlParser "p2p" 443
<*> logLevelParser
<*> migrationP
<*> migrationsFolderParser
-- We keep the p2p options around for backwards compatibility, but they're unused
<* ignoredP2pParser
where
ignoredP2pParser = ()
<$ strOption (long "p2p-host" <> internal <> value ("unused" :: String))
<* strOption (long "p2p-port" <> internal <> value ("unused" :: String))

migrationsFolderParser :: Parser (Maybe MigrationsFolder)
migrationsFolderParser = optional $ strOption
Expand Down