Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new node contract in registry #107

Merged
merged 1 commit into from
Aug 13, 2024
Merged

Use new node contract in registry #107

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ jobs:
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
args: --timeout=5m --config dev/.golangci.yaml
args: --timeout=5m --config .golangci.yaml
3 changes: 3 additions & 0 deletions dev/.golangci.yaml → .golangci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,6 @@ linters:
- ineffassign
- staticcheck
- unused
issues:
exclude-files:
- tools.go
3 changes: 3 additions & 0 deletions .mockery.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ mockname: "Mock{{.InterfaceName}}"
outpkg: mocks
filename: "mock_{{.InterfaceName}}.go"
packages:
github.com/xmtp/xmtpd/pkg/registry:
interfaces:
NodesContract:
github.com/xmtp/xmtpd/pkg/indexer/blockchain:
interfaces:
ChainClient:
Expand Down
42 changes: 21 additions & 21 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@

`xmtpd` (XMTP daemon) is an experimental version of XMTP node software. It is **not** the node software that currently forms the XMTP network.

After `xmtpd` meets specific functional requirements, the plan is for it to become the node software that powers the XMTP network.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think my editor has a default Markdown formatter. We can make that standard issue in the repo so we don't get diffs like this.

After `xmtpd` meets specific functional requirements, the plan is for it to become the node software that powers the XMTP network.

Some of these requirements include reaching functional parity with the current node software and reliably performing data replication without data loss.
Some of these requirements include reaching functional parity with the current node software and reliably performing data replication without data loss.

To keep up with and provide feedback about `xmtpd` development, see the [Issues tab](https://github.com/xmtp/xmtpd/issues) in this repo.

Expand Down Expand Up @@ -46,7 +46,7 @@ dev/down
To start the `xmtpd` node, run:

```sh
dev/start
dev/run
```

## Test the node
Expand All @@ -57,27 +57,27 @@ To run tests against the `xmtpd` node, run:
dev/test
```

These tests provide a full suite of unit and integration tests for the `xmtpd` repo to help ensure and maintain correctness of the code over time and to avoid regressions as the code evolves. You can explore the tests by taking a look at any files with the suffix `_test.go`.
These tests provide a full suite of unit and integration tests for the `xmtpd` repo to help ensure and maintain correctness of the code over time and to avoid regressions as the code evolves. You can explore the tests by taking a look at any files with the suffix `_test.go`.

## Monitor the node

The `xmtpd` node build provides two options for monitoring your node.

- To access your local Prometheus instance to explore node metrics, run:

```sh
open http://localhost:9090
```
```sh
open http://localhost:9090
```

To learn how to query node data in Prometheus, see [Metric Types in Prometheus and PromQL](https://promlabs.com/blog/2020/09/25/metric-types-in-prometheus-and-promql) and [The Anatomy of a PromQL Query](https://promlabs.com/blog/2020/06/18/the-anatomy-of-a-promql-query/).
To learn how to query node data in Prometheus, see [Metric Types in Prometheus and PromQL](https://promlabs.com/blog/2020/09/25/metric-types-in-prometheus-and-promql) and [The Anatomy of a PromQL Query](https://promlabs.com/blog/2020/06/18/the-anatomy-of-a-promql-query/).

- To access your local Grafana instance to explore and build node dashboards, run:

```sh
open http://localhost:3000
```
```sh
open http://localhost:3000
```

To learn how to visualize node data in Grafana, see [Prometheus Histograms with Grafana Heatmaps](https://towardsdatascience.com/prometheus-histograms-with-grafana-heatmaps-d556c28612c7) and [How to visualize Prometheus histograms in Grafana](https://grafana.com/blog/2020/06/23/how-to-visualize-prometheus-histograms-in-grafana/).
To learn how to visualize node data in Grafana, see [Prometheus Histograms with Grafana Heatmaps](https://towardsdatascience.com/prometheus-histograms-with-grafana-heatmaps-d556c28612c7) and [How to visualize Prometheus histograms in Grafana](https://grafana.com/blog/2020/06/23/how-to-visualize-prometheus-histograms-in-grafana/).

# Contributing

Expand All @@ -87,24 +87,24 @@ Please follow the [style guide](https://google.github.io/styleguide/go/decisions

Submit and land a PR to https://github.com/xmtp/proto. Then run:

```sh
dev/generate
```
```sh
dev/generate
```

## Modifying the database schema

Create a new migration by running:

```sh
dev/gen-migration
```
```sh
dev/gen-migration
```

Fill in the migrations in the generated files. If you are unfamiliar with migrations, you may follow [this guide](https://github.com/golang-migrate/migrate/blob/master/MIGRATIONS.md). The database is PostgreSQL and the driver is PGX.

## Modifying database queries

We use [sqlc](https://docs.sqlc.dev/en/latest/index.html) to generate the code for our DB queries. Modify the `queries.sql` file, and then run:

```sh
sqlc generate
```
```sh
sqlc generate
```
5 changes: 3 additions & 2 deletions cmd/replication/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"syscall"

"github.com/jessevdk/go-flags"
"github.com/xmtp/xmtpd/pkg/config"
"github.com/xmtp/xmtpd/pkg/registry"
"github.com/xmtp/xmtpd/pkg/server"
"github.com/xmtp/xmtpd/pkg/tracing"
Expand All @@ -18,7 +19,7 @@ import (

var Commit string

var options server.Options
var options config.ServerOptions

func main() {
if _, err := flags.Parse(&options); err != nil {
Expand Down Expand Up @@ -81,7 +82,7 @@ func fatal(msg string, args ...any) {
log.Fatalf(msg, args...)
}

func buildLogger(options server.Options) (*zap.Logger, *zap.Config, error) {
func buildLogger(options config.ServerOptions) (*zap.Logger, *zap.Config, error) {
atom := zap.NewAtomicLevel()
level := zapcore.InfoLevel
err := level.Set(options.LogLevel)
Expand Down
1 change: 1 addition & 0 deletions dev/abigen
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ rm -f ./pkg/abis/*.go
cd contracts

# Generate the abi files out of the solidity code
mkdir -p ../build
forge inspect ./src/Nodes.sol:Nodes abi > ../build/Nodes.abi.json
forge inspect ./src/GroupMessages.sol:GroupMessages abi > ../build/GroupMessages.abi.json

Expand Down
8 changes: 5 additions & 3 deletions dev/generate
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
#!/usr/bin/env sh
set -e
#!/usr/bin/env bash

set -euo pipefail

go generate ./...
mockery
rm -f pkg/mocks/*
./dev/abigen
mockery

rm -rf pkg/proto/**/*.pb.go pkg/proto/**/*.pb.gw.go pkg/proto/**/*.swagger.json
if ! buf generate https://github.com/xmtp/proto.git#subdir=proto; then
Expand Down
11 changes: 11 additions & 0 deletions dev/local.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/bin/bash

source dev/contracts/.env

export CHAIN_RPC_URL=$DOCKER_RPC_URL # From contracts/.env
export NODE_PRIVATE_KEY=$PRIVATE_KEY # From contracts/.env
Comment on lines +5 to +6
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason that dev/local.env and contracts/.env are not consolidated into a single file? For example, is the PRIVATE_KEY used anywhere else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use the stuff in contracts/local.env to deploy the smart contract locally. Could consolidate, but I'd like to figure out just how complicated all the local smart contract dev stuff is going to be with a full set of contracts and a more sophisticated deployer. My guess is that we're going to have a lot more configuration that is specific to the contracts.

export WRITER_CONNECTION_STRING="postgres://postgres:xmtp@localhost:8765/postgres?sslmode=disable"
NODES_CONTRACT_ADDRESS="$(jq -r '.deployedTo' build/Nodes.json)" # Built by contracts/deploy-local
export NODES_CONTRACT_ADDRESS
GROUP_MESSAGES_CONTRACT_ADDRESS="$(jq -r '.deployedTo' build/GroupMessages.json)" # Built by contracts/deploy-local
export GROUP_MESSAGES_CONTRACT_ADDRESS
12 changes: 12 additions & 0 deletions dev/run
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash

set -eu

. dev/local.env

go run cmd/replication/main.go \
--db.writer-connection-string=$WRITER_CONNECTION_STRING \
--private-key=${NODE_PRIVATE_KEY} \
--contracts.nodes-address=$NODES_CONTRACT_ADDRESS \
--contracts.messages-address=$GROUP_MESSAGES_CONTRACT_ADDRESS \
--contracts.rpc-url=$CHAIN_RPC_URL
5 changes: 4 additions & 1 deletion dev/up
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,18 @@
set -e

go mod tidy
git submodule update --init --recursive

if ! which forge &>/dev/null; then curl -L https://foundry.paradigm.xyz | bash ; fi
if ! which migrate &>/dev/null; then brew install golang-migrate; fi
if ! which golangci-lint &>/dev/null; then brew install golangci-lint; fi
if ! which shellcheck &>/dev/null; then brew install shellcheck; fi
if ! which mockery &>/dev/null; then brew install mockery; fi
if ! which mockery &>/dev/null; then go install github.com/vektra/mockery/v2; fi
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the tools.go file puts this in our go.mod, we can now get a deterministic version for the dependency when we do a Go install.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be simpler to just specify the version in the install command here, rather than updating the tools.go file?

if ! which sqlc &> /dev/null; then brew install sqlc; fi
if ! which buf &> /dev/null; then brew install buf; fi
if ! which golines &>/dev/null; then go install github.com/segmentio/golines@latest; fi
if ! which abigen &>/dev/null; then go install github.com/ethereum/go-ethereum/cmd/abigen; fi


dev/generate
dev/docker/up
Expand Down
89 changes: 89 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ require (
github.com/jackc/pgx/v5 v5.5.4
github.com/jessevdk/go-flags v1.6.1
github.com/pires/go-proxyproto v0.7.0
github.com/segmentio/golines v0.12.2
github.com/stretchr/testify v1.9.0
github.com/vektra/mockery/v2 v2.44.1
go.uber.org/zap v1.27.0
google.golang.org/genproto/googleapis/api v0.0.0-20240725223205-93522f1f2a9f
google.golang.org/grpc v1.65.0
Expand All @@ -25,56 +27,143 @@ require (
github.com/DataDog/go-libddwaf/v3 v3.2.1 // indirect
github.com/DataDog/go-tuf v1.0.2-0.5.2 // indirect
github.com/DataDog/sketches-go v1.4.5 // indirect
github.com/DataDog/zstd v1.4.5 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/StackExchange/wmi v1.2.1 // indirect
github.com/VictoriaMetrics/fastcache v1.12.2 // indirect
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect
github.com/alecthomas/units v0.0.0-20231202071711-9a357b53e9c9 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/bits-and-blooms/bitset v1.10.0 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.2.1 // indirect
github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/chigopher/pathlib v0.19.1 // indirect
github.com/cockroachdb/errors v1.11.3 // indirect
github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce // indirect
github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b // indirect
github.com/cockroachdb/pebble v1.1.1 // indirect
github.com/cockroachdb/redact v1.1.5 // indirect
github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 // indirect
github.com/consensys/bavard v0.1.13 // indirect
github.com/consensys/gnark-crypto v0.12.1 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/crate-crypto/go-ipa v0.0.0-20240223125850-b1e8a79f509c // indirect
github.com/crate-crypto/go-kzg-4844 v1.0.0 // indirect
github.com/dave/dst v0.27.3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/deckarep/golang-set/v2 v2.6.0 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
github.com/deepmap/oapi-codegen v1.6.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/ebitengine/purego v0.6.0-alpha.5 // indirect
github.com/ethereum/c-kzg-4844 v1.0.0 // indirect
github.com/ethereum/go-verkle v0.1.1-0.20240306133620-7d920df305f0 // indirect
github.com/fatih/structtag v1.2.0 // indirect
github.com/ferranbt/fastssz v0.1.2 // indirect
github.com/fjl/memsize v0.0.2 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/gballet/go-libpcsclite v0.0.0-20190607065134-2772fd86a8ff // indirect
github.com/getsentry/sentry-go v0.27.0 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/gofrs/flock v0.8.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/graph-gophers/graphql-go v1.5.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-bexpr v0.1.10 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/hcl v1.0.1-vault-5 // indirect
github.com/holiman/billy v0.0.0-20240216141850-2abb0c79d3c4 // indirect
github.com/holiman/bloomfilter/v2 v2.0.3 // indirect
github.com/holiman/uint256 v1.3.0 // indirect
github.com/huandu/xstrings v1.4.0 // indirect
github.com/huin/goupnp v1.3.0 // indirect
github.com/iancoleman/strcase v0.2.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/influxdata/influxdb-client-go/v2 v2.4.0 // indirect
github.com/influxdata/influxdb1-client v0.0.0-20220302092344-a9ab5670611c // indirect
github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
github.com/jackpal/go-nat-pmp v1.0.2 // indirect
github.com/jinzhu/copier v0.3.5 // indirect
github.com/klauspost/compress v1.17.1 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/lib/pq v1.10.9 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/pointerstructure v1.2.0 // indirect
github.com/mmcloughlin/addchain v0.4.0 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/outcaste-io/ristretto v0.2.3 // indirect
github.com/pelletier/go-toml/v2 v2.0.9 // indirect
github.com/peterh/liner v1.1.1-0.20190123174540-a2c9a5303de7 // indirect
github.com/philhofer/fwd v1.1.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.12.0 // indirect
github.com/prometheus/client_model v0.2.1-0.20210607210712-147c58e9608a // indirect
github.com/prometheus/common v0.32.1 // indirect
github.com/prometheus/procfs v0.7.3 // indirect
github.com/rivo/uniseg v0.4.4 // indirect
github.com/rogpeppe/go-internal v1.10.0 // indirect
github.com/rs/cors v1.7.0 // indirect
github.com/rs/zerolog v1.29.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/secure-systems-lab/go-securesystemslib v0.7.0 // indirect
github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spf13/afero v1.9.3 // indirect
github.com/spf13/cast v1.5.0 // indirect
github.com/spf13/cobra v1.6.1 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.15.0 // indirect
github.com/status-im/keycard-go v0.2.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.4.2 // indirect
github.com/supranational/blst v0.3.11 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d // indirect
github.com/tinylib/msgp v1.1.8 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/tyler-smith/go-bip39 v1.1.0 // indirect
github.com/urfave/cli/v2 v2.25.7 // indirect
github.com/x-cray/logrus-prefixed-formatter v0.5.2 // indirect
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.24.0 // indirect
golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
golang.org/x/mod v0.17.0 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/sync v0.7.0 // indirect
golang.org/x/sys v0.21.0 // indirect
golang.org/x/term v0.21.0 // indirect
golang.org/x/text v0.16.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240723171418-e6d459c13d2a // indirect
gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
rsc.io/tmplfunc v0.0.3 // indirect
)
Loading
Loading