v0.11.0

v0.11.0

This release contains 4 months of work moving towards a production-ready implementation of Celestia's data availability network.

---

Notable changes

Most notable changes (outside of general fixes) were focused around storage performance and API/RPC improvements and stability.

More granular changes can be found in the notes of all release candidates leading up to this release.

Some notable changes can be found below:

API + RPC changes and improvements

• introduction of the BlobModule
• refactor of the rpc-cli
• changes to the Getter interface
• deprecation of write-level endpoints on the Gateway
• elevation of permissions to minimum read-level for all endpoints on the JSON-RPC client
• new Namespace type

Storage performance optimizations and changes

This release includes several fixes, optimizations and breaks to the EDSStore (and the DAGStore) so it is incompatible with v0.10.4.

Network-level changes

This release contains a protocol-breaking change: a breaking fix to shrex-nd to allow blobs > 1MB in size.

Default network will now be celestia mainnet

The node will now point to the celestia main network by default. To connect to mocha-4 or arabica-10, use the --p2p.network flag.

---

What's Changed

• feat!: bump app v1.0.0-rc0 by @evan-forbes in #2253
• refactor(misc): pretty print namespace.iD and peer.ID by @walldiss in #2255
• test: delete Test_dataFromSharesBSR by @rootulp in #2271
• fix(share/discovery): timeout FindPeers operation by @Wondertan in #2263
• feat!(blob): blob module by @vgonkivs in #2273
• chore(deps): bump go.uber.org/fx from 1.19.2 to 1.19.3 by @dependabot in #2199
• chore(deps): bump golang.org/x/crypto from 0.7.0 to 0.9.0 by @dependabot in #2276
• chore(deps): bump codecov/codecov-action from 3.1.3 to 3.1.4 by @dependabot in #2235
• chore(deps): bump alpine from 3.17.3 to 3.18.0 by @dependabot in #2198
• chore!: bump arabica testnet params by @Bidon15 in #2281
• feat(nodebuilder): add ip resolving capabilities to core.ip by @sysrex in #2277
• chore(share/ipld): pretify error by @Wondertan in #2284
• feat(blob): add marshaling for blob and proof by @vgonkivs in #2286
• feat(nodebuilder): Add jwt.Signer to Node struct for auth token access by @renaynay in #2295
• chore(nodebuilder): decrease lifecycle timeout for LN by @Wondertan in #2290
• fix!(blob/module): fix api by changing variadic params to a slice by @vgonkivs in #2298
• fix(gateway): remove the GetByHeight hussle and rely on Module's logic by @Wondertan in #2302
• fix(blob): use default min gas price from core by @distractedm1nd in #2306
• feat(cmd/rpc): adding blob module to rpc cli and fixing namespace parsing by @distractedm1nd in #2308
• fix(getter/cascade): cascade getter should return fast on context cancel by @walldiss in #2310
• fix(nodebuilder): fix parsing url for core.ip by @walldiss in #2313
• chore(deps): bump github.com/filecoin-project/go-jsonrpc from 0.1.9 to 0.3.1 by @dependabot in #2318
• fix(das): limit amount of recent jobs by @walldiss in #2314
• chore(state): extend error handling in core accessor by @vgonkivs in #2307
• chore: bump celestia-app to v1.0.0-rc2 by @evan-forbes in #2305
• chore: bump common docker ci version pipeline by @Bidon15 in #2330
• !feat(api/gateway): return txdata from submitPFB if it is present by @walldiss in #2329
• fix(cmd/rpc): plaintext parsing of data for blob.Submit and state.SubmitPayForBlob by @distractedm1nd in #2340
• fix(dagstore): changing log level to WARN to shorten startup by @distractedm1nd in #2336
• fix(share): left pad nid in NewNamespaceV0 instead of right padding by @distractedm1nd in #2341
• fix(daser) don't count recent jobs in total sampled stats by @walldiss in #2342
• fix(state): fixing panic when err != nil but response is nil by @distractedm1nd in #2339
• fix!(state): use node blob type instead of app type to fix inconsistent unmarshalling by @distractedm1nd in #2338
• chore: mocha-2 support by @evan-forbes in #2326
• fix(nodebuilder): Fix dns lookup by @walldiss in #2332
• fix(rpc/client): stop overwriting existing clients by @distractedm1nd in #2361
• fix(state): JSON marshaling for sdk Address wrapper by @distractedm1nd in #2348
• chore!(gateway): add deprecation notices by @distractedm1nd in #2360
• chore(deps): bump github.com/pyroscope-io/client from 0.7.0 to 0.7.1 by @dependabot in #2319
• fix(header): GetByHeight requests for syncer head +1 shouldnt error by @distractedm1nd in #2362
• fix(share/eds): fix test flake by @Wondertan in #2364
• chore(deps): bump mheap/github-action-required-labels from 4 to 5 by @dependabot in #2354
• feat!(nodbuilder): making lifecycle timeouts configurable by @distractedm1nd in #2347
• chore(deps): bump golangci/golangci-lint-action from 3.4.0 to 3.6.0 by @dependabot in #2353
• fix(swamp): use synchronized map as container for nodes by @Wondertan in #2358
• chore(deps): bump github.com/imdario/mergo from 0.3.15 to 0.3.16 by @dependabot in #2355
• chore(deps): bump celestiaorg/.github from 0.1.1 to 0.2.0 by @dependabot in #2317
• chore(nodebuilder/tests): Clean up sync_test tests by @renaynay in #2162
• feat(share): collect absence proof if namespace is not present in merkle tree by @walldiss in #2242
• deps: bump libp2p by @Wondertan in #2359
• feat(node): implement node.AuthVerify and node.AuthNew by @distractedm1nd in #2370
• fix: parseNamespace for <= 10 byte namespace IDs by @rootulp in #2325
• add kind:testing label by @walldiss in #2375
• feat(libs/pidstore): Implement PeerIDStore by @renaynay in #2274
• Update ci_release.yml by @MSevey in #2384
• feat(gateway): enabling CORS * on gateway by @distractedm1nd in #2373
• share: introduce Namespace type by @Wondertan in #2367
• feat(header): Check app version in eh validation and return err on mismatch by @renaynay in #2138
• chore(share/getters): add unit test for ctxwithsplittimeout by @walldiss in #2374
• chore: bump celestia-app rc4 by @evan-forbes in #2371
• feat(share/getter) Add support for Non-inclusion proofs by @walldiss in #2256
• fix(api/gateway): Handle err from tx properly to avoid panic by @renaynay in #2393
• chore: bump celestia-app to v1.0.0-rc5 by @evan-forbes in #2395
• deps: bump libp2p by @Wondertan in #2399
• fix(libs/utils): Use valid ip4 address by @renaynay in #2394
• feat: adding spans to shrex getter by @distractedm1nd in #2404
• refactor!(share): integrate new Namespace type by @Wondertan in #2388
• chore(go.mod): Bump nmt by @renaynay in #2405
• tests(share/eds): add store basic benchmarks by @walldiss in #2407
• chore!: bump arabica-9 by @jrmanes in #2417
• feat: update Go base alpine - add branch in makefile by @jrmanes in #2414
• doc(share/p2p/shrexnd): Improve accuracy of RequestND description by @musalbas in #2418
• chore(deps): bump alpine from 3.18.0 to 3.18.2 by @dependabot in #2380
• chore(nodebuilder/p2p)! Upgrade to mocha-3 by @sysrex in #2432
• share/befp: add benchmarks for fraud proof creation and validation by @vgonkivs in #2400
• deps: bump go-header by @Wondertan in #2436
• chore: bump celestia-app to v1.0.0-rc8 by @evan-forbes in #2434
• deps: bump go-header by @Wondertan in #2438
• feat: use context in the common pipeline by @jrmanes in #2447
• fix(header): temporary disable skipping verification by @Wondertan in #2456
• share/befp: rework collecting and verifying the BEFP by @vgonkivs in #2408
• chore: bump celestia app v1.0.0-rc9 by @evan-forbes in #2458
• fix(lint): fix importing by @vgonkivs in #2467
• cmd: rpc.go fix error parsing height message by @tuxcanfly in #2469
• fix(nodebuilder): fix nil pointer exception and simplify synchronization by @Wondertan in #2466
• docs: clarify usage of len(merkleRowRoots) by @rootulp in #2431
• chore(share/p2p/discovery): demote disc peers log to debug by @renaynay in #2473
• chore(deps): bump otel, go-header, and go-fraud by @walldiss in #2472
• chore: move versioning outside the main package by @vgonkivs in #2468
• refactor(share/availability): don't log error canceled operation by @walldiss in #2475
• feat(nodebuilder): Invoke traces from fx by @vgonkivs in #2477
• fix: remove root from traces by @vgonkivs in #2483
• feat(share/byzantine): handle additional cases in befp validation by @vgonkivs in #2383
• chore(deps): bump celestiaorg/.github from 0.2.0 to 0.2.2 by @dependabot in #2484
• fix(nodebuilder/tests/swamp): Bump timeout by @renaynay in #2504
• dep: update nmt version by @vgonkivs in #2509
• feat: new metrics - deployment dashboard by @jrmanes in #2464
• test(share/utils): fix no deadline test for ctxWithSplitTimeout by @walldiss in #2502
• fix(share/availability): update availability ErrNotAvailable handling logic by @walldiss in #2445
• chore(deps): bump cosmossdk.io/errors from 1.0.0-beta.7 to 1.0.0 by @dependabot in #2486
• fix: check for namespace version 0 by @rootulp in #2514
• chore(deps): bump github.com/spf13/cobra from 1.6.1 to 1.7.0 by @dependabot in #2487
• test(share/p2p): fix namespace add operations in tests by @walldiss in #2422
• fix(share/eds): remove Has() and Sync() badger calls in inverted index by @walldiss in #2518
• fix(share/eds): turn off eds store dagstore gc by default by @walldiss in #2529
• feat(share/shrex/nd): add swamp test for shrex nd by @walldiss in #2227
• feat(share/eds): introduce List method for the Store by @Wondertan in #2494
• fix(share/eds): print error for stuck register shard by @walldiss in #2516
• chore(deps): bump celestia-app to rc10 by @cmwaters in #2508
• chore: bump app v1.0.0 rc11 by @evan-forbes in #2527
• fix(share/p2p): fix race in shrexeds test by @walldiss in #2534
• feat(share/store): add eds store metrics by @walldiss in #2536
• !feat(share/p2p/shrex-nd): rework shrex-nd to serve data in multiple messages by @walldiss in #2444
• chore: bump libp2p by @Wondertan in #2533
• feat(cmd): use rpc client instead of http.Request by @vgonkivs in #2521
• feat(eds/store): cache proofs after first eds recompute on sampling by @walldiss in #2429
• release!: Breaking release V0.11.0 rc9 by @walldiss in #2545
• misc(metrics)!: split global attributes into two by @walldiss in #2546
• fix(blob): ensure that the share sequence is not empty by @vgonkivs in #2547
• fix(rpc): fix commands handling by @vgonkivs in #2515
• fix(cmd): fix formatting during submit by @vgonkivs in #2551
• fix(share/eds): don't use in-memory buffer for mount after shard recover by @walldiss in #2554
• fix(header): Only bridge node should panic on data root mismatch in e… by @renaynay in #2558
• fix(share/eds): dagstore shard restore reflection workaround by @walldiss in #2559
• metrics(runtime): add runtime metrics by @walldiss in #2570
• fix(nodebuilder): set default log level for bitswap-client by @walldiss in #2576
• fix: TestShareAvailable_DisconnectedFullNodes by @distractedm1nd in #2560
• chore(deps): bump ipfs networking deps by @walldiss in #2478
• fix(share/discovery): discovery should try to connect to empty hosts peers by @walldiss in #2575
• feat(eds/store): add eds blockstore cache metrics by @walldiss in #2567
• chore: bump to rsmt2d v0.11.0 by @rootulp in #2580
• fix(share/ipld): Remove proofs from leaked ctx by @walldiss in #2574
• feat(nodebuilder/state): Provide stubbed state module if a core endpoint not provided by @renaynay in #2577
• chore!: update celestia-app to v1.0.0-rc12 by @evan-forbes in #2581
• chore: bump deps + go1.21 by @Wondertan in #2589
• fix(share/eds): scale up inverted_index badger levels by @walldiss in #2591
• feat(blob): improve gas estimation and track min gas price by @cmwaters in #2511
• fix(cmd/blob): fix response by @vgonkivs in #2600
• deps: bump dagstore by @walldiss in #2598
• nodebuilder/core: Set default ports for RPC and GRPC by @renaynay in #2596
• feat(cmd/node): add logs cmd to dynamically change the log level by @vgonkivs in #2601
• chore(deps): bump dagstore by @walldiss in #2602
• refactor: address namespace follow-ups from celestia-app v1.0.0-rc12 bump by @rootulp in #2597
• feat(edssser): introduce EDS Store Stresser and cel-shed utility by @Wondertan in #2482
• feat(share/p2p/discovery): Warning for discovery loop if haven't found wanted peers in 5 minutes by @nodersteam in #2573
• fix(eds/blockstore): Puts on EDSStore Blockstore by @distractedm1nd in #2532
• fix: fixing panic on GetShare with out of bounds indexes by @distractedm1nd in #2605
• chore: bump go-header and go-fraud by @Wondertan in #2619
• chore(nodebuilder/p2p)!: Bump arabica-10 by @jrmanes in #2639
• chore(deps): bump golanci-lint workflow by @distractedm1nd in #2641
• feat(share/eds) add dagstore shards status metric by @walldiss in #2642
• refactor(share)!: have NamespacedShares fields use snake_case in JSON by @distractedm1nd in #2633
• chore(deps): bump golangci/golangci-lint-action from 3.6.0 to 3.7.0 by @dependabot in #2586
• chore(deps): bump alpine from 3.18.2 to 3.18.3 by @dependabot in #2564
• chore(deps): bump cosmossdk.io/math from 1.1.1 to 1.1.2 by @dependabot in #2594
• tests(swamp): befp testing by @vgonkivs in #2584
• chore(deps): bump github.com/ipfs/go-ipld-cbor from 0.0.6 to 0.1.0 by @dependabot in #2622
• chore(deps): bump github.com/ipfs/go-ipld-format from 0.5.0 to 0.6.0 by @dependabot in #2621
• feat: local docker telemetry infra by @distractedm1nd in #2296
• chore(deps): bump github.com/ipfs/go-block-format from 0.1.2 to 0.2.0 by @dependabot in #2623
• chore!: bump celestia-app to v1.0.0-rc13 by @evan-forbes in #2646
• fix(blob): typo, sumitter -> submitter by @ramin in #2648
• chore(nodebuilder/tests): Clean up integration tests in p2p_test by @renaynay in #2166
• refactor(blob): removing jsonProof by @distractedm1nd in #2652
• feat(eds/store): remove corrupted blocks from store by @distractedm1nd in #2625
• chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp from 0.39.0 to 0.40.0 by @dependabot in #2635
• chore(deps): migrate to boxo by @Wondertan in #2595
• chore(deps): bump github.com/ipfs/go-ipfs-ds-help from 1.1.0 to 1.1.1 by @dependabot in #2624
• chore(go.mod): bump go-header by @renaynay in #2657
• feat(nodebuilder/header): Bootstrap from previously seen peers by @renaynay in #2507
• feat(modp2p): listen on WebTransport by default by @Wondertan in #2660
• fix(share/eds): skip concurrent writes on edsstore Put by @walldiss in #2613
• chore(lint/sort-imports): fix absolute path for sort-imports by @walldiss in #2640
• chore: cleanup usage of share.Root(DAH) by @Wondertan in #2481
• improvement!(blob/service): add options to submit by @vgonkivs in #2630
• chore!: bump mocha chain with new bootstrappers and genesis hash by @Bidon15 in #2668
• feat(headertest): Validate exact error types in header tests by @Manav-Aggarwal in #2644
• chore: bump celestia-app to v1.0.0-rc14 by @evan-forbes in #2662
• chore: bump libp2p by @Wondertan in #2676
• fix(modshare): start peer manager after Syncer by @Wondertan in #2677
• feat(cmd/blob): add fee and gasLimit flags by @vgonkivs in #2669
• chore(nodebuilder/p2p): BlockTime from 15 sec --> 10 sec by @renaynay in #2692
• refactor(nodebuilder/header): get rid of InitStore by @renaynay in #2678
• chore: fix unknown in ghcr registry by @Bidon15 in #2693
• chore: bump docker ci to v0.2.3 by @Bidon15 in #2696
• chore(deps): Bump actions/checkout from 3 to 4 by @dependabot in #2690
• chore(deps): Bump golang.org/x/crypto from 0.12.0 to 0.13.0 by @dependabot in #2688
• chore(deps): Bump github.com/ipfs/go-block-format from 0.1.2 to 0.2.0 by @dependabot in #2682
• chore(deps): Bump github.com/libp2p/go-libp2p-routing-helpers from 0.7.2 to 0.7.3 by @dependabot in #2687
• chore(deps): Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.42.0 to 0.43.0 by @dependabot in #2685
• chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.16.0 to 1.17.0 by @dependabot in #2683
• Stale issue github workflow by @ramin in #2667
• refactor(swamp): use RPC client instead of service pointers by @distractedm1nd in #2699
• Goreleaser by @MSevey in #2661
• fix(shrexeds): close stream by @Wondertan in #2705
• feat(share/eds): Rework accessor cache by @walldiss in #2612
• chore(deps): Bump actions/checkout from 3 to 4 by @dependabot in #2718
• chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp from 0.40.0 to 0.41.0 by @dependabot in #2713
• chore(deps): Bump github.com/hashicorp/golang-lru/v2 from 2.0.5 to 2.0.6 by @dependabot in #2711
• feat(share/store/cache): Split accessor cache into recent and blockstore by @walldiss in #2656
• chore(deps): Bump mathieudutour/github-tag-action from 6.0 to 6.1 by @dependabot in #2721
• chore(deps): Bump goreleaser/goreleaser-action from 4 to 5 by @dependabot in #2719
• chore(deps): Bump celestiaorg/.github from 0.2.2 to 0.2.3 by @dependabot in #2720
• chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.17.0 to 1.18.0 by @dependabot in #2714
• chore(dependabot): add otel group to dependabot by @walldiss in #2722
• chore(deps): Bump go.uber.org/zap from 1.25.0 to 1.26.0 by @dependabot in #2731
• chore(deps): Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.43.0 to 0.44.0 by @dependabot in #2732
• docs: Add share p2p documentation by @derrandz in #2008
• feat(share/eds/cache): Add accessor cache refs counter by @walldiss in #2723
• feat(share/eds/store): expose eds store params by @walldiss in #2724
• deps!: bump nmt, breaking JSON RPC return types including nmt Proofs by @distractedm1nd in #2728
• refactoring(full/availability): removing TeeGetter, storing via SharesAvailable by @distractedm1nd in #2726
• chore(deps): Bump google.golang.org/grpc from 1.58.0 to 1.58.1 by @dependabot in #2740
• chore(deps): Bump github.com/ipfs/boxo from 0.12.0 to 0.13.0 by @dependabot in #2734
• fix(full/availability): short-circuiting SharesAvailable on empty DAH by @distractedm1nd in #2746
• fix: improve ErrNoStateAccess message by @distractedm1nd in #2752
• refactor(availability): merging cache.ShareAvailability and light.ShareAvailability by @distractedm1nd in #2741
• refactor(share/discovery): decrease default Advertisement interval by @walldiss in #2758
• chore!: bump core and app by @cmwaters in #2753
• chore(deps): Bump github.com/ipfs/boxo from 0.13.0 to 0.13.1 by @dependabot in #2757
• chore(deps): Bump google.golang.org/grpc from 1.58.1 to 1.58.2 by @dependabot in #2756
• fix(share/p2p/peers): Prevent making pools for nil DataHash by @renaynay in #2761
• ignore dependabot in issue-label-automation/automate-labels workflow by @ramin in #2763
• test(share/eds/cache): benchmark eds cache eviction performance by @walldiss in #2778
• chore(deps): fix dependabot otel grouping by @walldiss in #2795
• chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace from 1.18.0 to 1.19.0 by @dependabot in #2784
• chore: bump celestia-app, celestia-core, and the cosmos-sdk by @evan-forbes in #2801
• refactoring!(rpc/cmd): rework rpc commands by @vgonkivs in #2812
• chore(deps): Bump the otel group with 2 updates by @dependabot in #2800
• chore(deps): Bump github.com/prometheus/client_golang from 1.16.0 to 1.17.0 by @dependabot in #2787
• chore(deps): Bump github.com/hashicorp/golang-lru/v2 from 2.0.6 to 2.0.7 by @dependabot in #2785
• chore(deps): Bump alpine from 3.18.3 to 3.18.4 by @dependabot in #2794
• (chore) ensure all dependency group PRs get the default label by @ramin in #2798
• deps: bump go-header by @renaynay in #2813
• fix(share/peer-manager): use mutex for peer-manager pool cleanup by @walldiss in #2817
• chore(deps): Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.44.0 to 0.45.0 by @dependabot in #2819
• feat(discovery)!: discover peers by tag by @walldiss in #2730
• chore(deps): Bump golang.org/x/crypto from 0.13.0 to 0.14.0 by @dependabot in #2821
• fix(docs): fix GetVerifiedRange doc by @vgonkivs in #2826
• chore(deps): Bump golang.org/x/sync from 0.3.0 to 0.4.0 by @dependabot in #2820
• fix(rpc)!: various rpc breaks by @distractedm1nd in #2825
• chore(share/ipld): obliterate expensive ipld getter tracing by @Wondertan in #2832
• fix(share/p2p/peer-manager) fix race for hasPeerCh pointer read by @walldiss in #2835
• chore: bump to the official v1.0.0 tag of celestia-app by @evan-forbes in #2831
• fix(share/eds): fix races in cache tests by @walldiss in #2834
• chore: Update docker-build-publish.yml by @MSevey in #2838
• fix(modp2p): temporary disable quic by @Wondertan in #2837
• chore: adding mainnet chain id and setting as default by @distractedm1nd in #2840
• bug: fix duplicate docker build stages by @MSevey in #2845
• fix(blob/service): fix getByCommitment by @vgonkivs in #2828
• fix: removing chainid from default store name for mainnet by @distractedm1nd in #2841
• deps!: Bumps go-header by @renaynay in #2844
• misc(blob): Add debug logs for the GetAll method by @rise2semi in #2749
• chore(deps): Bump tspascoal/get-user-teams-membership from 2 to 3 by @dependabot in #2827
• chore(deps): Bump github.com/celestiaorg/celestia-app from 1.0.0 to 1.1.0 by @dependabot in #2850
• fix(share/getters): short-circuit on empty root in ShrexGetter by @distractedm1nd in #2846

New Contributors

• @musalbas made their first contribution in #2418
• @tuxcanfly made their first contribution in #2469
• @cmwaters made their first contribution in #2508
• @nodersteam made their first contribution in #2573
• @Manav-Aggarwal made their first contribution in #2644
• @rise2semi made their first contribution in #2749

Full Changelog: 0.10.1...v0.11.0