Release v2.10.8

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.6

Dependencies

• golang.org/x/crypto v0.18.0
• golang.org/x/sys v0.16.0
• github.com/nats-io/nkeys v0.4.7

Added

TLS

• Add 'certs' option to TLS block for multi-cert support (#4889)

Improved

General

• Random number generation now uses a faster lock-free algorithm (#4901)
• no_auth_user is now allowed to be an nkey (#4938)

JetStream

• Improve matching efficiency of filter subjects in consumer (#4864) Thanks to @svenfoo for the contribution!
• Optimize JetStream metalayer snapshots by reducing allocations and simplifying marshaling (#4925)
• Micro-optimization where subject tokenization occurs (#4880) Thanks to @svenfoo for the contribution!
• Prevent backing up internal JS API requests in large-scale source and mirror setups (#4884)
• Optimize catchups for replicas and mirrors where there are a significant number of interior deletes (#4929)
• Reduce lock contention on the stream lock for some operations that could block routes & gateways (#4933)
• Do not load all blocks for NumPending when delivery is LastPerSubject (#4885)
• Call stream update only if the config has changed (#4898)
• Prevent large memory buildup in the apply queue for NRGs during startup (#4895)
• Finding the last sourced message for each source of a stream is now much faster (#4935)

MQTT

• Retained messages will now be fetched concurrently for a new subscription (#4835)

Fixed

Accounts

• Guard account random number generator with mutex (#4894) Thanks to @igorrius for the report!

JetStream

• Fix accounting for replicas and tier limits (#4868, #4909)
• Ensure all filter subjects across consumers are accounted for when purging a stream (#4873) Thanks to @svenfoo for the contribution!
• Detect corrupt psim subjects during recovery of index.db (#4890)
• Don’t allow writing snapshots to disk before recovery has completed (#4927)
• Reduce memory usage during purge operations by flushing cache (#4905)
• Return an “Account not enabled” error when trying to access JetStream via the system account (#4910)
• Reduce the number of blocks loaded into memory when doing linear scans (#4916)

Leafnodes

• Mapping updates on reload for the global account are now propagated to leafnodes correctly (#4937)
• Leafnode authorization now supports nkeys (#4940)

MQTT

• Fixed an out-of-date error message on unsupported characters in MQTT topics (#4903)

OCSP

• Default to Unknown status instead of Good for unknown status assertions (#4917)
• Fixed OCSP Stapling not resuming for gateways on reload after certs change (#4943)

Complete Changes

v2.10.7...v2.10.8

Release v2.10.7

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.5

Dependencies

• github.com/klauspost/compress v1.17.4
• golang.org/x/crypto v0.16.0
• golang.org/x/sys v0.15.0
• golang.org/x/time v0.5.0

Improved

JetStream

• Increase minimum interval for full index.db state writes to reduce contention for high-speed ingest in large streams (#4858)

Fixed

JetStream

• Corruption of per-subject tracking on recover of bad or missing index.db (#4851) Thanks to @oscarwcl for the report!
• Prevent GetSeqFromTime panic in memstore if the last sequence was deleted (#4853) Thanks to @andreib1 for the report!
• Protect against bad consumer state and high-memory usage during rollback (#4857)

Complete Changes

v2.10.6...v2.10.7

Release v2.10.6

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.4

Dependencies

• github.com/klauspost/compress v1.17.3

Improved

JetStream

• Added in internal filestore state checks on write and recover (#4804)
• Reduce memory usage for streams with a large subject space (#4806)
• Only gather subject filters if we need them (#4820) Thanks to @svenfoo for the contribution!
• Add pre-check when expected-last-subject-sequence header is present (#4827)
• Avoid resetting WAL in RAFT layer if we already processed the message (#4830)

Monitoring

• Remove ocsp_peer_cache from varz response when not applicable (#4829)

Fixed

JetStream

• Only drop firstSeq under DiscardOld policy (#4802) Thanks to @davidmcote for the report and contribution!
• Do not allow consumers to be updated if they have been deleted (#4818) Thanks to @matevzmihalic for the report!
• Fix potential race when starting the consumer monitor (#4828)
• Fix race condition in debug print (#4833)

MQTT

• Fix typo in README (#4791) Thanks to @testwill for the contribution!
• Improved large number of MQTT clients on reconnect with retain messages and larger scoped subscriptions (#4810)

WebSockets

• Fix potential data race in overlapping re-use of buffers (#4811) Thanks to @oscarwcl for the report!

Complete Changes

v2.10.5...v2.10.6

Release v2.10.5

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.4

Dependencies

• golang.org/x/crypto v0.15.0
• golang.org/x/sys v0.14.0
• golang.org/x/time v0.4.0
• github.com/nats-io/jwt/v2 v2.5.3

Improved

General

• Remove places where using time.After could cause GC pressure (#4756)

JetStream

• Remove unused Observer const, add unit test to check observer applies (#4727)
• Throttle writeFullState from separate goroutine (#4731)
• Reduce memory usage with lots of subjects in filestore (#4742)
• Resiliency when doing lots of conditional updates to a KV and restarting servers (#4764)
• ​​General stability and consistency improvements for clustered streams with failure offsets during server restarts (#4777)
• Improve code comments for Raft subsystem (#4724)
• Optimize linear scan when looking by comparing the first seq in a block (#4780)
• Move filestore cleanup to separate goroutine to make non-blocking (#4782)
• Move deletion of filestore files to separate goroutine to make non-blocking (#4783)

Monitoring

• Better check for standalone mode when determining to send statsz (#4757)

MQTT

• Add "clean" flag in trace message (#4740)

WebSocket

• ​​Check for /leafnode suffix path on leaf WebSocket connection (#4774)

Fixed

Accounts

• Fix panic in JWT permissions template handling (#4730)

Leafnode

• Fix subpath concatenation used for WebSocket remote connect URL (#4770) Thanks to @yoadey for the report!

JetStream

• Remove the state check in the runAs loops except for runAsLeader (#4725)
• Make sure to properly remove meta files for filestore after conversion from 2.9.x to 2.10.x (#4733)
• Make sure we check limits when scaling up a stream (#4738)
• Improve estimation on full state allocations to avoid reallocations in filestore (#4743)
• Make access to message block first and last sequence consistently use atomics (#4744)
• Fix DiscardNew exceed bytes calculation (#4772) Thanks to @MauriceVanVeen for the contribution! Thanks to @davidmcote for the report!
• Fix data race and possible panic when compacting (#4773, #4776)
• Fix panic in fileStore.Stop() (#4779)

MQTT

• Rapid load-balanced (re-)CONNECT to cluster causes races (#4734)
• Potential deadlock between JS API and mqttDeliverMsgCbQoS0 (#4760)

WebSocket

• Partial writes may lead to disconnect (#4755)

Complete Changes

v2.10.4...v2.10.5

Release v2.9.24

Changelog

Go Version

• 1.20.11

Improved

JetStream

• Stricter management of Raft state, which should improve recovery from a leaderless state (#4684 backport via #4737)
• Remove unused Observer const, add unit test to check observer applies (#4727 backport via #4737)

Fixed

Accounts

• Fix panic in JWT permissions template handling (#4730 backport via #4759)

WebSocket

• Partial writes may lead to disconnect (#4755 backport via #4759)

Complete Changes

v2.9.23...v2.9.24

Release v2.10.4

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

CVEs

• CVE-2023-46129 - nkeys: xkeys seal encryption used fixed key for all encryption

Go Version

• 1.21.3

Dependencies

• github.com/nats-io/nats.go v1.31.0
• github.com/nats-io/nkeys v0.4.6
• github.com/klauspost/compress v1.17.2
• golang.org/x/crypto v0.14.0
• golang.org/x/sys v0.13.0

Added

JetStream

• Report Raft group name in stream and consumer info responses (#4661)

MQTT

• Add config options to disable QoS 2 support (#4705)

TLS

• Add opt-in TLS handshake first for client connections (#4642)

Improved

Dependencies

• Remove unnecessary constraints dependency for ordered constraint (#4709) Thanks to @misterpickypants for the contribution!

JetStream

• Add internal pprof labels as metadata to the stream config for improved debuggability (#4662)
• Stricter management of Raft state, which should improve recovery from a leaderless state (#4684)
• Avoid unnecessary reallocations when writing the full filestore state to disk (#4687)
• Improve recovery of blocks that are being updated midway (#4692)
• Recycle filestore buffers on rebuild and write out full state prior to snapshotting (#4699)
• Extend AckTerm advisory event to support a reason (#4697)
• Improve time to select skip list and starting sequence number for deliver last by subject (#4712, #4713) Thanks to @StanEgo for the report!
• Optimize loading messages on last by subject if max messages per subject is one (#4714)

MQTT

• No longer require a server name to be set for a standalone server (#4679)

Routes

• Remove unnecessary account lookups for pinned accounts (#4686)
• Upgrade non-solicited routes if present in config (#4701, #4708)

Systemd

• Use correct network target to prevent host-dependent race conditions when establishing external connections (#4676)

Fixed

Configuration

• Fix possible panic during configuration reload during a server shutdown (#4666)

Exports/imports

• Prevent service import from duplicating MSG as HMSG with a remapped subject (#4678) Thanks to @izwerg for the report!

JetStream

• Fix panic if store error occurs when requesting consumer info (#4669)
• Fix incorrect calculation of num pending with a filtered subject (#4693) Thanks to @a-h for the report!
• Prevent purge of entire stream when targeting a sequence of 1 (#4698) Thanks to @john-bagatta for the report!
• Ensure there is a valid messages queue prior to processing within a mirror (#4700)
• Avoid concurrent consumer setLeader calls resulting in chance of multiple leaders (#4703)

MQTT

• Fix memory leak for retained messages (#4665) Thanks to @pricelessrabbit for the contribution!

Windows

• Ensure signal handler is stopped when shutting down on Windows to prevent goroutine leak (#4690)

Complete Changes

v2.10.3...v2.10.4

Release v2.10.3

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.3

Fixed

JetStream

• Reclaim more space with streams having many interior deletes during compaction with compression enabled (#4645)
• Fixed updating a non unique consumer on workqueue stream not returning an error. Thanks to @mdawar for the contribution (#4654)
• Stream / KV lookups fail after decreasing history size (#4656)
• Only mark fs as dirty vs full write on mb compaction (#4657)

MQTT

• Fix crash in MQTT layer with outgoing PUBREL header (#4646)

Complete Changes

v2.10.2...v2.10.3

Release v2.9.23

Changelog

Go Version

• 1.20.10

Fixed

Accounts

• Prevent bypassing authorization block when enabling system account access in accounts block (#4605). Backport from v2.10.2

Leafnodes

• Prevent a leafnode cluster from receiving a message multiple times in a queue subscription (#4578). Backport from v2.10.2

JetStream

• Hold lock when calculating the first message for subject in a message block (#4531). Backport from v2.10.0
• Add self-healing mechanism to detect and delete orphaned Raft groups (#4647). Backport from v2.10.0
• Prevent forward proposals in consumers after scaling down a stream (#4647). Backport from v2.10.0
• Fix race condition during leader failover scenarios resulting in potential duplicate messages being sourced (#4592). Backport from v2.10.2

Complete Changes

v2.9.22...v2.9.23

Release v2.10.2

Changelog

Downgrade compatibility note

2.10.x brings on-disk storage changes which bring significant performance improvements. Upgrade existing server versions will handle the new storage format transparently. However, if a downgrade from 2.10.x occurs, the old version will not understand the format on disk with the exception 2.9.22 and any subsequent patch releases for 2.9. So if you upgrade from 2.9.x to 2.10.0 and then need to downgrade for some reason, it must be back to 2.9.22+ to ensure the stream data can be read correctly.

Go Version

• 1.21.2

Dependencies

• github.com/nats-io/nats.go v1.30.2

Added

Profiling

• Add prof_block_rate config option for configuring the block profile (#4587)
• Add more pprof labels to consumers, sources, and mirrors (#4609)

Improved

Core

• Reduce contention when pattern matching subjects when the sublist cache is disabled (#4586)
• Various service import reply optimizations (#4591)
• Remove unnecessary lock on subscription list if cache is disabled (#4594)

Docs

• Fix links in various repo markdown files (#4590) Thanks to @jdhenke for the contribution!

Leafnodes

• Set S2 writer concurrency to 1 rather than the default of GOMAXPROCS to improve performance (#4570)

JetStream

• Make install snapshot errors rate limited when catching up (#4574)
• Log a warning on reset if bad stream state is detected (#4583)
• Change some contended locks to atomic swap operations (#4585)
• Log a warning if filestore recovery fails on the happy path (#4599)
• Ensure concurrent stream of the same stream does not return not found (#4600)
• Add additional markers for indicating unflushed state (#4601)
• Log a warning when subject skew is detected in the filestore (#4606)
• Reduce contention for a high number of connections in JetStream enabled account (#4613)
• Reduce contention in the consumer info API (#4615)
• Reduce contention and increase throughput of replica synchronization (#4621)

Systemd

• Update systemd scripts to use SIGUSR2 (lame duck model) for shutdown (#4603)

WebSocket

• Minimize memory growth for compressed WebSocket connections (#4620)
• Significantly reduce allocations in WebSocket interface (#4623)

Fixed

Accounts

• Fix inversion of lock on startup when setting up the account resolver (#4588)
• Prevent bypassing authorization block when enabling system account access in accounts block (#4605) Thanks to @alexherington for the report!

Leafnodes

• Prevent a leafnode cluster from receiving a message multiple times in a queue subscription (#4578) Thanks to @pcsegal for the report!

JetStream

• Fix possible panic due to message block unlock occurring prematurely (#4571)
• Guard against an accounting error resulting in a negative message count (#4575)
• Skip enabling direct gets if no commits (#4576)
• In lame duck mode, shutdown JetStream at the start to signal transfer of leadership if the leader (#4579)
• Fix possible stream assignment race condition (#4589)
• Fix race condition during leader failover scenarios resulting in potential duplicate messages being sourced (#4592)
• Respond with “not found” for consumer info if consumer is closed (#4610)
• Prevent processing of consumer assignments after JetStream shutdown occurs (#4625)
• Fix possibly lookup misses when MaxMsgsPerSubject=1 leading to excess messages in stream (#4631)

MQTT

• Fix PUBREL header incompatibility (#4616)

Routes

• Fix potential of pinned accounts not establishing a route on connect (#4602)

Complete Changes

v2.10.1...v2.10.2

Release v2.10.1

Changelog

Downgrade compatibility note

2.10.x brings on-disk storage changes which bring significant performance improvements. Upgrade existing server versions will handle the new storage format transparently. However, if a downgrade from 2.10.x occurs, the old version will not understand the format on disk with the exception 2.9.22 and any subsequent patch releases for 2.9. So if you upgrade from 2.9.x to 2.10.0 and then need to downgrade for some reason, it must be back to 2.9.22+ to ensure the stream data can be read correctly.

Go Version

• 1.21.1

Fixed

Leafnode

• Fix TLS handshake being prevented if remote (leaf) does not have a TLS block configured (#4565)

JetStream

• Ensure a single filter in new consumer SubjectFilters or stream SubjectTransforms block uses the extended consumer subject format as it did with SubjectFilter (#4564)
• Ensure stream-specified consumer limits are correctly applied in combination with the explicit ack policy (#4567)

Complete Changes

v2.10.0...v2.10.1