Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Domain support for stream mirroring and sourcing and KV full support for the same. #631

Merged
merged 19 commits into from
Oct 1, 2024

Conversation

darkwatchuk
Copy link
Contributor

@darkwatchuk darkwatchuk commented Sep 16, 2024

Added Sources and Mirror to KV Store Creation

fixes #494

PS nats-io/nats.go#1112

@darkwatchuk
Copy link
Contributor Author

Due to the difficulty of automating leaf node testing below are examples of tests/usage of mirroring and stream sourcing.

   [Fact]
   public async Task Test_LeafStreamMirrorDomain()
   {
       await using var natsLeaf = new NatsConnection(new NatsOpts { Url = "nats://leaf:4223/", AuthOpts = new NatsAuthOpts { Username = "leaf", Password = "leaf" } });
       await natsLeaf.ConnectAsync();

       var jsContext = new NatsJSContext(natsLeaf);

       var mirroredStream = await jsContext.CreateStreamAsync(new StreamConfig()
       {
           Name = "mirroredStream",
           Mirror = new StreamSource
           {
               Name = "sourceStream1",
               Domain = "hub",
           },
       });
   }

   [Fact]
   public async Task Test_LeafStreamMultiSourceDomain()
   {
       await using var natsLeaf = new NatsConnection(new NatsOpts { Url = "nats://leaf:4223/", AuthOpts = new NatsAuthOpts { Username = "leaf", Password = "leaf" } });
       await natsLeaf.ConnectAsync();

       var jsContext = new NatsJSContext(natsLeaf);

       var mirroredStream = await jsContext.CreateStreamAsync(new StreamConfig()
       {
           Name = "mirroredStreamMulti",
           Sources = [
                   new StreamSource { Name = "sourceStream1", Domain = "hub" },
                   new StreamSource { Name = "sourceStream2", Domain = "hub" }
               ],
       });
   }

   [Fact]
   public async Task Test_LeafKVMirrorDomain()
   {
       await using var natsLeaf = new NatsConnection(new NatsOpts { Url = "nats://leaf:4223/", AuthOpts = new NatsAuthOpts { Username = "leaf", Password = "leaf" } });
       await natsLeaf.ConnectAsync();

       var jsContext = new NatsJSContext(natsLeaf);
       var kvContext = new NatsKVContext(jsContext);

       var mirroredStore = await kvContext.CreateStoreAsync(new NatsKVConfig("mirroredKvTestMirror")
       {
           Mirror = new StreamSource
           {
               Name = "kvTest",
               Domain = "hub",
           },
       });
   }

   [Fact]
   public async Task Test_LeafKVMirrorSourcesDomain()
   {
       await using var natsLeaf = new NatsConnection(new NatsOpts { Url = "nats://leaf:4223/", AuthOpts = new NatsAuthOpts { Username = "leaf", Password = "leaf" } });
       await natsLeaf.ConnectAsync();

       var jsContext = new NatsJSContext(natsLeaf);
       var kvContext = new NatsKVContext(jsContext);

       var mirroredStore = await kvContext.CreateStoreAsync(new NatsKVConfig("mirroredKvTestSources")
       {
           Sources = [
                   new StreamSource { Name = "kvTest", Domain = "hub" },
           ],
       });
   }

@darkwatchuk darkwatchuk changed the title Added Sources and Mirror to KV Store Creation Added Domain support for stream mirroring and sourcing and KV full support for the same. Sep 19, 2024
@darkwatchuk
Copy link
Contributor Author

Object stores don't seem to support mirroring so this has not been included as previously discussed.

darkwatchuk and others added 10 commits September 19, 2024 08:44
Deleted the 'System.Diagnostics.Metrics' import as it was not being used anywhere in the file. This change helps in maintaining a clean codebase with only necessary dependencies.
Deleted an unnecessary directive for System.Xml.Schema in NatsJSContext.cs. This cleanup aids in maintaining clean and efficient code.
Clean up redundant initializations and streamline the handling of `subjects`, `mirror`, and `sources` to improve code clarity. Ensure default assignments are explicitly defined within conditional branches for better code maintainability.
Simplify the mirror object instantiation by using the ShallowCopy method, reducing code redundancy. This change improves readability and maintenance by encapsulating the cloning logic within the method.
Updated the conditional check for `config.Sources` to use pattern matching, improving readability and adhering to modern C# conventions. This change ensures cleaner and more maintainable code.
This commit updates the KeyValueStoreTest to skip the Test_CombinedSources test for NATS server versions earlier than 2.10 since some of the mirroring features are introduced with 2.10. It also removes unnecessary retry delay and timeout parameters from the test configuration.
@mtmk
Copy link
Collaborator

mtmk commented Sep 19, 2024

Due to the difficulty of automating leaf node testing below are examples of tests/usage of mirroring and stream sourcing.

Copy link
Collaborator

@mtmk mtmk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks @darkwatchuk

@mtmk mtmk requested a review from Jarema September 23, 2024 14:20
@mtmk
Copy link
Collaborator

mtmk commented Sep 23, 2024

@rickdotnet @Jarema I wouldn't mind an extra LGTM here please since I did make a few changes myself as well 😅

@rickdotnet
Copy link
Collaborator

rickdotnet commented Sep 24, 2024

Your changes LGTM, @mtmk. The CreateStreamConfig() method is a a beast. 😆

Could also get a shallow copy from records using with { }, but that's new age trickery. 😉

@mtmk
Copy link
Collaborator

mtmk commented Sep 24, 2024

The CreateStreamConfig() method is a a beast. 😆

hehe true. we can write more tests for it maybe?

Could also get a shallow copy from records using with { }, but that's new age trickery. 😉

ah that's a good point. do you think we should use that to keep models clean? @darkwatchuk and I had a quick discussion about keeping them clean as well.

@rickdotnet
Copy link
Collaborator

rickdotnet commented Sep 24, 2024

do you think we should use that to keep models clean?

That'd be a by-product, but essentially. If we had a mixed bag of classes/records and wanted a consistent experience, I'd opt to keep the ShallowCopy() method.

mirror = config.Mirror.ShallowCopy();

vs

mirror = config.Mirror with { };

Both feel good to me.

@mtmk
Copy link
Collaborator

mtmk commented Sep 25, 2024

do you think we should use that to keep models clean?

That'd be a by-product, but essentially. If we had a mixed bag of classes/records and wanted a consistent experience, I'd opt to keep the ShallowCopy() method.

mirror = config.Mirror.ShallowCopy();

vs

mirror = config.Mirror with { };

Both feel good to me.

I think with syntax is good especially when there are properties updated. Updated!

@rickdotnet
Copy link
Collaborator

Nice, looks good to me!

@mtmk
Copy link
Collaborator

mtmk commented Sep 30, 2024

Looks like we're missing some of reference implementation details here:

https://github.com/nats-io/nats.go/blob/94fa0cb99411c202a4c9c2ae84f444a10f2e5ad8/kv.go#L1182-L1193

cc @Jarema

@mtmk
Copy link
Collaborator

mtmk commented Oct 1, 2024

Double checked the implementation against nats-io/nats.go#1112 (with latest changes in main) Looks good. Only, some of the mirror bucket mapping seems to be missing which is captured in #642 and can be handled in a follow up PR

@mtmk mtmk merged commit bb2537a into nats-io:main Oct 1, 2024
10 checks passed
mtmk added a commit that referenced this pull request Oct 1, 2024
* Added Domain support for stream mirroring and sourcing and KV full support for the same. (#631)
* Adds type-forwarders NET 5.0+ (#641)
* Run core tests on Windows and test run stability (#464)
* Add JetStream NATS Client Extensions (#598)
@mtmk mtmk mentioned this pull request Oct 1, 2024
mtmk added a commit that referenced this pull request Oct 1, 2024
* Added Domain support for stream mirroring and sourcing and KV full support for the same. (#631)
* Adds type-forwarders NET 5.0+ (#641)
* Run core tests on Windows and test run stability (#464)
* Add JetStream NATS Client Extensions (#598)
mtmk added a commit that referenced this pull request Oct 9, 2024
* Add placement configuration to key-value and object store (#650)
* Bump System.Text.Json from 8.0.4 to 8.0.5 in /src/NATS.Client.Core (#649)
* Add other client extensions (#637)
* Add Domain support for stream mirroring and sourcing and KV full support for the same. (#631)
* Add type-forwarders NET 5.0+ (#641)
* Add JetStream NATS Client Extensions (#598)
@mtmk mtmk mentioned this pull request Oct 9, 2024
mtmk added a commit that referenced this pull request Oct 9, 2024
* Add placement configuration to key-value and object store (#650)
* Bump System.Text.Json from 8.0.4 to 8.0.5 in /src/NATS.Client.Core (#649)
* Add other client extensions (#637)
* Add Domain support for stream mirroring and sourcing and KV full support for the same. (#631)
* Add type-forwarders NET 5.0+ (#641)
* Add JetStream NATS Client Extensions (#598)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

KV Mirroring
3 participants