Skip to content
This repository has been archived by the owner on Jan 8, 2022. It is now read-only.

feat: added @discordjs/sharder module #87

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

kyranet
Copy link
Member

@kyranet kyranet commented Dec 2, 2021

Please describe the changes this PR makes and why it should be merged:

I heard you wanted a nice and complete sharder. 👀

Sadly, this system is not compatible with [email protected], but it will be for a future version! This sharder draws inspiration from the strategy and plug-in system Sapphire has largely adopted over the months, the way Kurasuta handles messages and shards separately, and the structure of Discord.js's built-in sharding manager to make the update easy, alongside a few sprinkles of my own based on my experience when building a cross-process intercommunication system.

⚠️ This PR is under heavy WIP: docs and tests are missing, they'll be done as I progress. The API may also change as new needs arise when building more code.

Let's talk about this sharder's structure! It is divided in three components:

  • The Sharding Manager: it's the central class that handles the shards, from starting them up, to closing.
  • The Shards: they're strategies that define how the "channel" should be handled from the sharding manager, they can be of any kind, from workers (WorkerShardHandler) and forks (ForkProcessShardHandler) to cross-server clusters going thru local clusters (ClusterProcessShardHandler). Yes, you heard that. Although we'll most likely ship only the 3 local ones. You can make your own custom shard strategies.
  • The Messengers: they're also strategies, but they define how data must be serialized and/or deserialized as well as handling message queues, we'll provide RawMessageHandler (data as-is, without converting to a Buffer or a string), JsonMessageHandler (JSON), and BinaryMessageHandler (V8 serder)... You can plug in your own ones, such as the Erlang External Term Format (ETF), YAML, TOML, etc.

This clear separation between responsibilities allow for greater customization of the core functionality of the library, reducing the need to use external libraries.

And coming in the future, we'll probably release The Message Mappers, which give a final touch-up to a message, such as appending a header/footer or even, using OpenSSL to encrypt messages over the network.

Issues to handle:

Todo:

  • Consolidate the sharder's API, make everything more consistent.
  • Make ProcessShardHandler abstract, for the two classes below.
  • Implement ForkProcessShardHandler.
  • Implement CusterProcessShardHandler.
  • Implement WorkerShardHandler.
  • Implement RawMessageHandler.
  • Implement BinaryMessageHandler.
  • Implement built-in cluster sharding (e.g. 2 processes with 5 shards each to have 10 shards) support.
  • Implement fetchRecommendedShards.
  • Implement shard client to interact with the shard manager, ShardTunnel? ShardClient?
  • Type the events.
  • Add documentation.
  • Add tests.

Status and versioning classification:

  • I know how to update typings and have done so, or typings don't need updating
  • This PR changes the library's interface (methods or parameters added)

@codecov
Copy link

codecov bot commented Dec 2, 2021

Codecov Report

Merging #87 (daf9d67) into main (c22081d) will decrease coverage by 0.08%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #87      +/-   ##
==========================================
- Coverage   89.03%   88.95%   -0.09%     
==========================================
  Files           9        9              
  Lines        1487     1566      +79     
  Branches      153      162       +9     
==========================================
+ Hits         1324     1393      +69     
- Misses         12       13       +1     
- Partials      151      160       +9     
Impacted Files Coverage Δ
packages/rest/src/lib/REST.ts 95.57% <0.00%> (-0.98%) ⬇️
packages/rest/src/lib/errors/DiscordAPIError.ts 81.81% <0.00%> (-0.66%) ⬇️
packages/rest/src/lib/RequestManager.ts 93.46% <0.00%> (-0.50%) ⬇️
packages/rest/src/lib/utils/constants.ts 100.00% <0.00%> (ø)
...ackages/rest/src/lib/handlers/SequentialHandler.ts 82.98% <0.00%> (+0.54%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c22081d...daf9d67. Read the comment docs.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants