Skip to content

Commit

Permalink
docs: Reworked replication section
Browse files Browse the repository at this point in the history
  • Loading branch information
@jruaux
jruaux committed Nov 4, 2024
1 parent 10110b9 commit b1863ad
Show file tree
Hide file tree
Showing 2 changed files with 73 additions and 74 deletions.
84 changes: 32 additions & 52 deletions docs/guide/src/docs/asciidoc/concepts.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,45 +5,6 @@

image::architecture.svg[]

[[_concepts_redis_uri]]
== Redis URI

{project-title} follows the https://github.com/redis/redis-specifications/tree/master/uri[Redis URI] specification, which supports standalone, sentinel and cluster Redis deployments with plain, SSL, TLS and unix domain socket connections.

TIP: You can use the `host:port` short hand for `redis://host:port`.

TIP: You can provide the database, password and timeouts within the Redis URI.

Redis Standalone::
`redis :// [[username :] password@] host [:port][/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName]
[&libraryName=libraryName] [&libraryVersion=libraryVersion] ]`
Redis Standalone (SSL)::
`rediss :// [[username :] password@] host [: port][/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName]
[&libraryName=libraryName] [&libraryVersion=libraryVersion] ]`
Redis Standalone (Unix Domain Sockets)::
`redis-socket :// [[username :] password@]path
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&database=database]
[&clientName=clientName] [&libraryName=libraryName]
[&libraryVersion=libraryVersion] ]`
Redis Sentinel::
`redis-sentinel :// [[username :] password@] host1[:port1] [, host2[:port2]] [, hostN[:portN]] [/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&sentinelMasterId=sentinelMasterId]
[&clientName=clientName] [&libraryName=libraryName]
[&libraryVersion=libraryVersion] ]`

.Timeout Units
[horizontal]
`d`:: Days
`h`:: Hours
`m`:: Minutes
`s`:: Seconds
`ms`:: Milliseconds
`us`:: Microseconds
`ns`:: Nanoseconds


[[_concepts_batching]]
== Batching

Expand Down Expand Up @@ -132,22 +93,41 @@ For example this filter will only keep records where the `value` field is a seri
riot file-import --filter "value matches '\\d+'" ...
----

[[_concepts_replication]]
== Replication
Most Redis migration tools available today are offline in nature.
Migrating data from AWS ElastiCache to Redis Enterprise Cloud for example means backing up your Elasticache data to an AWS S3 bucket and importing it into Redis Enterprise Cloud using its UI.
Redis has a replication command called https://redis.io/commands/replicaof[REPLICAOF] but it is not always available (see https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/RestrictedCommands.html[ElastiCache restrictions]).

Instead, {project-title} implements using *dump & restore* or *type-based read & write*.
Both snapshot and live replication modes are supported.
[[_concepts_redis_uri]]
== Redis URI

image::replication-architecture.svg[]
{project-title} follows the https://github.com/redis/redis-specifications/tree/master/uri[Redis URI] specification, which supports standalone, sentinel and cluster Redis deployments with plain, SSL, TLS and unix domain socket connections.

The basic replication mechanism is as follows:
TIP: You can use the `host:port` short hand for `redis://host:port`.

1. Identify source keys to be replicated using scan and/or keyspace notifications depending on the <<_replication_mode,replication mode>>.
TIP: You can provide the database, password and timeouts within the Redis URI.

2. Read data associated with each key using <<_replication_type_dump,dump>> or <<_replication_type_struct,type-specific commands>>.
Redis Standalone::
`redis :// [[username :] password@] host [:port][/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName]
[&libraryName=libraryName] [&libraryVersion=libraryVersion] ]`
Redis Standalone (SSL)::
`rediss :// [[username :] password@] host [: port][/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName]
[&libraryName=libraryName] [&libraryVersion=libraryVersion] ]`
Redis Standalone (Unix Domain Sockets)::
`redis-socket :// [[username :] password@]path
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&database=database]
[&clientName=clientName] [&libraryName=libraryName]
[&libraryVersion=libraryVersion] ]`
Redis Sentinel::
`redis-sentinel :// [[username :] password@] host1[:port1] [, host2[:port2]] [, hostN[:portN]] [/database]
[?[timeout=timeout[d|h|m|s|ms|us|ns]] [&sentinelMasterId=sentinelMasterId]
[&clientName=clientName] [&libraryName=libraryName]
[&libraryVersion=libraryVersion] ]`

3. Write each key to the target using <<_replication_type_dump,restore>> or <<_replication_type_struct,type-specific commands>>.
.Timeout Units
[horizontal]
`d`:: Days
`h`:: Hours
`m`:: Minutes
`s`:: Seconds
`ms`:: Milliseconds
`us`:: Microseconds
`ns`:: Nanoseconds

63 changes: 41 additions & 22 deletions docs/guide/src/docs/asciidoc/replication.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@

The `replicate` command reads data from a *source* Redis database and writes to a *target* Redis database.

image::replication-architecture.svg[]

The replication mechanism is as follows:

1. Identify source keys to be replicated using scan and/or keyspace notifications depending on the <<_replication_mode,replication mode>>.

2. Read data associated with each key using <<_replication_type_dump,dump>> or <<_replication_type_struct,type-specific commands>>.

3. Write each key to the target using <<_replication_type_dump,restore>> or <<_replication_type_struct,type-specific commands>>.

The basic usage is:

[source,console]
Expand All @@ -21,18 +31,33 @@ riot replicate --help
TIP: To replicate a Redis logical database other than the default (`0`), specify the database in the source Redis URI.
For example `riot replicate redis://source:6379/1 redis://target:6379` replicates database `1`.

A status bar shows progress with a percentage of keys that have been replicated.
The total number of keys is estimated when the process starts and it can change by the time it is finished, for example if keys are deleted or added during replication.

[[_replication_mode]]
== Replication Modes
== Replication Mode

Replication starts with identifying keys to be replicated from the source Redis database.
The `--mode` option allows you to specify how {project-title} identifies keys to be replicated:

The `--mode` option allows you to specify how {project-title} identifies keys to be replicated.
Possible values: `SCAN` (default), `LIVE`, `LIVEONLY`.
* iterate over keys with a key scan (`--mode scan`)
* received by a keyspace notification subscriber (`--mode liveonly`)
* or both (`--mode live`)

[[_replication_mode_scan]]
=== `SCAN`
This mode uses the `SCAN` command to identify keys to propagate to the target Redis database.
=== Scan

This key reader scans for keys using the Redis https://redis.io/docs/latest/commands/scan/[`SCAN` command]:

```
SCAN cursor [MATCH pattern] [COUNT count] [TYPE type]
```

`MATCH pattern`:: configured with the `--key-pattern` option
`TYPE type`:: configured with the `--key-type` option
`COUNT count`:: configured with the `--scan-count` option

INFO: In cluster mode keys are scanned in parallel across cluster nodes.

The status bar shows progress with a percentage of keys that have been replicated.
The total number of keys is estimated when the replication process starts and it can change by the time it is finished, for example if keys are deleted or added during replication.

.Scan replication example
[source,console]
Expand All @@ -41,17 +66,9 @@ include::{testdir}/replicate[]
----

[[_replication_mode_live]]
=== `LIVE`
This mode builds upon scan replication by also listening for changes on the source Redis database.
Whenever a key is modified its corresponding value is read and propagated to the target Redis database.
=== Live

.Live replication example
[source,console]
----
include::{testdir}/replicate-live[]
----

Live replication relies on keyspace notifications for capturing these changes.
The key notification reader listens for key changes using keyspace notifications.

**Make sure the source database has keyspace notifications enabled** using:

Expand All @@ -60,6 +77,12 @@ Live replication relies on keyspace notifications for capturing these changes.

For more details see {link_redis_notif}.

.Live replication example
[source,console]
----
include::{testdir}/replicate-live[]
----

[WARNING]
====
The live replication mechanism does not guarantee data consistency.
Expand All @@ -74,10 +97,6 @@ For those potentially problematic migrations it is recommend to perform some pre
If you need assistance please contact your Redis account team.
====

[[_replication_mode_liveonly]]
=== `LIVEONLY`
In this mode only keyspace notifications are used.

== Replication Types

{project-title} offers two different mechanisms for reading and writing keys:
Expand Down

0 comments on commit b1863ad

Please sign in to comment.