Skip to content

Commit

Permalink
docs: Added elasticache migration chapter
Browse files Browse the repository at this point in the history
  • Loading branch information
Julien Ruaux committed Jun 1, 2023
1 parent 6682741 commit 1301d71
Show file tree
Hide file tree
Showing 11 changed files with 94 additions and 76 deletions.
4 changes: 2 additions & 2 deletions docs/guide/src/docs/asciidoc/_links.adoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
:link_riot_doc: link:https://developer.redis.com/riot[RIOT Documentation]
:link_riot_release: link:https://github.com/redis-developer/riot/releases/latest[RIOT latest release]
:link_riot_doc: link:https://developer.redis.com/riot[{project-title} Documentation]
:link_riot_release: link:https://github.com/redis-developer/riot/releases/latest[{project-title} latest release]
:link_redis_pipeline: link:https://redis.io/topics/pipelining[Redis Pipelining]
:link_pipeline_tuning: link:https://stackoverflow.com/a/32165090[Redis Pipeline Tuning]
:link_redis_enterprise: link:https://redis.com/redis-enterprise-software/overview/[Redis Enterprise]
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/src/docs/asciidoc/db-import.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

The `db-import` command imports data from a relational database into Redis.

NOTE: Ensure RIOT has the relevant JDBC driver for your database.
NOTE: Ensure {project-title} has the relevant JDBC driver for your database.
See the <<_database_drivers,Drivers>> section for more details.

[source]
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/src/docs/asciidoc/dump-import.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[[_dump_import]]
= Dump Import

RIOT can import Redis data structure files in JSON or XML formats. See <<_file_export, File Export>> section to generate such files.
{project-title} can import Redis data structure files in JSON or XML formats. See <<_file_export, File Export>> section to generate such files.

.Example
[source]
Expand Down
15 changes: 7 additions & 8 deletions docs/guide/src/docs/asciidoc/elasticache.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Use `redis-cli` to check connectivity with the Elasticache database:

[source]
----
redis-cli -h <elasticache primary endpoint> -p 6379
redis-cli -h <ec primary endpoint> -p 6379
----

Ensure that the above command allows you to connect to the remote Elasticache database successfully.
Expand All @@ -75,7 +75,6 @@ We are now all set to begin the migration process.
The options you will use depend on your source and target databases, as well as the replication mode (snapshot or live).

.EC Single Master -> RE

[source]
----
riot -h <source EC host> -p <source EC port> replicate -h <target RE host> -p <target RE port> --pass <RE password>
Expand All @@ -84,23 +83,23 @@ riot -h <source EC host> -p <source EC port> replicate -h <target RE host> -p <t
.Live EC Single Master -> RE
[source]
----
riot -h <source EC host> -p <source EC port> replicate *[blue]#--mode live#* -h <target RE host> -p <target RE port> --pass <RE password>
riot -h <source EC host> -p <source EC port> replicate --mode live -h <target RE host> -p <target RE port> --pass <RE password>
----

[IMPORTANT]
====
In case ElastiCache is configured with https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/auth.html[AUTH TOKEN enabled], you need to pass `--tls` as well as `--pass` option:
[source,console,subs="+quotes"]
----
riot -h <source EC host> -p <source EC port> *[blue]#--tls --pass <token>#* replicate -h <target RE host> -p <target RE port> --pass <RE password>
riot -h <source EC host> -p <source EC port> --tls --pass <token> replicate -h <target RE host> -p <target RE port> --pass <RE password>
----
====

.EC Cluster -> RE

[source]
----
riot -h <source EC host> -p <source EC port> *[blue]#--cluster#* replicate -h <target RE host> -p <target RE port> --pass <RE password>
riot -h <source EC host> -p <source EC port> --cluster replicate -h <target RE host> -p <target RE port> --pass <RE password>
----

NOTE: `--cluster` is an important parameter used ONLY for ElastiCache whenever cluster-mode is enabled.
Expand All @@ -110,20 +109,20 @@ Do note that the source database is specified first and the target database is s

[source]
----
riot -h <source EC host> -p <source EC port> *[blue]#--db <index>#* replicate -h <target RE host> -p <target RE port> --pass <RE password>
riot -h <source EC host> -p <source EC port> --db <index> replicate -h <target RE host> -p <target RE port> --pass <RE password>
----

.EC Single Master -> RE with OSS Cluster
[source]
----
riot -h <source EC host> -p <source EC port> replicate -h <target RE host> -p <target RE port> --pass <RE password> *[blue]#--cluster#*
riot -h <source EC host> -p <source EC port> replicate -h <target RE host> -p <target RE port> --pass <RE password> --cluster
----

.Live EC Cluster -> RE with OSS Cluster

[source]
----
riot -h <source EC host> -p <source EC port> *[blue]#--cluster#* replicate *[blue]#--mode live#* -h <target RE host> -p <target RE port> --pass <RE password> *[blue]#--cluster#*
riot -h <source EC host> -p <source EC port> --cluster replicate --mode live -h <target RE host> -p <target RE port> --pass <RE password> --cluster
----

== Important Considerations
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/src/docs/asciidoc/faker-import.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[[_faker]]
[[_faker_import]]
= Faker

The `faker-import` command generates data using {link_datafaker}.
Expand Down
6 changes: 3 additions & 3 deletions docs/guide/src/docs/asciidoc/file-import.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ riot file-import my.json hset -h myredis.com -p 6380
== Paths
Paths can include https://man7.org/linux/man-pages/man7/glob.7.html[wildcard patterns].

RIOT will try to determine the file type from its extension (e.g. `.csv` or `.json`), but you can specify it with the `--filetype` option.
{project-title} will try to determine the file type from its extension (e.g. `.csv` or `.json`), but you can specify it with the `--filetype` option.

Gzipped files are supported and the extension before `.gz` is used (e.g. `myfile.json.gz` -> JSON type).

Expand Down Expand Up @@ -153,7 +153,7 @@ include::{testdir}/file-import-json[]

JSON records are trees with potentially nested values that need to be flattened when the target is a Redis hash for example.

To that end, RIOT uses a field naming convention to flatten JSON objects and arrays:
To that end, {project-title} uses a field naming convention to flatten JSON objects and arrays:

.Nested object
[cols="45%m,10%,45%m",frame="none",grid="none"]
Expand All @@ -174,7 +174,7 @@ To that end, RIOT uses a field naming convention to flatten JSON objects and arr
[[_file_import_xml]]
=== XML

Here is a sample XML file that can be imported by RIOT:
Here is a sample XML file that can be imported by {project-title}:

[source,xml]
----
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/src/docs/asciidoc/files.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[[_files]]
= Files

RIOT can import from and export to files in various formats:
{project-title} can import from and export to files in various formats:

* Delimited (CSV, TSV, PSV)
* Fixed-length (also known as fixed-width)
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/src/docs/asciidoc/generators.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[[_generators]]
= Generators

RIOT includes two data generators that can be used to quickly mock up a dataset in Redis.
{project-title} includes two data generators that can be used to quickly mock up a dataset in Redis.

:leveloffset: 2
include::{includedir}/faker-import.adoc[]
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/src/docs/asciidoc/introduction.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[[_introduction]]
= Introduction

RIOT is a series of command-line utilities designed to help you get data in and out of Redis.
Redis Input/Output Tools ({project-title}) is a suite of command-line utilities designed to help you get data in and out of Redis.

It supports many different sources and targets:

Expand Down
127 changes: 73 additions & 54 deletions docs/guide/src/docs/asciidoc/replicate.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ include::{testdir}/replicate-live[]
----

[[_replicate_options]]
== Source/Target Options
=== Source/Target Options
`-h <host>`::
Redis server hostname
`-p <port>`::
Expand All @@ -43,18 +43,25 @@ include::{testdir}/replicate-live[]
Enable cluster mode

[[_replicate_mode]]
== Replication Mode
=== Replication Mode
`--mode snapshot`::
Initial replication using key scan
`--mode liveonly`::
Continuous replication using keyspace notifications (only keys that change are replicated)
`--mode live`::
Initial + continuous replication using key scan and keyspace notifications in parallel

[[_replicate_type]]
=== Replication Type
`--type dump`::
<<_replicate_dump_and_restore,DUMP & RESTORE>>
`--type ds`::
<<_replicate_type_based,Type-based>>

TIP: Make sure the source database has keyspace notifications enabled using `notify-keyspace-events = KA` in `redis.conf` or via CONFIG SET.

[[_replicate_reader_options]]
.Reader options
=== Reader options
`--scan-count`::
How many keys to read at once on each call to https://redis.io/commands/scan#the-count-option[SCAN]
`--scan-match`::
Expand All @@ -77,9 +84,9 @@ Queue size should be at least *#threads * batch*, for example `--reader-threads
Can be smaller than the number of threads

[[_replicate_performance_tuning]]
== Performance Tuning
=== Performance Tuning

Performance tuning is an art but RIOT offers some options to identify potential bottlenecks.
Performance tuning is an art but {project-title} offers some options to identify potential bottlenecks.
In addition to the <<_architecture_batch,batch>> and <<_architecture_threads,threads>> options you have the `--dry-run` option which disables writing to the target Redis database so that you can tune the <<_replicate_reader_options,reader>> in isolation.
Add that option to your existing `replicate` command-line to compare replication speeds with and without writing to the target Redis database:

Expand All @@ -88,29 +95,76 @@ Add that option to your existing `replicate` command-line to compare replication
riot <source> replicate <target> --dry-run
----

[[_replicate_verification]]
=== Verification

Once replication is complete {project-title} will perform a verification step by iterating over keys in the source database and comparing values and TTLs between source and target databases.

The verification step happens automatically after the scan is complete (snapshot replication), or for live replication when keyspace notifications have become idle (see <<_replicate_usage, Usage>> section).

Verification can also be run on-demand using the `compare` mode:
[source]
----
riot <source> replicate --mode compare <target>
----

The output looks like this:

----
>1,234 T2,345 ≠3,456 ⧗4,567 <5,678
----

* `>`: # keys only present in source database
* `T`: # mismatched data structure types
* `≠`: # mismatched values
* `⧗`: # keys with TTL delta greater than tolerance
* `<`: # keys only present in target database

To show which keys are different use the `--show-diffs` option:

[source]
----
riot <source> replicate <target> --show-diffs
----

[[_replicate_progress]]
=== Progress

Each process (scan iterator and/or event listener in case of live replication) has a corresponding status bar that shows the process name and its progress:

Scanning::
Percentage of keys that have been replicated => replicated / total.
The total number of keys is calculated when the process starts and it can change by the time it is finished (for example if keys are deleted or added during the replication).
The progress bar is only a rough indicator.
Listening::
Progress is indefinite as total number of keys is unknown.

[[_replicate_architecture]]
== Architecture

[[_replicate_live]]
== Live Replication
=== Live Replication

In live replication mode {project-title} listens for changes happening on the source database using keyspace notifications.
Each time a key is modified, RIOT reads the corresponding value and propagates that change to the target database.
Each time a key is modified, {project-title} reads the corresponding value and propagates that change to the target database.

[WARNING]
====
The live replication mechanism does not guarantee data consistency.
Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery.
It is possible that RIOT can miss some notifications in case of network failures for example.
It is possible that {project-title} can miss some notifications in case of network failures for example.
Also, depending on the type, size, and rate of change of data structures on the source it is possible that RIOT cannot keep up with the change stream.
For example if a big set is repeatedly updated, RIOT will need to read the whole set on each update and transfer it over to the target database.
With a big-enough set, RIOT could fall behind and the internal queue could fill up leading up to updates being dropped.
Also, depending on the type, size, and rate of change of data structures on the source it is possible that {project-title} cannot keep up with the change stream.
For example if a big set is repeatedly updated, {project-title} will need to read the whole set on each update and transfer it over to the target database.
With a big-enough set, {project-title} could fall behind and the internal queue could fill up leading up to updates being dropped.
Some preliminary sizing using Redis statistics and big-keys is recommended for these migrations.
If you need assistance please contact your Redis account team.
====

[[_replicate_dump_and_restore]]
== Dump & Restore
=== Dump & Restore

The default replication mechanism in RIOT is DUMP & RESTORE:
The default replication mechanism in {project-title} is DUMP & RESTORE:

image::dump-and-restore.svg[]

Expand All @@ -123,9 +177,9 @@ If live replication is enabled the reader also subscribes to keyspace notificati


[[_replicate_type_based]]
== Type-Based Replication
=== Type-Based Replication

RIOT includes another replication strategy called *Type-Based Replication* in case the target Redis database does not support the RESTORE command (for example {link_redis_crdb}).
{project-title} includes another replication strategy called *Type-Based Replication* in case the target Redis database does not support the RESTORE command (for example {link_redis_crdb}).
With this strategy each type of Redis data structure has a corresponding pair of read/write commands:

[%header,cols="h,1,1"]
Expand All @@ -149,44 +203,9 @@ To select this replication mechanism use the `--type ds` option:
include::{testdir}/replicate-ds-live[]
----

WARNING: This replication strategy is more intensive in terms of CPU, memory, and network for the machines running RIOT.
WARNING: This replication strategy is more intensive in terms of CPU, memory, and network for the machines running {project-title}.
Adjust number of threads, batch, and queue sizes accordingly.

[[_replicate_verification]]
== Verification

Once replication is complete {project-title} will perform a verification step by iterating over keys in the source database and comparing values and TTLs between source and target databases.

The verification step happens automatically after the scan is complete (snapshot replication), or for live replication when keyspace notifications have become idle (see <<_replicate_usage, Usage>> section).

Verification can also be run on-demand using the `compare` mode:
[source]
----
riot <source> replicate --mode compare <target>
----

The output looks like this:

----
>1,234 T2,345 ≠3,456 ⧗4,567 <5,678
----

* `>`: # keys only present in source database
* `T`: # mismatched data structure types
* `≠`: # mismatched values
* `⧗`: # keys with TTL delta greater than tolerance
* `<`: # keys only present in target database

To show which keys are different use the `--show-diffs` option:

[source]
----
riot <source> replicate <target> --show-diffs
----

[[_replicate_progress]]
== Progress

Each process (scan, and event listener in case of live replication) has a corresponding status bar that shows the process name and its progress:
* Scanning: percentage of keys that have been replicated => replicated / total. The total number of keys is calculated when the process starts and it can change by the time it is finished (for example in case keys are deleted or added during the replication), so the progress bar is only a rough indicator.
* Listening: progress is indefinite as total number of keys is unknown
:leveloffset: 2
include::{includedir}/elasticache.adoc[]
:leveloffset: 1
6 changes: 3 additions & 3 deletions docs/guide/src/docs/asciidoc/usage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

== Install

RIOT can be installed in different ways depending on your environment and preference.
{project-title} can be installed in different ways depending on your environment and preference.

[[_macos_install]]
.Homebrew (macOS)
Expand All @@ -25,7 +25,7 @@ scoop install riot
[[_linux_install]]
.Download and run (Linux)

RIOT requires Java and the easiest option is to use the version packaged with Ubuntu.
{project-title} requires Java and the easiest option is to use the version packaged with Ubuntu.
By default Ubuntu 18.04 includes Open JDK 11.

To install this version, first update the package index:
Expand Down Expand Up @@ -89,7 +89,7 @@ docker run fieldengineering/riot [OPTIONS] [COMMAND]

== Run

You can launch RIOT with the following command:
You can launch {project-title} with the following command:

[source]
----
Expand Down

0 comments on commit 1301d71

Please sign in to comment.