Skip to content

Commit

Permalink
docs: Updated guide structure and tweaked for PDF rendering
Browse files Browse the repository at this point in the history
  • Loading branch information
jruaux committed Mar 17, 2024
1 parent f1c17e5 commit 7aea74e
Show file tree
Hide file tree
Showing 33 changed files with 865 additions and 3,356 deletions.
328 changes: 328 additions & 0 deletions docs/guide/images/Diagrams.drawio

Large diffs are not rendered by default.

Binary file added docs/guide/images/architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 3 additions & 0 deletions docs/guide/images/architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/guide/images/mapping.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 3 additions & 0 deletions docs/guide/images/mapping.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/guide/images/replication-architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 3 additions & 0 deletions docs/guide/images/replication-architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
147 changes: 147 additions & 0 deletions docs/guide/images/riot.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/guide/src/docs/asciidoc/_links.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,4 @@
:link_jdbc_mssql: link:https://docs.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url?view=sql-server-2017[MS SQL Server]
:link_jdbc_mysql: link:https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-jdbc-url-format.html[MySQL]
:link_jdbc_postgres: link:https://www.postgresql.org/docs/7.4/jdbc-use.html[PostgreSQL]
:link_jdbc_sqlite: link:https://www.sqlitetutorial.net/sqlite-java/sqlite-jdbc-driver/[SQLite]
:link_jdbc_sqlite: link:https://www.sqlitetutorial.net/sqlite-java/sqlite-jdbc-driver/[SQLite]
Original file line number Diff line number Diff line change
@@ -1,25 +1,12 @@
[[_overview]]
= Overview

Redis Input/Output Tools ({project-title}) is a command-line utility to get data in and out of any Redis-compatible database like Redis OSS, Redis Cluster, Redis Enterprise, Redis Cloud, or <<_elasticache, Amazon ElastiCache>>.

{project-title} allows you to:

* Import/export data between a Redis database and
** Files (CSV, JSON, XML)
** Data generators (data structures, Faker)
** Relational databases
* Replicate (migrate) data between two Redis databases

[[_architecture]]
== Architecture
= Architecture

{project-title} is essentially an {link_etl} tool where data is extracted from the source system, transformed (see <<_processing,Processing>>), and loaded into the target system.

image::architecture.svg[]

[[_batching]]
=== Batching
== Batching

Processing in {project-title} is done in batches: a fixed number of records is read from the source, processed, and written to the target.
The default batch size is `50`, which means that an execution step reads 50 items at a time from the source, processes them, and finally writes then to the target.
Expand All @@ -28,14 +15,8 @@ If the target is Redis, writing is done in a single command ({link_redis_pipelin
You can change the batch size (and hence pipeline size) using the `--batch` option.
The optimal batch size in terms of throughput depends on many factors like record size and command types (see {link_pipeline_tuning} for details).

.Batching example
[source]
----
include::{testdir}/faker-import-tsadd[]
----

[[_threads]]
=== Multi-threading
== Multi-threading

It is possible to parallelize processing by using multiple threads.
In that configuration, each chunk of items is read, processed, and written in a separate thread of execution.
Expand All @@ -51,7 +32,7 @@ include::{testdir}/db-import-postgresql-multithreaded[]
----

[[_processing]]
=== Processing
== Processing

{project-title} lets you transform incoming records using processors.
These processors allow you to create/update/delete fields using the {link_spel} (SpEL).
Expand Down Expand Up @@ -89,7 +70,7 @@ include::{testdir}/file-import-process-var[]
----

[[_filters]]
=== Filtering
== Filtering

Filters allow you to exclude records that don't match a SpEL boolean expression.

Expand All @@ -98,4 +79,17 @@ For example this filter will only keep records where the `value` field is a seri
[source]
----
riot file-import --filter "value matches '\\d+'" ...
----
----

[[_replication_architecture]]
== Replication

image::replication-architecture.svg[]

The basic replication mechanism is as follows:

1. Identify source keys to be replicated using scan and/or keyspace notifications depending on the <<_replication_mode,replication mode>>.

2. Read data associated with each key using <<_replication_dump_restore,dump>> or <<_replication_type_based,type-specific commands>>.

3. Write each key to the target using <<_replication_dump_restore,restore>> or <<_replication_type_based,type-specific commands>>.
6 changes: 3 additions & 3 deletions docs/guide/src/docs/asciidoc/cookbook.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

Here are various recipes using {project-title}.

:leveloffset: +1
include::{includedir}/connectivity-test.adoc[]
:leveloffset: 2
include::{includedir}/ping.adoc[]
include::{includedir}/elasticache.adoc[]
:leveloffset: -1
:leveloffset: 1
6 changes: 2 additions & 4 deletions docs/guide/src/docs/asciidoc/databases.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ riot db-import "SELECT * FROM customers" hset -h myredis.com -p 6380

The keys that will be written are constructed from input records by concatenating the keyspace prefix and key fields.

image::mapping.png[]
image::mapping.svg[]

.PostgreSQL Import Example
[source]
Expand Down Expand Up @@ -77,9 +77,7 @@ To show the full usage, run:
riot db-export --help
----

[[db_export_examples]]
=== Examples
.Export to PostgreSQL
.Example: export to PostgreSQL
[source]
----
include::{testdir}/db-export-postgresql[]
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/src/docs/asciidoc/faq.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This could be due to concurrency issues in the terminal when refreshing the prog
Try running with job options `--progress log`.

Unknown options: '--keyspace', '--keys'::
You must specify one or more Redis commands with import commands (<<_file_import,`file-import`>>, <<_faker_import,`faker-import`>>, <<_db_import,`db-import`>>).
You must specify one or more Redis commands with import commands (<<_file_import,`file-import`>>, <<_faker,`faker-import`>>, <<_db_import,`db-import`>>).

ERR DUMP payload version or checksum are wrong::
Redis 7 DUMP format is not backwards compatible with previous versions.
Expand Down
40 changes: 0 additions & 40 deletions docs/guide/src/docs/asciidoc/file-export.adoc

This file was deleted.

Loading

0 comments on commit 7aea74e

Please sign in to comment.