Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RC: RDI in the cloud #570

Draft
wants to merge 51 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
c4d39e6
begin index and setup drafts
kaitlynmichael Aug 19, 2024
d586971
overview draft
kaitlynmichael Aug 22, 2024
b5beb6a
edit desc
kaitlynmichael Aug 22, 2024
10d5de8
fix relref
kaitlynmichael Aug 22, 2024
c9e5161
get started, doc link
kaitlynmichael Aug 22, 2024
42fcff9
prepare source database with link
kaitlynmichael Aug 22, 2024
17270d8
setup connectivity
kaitlynmichael Aug 22, 2024
9e0de8e
connectivity steps
kaitlynmichael Aug 22, 2024
efc8b0b
pane > box
kaitlynmichael Aug 22, 2024
20c3050
Limitations draft
kaitlynmichael Aug 23, 2024
c3b7bca
Apply suggestions from code review
cmilesb Aug 30, 2024
85c6d08
Merge branch 'main' into DOC-4137
cmilesb Sep 3, 2024
e938762
Edits and DB credentials changes
cmilesb Sep 3, 2024
a5977a2
Add setup step and more edits
cmilesb Sep 6, 2024
289f0e4
Move last step to provision
cmilesb Sep 11, 2024
1d6fce4
Merge branch 'main' into DOC-4137
cmilesb Sep 13, 2024
272e518
Add some of Yaron's suggestions to intro
cmilesb Sep 13, 2024
9d12553
Apply suggestions from code review
cmilesb Sep 19, 2024
1551e1d
Merge branch 'main' into DOC-4137
cmilesb Sep 19, 2024
a32204e
Add define steps
cmilesb Sep 20, 2024
07bc249
Merge branch 'main' into DOC-4137
cmilesb Sep 20, 2024
bc41625
Merge branch 'main' into DOC-4137
cmilesb Sep 20, 2024
71f431c
Fix relrefs
cmilesb Sep 20, 2024
e8ea310
stash commit
cmilesb Sep 23, 2024
860944a
Add security information
cmilesb Sep 24, 2024
639fc50
Merge branch 'main' into DOC-4137
cmilesb Sep 27, 2024
697d665
Merge branch 'main' into DOC-4137
cmilesb Oct 7, 2024
fa87f86
Add View/Edit
cmilesb Oct 7, 2024
49d1f54
Incorporate Yaron's feedback, part 1
cmilesb Oct 11, 2024
4abac23
Merge branch 'main' into DOC-4137
cmilesb Oct 11, 2024
4d69274
stash commit
cmilesb Oct 15, 2024
129d6af
More suggestions from Yaron
cmilesb Oct 21, 2024
99229b1
Merge branch 'main' into DOC-4137
cmilesb Oct 21, 2024
c3e501e
Apply suggestions from code review
cmilesb Oct 22, 2024
01907d8
Merge branch 'main' into DOC-4137
cmilesb Oct 24, 2024
c9469e0
Combine define and provision and remove errors section
cmilesb Oct 25, 2024
d56188a
Merge branch 'main' into DOC-4137
cmilesb Nov 5, 2024
f817848
Add secret permissions and keys
cmilesb Nov 5, 2024
8f274f5
Update content/operate/rc/databases/rdi/define.md
cmilesb Nov 5, 2024
2d2820d
Fix note
cmilesb Nov 5, 2024
46603a6
replace account ID in resource permissions
cmilesb Nov 5, 2024
89e0c4f
DOC-4548 Setup and define screenshots
cmilesb Nov 12, 2024
962c1e9
Merge branch 'main' into DOC-4137
cmilesb Nov 12, 2024
2b110b4
stash commit
cmilesb Nov 12, 2024
a51743b
Merge branch 'main' into DOC-4137
cmilesb Nov 12, 2024
35c8dc5
Fix screenshot widths
cmilesb Nov 12, 2024
52412cc
add edit pipeline images
cmilesb Nov 12, 2024
7ef44a2
Apply suggestions from code review
cmilesb Nov 25, 2024
d0bf623
Merge branch 'main' into DOC-4137
cmilesb Nov 25, 2024
07b45a0
Merge branch 'main' into DOC-4137
cmilesb Nov 27, 2024
ff2c061
Added metrics
cmilesb Nov 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions content/operate/rc/databases/rdi/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
Title: Data Integration
alwaysopen: false
categories:
- docs
- operate
- rc
description: Use Redis Data Integration with Redis Cloud.
hideListLinks: true
weight: 99
---

Redis Cloud now supports [Redis Data Integration (RDI)]({{<relref "integrate/redis-data-integration">}}), a fast and simple way to bring your data into Redis.

The applications that write data to your primary database are not always the same application that need faster access to the data. Redis Cloud uses [ingest]({{<relref "/integrate/redis-data-integration/ingest/">}}) to help you offload all read queries from the application database to Redis.
cmilesb marked this conversation as resolved.
Show resolved Hide resolved

RDI's helps Redis customers sync Redis Cloud with live data from their primary databases to:
- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
- Save resources and time when building pipelines and coding data transformations.
- Reduce the total cost of ownership by saving money on expensive database read replicas.
cmilesb marked this conversation as resolved.
Show resolved Hide resolved

Using RDI with Redis Cloud simplifies managing your data integration pipeline. No need to worry about hardware or underlying infrastructure, as Redis Cloud manages that for you. Creating the data flow from source to target is much easier, and there are validations in place to reduce errors.

## Get started

To start creating your new data pipeline, see [Prepare source database]({{<relref "content/operate/rc/databases/rdi/setup.md">}}).
cmilesb marked this conversation as resolved.
Show resolved Hide resolved

## Limitations

Only the following are supported at this time:

- One source database connected to one target database
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would go with these limitations in this order:

  • Redis Cloud Pro tier
  • Ingest use case
  • AWS platform (source and target database)
  • One source database connected to one target database

- AWS source databases
- Ingest use case
42 changes: 42 additions & 0 deletions content/operate/rc/databases/rdi/setup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
Title: Prepare source database
alwaysopen: false
categories:
- docs
- operate
- rc
description: null
hideListLinks: true
weight: 1
---

## Create new data pipeline

1. Log in to Redis Cloud.
1. (Create a new database){{<relref "/operate/rc/databases/create-database/create-pro-database-existing/">}}.
1. Once your database is created and running, navigate to the **Data pipeline** tab.
1. Select **Create data pipeline**.
1. Select your database type (currently, only **TBD** is supported).
1. If you know the size of your source database, enter it into the **Source dataset size** field.

## Prepare source database

Before using the pipeline you must first prepare your source database to use
the Debezium connector for change data capture (CDC).

See [Prepare source databases]({{<relref "/integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/">}}) to find steps for your database type.
cmilesb marked this conversation as resolved.
Show resolved Hide resolved

See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/ingest/architecture#overview" >}}) for more information about CDC.
cmilesb marked this conversation as resolved.
Show resolved Hide resolved

## Source database credentials

## Setup connectivity

To expose your source database to Redis, you need to [edit your AWS PrivateLink VPC permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions).

1. Open your [Amazon VPC console]() and navigate to **Endpoint services**.
1. Copy the Amazon Resource Name (ARN) provided in **Setup connectivity** pane.
1. Navigate to **Allow principals** tab.
1. Add the Redis Cloud ARN and choose **Allow principals**.

For more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html).
Loading