Messages User Guide #2493

sougou · 2017-01-23T08:00:12Z

No description provided.

derekperkins

Overall, there are some pretty great design choices that I hadn't expected that really make this stand out.

Rather than a separate keyspace, the fact that queues can live inside the keyspace of relevant existing data, allowing for it to be truly transactional. That is awesome.

It does have the tradeoff of being more prone to hotspots, though I can't think of a way around that without having a task keyspace sharded on the ID, which would lose most of the other guarantees.

derekperkins · 2017-01-23T17:14:24Z

doc/Messaging.md

+* `vt_ack_wait=30`: Wait for 30s for the first message ack. If one is not received, resend.
+* `vt_purge_after=86400`: Purge acked messages that are older than 86400 seconds (1 day).
+* `vt_batch_size=10`: Send up to 10 messages per batch.
+* `vt_cache_seze=10000`: Store up to 10000 messages in the cache.


should be vt_cache_size

derekperkins · 2017-01-23T17:14:46Z

doc/Messaging.md

+insert into my_message(id, message) values(1, 'hello world')
+```
+
+These inserts can be part of a regular transaction. Multiple messages can be inserted to different tables. Avoid accumulating too many big messages within a transaction as it consumes memory on the VTTablet side. At the time of commit, memoery permitting, all messages are instantly enqueued to be sent.


memoery to memory

derekperkins · 2017-01-23T17:15:38Z

doc/Messaging.md

+  epoch bigint,
+  time_created bigint,
+  time_acked bigint,
+  message varchar(128),


Why varchar instead of varbinary? Is the size configurable?

As mentioned below, the message can be any MySQL type.

derekperkins · 2017-01-23T17:16:30Z

doc/Messaging.md

+
+# Creating a message table
+
+The current implementation requires a fixed schema. This will be made more flexible in the future. There will also be a custom DDL syntax. For now, a message table must be created like this:


1 table ~ 1 queue, so inserts / acks will be limited by your single node performance, correct?

The table can be sharded. So, there should be no performance bottleneck. Or did I misunderstand the question?
Also, let me know if the statement can be better rephrased.

derekperkins · 2017-01-23T17:17:22Z

doc/Messaging.md

+* `vt_ack_wait=30`: Wait for 30s for the first message ack. If one is not received, resend.
+* `vt_purge_after=86400`: Purge acked messages that are older than 86400 seconds (1 day).
+* `vt_batch_size=10`: Send up to 10 messages per batch.
+* `vt_cache_seze=10000`: Store up to 10000 messages in the cache.


What are vt_batch_size and vt_cache_size?

Batch size is the number of message rows sent per RPC packet.
Cache size is the number of messages that will be kept in the send queue in memory. The rest will have to wait for the next poller cycle.
I'll add these clarifications.

derekperkins · 2017-01-23T17:21:48Z

doc/Messaging.md

+insert into my_message(id, message) values(1, 'hello world')
+```
+
+These inserts can be part of a regular transaction. Multiple messages can be inserted to different tables. Avoid accumulating too many big messages within a transaction as it consumes memory on the VTTablet side. At the time of commit, memoery permitting, all messages are instantly enqueued to be sent.


What does that VTTablet memory limitation look like? It's pretty common for us to add > 1M tasks in a single transaction, with the payload being a ~100 character url.

If you have 100 shards, you'll need about 1M per shard.
Is there a reason why all DMLs have to be in the same transaction? Can you not break it up into smaller transactions?

derekperkins · 2017-01-23T17:26:24Z

doc/Messaging.md

+* Dropped tables: The message engine does not currently detect dropped tables.
+* Changed properties: Although the engine detects new message tables, it does not refresh properties of an existing table.
+* A `SELECT` style syntax for subscribing to messages.
+* No rate limiting.


Rate limiting would be my #1 request

The rate limiting design needs to be discussed. You could add the ability for each vttablet to rate-limit. However, it's hard to manage because of possible size differences, resharding, etc. A more global solution may need to be brainstormed.
In the meantime, you can consider pulling messages at a controlled rate (per subscriber), which should effectively achieve the same thing.

derekperkins · 2017-01-23T17:27:06Z

doc/Messaging.md

+
+The message feature is currently in alpha, and can be improved. Here is the list of possible limitations/improvements:
+
+* Flexible columns: Allow any number of application defined columns to be in the message table.


Flexible columns is a close second on the feature list

This won't be too hard to add on (after I take care of a few unrelated high-priority items).

derekperkins · 2017-01-23T17:28:24Z

doc/Messaging.md

+
+## Exponential backoff
+
+A message that was successfully sent will wait for the specified ack wait time. If no ack is received by then, it will be resent. The next attempt will be 2x the previous wait, and this delay is doubled for every attempt.


Would it be possible to set a queue level min/max?

We debated not backing off beyond a certain time period. However, that will just lead to hotspots in the future for messages that have chronic failures.

It's better to provide for manual control over such messages, so they can be tried only after the root cause is addressed.

derekperkins · 2017-01-23T17:32:10Z

doc/Messaging.md

+* Proactive scheduling: Upcoming messages can be proactively scheduled for timely delivery instead of waiting for the next polling cycle.
+* Monitoring support: To be added.
+* Dropped tables: The message engine does not currently detect dropped tables.
+* Changed properties: Although the engine detects new message tables, it does not refresh properties of an existing table.


Does properties mean the columns or the rows themselves? Is it possible to UPDATE the table? For example, sometimes we've had situations where exponential backoff has pushed the next execution time back hours or days, but it is especially helpful during debugging to have a message run now, and then update all those messages to run now after the bug has been fixed.

Right now, it's all properties: columns, types as well as config parameters like cache size, etc.

It's actually possible to update the table. I've mentioned this in the Advanced Usage section. I'll add instructions for rescheduling messages in there.

sougou · 2017-01-23T22:04:59Z

About your comment related to making your message rows co-exist with a parent row of another table, you can still get that using a custom sharding scheme. However, it could lead to hotspots if some root rows generate more messages than others.

Let's say the parent table has user_id as the primary key, the message table can have an id column that contains the user_id in its bits, and you can have it use the same sharding, but only using the user_id parts as input.

Review comments addressed.

sougou · 2017-01-30T23:58:46Z

Ping. Need lgtm.

alainjobart · 2017-01-31T00:00:58Z

doc/Messaging.md

+
+# Overview
+
+Vitess messaging gives the application an easy way to schedule and manage work that needs to be performed asynchronously. Under the covers, messages are stored in a traditional MySQL table and therefore enjoy the following properties:


Can you format this file with < 80 characters per line? My emacs does that for me...

alainjobart · 2017-01-31T00:01:39Z

doc/Messaging.md

+
+Vitess messaging gives the application an easy way to schedule and manage work that needs to be performed asynchronously. Under the covers, messages are stored in a traditional MySQL table and therefore enjoy the following properties:
+
+* **Transactional**: Messages can be created or acked as part of an existing transaction. The action will complete only if the commit succeeds.


This one is not the main attraction, I would move it down, keep Scalable as the first one in the list.

alainjobart · 2017-01-31T00:16:38Z

LGTM

….0 query (vitessio#2493) * cherry pick of 13375 * fix xtrabackup setup Signed-off-by: Shlomi Noach <[email protected]> * either MySQL or Percona Server Signed-off-by: Shlomi Noach <[email protected]> * fixing xtrabackup/percona-server installation Signed-off-by: Shlomi Noach <[email protected]> * again, fixing xtrabackup code Signed-off-by: Shlomi Noach <[email protected]> * reverting workflow changes Signed-off-by: Shlomi Noach <[email protected]> * Support views in BaseShowTablesWithSizes for MySQL 8.0 (vitessio#13394) * Support views in BaseShowTablesWithSizes for MySQL 8.0 Signed-off-by: Shlomi Noach <[email protected]> * added test Signed-off-by: Shlomi Noach <[email protected]> * fixed test Signed-off-by: Shlomi Noach <[email protected]> * schema tracker: use null safe comparison Signed-off-by: Andres Taylor <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Co-authored-by: Andres Taylor <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> Signed-off-by: Andres Taylor <[email protected]> Co-authored-by: Shlomi Noach <[email protected]> Co-authored-by: Andres Taylor <[email protected]>

sougou added 4 commits January 22, 2017 21:48

Create Messaging.md

f7f676a

Update Messaging.md

94c0758

Update Messaging.md

236dddb

Update Messaging.md

4ee1d22

googlebot added the cla: yes label Jan 23, 2017

sougou requested a review from alainjobart January 23, 2017 08:00

derekperkins reviewed Jan 23, 2017

View reviewed changes

Update Messaging.md

38302a0

Review comments addressed.

alainjobart reviewed Jan 31, 2017

View reviewed changes

Messaging.doc: line-wrap.

8c618a2

sougou merged commit 6297b03 into vitessio:master Jan 31, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Messages User Guide #2493

Messages User Guide #2493

sougou commented Jan 23, 2017

derekperkins left a comment

derekperkins Jan 23, 2017

derekperkins Jan 23, 2017

derekperkins Jan 23, 2017

sougou Jan 23, 2017

derekperkins Jan 23, 2017

sougou Jan 23, 2017

derekperkins Jan 23, 2017

sougou Jan 23, 2017

derekperkins Jan 23, 2017

sougou Jan 23, 2017

derekperkins Jan 23, 2017

sougou Jan 23, 2017

derekperkins Jan 23, 2017

sougou Jan 23, 2017

derekperkins Jan 23, 2017

sougou Jan 23, 2017

derekperkins Jan 23, 2017

sougou Jan 23, 2017

sougou Jan 23, 2017

sougou commented Jan 23, 2017

sougou commented Jan 30, 2017

alainjobart Jan 31, 2017

alainjobart Jan 31, 2017

alainjobart commented Jan 31, 2017 •

edited

Loading


		# Creating a message table

		The current implementation requires a fixed schema. This will be made more flexible in the future. There will also be a custom DDL syntax. For now, a message table must be created like this:


		The message feature is currently in alpha, and can be improved. Here is the list of possible limitations/improvements:

		* Flexible columns: Allow any number of application defined columns to be in the message table.


		## Exponential backoff

		A message that was successfully sent will wait for the specified ack wait time. If no ack is received by then, it will be resent. The next attempt will be 2x the previous wait, and this delay is doubled for every attempt.


		# Overview

		Vitess messaging gives the application an easy way to schedule and manage work that needs to be performed asynchronously. Under the covers, messages are stored in a traditional MySQL table and therefore enjoy the following properties:


		Vitess messaging gives the application an easy way to schedule and manage work that needs to be performed asynchronously. Under the covers, messages are stored in a traditional MySQL table and therefore enjoy the following properties:

		* Transactional: Messages can be created or acked as part of an existing transaction. The action will complete only if the commit succeeds.

Messages User Guide #2493

Messages User Guide #2493

Conversation

sougou commented Jan 23, 2017

derekperkins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sougou commented Jan 23, 2017

sougou commented Jan 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alainjobart commented Jan 31, 2017 • edited Loading

alainjobart commented Jan 31, 2017 •

edited

Loading