What are the primary considerations re scaling proton?

[marbemac]

Basically I'm curious re how to think about the performance characteristics of proton - what considerations are there for scaling it (like general best practices, dos/donts kind of thing)?

The more specific question:

Let's say we're building a multi-tenant app, where each tenant can (in the product UI) define "monitors". Each monitor is supposed to watch the event stream and track when events match the conditions specified when the user defined the monitor. The product has thousands of tenants (customers), each of whom can define any number of monitors.

Are proton streams a good solution for this use case? If creating/removing and running streams (assuming each stream query filters down incoming source by tenant key, which is part of the order by) are not expensive - stream per monitor?

Thanks!

[jovezhong]

Hi @marbemac , thanks the question and sharing your use case. The short answer is Timeplus can support such monitoring case

• the original data arrive in your system via something like Kafka or WebSocket. Maybe there is a single stream, or multiple streams (one for each topic, for example)
• Each tenant of your platform can define monitor rules. Each rule will be created as a Materialized View in Timeplus. It's a long running streaming SQL. You can query directly on the Materialized View with REST API or SQL/JDBC. Or the Materialized View can send events/alerts to Kafka or Webhook
• You can use our REST API or SDK for different languages to programmatically create/delete such Materialized View
• Timeplus can support hundreds or thousand of Materialized View on one node and you can setup cluster for HA and scalability
• If there are some monitor rules are identical for some customers (for example you predefined some rules, only need your tenants to turn on/off it). You may also create the Materialized View (group by tenant), then the number of Materialized View can be greatly reduced. In this case, the Materialized View will fire events to the downstream (Kafka or Webhook) for different tenants. Your downstream need to info the right tenant.

When we built our platform, we focus on performance and efficiency a lot. So a single node can achieve great performance. You don't need to setup a large cluster with many nodes. There are many tuning settings. You can start from Timeplus Proton, our core engine, or try Timeplus Enterprise with web UI in the cloud, or reach out to us if you want to setup multi-node cluster on-prem.

Happy to discuss more.

[chenziliang]

Thanks @marbemac for the questions. In general, scaling out by setting up a Timeplus cluster. And application realtime monitors can get distributed to different nodes in the cluster. If necessary an application tenant to Timeplus namespace mapping can be done as well and monitors can be placed into separate namespaces in Timeplus. Happy to setup a quick zoom call to chat more with your use cases if you are interested. Thanks again.