Skip to content
This repository has been archived by the owner on Dec 4, 2023. It is now read-only.

[Shared infra] Add support for all address types #4469

Open
4 of 9 tasks
lulf opened this issue May 6, 2020 · 1 comment
Open
4 of 9 tasks

[Shared infra] Add support for all address types #4469

lulf opened this issue May 6, 2020 · 1 comment
Labels
component/enmasse-operator kind/task size/medium
Milestone
1.0.0

Comments

@lulf
Copy link
Member

lulf commented May 6, 2020 •
edited
Loading

Description

Add support for all address types.

Tasklist

  • Add support for partitioned queues
  • Add support for migrating messages for queues that get rescheduled
  • Add support for system-wide queues (present on all brokers)
  • Add support for topics
  • Add support for defining DLQ and expiry queue for an individual queue
  • Add support for subscriptions
  • Add support for brokered queues and topics
  • Add monitoring metrics on addresses not configured, in migration, failed to migrate
  • Fix failing systemtests for transactions and wildcards. (Transactions depend on https://issues.apache.org/jira/browse/DISPATCH-1703)
@lulf lulf added this to the 1.0.0 milestone May 6, 2020
@lulf lulf changed the title Add support for topics and subscription [Shared infra] Add support for topics and subscription May 6, 2020
@lulf lulf changed the title [Shared infra] Add support for topics and subscription [Shared infra] Add support for all address types May 6, 2020
lulf added a commit to lulf/enmasse that referenced this issue Jun 6, 2020
@lulf
Add support for remaining address types
8e407c5 
Fixes EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 6, 2020
@lulf
Add support for remaining address types
2c16bfa 
Fixes EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 6, 2020
@lulf
Add support for remaining address types
7f98a19 
Fixes EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 7, 2020
@lulf
Add support for topic and subscriptions
52a2797 
* Add support for non-durable and durable topics
* Add systemtest for all supported address types
* Fix bug in state reconnect not being correctly initialized

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 7, 2020
@lulf
Add support for topic and subscriptions
5f54266 
* Add support for non-durable and durable topics
* Add systemtest for all supported address types
* Fix bug in state reconnect not being correctly initialized

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 7, 2020
@lulf
Add support for topic and subscriptions
3283ad0 
* Add support for non-durable and durable topics
* Add systemtest for all supported address types
* Fix bug in state reconnect not being correctly initialized

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 7, 2020
@lulf
Add support for topic and subscriptions
11d12ac 
* Add support for non-durable and durable topics
* Add systemtest for all supported address types
* Fix bug in state reconnect not being correctly initialized

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 9, 2020
@lulf
Add support for topic and subscriptions
8ec803d 
* Add support for non-durable and durable topics
* Add systemtest for all supported address types
* Fix bug in state reconnect not being correctly initialized

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 14, 2020
@lulf
Add support for topic and subscriptions
277f8ca 
* Add support for non-durable and durable topics
* Add systemtest for all supported address types
* Fix bug in state reconnect not being correctly initialized

Issue EnMasseProject#4469
lulf pushed a commit that referenced this issue Jun 15, 2020
[Shared infra] Add support for topics (#4693)
34e2250 
* Add support for topic and subscriptions

* Add support for non-durable and durable topics
* Add systemtest for all supported address types
* Fix bug in state reconnect not being correctly initialized

Issue #4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 18, 2020
@lulf
Add support for deadletter and expiry settings
84d94b4 
Default settings are set to broker defaults, except broker full
policy, which is set to FAIL. More settings will be exposed at a later
point.

Add systemtests for deadletter and expiry behavior, modify the AMQP
client to be able to reject messages so that they end up in the dead
letter queue.

Make more tests run in multiple Kubernetes environments by retrieving
a node ip and using nodeport to expose service.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 18, 2020
@lulf
Add support for deadletter and expiry settings
4320bed 
Default settings are set to broker defaults, except broker full
policy, which is set to FAIL. More settings will be exposed at a later
point.

Add systemtests for deadletter and expiry behavior, modify the AMQP
client to be able to reject messages so that they end up in the dead
letter queue.

Make more tests run in multiple Kubernetes environments by retrieving
a node ip and using nodeport to expose service.

Issue EnMasseProject#4469
@lulf lulf mentioned this issue Jun 18, 2020
Merged
7 tasks
lulf added a commit to lulf/enmasse that referenced this issue Jun 18, 2020
@lulf
Add support for deadletter and expiry settings
24121b6 
Default settings are set to broker defaults, except broker full
policy, which is set to FAIL. More settings will be exposed at a later
point.

Add systemtests for deadletter and expiry behavior, modify the AMQP
client to be able to reject messages so that they end up in the dead
letter queue.

Make more tests run in multiple Kubernetes environments by retrieving
a node ip and using nodeport to expose service.

Issue EnMasseProject#4469
@lulf
Copy link
Member Author

lulf commented Jun 19, 2020 •
edited
Loading

In the shared infra design doc, the ability to create 'transactional' queues and topics are indicated by an array of capabilities on the MessagingPlan:

kind: MessagingPlan
spec:
  capabilities:
  - transactional

In addition to enforcing this at the messaging plan, I wonder if it should be possible to specify this property on the MessagingTenant as well. This allows making use of transactional addresses without having to create a MessagingPlan. It could be exposed like this:

kind: MessagingTenant
spec:
  # For user setting the capability
  capabilities:
  - transactional
status:
  # The capabilities as inferred by the operator, either based on plan or on the setting in the spec.
  capabilities:
  - transactional

Question 1: Should it be possible to override the plan in the MessagingTenant ?

If a MessagingTenant has the transactional capability, the MessagingAddress controller can:

  • Use a 'transactional scheduler' to enforce all addresses for that tenant to be created on the same broker
  • Configure the linkRoute in the router for all addresses with prefix of the tenant namespace

I think the simplest way to implement this in the operator is to create this link route when the first address is created belonging to a tenant with transactional capability is created. Once that address has been scheduled, the operator can then know what parameters the linkRoute should have.

Question 2: Does this sufficiently cover the use cases of a JMS/Transactional broker? I.e. do we need to allow clients to use this virtual broker without any addresses configured?

@k-wall @rgodfrey Any concerns/comments to this?

lulf added a commit to lulf/enmasse that referenced this issue Jun 19, 2020
@lulf
Add support for brokered queues and topics
b3bae4f 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add 'transactional' scheduler that will schedule all addresses in a
namespace on the same broker, if they belong to a tenant with
transactional capability.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and topics.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 22, 2020
@lulf
Add support for deadletter and expiry settings
686f4a3 
Default settings are set to broker defaults, except broker full
policy, which is set to FAIL. More settings will be exposed at a later
point.

Add systemtests for deadletter and expiry behavior, modify the AMQP
client to be able to reject messages so that they end up in the dead
letter queue.

Make more tests run in multiple Kubernetes environments by retrieving
a node ip and using nodeport to expose service.

Issue EnMasseProject#4469
lulf pushed a commit that referenced this issue Jun 23, 2020
Add support for deadletter and expiry settings (#4767)
97de89a 
Default settings are set to broker defaults, except broker full
policy, which is set to FAIL. More settings will be exposed at a later
point.

Add systemtests for deadletter and expiry behavior, modify the AMQP
client to be able to reject messages so that they end up in the dead
letter queue.

Make more tests run in multiple Kubernetes environments by retrieving
a node ip and using nodeport to expose service.

Issue #4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 23, 2020
@lulf
Add support for brokered queues and topics
2a08478 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add 'transactional' scheduler that will schedule all addresses in a
namespace on the same broker, if they belong to a tenant with
transactional capability.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and topics.

Issue EnMasseProject#4469
@lulf lulf mentioned this issue Jun 24, 2020
Closed
lulf added a commit to lulf/enmasse that referenced this issue Jun 25, 2020
@lulf
Add support for brokered queues and topics
a8c4e22 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add 'transactional' scheduler that will schedule all addresses in a
namespace on the same broker, if they belong to a tenant with
transactional capability.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and topics.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 26, 2020
@lulf
Add support for brokered queues and topics
6e6cd9b 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add 'transactional' scheduler that will schedule all addresses in a
namespace on the same broker, if they belong to a tenant with
transactional capability.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and topics.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 26, 2020
@lulf
Add support for brokered queues and topics
44f14af 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add 'transactional' scheduler that will schedule all addresses in a
namespace on the same broker, if they belong to a tenant with
transactional capability.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and topics.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 26, 2020
@lulf
Add support for brokered queues and topics
54fc085 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add 'transactional' scheduler that will schedule all addresses in a
namespace on the same broker, if they belong to a tenant with
transactional capability.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and
topics. Tests for durable subscriptions are disabled for now.
* Prevent deletion of brokers that are in use.

Issue EnMasseProject#4469
@lulf lulf mentioned this issue Jun 26, 2020
Merged
lulf added a commit to lulf/enmasse that referenced this issue Jun 26, 2020
@lulf
Add support for brokered queues and topics
901a3d0 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add scheduling of tenants that are transactional to ensure that addresses
for that tenant end up on the same broker.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and
topics. Tests for durable subscriptions are disabled for now.
* Prevent deletion of brokers that are in use.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 29, 2020
@lulf
Add support for brokered queues and topics
46d2952 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add scheduling of tenants that are transactional to ensure that addresses
for that tenant end up on the same broker.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and
topics. Tests for durable subscriptions are disabled for now.
* Prevent deletion of brokers that are in use.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 29, 2020
@lulf
Add support for brokered queues and topics
db6bb9b 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add scheduling of tenants that are transactional to ensure that addresses
for that tenant end up on the same broker.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and
topics. Tests for durable subscriptions are disabled for now.
* Prevent deletion of brokers that are in use.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 29, 2020
@lulf
Add support for brokered queues and topics
740576b 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add scheduling of tenants that are transactional to ensure that addresses
for that tenant end up on the same broker.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and
topics. Tests for durable subscriptions are disabled for now.
* Prevent deletion of brokers that are in use.

Issue EnMasseProject#4469
lulf added a commit to lulf/enmasse that referenced this issue Jun 30, 2020
@lulf
Add support for brokered queues and topics
ec65778 
* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add scheduling of tenants that are transactional to ensure that addresses
for that tenant end up on the same broker.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and
topics. Tests for durable subscriptions are disabled for now.
* Prevent deletion of brokers that are in use.

Issue EnMasseProject#4469
lulf pushed a commit that referenced this issue Jun 30, 2020
[Shared infra] Add support for brokered queues and topics (#4841)
6135df5 
* Add support for brokered queues and topics

* Synchronize and store messaging tenants in infra state. This will be
used to lookup the tenant of an address to determine how that address
will be configured. It will also later be used to drive the creation
of vhost policies.
* Add scheduling of tenants that are transactional to ensure that addresses
for that tenant end up on the same broker.
* If a tenant is transactional, instead of creating multiple autolinks
and linkroutes on the router, create a single in+out link route pair
with the prefix of the tenant (per endpoint for now) so that all
links for that tenant go directly to the broker.
* Add systemtest that verifies basic transactions work for queues and
topics. Tests for durable subscriptions are disabled for now.
* Prevent deletion of brokers that are in use.

Issue #4469
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
component/enmasse-operator kind/task size/medium
Projects
None yet
Milestone
1.0.0
Development

No branches or pull requests
1 participant
@lulf