Skip to content

Latest commit

 

History

History
205 lines (176 loc) · 8.74 KB

5.1.1. Azure Queues (Storage Queues & Event Grid & Service Bus & Event Hubs & IoT Hub).md

File metadata and controls

205 lines (176 loc) · 8.74 KB
Raw

Azure Queues

Storage Queues

Implement Azure storage

Send message

  CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
  CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
  CloudQueue queue = queueClient.GetQueueReference("myqueue");
  queue.CreateIfNotExists();
  CloudQueueMessage message = new CloudQueueMessage("Hello, World");
  queue.AddMessage(message);

Receive message

  • Peek: CloudQueueMessage peekedMessage = queue.PeekMessage();
  • Read & dequeue
    • Get message (message becomes invisible for 30 seconds): CloudQueueMessage retrievedMessage = queue.GetMessage();
    • Delete message to dequeue: retrievedMessage.DeleteMessage(retrievedMessage);

Service Bus

  • PaaS for web applications.
  • Managed messaging infrastructure across private & public cloud.
  • Uses pull model, can be integrated with Event Grid to have push model
  • Extends Storage queues with middlewares, publish/subscribe messaging, load balancing, FIFO.
  • Can be used to connect on-prem with cloud or cloud to cloud.

Service Bus Concepts

  • Communication mechanisms
    • Queues (simple queues)
      • Each queue acts as broker (intermediary) that stores messages until sent.
      • One directional broker: Message is sent to single recipients.
    • Topics (publish-and-subscribe)
      • One-directional broker: single topic can have multiple subscriptions.
      • Can use filters
    • Relays (connects with direct communication)
      • Bidirectional communications
      • No storage or broker: just passes messages to the destination.
  • Namespaces
    • Application scoped container for all messaging components.
    • Can have multiple queues & topics.
    • Can have different communication mechanisms
  • Messages
    • Decouples applications.
    • Enables 1:n relationships with topics & subscriptions
    • Message sessions enables message ordering or deferral.
    • Decouples applications: improves scalability and reliability.
  • Queues
    • Offers FIFO with ordering + timestamps or more competing consumers.
    • Messages are delivered in pull mode, held in redundant storage.

Implement Azure Service Bus

  • Using C#

    • Send batch to queue

        QueueClient queueClient = new QueueClient(ServiceBusConnectionString, QueueName);
  string messageBody = $"First Message";
  Message message = new Message(Encoding.UTF8.GetBytes(messageBody));
  await queueClient.SendAsync(message);
  var messages = new List<Message>();
  for (int i = 0; i < 10; i++)
  {
    var message = new Message(Encoding.UTF8.GetBytes($"Message {i:00}")};
    messages.Add(message);
  }
  await queueClient.SendBatchAsync(messages);

    • Handle a message

      • In client SDK, you have handler callback you call either CompleteAsync or AbandonAsync (you receive the message again)
  • Create namespaces where pub/subs will meet
    • Provides a service + security boundary.
    • Can be integrated with Azure Relay
      • Integrates on-premises communication easily.
      • Less intrusive than VPN.
      • Traditional one-way, request/response, and peer-to-peer communication
      • Event distribution at internet-scope to enable publish/subscribe scenarios
      • Bi-directional and unbuffered socket communication across network boundaries.
    • The name provides a unique identifier for the object. For example, sbces12345.servicebus.windows.net
  • Create a queue
    • You can select: • Message time to live • Lock duration • Duplicate detection (duplicates won't be accepted) • Dead lettering (hold messages can't be delivered in another queue) • Sessions (guarantees FIFO) • Partitioning.
  • 📝 Receiving an event
    • ReceivesAndDelete
      • Simplest, ok if system can tolerate if a message is missing if it can't be handled
    • PeekLock
      • CompleteAsync => Message is handled & deleted
      • AbandonAsync => Re-queued

Publish/Subscribe

  • Temporal decoupling
    • Producers (senders) and consumers (receivers) do not have to be sending and receiving messages at the same time, because messages are stored durably in the queue.
  • Load leveling
    • Producers consumers send & receive messages at different rates.
    • You don't have to pay for a system that is underutilized part of the time.
  • Loose coupling
    • Resilience because messages are durable until they reach the worker.
  • Load balancing
    • Bring on more workers as the queue increases.
  • Topics and subscriptions
    • One to many (receivers)
    • Messages can be filtered in topics
  • Other features
    • Auto-forwarding
    • Batching
    • Scheduled delivery (delayed processing)
    • Nessage deferral

Azure Service Bus Pricing

  • Premium
    • Fixed size
    • Predictable performance
    • Up-down scaling
    • Messages up to 1 MB
  • Standard
    • Elastic
    • Auto-scaling
    • Message size up to 256 KB

Service Bus Monitoring

  • Metrics
    • Request metrics counts the number of requests.
    • Message metrics counts the messages (active, incoming, outgoing etc)
    • Connection metrics active/opened/closed messages
    • Resource usage metrics in Premium: CPU/memory size usage per namespace.
  • Diagnostics Logs
    • E.g. • ActivityId • EventName • resourceId • SubscriptionId • EventTimeString • EventProperties • Status • Caller • category (= operationalLogs)

Service Buses vs Storage Queues

  • Service buses are built on top of storage queues and are more advanced:
    • Clients can use AMQP (ISO standard for queueing)
    • FIFO is guaranteed with sessions.
    • And other features such as batch send, automatic dead lettering, message auto-forwarding, message groups, duplicate detection, sessions, transactions, duplicate detection, durable publish/subscribe.
  • In storage queues is REST only, FIFO is not guaranteed, lease/lock is on message level (while it's on queue level in service buses).

Event Grid

  • iPaaS: Integration Platform as a Service
  • Fully managed, http-based event routing service for events.
  • It connects and integrates all Azure services.
    • You can create events in Events blade of resources (e.g. blob storage)
  • Eliminate polling because it's expensive.
    • Long polling: Server waits until the data is available instead of empty directly.
      • it is very expensive in terms of CPU, memory and bandwidth
  • Concepts
    • Event Publishers (sources) => (through topics) Event Grid => (through subscriptions) Event Handlers
    • Topic: Endpoint where the source sends events.
    • Event Subscription: what you're interested in receiving
      • Can be filtered
      • Can have expiration date.
  • Batching is supported (array of events) and recommended.
  • Use case example:
    • Ops automation: New VM created or SQL DB spun up => check whether configurations are compliant, tag, file work items etc.
  • You can use custom events
    • Get easy to use UI
    • Native integrated event handlers
    • Uniform consumption of events
  • Uses pub/sub model.
    • Can send events to multiple recipients
  • Pricing:
    • Pay by number of operations
    • Published events + delivery attempts - monthly free grant (100k) = total operations x $0.60

Event Hubs

  • Big data streaming platform and event ingestion service.
    • Event ingestion
      • "Front door" for an event pipeline
      • Sits between publishers and consumers.
  • Seamless integration with data + analytics services inside/outside Azure.
  • Key components
    • Event Producers
      • Can publish using HTTPS, AMQP 1.0 or Apache Kafka.
    • Partitions
      • Views of consumers (state + position + offsets)
    • Throughput units
      • Prepurchased units to control capacity of EventHubs
    • Event receivers
      • Connect via AMQP 1.0.
  • Can be integrated Azure Stream Analytics
    • Azure Stream Analytics: PaaS for parallel Complex Event Processing (CEP) pipelines for BI.

Azure IoT Hub

  • Central message hub for bidirectional communication between IoT application and the devices it manages.
  • Use cases include device to cloud telemetry, file upload from devices, request-reply.
  • You can run Azure services (Azure Functions, Azure Stream Analytics, Azure Machine Learning, ..), or your own code on-prem on devices with remote cloud monitoring & management
  • EventHubs vs Azure IoT Hub
    • Azure IoT Hub leverages Event Hubs for its telemetry flow path
    • Both support the ingestion of data with low latency and high reliability