System Design: Building Blocks - Pub-Sub

What is pub-sub system?

Message can be sent asynchronously to different subsystems using the pub-sub system. All the services subscribed to the pub-sub model receive the message that’s pushed to system.

Design

Requirements

• Topic wise:
  • create topic: producer should be able to create a topic
  • subscribe topic: consumer should be able to subscribe the topic
• Message wise:
  • write message: producers should be able to write message to topic
  • read message: consumer read the message from topic
  • delete message: message should be able to delete from topic after a certain retention time

First Design

Components

• topic queue: each topic will be a distributed message queue, producer can create this queue, and write the queue into this topic
• DB: we should have DB to store the subscription details.
• Message director: this service will read the message from the topic queue, fetch the consumers from the database, and send the message to the consumer queue
• Consumer Queue: the message from topic queue will be copied to the consumer queue, so consumers can read the message
• subscriber: when consumer request a subscription to a topic, this service will add an entry into the database.

Integrations

• consumer subscribe the topic, subscriber will add the entry into the database
• producer write message into the topic
• message director read the message from queue, and fetch the details who it should add the message
• consumer will consume the message from their queue.

Evaluation

• performance: huge number of queues needed is a significant concern.
• we copy the same message into all subscribers queues, which is unnecessary duplication and takes up spaces.

Improved Design

to resolve above problem, we come up a new design with following components

Components

• broker: this service will handle the message, it will store the message sent from producer, and allow consumer to read them
• cluster manager: we will have multiple brokers, so we need a manager to supervise their health.
• storage: we will use relational DB to store consumer details, such as subscription and retention period.
• consumer manager: manager consumers, like the authentication etc.

Integrations

1. A broker will have multiple topics, and each topics have multiple partitions.
   1. different partitions of same topic will be in different brokers.
2. consumer consume the message of their choice from a partition by reading from a specific offset
   1. consumers are independent so they can read messages anywhere from the file using the necessary function

Retrospect

• this article only cover the basics of pub-sub system, this system structure is totally new compare to previous system.
• two ways to make consumer workable, either consumer themselves fetch the information frequently, or brokers notify the consumers to work, no magic thing happen