Designing Scalable Queues for Personal Projects

This is a post is to give an overview of how I designed the queues I plan to use for all my projects (GitHub cleaner, Runs, Techrecipes, etc.).

This post, I want to talk about the design tenants I navigated. I will explain how I ended up creating a RabbitConfig with a design. Additionally, I will discuss how I plan to scale when I add more “sathish-projects”.

I want to think through an important design tenant. It is to have a central dashboard of events of all the projects. This dashboard shows all the events happening across different projects that am presently working. Githubcleaner is an auto running module. It matches up the Github clearing task. Runs app is about the running project that I have. I deploy both the apps in various cloud environments due to cost. So these projects function based on specific needs. Githubcleaner runs on a fixed schedule. The runs project operates 24/7. The events in this project are independent. I work on both projects. Hence, I want a dashboard that provides an overview of events happening.

Here is an overview design process on how I see the sathish-projects. sathish-projects acts as a parent for all the events in the sub-project.

Any event that happens in GitHub cleaner should be reflected in the source project (GitHub Cleaner). It should also be reflected in the parent (sathish-projects). There is no point in making these project interdependent with a HTTP call or a microservice dependent deployment. Similarly, if I have an issue in processing GitHub Cleaner events, I would want that issue handled separately. It should be independent of the Dashboard “sathish-projects”.

So with this in mind and I wanted to lean towards either Kafka or Rabbit MQ. With ease and the guarantee of deployment steps far less, I leaned towards Rabbit MQ. I don’t want to deep dive on lot about Rabbit MQ, there are some good information and courses available online. I want to focus on the design that I was doing for this above and beyond need that I had. From all the design tenants, I decided to use Rabbit’s fanout exchange and drive the need from there. Let me put the high level design diagram here first.

We will start with a deep dive from the left. Then, we will talk about the DLQ (Dead letter Queues) queues. To start processing, I have the exchange configured as Fanout. Reason for the fanout is, when I send a payload running either the Github project or GarminRuns project. I want the data sent to be a default sathish-projects queue -“q.sathishprojects.events“. Additionally, I want the same payload to be sent to project specific topics or exchanges. In this case the discussion is about the Github projects and is designed to be a Topic Exchange (x.sathishprojects.github.events.exchange). This matches with the initial design we talked about of having sathishprojects span across all the projects.

I am now doing a deep dive into the Github project. I have two different queues. One queue is tailored towards the operations events – “q.sathishprojects.github.ops.events” of the project. The other one is related to the API – “q.sathishprojects.github.api.events” events within the project. To keep things simple, the routing key for ops queue is “sathishprojects.github.ops.*” and api queue is “sathishprojects.github.api.*”

Finally- the error or exception conditions to be considered in the design above, I have two DLQ exchanges. The default catch-all exchange for all system and project specific errors is called “x.sathishprojects.dlx.exchange” and the queue for this exchange is “dlq.sathishprojects.events.” For now, I will route all the error condition messages to this “dlq.sathishprojects.events”. I am notating the routing to be “#”. For project specific – Github in this case we have a Topic exchange – “x.sathishprojects.github.events.dlx.exchange” that routes specific errors to underlying queues – Ops- “dlq.saathishprojects.github.ops.events” and APIs – “dlq.saathishprojects.github.api.events” . The routing keys are the same as in queues – “sathishprojects.github.ops.*” and “sathishprojects.github.api.*”

So that sums up the queues, exchanges and routing keys that is in the design. Now if we drive further, some of the conventions to keep in mind. In this case, all the projects are related to my personal ones. All the assets that were created have the common convention of “-sathishprojects.” This gives me an overview that these are related to my personal projects. For the exchange names, all the names start with a convention of “x.” This prefix indicates that the asset created is an exchange.

Now to tie all of this, the power of Spring support is the best of it. Here is the RabbitConfig class that ties all the design that we talked into one simple Declarable.

package me.sathish.my_github_cleaner.base.config;

import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.*;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@Slf4j
public class RabbitSchemaConfig {
    // Exchange names
    public static final String FANOUT_EXCHANGE = "x.sathishprojects.fanout";
    public static final String DLX_EXCHANGE = "x.sathishprojects.dlx.exchange";
    public static final String GITHUB_EVENTS_EXCHANGE = "x.sathishprojects.github.events.exchange";
    public static final String GITHUB_EVENTS_DLX_EXCHANGE = "x.sathishprojects.github.events.dlx.exchange";

    // Queue names
    public static final String SAT_PROJECTS_EVENTS_QUEUE = "q.sathishprojects.events";
    public static final String DLQ_SAT_PROJECTS_EVENTS_QUEUE = "dlq.sathishprojects.events";
    public static final String GITHUB_API_EVENTS_QUEUE = "q.sathishprojects.github.api.events";
    public static final String GITHUB_OPS_EVENTS_QUEUE = "q.sathishprojects.github.ops.events";
    public static final String DLQ_GITHUB_API_EVENTS_QUEUE = "dlq.sathishprojects.github.api.events";
    public static final String DLQ_GITHUB_OPS_EVENTS_QUEUE = "dlq.sathishprojects.github.ops.events";

    // Routing keys
    public static final String GITHUB_API_ROUTING_KEY = "sathishprojects.github.api.*";
    public static final String GITHUB_OPS_ROUTING_KEY = "sathishprojects.github.ops.*";
    public static final int MESSAGE_TTL_MS = 10000; // 10 seconds

    @Bean
    public Declarables declarables() {
        // Exchanges
        FanoutExchange fanoutExchange = new FanoutExchange(FANOUT_EXCHANGE);
        TopicExchange dlxExchange = new TopicExchange(DLX_EXCHANGE);
        TopicExchange gitHubEventsExchange = new TopicExchange(GITHUB_EVENTS_EXCHANGE);
        TopicExchange gitHubEventsDlxExchange = new TopicExchange(GITHUB_EVENTS_DLX_EXCHANGE);

        // Queues
        Queue satProjectsEventsQueue = QueueBuilder.durable(SAT_PROJECTS_EVENTS_QUEUE)
                .withArgument("x-dead-letter-exchange", DLX_EXCHANGE)
                .withArgument("x-message-ttl", MESSAGE_TTL_MS)
                .build();

        Queue dlqSatProjectsEventsQueue = QueueBuilder.durable(DLQ_SAT_PROJECTS_EVENTS_QUEUE).build();
        Queue dlqGitHubApiEventsQueue = QueueBuilder.durable(DLQ_GITHUB_API_EVENTS_QUEUE).build();
        Queue dlqGitHubOpsEventsQueue = QueueBuilder.durable(DLQ_GITHUB_OPS_EVENTS_QUEUE).build();

        Queue gitHubApiEventsQueue = QueueBuilder.durable(GITHUB_API_EVENTS_QUEUE)
                .withArgument("x-dead-letter-exchange", GITHUB_EVENTS_DLX_EXCHANGE)
                .withArgument("x-message-ttl", MESSAGE_TTL_MS)
                .withArgument("x-dead-letter-routing-key", GITHUB_API_ROUTING_KEY)
                .build();

        Queue gitHubOpsEventsQueue = QueueBuilder.durable(GITHUB_OPS_EVENTS_QUEUE)
                .withArgument("x-dead-letter-exchange", GITHUB_EVENTS_DLX_EXCHANGE)
                .withArgument("x-message-ttl", MESSAGE_TTL_MS)
                .withArgument("x-dead-letter-routing-key", GITHUB_OPS_ROUTING_KEY)
                .build();

        // Bindings
        return new Declarables(
                // Exchanges
                fanoutExchange,
                dlxExchange,
                gitHubEventsExchange,
                gitHubEventsDlxExchange,

                // Queues
                satProjectsEventsQueue,
                dlqSatProjectsEventsQueue,
                gitHubApiEventsQueue,
                gitHubOpsEventsQueue,
                dlqGitHubApiEventsQueue,
                dlqGitHubOpsEventsQueue,

                // Bindings
                BindingBuilder.bind(satProjectsEventsQueue).to(fanoutExchange),
                BindingBuilder.bind(dlqSatProjectsEventsQueue).to(dlxExchange).with("#"),
                BindingBuilder.bind(gitHubApiEventsQueue).to(gitHubEventsExchange).with(GITHUB_API_ROUTING_KEY),
                BindingBuilder.bind(gitHubOpsEventsQueue).to(gitHubEventsExchange).with(GITHUB_OPS_ROUTING_KEY),
                BindingBuilder.bind(dlqGitHubApiEventsQueue).to(gitHubEventsDlxExchange).with(GITHUB_API_ROUTING_KEY),
                BindingBuilder.bind(dlqGitHubOpsEventsQueue).to(gitHubEventsDlxExchange).with(GITHUB_OPS_ROUTING_KEY),
                BindingBuilder.bind(gitHubEventsExchange).to(fanoutExchange)
        );
    }
}

The initial code block defines all the exchanges (33-36), that we discussed in the design. The next code block focuses on the queues that are involved. It starts with the DLQ’s, which are simpler to define. Then, it is followed by the project-specific queues. Finally to tie all this we are defining it as a declarable object and returning it to the AppContext. Here is the final dashboard look in RabbitMQ for the entire discussion.

I had fun putting this design, code, and final output together. The biggest takeaway was to highlight how a good upfront design helps build an application that fits my project needs. Until next time.