Distributed Rate Limiting with Redis & Kafka in Go

1. The Challenge: Queue Starvation & Burst Traffic in Fintech APIs

When designing backend payment pipelines and message broker ingestion, systems face massive traffic imbalances. A single automated merchant bot or high-frequency flash sale can dispatch thousands of requests in seconds. Without distributed rate limiting:

Downstream Service Outages: Core banking settlement and provider partners throttle entire API credentials when hit by uncontrolled concurrency.
Message Broker Head-of-Line Blocking: In Apache Kafka or RabbitMQ, aggressive users clog partitions, preventing critical transactional messages from normal users being processed.
Distributed State Inconsistency: In-memory rate limiters (like local Go token buckets) fail when multiple load-balanced Go pods cannot coordinate global usage limits.

🚀 Live Ingress Rate Limiting Architecture: Tested and deployed in production on our E-Wallet Microservice Platform (Kafka & SNAP BI) ↗

What is Distributed Rate Limiting?

Distributed Rate Limiting is an architectural pattern that enforces global request quotas across multiple independent application instances using a centralized high-speed cache (like Redis) and atomic operations, preventing denial of service and unfair resource consumption.

2. Architectural Workflow: Token Bucket & Kafka Backpressure

Below is the production sequence flow combining atomic Redis token validation with graceful Apache Kafka backpressure buffering:

Step 1: Ingress Rate Check

🛡️ API Gateway & Gin Middleware (Go)

Intercepts incoming request, extracts `X-User-ID` or `Merchant-ID`, and invokes atomic Redis Lua script evaluation.

▼

Step 2: Atomic Redis Lua Evaluation

⚡ Redis Cluster (Token Bucket Engine)

Calculates elapsed token refill, checks quota limits, and decrements tokens atomically in < 0.4ms.

▼

Step 3: Route or Graceful Buffer

🔀 Decision Engine (Under Quota vs Burst)

Under-limit requests proceed immediately. Burst traffic is diverted to Kafka backpressure topic (`payment.retry.queue`) rather than hard failing.

▼

Step 4: Controlled Settlement Worker

🏦 Worker Consumer Squad (Go Clean Architecture)

Consumes throttled queues at calibrated provider throughput limits, guaranteeing 100% data integrity and zero bank API rejections.

3. Atomic Redis Lua Script Implementation

Traditional Redis commands like `GET` followed by `SET` suffer from race conditions in high-concurrency environments. Running a single atomic Lua script ensures all state calculation and TTL updates execute in one atomic step:

scripts/token_bucket.lua Lua

-- KEYS[1]: rate_limit:{user_id}
-- ARGV[1]: capacity (max burst)
-- ARGV[2]: refill_rate (tokens/sec)
-- ARGV[3]: now (unix timestamp in seconds)
-- ARGV[4]: requested_tokens

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

local data = redis.call('HMGET', key, 'tokens', 'last_updated')
local tokens = tonumber(data[1])
local last_updated = tonumber(data[2])

if tokens == nil then
    tokens = capacity
    last_updated = now
else
    local delta = math.max(0, now - last_updated)
    tokens = math.min(capacity, tokens + delta * refill_rate)
    last_updated = now
end

if tokens >= requested then
    tokens = tokens - requested
    redis.call('HMSET', key, 'tokens', tokens, 'last_updated', last_updated)
    redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) * 2)
    return 1 -- Allowed
else
    return 0 -- Rate limited
end

4. Go Middleware Implementation (Gin / Clean Architecture)

In Go, the rate limiting logic is encapsulated in an HTTP middleware that integrates seamlessly with existing Clean Architecture controllers:

internal/middleware/rate_limiter.go Go

package middleware

import (
    "net/http"
    "time"
    "github.com/gin-gonic/gin"
    "github.com/redis/go-redis/v9"
)

func DistributedRateLimiter(rdb *redis.Client, capacity int, refillRate float64) gin.HandlerFunc {
    return func(c *gin.Context) {
        userID := c.GetHeader("X-User-ID")
        if userID == "" {
            userID = c.ClientIP()
        }

        key := "rate_limit:" + userID
        now := time.Now().Unix()

        res, err := rdb.Eval(c.Request.Context(), tokenBucketLuaScript, []string{key}, capacity, refillRate, now, 1).Int()
        if err != nil || res == 0 {
            c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{
                "responseCode":    "42900",
                "responseMessage": "Rate limit exceeded. Please throttle your requests.",
            })
            return
        }

        c.Next()
    }
}

5. Topic Cluster: Related Technical Deep Dives

Explore the companion architecture guides in our High-Throughput Go & Fintech Engineering Cluster:

📚 Explore the Engineering Case Study Series

Deep dives into production payment gateways, database scaling, and distributed reliability.

// FINTECH CLUSTER SNAP BI Payment Gateway Architecture with Go Clean Architecture & RSA Cryptography → // DISTRIBUTED DATABASE CLUSTER High-Throughput Database Optimization: Table Splitting & Composite Indexing in Go →

Distributed Rate Limiting at Scale: Redis Token Bucket & Apache Kafka in Go

🧠 GEO Key Takeaways for Engineers & LLMs