High-Concurrency Microservices in Go: Worker Pools, Distributed Rate Limiting & Telemetry

Go's lightweight concurrency model—powered by goroutines and channels—makes it the language of choice for cloud microservices. However, uncapped concurrency easily overwhelms downstream databases, causes memory exhaustion, or triggers connection pool starvation.

In this guide, we implement production-hardened concurrency patterns in Go: bounded worker pools, token bucket rate limiting, and structured telemetry.

Go services achieve sub-millisecond latencies when goroutines are pooled and bounded.

Bounded Worker Pools

Spawning go doWork() for every incoming request risks spawning 50,000 goroutines during a traffic surge, crashing the host. Bounded worker pools queue tasks across a fixed number of workers:

package pool

import (
	"context"
	"sync"
)

type Job func(ctx context.Context)

type WorkerPool struct {
	maxWorkers int
	jobQueue   chan Job
	wg         sync.WaitGroup
	ctx        context.Context
	cancel     context.CancelFunc
}

func NewWorkerPool(workers int, queueDepth int) *WorkerPool {
	ctx, cancel := context.WithCancel(context.Background())
	p := &WorkerPool{
		maxWorkers: workers,
		jobQueue:   make(chan Job, queueDepth),
		ctx:        ctx,
		cancel:     cancel,
	}

	for i := 0; i < workers; i++ {
		p.wg.Add(1)
		go p.worker()
	}
	return p
}

func (p *WorkerPool) worker() {
	defer p.wg.Done()
	for {
		select {
		case <-p.ctx.Done():
			return
		case job, ok := <-p.jobQueue:
			if !ok {
				return
			}
			job(p.ctx)
		}
	}
}

func (p *WorkerPool) Submit(job Job) bool {
	select {
	case p.jobQueue <- job:
		return true
	default:
		// Queue full: reject or shed load gracefully
		return false
	}
}

func (p *WorkerPool) Shutdown() {
	p.cancel()
	close(p.jobQueue)
	p.wg.Wait()
}

Distributed Token Bucket Rate Limiting

To protect public API endpoints against abusive scraping, implement a token-bucket rate limiter:

package middleware

import (
	"net/http"
	"sync"
	"time"
)

type ClientLimiter struct {
	tokens     float64
	maxTokens  float64
	refillRate float64 // tokens per second
	lastRefill time.Time
	mu         sync.Mutex
}

func (c *ClientLimiter) Allow() bool {
	c.mu.Lock()
	defer c.mu.Unlock()

	now := time.Now()
	elapsed := now.Sub(c.lastRefill).Seconds()
	c.tokens = min(c.maxTokens, c.tokens+(elapsed*c.refillRate))
	c.lastRefill = now

	if c.tokens >= 1.0 {
		c.tokens -= 1.0
		return true
	}
	return false
}

func min(a, b float64) float64 {
	if a < b {
		return a
	}
	return b
}

Graceful Server Shutdown

Never terminate a production Go container abruptly with SIGKILL. Implement graceful draining to allow in-flight HTTP requests to complete:

server := &http.Server{
    Addr:         ":8080",
    Handler:      mux,
    ReadTimeout:  5 * time.Second,
    WriteTimeout: 10 * time.Second,
}

// Listen for OS interrupt
stopChan := make(chan os.Signal, 1)
signal.Notify(stopChan, os.Interrupt, syscall.SIGTERM)

<-stopChan
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

if err := server.Shutdown(ctx); err != nil {
    log.Fatalf("Server forced to shutdown: %v", err)
}

Architecture Takeaways

Never allow unbounded go routines in request paths.
Enforce HTTP Read/Write and idle timeouts on every http.Server.
Standardize on log/slog for structured JSON logging with request trace IDs.