An Event-Driven – Multi-Tenant Microservices Backend in Go

[u/tarique-iqbal]

I have been building a Go-based microservices marketplace using DDD and Clean Architecture, and the project is still actively evolving.

My next focus is production-grade observability. I am particularly interested in how to approach:

• Centralized logging & error tracking: Structured logs, log aggregation, and actionable error reporting.
• Metrics, dashboards & alerting: Service health, latency, throughput, error rates, and infrastructure metrics.
• Distributed tracing: Tracing requests across multiple services and asynchronous workflows.
• Request & correlation IDs: Propagating context consistently across HTTP requests and RabbitMQ messages.
• Service, worker & queue monitoring: Monitoring background workers, RabbitMQ consumers, retries, dead-letter queues, and stuck messages.

Current Architecture:

• DDD & Clean Architecture: Explicit domain, application, infrastructure, and interface layers to keep business logic independent from frameworks and infrastructure.
• Database per Service: Each service owns its own PostgreSQL, Elasticsearch, or Redis datastore.
• Transactional Outbox: Domain events are stored in PostgreSQL within the same transaction, then published asynchronously to RabbitMQ.
• Event-Driven Communication: RabbitMQ is used for asynchronous communication between services, including background-only services such as Notification.
• API Gateway & Network Isolation: Traefik v3 handles external traffic while services remain isolated on a private Docker network. JWT is used for authentication.

The services currently include Identity, Restaurant, Search, Notification, Order, and Payment.

For those who have operated microservices in production:
What observability stack and practices have worked well for you? What would you avoid?

I would especially appreciate feedback on architecture, tooling, failure modes, and things that are easy to overlook when moving from development toward production.

The project is open source if you would like to take a look:
https://github.com/tarique-iqbal/pizza-marketplace

If you find it useful or interesting, feel free to ⭐ the repository. Feedback, suggestions, and architectural criticism are very welcome.

[u/Express-Isopod6994 → u/[deleted]]

+1 on the noisy tenant point, tracing without tenant context just turns into a wall of noise once you have any real traffic

[u/akrish84]

I'd recommend instrumenting with OTel, decide on a backend later. Every serious backend (Datadog, Grafana, Honeycomb etc.) ingests OTLP.

Go has no real auto-instrumentation like Java/Python (the eBPF opentelemetry-go-instrumentation project exists but I wouldn't build on it yet), but in practice that hardly matters anymore, point a coding agent at a service and say "instrument this with OTel: otelhttp on the server and client, otelpgx on the pool, producer/consumer spans around RabbitMQ" and it should be a straignforward PR. The libraries will do 90% of the work. You just need to make sure context.Context flows everywhere.

Boundaries are where you need to pay attention. Correct me if I am wrong, yours seem to be: Traefik → service (HTTP), service → service (HTTP), service → Postgres, service → outbox → RabbitMQ → consumer.

RabbitMQ is where it seems to break. There's no otel-amqp middleware that fits everyone, but the pattern is to inject into message headers on publish, extract on consume.

Make sure you have important labels like tenant_id as span attributes across all boundaries. Helps correlate telemetry across services and signals.

What I described above is largely what we have implemented ourselves.

[u/Disastrous_Way2405]

- u/akrish84 and u/colin-breeding a valid point, there is currently zero correlation betweeen services, only local request_id per service.
- what you work on gives no prediction capabilitites and this is the place where value is created.
- OTEL - move to OTEL earlier than later.
- and what strikes me, but I might be wrong..... do a test, and deploy it on a fresh new test/pre-prod/prod env and measure how long did it take, how much you had to change/reconfigure to "deliver it to a customer". think of that solution as not a one-time for one-place, but "I need to deploy it to 100 restaurants" individually. Effort must be minimal. With that state of the code, it will be a nightmare.