meldestelle/docs/architecture/adr/0005-polyglot-persistence.md

ADR-0005: Polyglot Persistence

Status

Accepted

Context

As part of our microservices architecture (ADR-0003), we needed to determine the most appropriate data storage strategy. Different parts of our system have different data storage requirements:

1. Some data requires strong consistency and complex relationships
2. Some data needs to be accessed with very low latency
3. Some data is event-based and needs to be stored in a time-series format
4. Different services have different data access patterns

A one-size-fits-all approach to data storage would force compromises that could impact performance, scalability, or development productivity.

Decision

We decided to implement a polyglot persistence strategy, using different data storage technologies for different use cases:

1. PostgreSQL: As the primary relational database for storing structured data with complex relationships

   • Used by all domain services for their primary data storage
   • Each service has its own database schema to maintain isolation

2. Redis: As a distributed cache for high-speed data access

   • Used for caching frequently accessed data
   • Used for session storage
   • Used for rate limiting

3. Kafka: As an event store for event sourcing

   • Used to store domain events for event sourcing
   • Enables event replay for rebuilding state

4. Elasticsearch (planned): For full-text search capabilities

   • Will be used for advanced search features across multiple domains

Each service is responsible for managing its own data storage, and services are not allowed to access each other's databases directly.

Consequences

Positive

• Optimized performance: Each type of data is stored in the most appropriate storage technology
• Scalability: Different storage technologies can be scaled independently based on their specific requirements
• Flexibility: Teams can choose the best storage technology for their specific use cases
• Resilience: Issues with one storage technology don't necessarily affect others

Negative

• Operational complexity: Multiple storage technologies need to be deployed, monitored, and maintained
• Data consistency challenges: Maintaining consistency across different storage technologies requires careful design
• Learning curve: Teams need to be familiar with multiple storage technologies
• Backup and recovery complexity: Different storage technologies have different backup and recovery procedures

Neutral

• Data governance: Comprehensive data governance is required across all storage technologies
• Monitoring needs: Each storage technology requires its own monitoring approach

Alternatives Considered

Single Database for All Services

We considered using a single PostgreSQL database with separate schemas for each service. This would have simplified operations but would have created a single point of failure and would not have allowed us to optimize for different data access patterns.

Database per Service, Same Technology

We considered using PostgreSQL for all services but with separate databases. This would have provided service isolation while simplifying operations, but would not have allowed us to optimize for different data access patterns.

Fully Distributed NoSQL Approach

We considered using a fully distributed NoSQL approach with technologies like Cassandra or MongoDB for all data storage. This would have provided excellent scalability but would have made it harder to model complex relationships and would have required significant changes to our development practices.

References

• Polyglot Persistence by Martin Fowler
• PostgreSQL Documentation
• Redis Documentation
• Apache Kafka Documentation
• Elasticsearch Documentation