Databases in AWS — Aurora, DynamoDB, RDS, Caching Services
Exploring the Databases offerings in AWS and their Serverless Solutions
Table of content
- Amazon Aurora
- Aurora Writer and Reader endpoints
- Aurora Availability and Scaling
- Aurora Serverless and Global Aurora
- Aurora Backups and Cloning - Amazon RDS
- RDS Read Replica
- Availability and Fault Tolerance
- Storage type
- RDS Backups and Recovery
- Encryption and Security - DynamoDB
- Provisioning DynamoDB
- Features offered by DynamoDB - Amazon DocumentDB
- Caching Services
AWS offers various opinionated Database options each designed for a specific access pattern and scale profile from their proprietary solutions such as Aurora to those built on third-party solutions such as DocumentDB for MongoDB.
Below is a list of Database solutions offered by AWS grouped accordingly
SQL Databases
Amazon Aurora, Amazon RDS
NOSQL Database
DynamoDB, DocumentDB
Caching technology
ElastiCache for Valkey, Redis and Memcached, DAX
Special Databases
Amazon Timeseries, Amazon Neptune, Amazon OpenSearch (more on OpenSearch here)
Amazon Aurora
Aurora, like RDS, is compatible with PostgreSQL and MySQL. Its high availability and self-healing are substantial pros. Aurora has a lot of features that make it useful in terms of performance, budgeting and security. Aurora has a faster failover and better read scaling compared to RDS.
Some of the features of Aurora include:
- Aurora supports up to 15 Read replicas
- Separate Writer and Reader endpoints
- Aurora is distributed, with 6 copies replicated across 3 AZs
- Aurora Serverless for zero capacity planning. So instantiation and autoscaling are based on actual usage
- Aurora Global for high availability. It achieves this by creating a large number of read instance across regions.
Aurora Writer and Reader endpoints
In the Fig. 1 above, the Primary instance lives in the AZ-a with three Read Replicas across AZ-b and AZ-c. When the primary instance fail, Aurora has a quick failover and promotes one of the Aurora Read Replicas as the new primary instance. If no Read Replicas are available, then it creates a new primary instance.
Aurora Availability and Scaling
Aurora leverages read replicas, up to 15, to distribute read loads and therefore has separate read and write endpoints. It automatically scales storage in 10 GB increments and writes copies of data across AZs, making it highly available.
Aurora has self-healing through peer-to-peer replication and supports quick failover in case the default master fails. It provides writer and reader endpoints that handle load balancing at the connection level.
Aurora Serverless and Global Aurora
Aurora Serverless removes instance management entirely. Capacity scales based on demand, which is ideal for spiky or unpredictable workloads.
Aurora Global Database enables low-latency global reads and fast disaster recovery by replicating data across regions with sub-second lag.
Aurora Backups and Cloning
When it comes to backups, Aurora offers a couple of options:
- automated backups. This has a retention period of 1–35 days
- manual DB snapshots. Retention for manual snapshots can be as long as the owner wants.
Aurora database cloning on the other hand creates a new database from an existing one. It uses a copy-on-write protocol, meaning the new database initially uses the same data volume as the original. When updates are made, additional storage is allocated and data is copied. The main advantage is that this is faster than snapshot and restore.
Amazon RDS
RDS as a Database supports most popular DB engines such as MySQL, PostgreSQL, MariaDB, Oracle and SQL Server. It is not a fully managed service therefore manual provisioning of size and EBS volume type and maintenance is required. It does however has an Autoscaling capacity. RDS provides a single primary instance that handles writes operation and optional Read replicas for scaling reads.
Below are some of the features of RDS (or limitations based on your perspective):
- With RDS, Multi-AZ must be manually enabled for managed failover.
- It uses SSL and KMS for encryption in transit and at rest respectively.
- Provisioning of Size and EBS volume is done manually.
- It has an Autoscaling capacity.
- It includes optional Read Replicas which are asynchronous. This means they are eventually consistent by nature.
RDS Read Replica
RDS Read Replica offers some benefits including:
- Enhanced performance, as the load on the source DB instance can be reduced by routing Read queries from the application to the Read Replicas. RDS for MySQL especially allows you to add table indexes directly to Read Replicas without those indexes being present on the primary instance. The implication? Read performance is further maximized.
- Increased availability, as a Read Replica can be promoted to the primary instance if the source DB instance fails.
This is not active-active. Which means they are eventually consistent.
Availability and Fault Tolerance
RDS achieves high availability through Multi-AZ deployments. This must be manually enabled. AWS maintains a synchronous standby replica in another Availability Zone. Writes go to the primary, changes are replicated, and if the primary fails, RDS automatically promotes the standby.
Storage type
RDS offers several Storage options which has to be manually provisioned.
Some include:
- General Purpose SSD (gp2)
- General Purpose SSD (gp3), where performance scales independently from storage
- Provisioned IOPS SSD (io1)
- Provisioned IOPS SSD (io2), for low latency and I/O intensive operations
RDS Backups and Recovery
RDS supports taking DB snapshots and these snapshots can be exported into an S3 buckets.
- Automated backups, which allows point-in-time recovery (PITR), up to 35 days
- Manual Snapshots
Encryption and Security
RDS is one of those AWS Services that integrates directly into the VPC. i.e., it lives directly in the VPC Subnet.
SGs can be attached to the RDS which effectively becomes the database Firewall.
At rest, storage is encrypted using KMS. In transit, TLS is supported across engines.
Authentication can be traditional database credentials or IAM-based authentication, which replaces static passwords with short-lived tokens.
RDS integrates cleanly with Secrets Manager for credentials or environment variables.
Some RDS Limitations as a Cloud Service to note
Write-heavy workloads expose its biggest weakness. Scaling writes usually means either upgrading the instance or sharding manually.
Read replicas help, but replication lag can introduce subtle bugs if the application assumes immediate consistency.
DynamoDB
At its core, DynamoDB is a fully managed, serverless key-value and document database. It scales automatically and is designed for high performance at any scale. You rarely worry about servers or storage — AWS handles it since it is fully Serverless. DynamoDB forces you to think about how you access data first, not how you store it. i.e., things like the Partition key, secondary indexes have to be properly designed from the start.
Provisioning DynamoDB
DynamoDB offers a couple of options for provisioning the Database.
- On-demand
- Provisioned mode for read and write capacity
Features offered by DynamoDB
DynamoDB as a Database offers an array of features which can be useful for a Solutions Architect when designing a Database solution. Some of the features include:
- DAX, a fully managed in-memory cache service for DynamoDB
- DynamoDB Streams
- Global Tables
- TTL
- Deletion Protection
- Integration with S3
- Security and Encryption
- Backup and Disaster Recovery
- Integration with VPC Gateway endpoint
DAX
This is DynamoDB’s fully managed in-memory cache service. The DAX cluster is there for read caching. It offers various family types depending on the vCPU, Memory and Network Performance required. DAX like every Read cache provides low-latency reads of single-digit microseconds.
DynamoDB Streams and Event Processing
DynamoDB stream processing has use cases such as real-time usage analytics, reacting to changes in real time, and invoking Lambda on any changes to the DynamoDB table.
A limitations of DynamoDB Streams is the short retention period and limited number of consumers.
I looked into a couple of architectures that integrate this service and compared them to a Kinesis Data Streams architecture. For example, after performing a CRUD operation on the table, data can be streamed into Kinesis Data Streams, then into Data Firehose, and finally into Redshift for analytics, OpenSearch for indexing, or S3 for archiving, among other destinations.
Global Tables
DynamoDB also has the concept of Global Tables, which provide two-way replication across regions. Since Global Tables are active-active, the application can read and write to the table in any region.
It also integrates with DynamoDB Streams, which are required to stream data in real time.
Global Tables offers the benefits such as:
- High availability and Disaster recovery, since copies of Tables are replicated and kept across Regions. This can be used in combination with Route53 for automatic failovers.
- Reduced latency. Using Global Tables with Route53 Latency routing, user request can always be directed to DB instance closest to them.
TTL
DynamoDB TTL, which automatically deletes items after an expiry timestamp. This is useful when data only needs to be retained for a specific amount of time and eliminates operational overhead such as using a Lambda function or Monitoring applications to expire items.
Deletion Protection
When turned on, the DynamoDB Table cannot be deleted. Thus providing some extra security against accidental deletion.
Integration with S3
DynamoDB integrates with S3. Data can be exported from DynamoDB to S3, where operations such as querying with Athena can be performed. The advantage is that these queries do not affect DynamoDB’s Table Read capacity.
S3 data can also be imported into DynamoDB, and this operation does not consume DynamoDB write capacity. A common use case is performing an ETL job on the data before importing it into DynamoDB.
Security and Encryption
DynamoDB integrates with AWS security layers seamlessly. At the network level, tables exist in your VPC endpoints. At the access layer, IAM policies define who can read, write, or manage tables.
Encryption at rest is automatic via KMS, and TLS is used in-transit for secure transfer. For compliance-heavy environments, DynamoDB supports HIPAA, PCI DSS, and ISO standards, which makes it easier to justify moving sensitive data.
Backup and Disaster Recovery
Point-in-time recovery (PITR), which provides continuous backups and provides up to 35 days recovery. It protects DynamoDB tables from accidental writes and deletes.
On-demand backup, this can provide longer retention period and further custom configuration such as transitioning to cold storage, defining backup vault in AWS Backup and specifying retention period.
Integration with VPC Gateway endpoint
DynamoDB is one of the services that support VPC Gateway endpoint. This makes compute instances from within VPC Subnets connect seamlessly with DynamoDB using the fast, consistent and cheaper private network.
Amazon DocumentDB
DocumentDB is built on MongoDB and has similar deployment concepts to Aurora. It is fully managed, highly available, and follows the same operational model.
Instance-based cluster
This DocumentDB cluster type requires more provisioning but also give the flexibility to choose your Cluster Storage type and Instance type based on your requirement.
DocumentDB offer various Cluster Storage configuration:
- Amazon DocumentDB Standard, which is effective for applications with low to moderate I/O requirement
- Amazon DocumentDB I/O Optimized, for I/O intensive applications
The DB instance class, which is the vCPU and RAM size vary across the predefined Memory optimized and NVMe-backed instance classes and the Serverless class.
Memory optimized and NVMe-backed are just the group names for the numerous instance class that goes from modest 2 vCPUs compute power + 4 GiB RAM memory capacity all the way to the large instance with 64 vCPUs + 512GiB RAM.
The Serverless instance provides a bit more flexibility to choose the exact configuration you want.
DocumentDB has a constraint of a maximum of 15 replica instances per cluster.
Elastic cluster
With Elastic cluster on the otherhand, you do not need to choose, manage or upgrade instances. However, you can configure your Backup retention period, Backup and Maintenance window.
Caching Services
AWS offers ElastiCache which is a fully managed caching services built on Valkey, Memcached and Redis OSS engine. i.e. instances of the ElastiCache can be provisioned using any of the engines listed above.
Highly available, as the Cache when created will be available in up to 3 default subnets making it highly available.
Encryption is done in Transit using SSL and at rest using AWS KMS.
ElastiCache Global Datastore offers local reads through cross-Region replication for multi region application. This is also fully managed by AWS.
Auto Discovery, which is a service supported by ElastiCache running on Memcached engine. This gives the client programs the ability to automatically identify all the nodes in a cluster and initiate and maintain connections to all of the identified nodes. The purpose of this is that , your application does not need to manually connect to individual cache nodes. Instead, the application connects to one Memcached node and from that node, the application is aware of the rest of the nodes in the cluster and thus can connect to any of them.
From the architecture above, the Lambda function in 4 populates predictions from S3 to ElastiCache. The second Lambda function gets prediction based on user ID. Now when a user makes a request to this Lambda function via the API Gateway, the request does not have to go all the way to S3 bucket. It simply checks the ElastiCache for the prediction.
My website: https://www.charlesikulayo.dev
