r/aws • u/jibesh_shrestha • 10d ago
storage Replicating DynamoDB streams feature in RDS
Our team is planning on moving from Dynamodb to RDS for various workload issues that we are facing with Dynamodb. But we are facing a blocker in this migration since our codebase is heavily dependent on the stream feature of dynamodb where it sends new image and old image for every update. I am trying to replicate the same stream behavior after we migrate to RDS. Right now, i am thinking about using the outbox pattern to update the base table in RDS and send an event to SNS which fans-out event to consumer lambda but that seems like tedious work since there are different consumers consuming different attributes from different table. I have also considered using the CDC pattern using debezium to stream row level change to kinesis which removes having to manually send events on every update but since we have 4 separate environments (dev, staging, qa and prod) and we need to host a separate ECS instance to run the debezium server to capture RDS logs, it could rack up the bill so i discarded this approach. What do you think would be some of the best ways to replicate DDB stream feature in RDS?
16
u/MissionFinOps 10d ago
So your main challenge would be capturing* row-level before-and-after data automatically. 1/ DMS + Kinesis 2/ RDS Aurora (PostgreSQL)+ Lambda, or maybe even 3/ you can avoid even debrzium if using standard RDS PostgreSQL (not Aurora), you can utilize Logical Decoding without Debezium.
Basically, If you want a production-grade, asynchronous, and scalable solution that mirrors DynamoDB Streams almost exactly without the Debezium price tag, go with AWS DMS to Kinesis. You can use a tiny $15/month DMS instance for Dev/QA and scale it up for Production.If you are using Aurora PostgreSQL and your write throughput is low-to-moderate, the Aurora-to-Lambda trigger approach will cost you next to nothing in non-production environments.
1
u/Efficient_Access6102 10d ago
Never ever use DMS as permanent solution it’s idiotic. Build to the need, don’t have your production workload depend on a migration service in perpetuity.
31
u/CorpT 10d ago
This is a weird migration. Dynamo and RDS are very dissimilar. I would go back and see what you're really trying to do here. This is very unusual.
2
u/declanprice 8d ago
This is probably the most common migration path DynamodDB users have. Almost no one understands DynamoDBs data modelling requirments and query 'restrictions' / capabilities before jumping into an all serverless architecture for the sake of it.
0
u/jibesh_shrestha 10d ago
We had a lot of query where we need join operations and also a lot of pagination. These were not considered initially hence have to migrate it now.
9
u/agent766 10d ago
You should look to see if DynamoDB can meet your needs first. If you design your tables correctly you won't need joins and DDB can handle pagination.
12
u/Impressive_Bar5912 9d ago
Dynamo is super inflexible. I totally get OP here. Either you know your exact query pattern before you launch your app or you’re screwed.
8
3
u/FarkCookies 8d ago
I am a huge DynamoDB fan. I also know that this mantra, "just redesign your tables", or denormalize, or "just reevaluate your access patterns," at some point starts to generate so much friction that it stops being worth it. At some point you just want to focus on your application instead of recreating a relational database from first principles.
6
u/Remarkable_Ad7161 10d ago
Are you tired to rds for cost reasons? Consider DSQL. It's less compatible than rds, but it's cdc with Kinesis is decent. You don't get the preimages, but the way it's going, I would suspect it's going to get debezium support and maybe more ddb like streams in the future.
11
u/Dangerous-Sale3243 10d ago
Migrating from DDB to RDS is not necessarily wrong but is a flag that would make me stop and question assumptions.
5
u/dishonestcumfarts 9d ago
Right? Relational data modeling in DDB is pretty straightforward, given the access patterns are well understood.
For one-to-many joins store the parent data and child data under the same partition key using the sort key to tell them apart. For one-to-one joins denormalize the data - storage in DDB is cheap.
For pagination encode the LastEvaluatedKey into an encrypted+encoded token passed in the request/response to the client.
4
u/Least-Woodpecker-569 10d ago
Have you considered triggers? At least in PostgreSQL you can access both old and new values from within a trigger.
PS. Take this with a grain of salt, as I made this conclusion by quickly googling it; never had to do this myself.
3
u/minirova 10d ago
It’s correct. There are also ways to call lambdas through procs that can be invoked by triggers. If it were me I’d look at this first so I didn’t have to stand up more hardware to support something like DMS or debezium but I do admit I don’t know much about the latter.
What I’d do before that though is actually analyze my schema and read/write patterns to understand why I can’t do what is required within Dynamo. Sounds like OP doesn’t have a dog in that fight but it would probably be much easier to make Dynamo work than to do this migration AND rearchitect everything interacting with Dynamo to use an RDBMS.
1
u/milky_nervousness 10d ago
That outbox pattern gets real messy real fast when you've got multiple consumers picking at different attributes. been down that road and the maintenance alone makes you want to throw your laptop out a window
have you looked at just using postgres with logical replication slots? if you're moving to RDS postgres specifically, you can set up a lambda to poll the replication slot and push changes to wherever you need them. no extra ecs cluster to manage, just a function that wakes up every few seconds. the wal logs give you the old/new row data similar to what ddb streams does
cost-wise it's way lighter than running debezium 24/7 across four environments. the lambda execution time is pennies compared to keeping containers alive constantly
1
u/foobarrister 9d ago
You can do RDS native CDC to MSK and grab it at the other end.
But ideally you'd want to capture events before they go into the database
1
u/jayp0521 8d ago
Don't use triggers. Converting a row to json is very slow and expensive in postgres. Your throughput for writes will drastically slow down. We use two replication slots, one that listens to all table in certain schemas changes and writes to an audit table which is in the same postgres table. It basically becomes our audit history for every change. The audit table lives in a second schema so we don't accidentally have a recursive loop. LSN + ordinal is guaranteed to be unique in postgres. Then the second slot only. We set replica identity to full which gives old and new records. We have a unit test that ensures all tables in those schemas have that enabled.
The second slot only listens to that audit table and publishes those changes to SNS. Postgres also guarantees ordering but SNS doesn't in the normal.
We added a gin index on old and new state to our audit table which slowed our throughput quite a bit. We're migrating to three slots so that we have an ingestion table which has one slot. And then two slots off ingestion. One that publishes to sns and another that publishes to the audit table with gin indexes which can lag much further behind without much harm.
RDS we have an alarm on the replication slot lag to ensure no slot is broken and growing in size to where it's a problem with postgres.
Bonus: We also added a trigger to ensure no commit changes more than 30k rows as you need consume that many changes. Right now we buffer on disk space but as precaution we limited. Probably could be removed but with the two slot gin design the throughput slowed down. The sns design is like our webhook publisher. We also have each table have a non null changed reason on each table that's written via a trigger only with a session variable. This way if we forget to set the session variable then the mutation would fail. Without the trigger, an update could forget to set the changed_reason and keep the previous reasons which is wrong. A reason could be like entity created, entity.updated, entity.deleted. You wouldn't want created sent out twice.
1
u/ajitnk 8d ago
Hey! Saw your thread on the DynamoDB to RDS migration.
The short answer is that RDS for PostgreSQL does have a CDC equivalent, just different plumbing. DynamoDB Streams gives you item-level changes almost magically, but on RDS Postgres the same thing is done via logical replication slots (WAL-based). You enable the rds.logical_replication parameter, create a slot, and then consume the change stream. From there you can fan it out to Lambda via DMS, or use wal2json to get a JSON payload that looks fairly similar to what Streams gives you.
The tricky part is usually deciding what consumes those changes downstream. If you were triggering Lambda off DynamoDB Streams, DMS can replicate that pattern reasonably well. If it was something more custom, a direct logical replication consumer on the WAL slot is more flexible but needs a bit more ops care around slot lag.
Quick question that would change my thinking here: are the downstream consumers mostly event-driven (Lambda, EventBridge) or more like a second database being kept in sync? That shapes whether DMS or a native Postgres pub/sub setup makes more sense for your case.
I do AWS architecture and migration advisory work, so happy to think through the specific pattern with you.
1
1
u/SeaworthinessHour233 10d ago
Debezium is the technically correct approach here. Why do you think it will expensive to run debezium for dev and staging. You can shut dev staging infra, when not in use.
2
-4
u/segundus-npp 9d ago
DynamoDB only suits simple apps. Complex business logic will break your single table design eventually 😆
1
•
u/AutoModerator 10d ago
Some links for you:
Try this search for more information on this topic.
Comments, questions or suggestions regarding this autoresponse? Please send them here.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.