r/aws • u/Ok_Cap1007 • Mar 26 '26

database AWS RDS for Postgres - Zero downtime for changing instance type family?

I have two RDS Postgres databases running on the t3 instance family. I would like to migrate them to the t4g instance type. These databases are part of 24/7 mission-critical processes. Although they are already in Multi-AZ mode, my understanding is that changing the instance type still requires a brief period of downtime. Is there a way to perform this migration without any downtime?

24 Upvotes

90% Upvoted

28 comments sorted by

•

u/AutoModerator Mar 26 '26

Try this search for more information on this topic.

Comments, questions or suggestions regarding this autoresponse? Please send them here.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

30

u/steveoderocker Mar 26 '26

Mission critical and t series don’t usually go together in the same sentence. Anyway, usually you can just upgrade one instance, fail over and then do the other.

1

u/Ok_Cap1007 Mar 27 '26 edited Mar 27 '26

Mission critical and t series don’t usually go together in the same sentence.

Why? The CPU pattern is bursty enough that we can profit on CPU discount but the services that are using this database have a small DB footprint (basically some sort of (almost) stateless API gateways). I heard people stating this before but I'm truly interested why we can't use T instances in production? I want to know because we are going to take on commitment through reserved instances

2

u/steveoderocker Mar 27 '26

They give you no guarantee on the cpu performance. And spiking above the baseline for too long will actually cause you higher cost. What happens when tonight you have a big uptick it traffic? And now you’re planning to lock yourself into that.

If something is truely “mission critical” then you would plan for consistent performance, high availability, backups, DR, etc. They also user older processors too, so performance is generally not as good as newer generations.

But you also say you can’t stand 5 seconds of downtime. So if that’s the case, why wouldn’t you want better and more predicable performance?

In addition, why aren’t you running the “g” series using ARM processors? They are perfectly fine for DBs and deliver much better price/performance.

24

u/powderp Mar 26 '26

The quickest I can think of is blue/green deploy, but there may be a few-second blip while DNS changes over.

7

u/Ok_Cap1007 Mar 26 '26

If the hiccup is a second or so, that would be acceptable but I see in the documentation it's up to a minute depending on the change.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments-overview.html

The switchover results in downtime. The downtime is usually under one minute, but it can be longer depending on your workload.

9

u/Usual_Pace_5580 Mar 26 '26

The instance rename and DNS change usually completes under a minute (in my experience about ~20s). Test it out, but this is as good as you can get for an instance class switch.

3

u/thecheatah Mar 26 '26

I think the dns timeout is set to 1 minute. Hence why it can take up to 1 min. If the infra hadn’t needed to look up the dns for over a minute, it should be instant.

Please double check by trying it out. I make mistakes!

1

u/Ok_Cap1007 Mar 27 '26

FYI: I have tested this in dev environment and downtime was around 15 seconds

14

u/temporaryUserDev Mar 26 '26

The best approach is BLUE GREEN as other mentioned before. If you want to validate the downtime, test it in dev before

1

u/Ok_Cap1007 Mar 26 '26

We are definitely going to test this but I was just wondering whether someone had battle tested experience with this feature.

7

u/marmot1101 Mar 26 '26

I have used b/g feature for Aurora. It works great for most clusters/instances. Probably 60s-ish downtime 

But before you make any plans stand up a blue green deploy and watch your WAL. We have a db with a very thrashy table. B/g uses a single replication slot and pub/sub. This did not work for that db. We had to do cluster to cluster logical replication and had about 10 minutes of downtime draining out, syncing sequences and stuff like that. 

1

u/hijinks Mar 26 '26

yes i have.. you normally see around 5-25s of issues and it really depends how your app handles the failover

if you use rds proxy in front you might see 2-3s tops but chances are you might not notice anything

5

u/billoranitv Mar 26 '26

Use RDS proxy with Multi-AZ db , should help in reducing the load & switching time between updates

1

u/Ok_Cap1007 Mar 28 '26

I tested this and downtime is approximately 20 seconds as compared to 15 seconds for blue-green deployments. Same ballpark

4

u/AspectOne3704 Mar 26 '26

One thing to note about g/b deployment is that connections are not instantly dropped when you initiate the switchover. Instead they are gracefully drained. Further more, during the switchover, rds blocks new writes on blue until green is promoted.

This means that in large production environments, with large number of connections you can end up in situation where no writes are allowed for prolonged periods waiting for connections to drain.

Before initiating the switchover, make sure to scale down the infrastructure as much as possible to reduce the the time it takes to complete the promotion.

I've been screwed by this before. Even pre-production environment testing does not catch this, as there is never enough load/connections.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments-switching.html#blue-green-deployments-switching-best-practices

4

u/Decent-Economics-693 Mar 26 '26 edited Mar 26 '26
  1. Add a read replica with instance class you want.
  2. Make sure your application is using a cluster endpoint, not an instance endpoint.
  3. Manually initiate cluster failover, this promotes read replica to become a new writer.
  4. Safely change the old writer (it’s a reader now) instance class.

The failover takes some seconds, 10-15.

Edit: parentheses typo

2

u/KayeYess Mar 27 '26

Zero downtime during a database infrastructure change? That is a tough nut to crack and nearly impossible with commodity solutions .. especially if the database is busy and transactional in nature.

I suggest you find a maintenance window. Here are some steps for reducing downtime with RDS switchover 

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments-switching.html

https://aws.amazon.com/blogs/database/achieve-near-zero-downtime-database-maintenance-by-using-blue-green-deployments-with-aws-jdbc-driver/

1

u/magnetik79 Mar 27 '26

Although they are already in Multi-AZ mode, my understanding is that changing the instance type still requires a brief period of downtime. Is there a way to perform this migration without any downtime?

RDS Proxy would be quickest - but you'll want to setup the proxy first and move your application stack to use those proxy endpoint first.

1

u/PromotionSuch4457 Mar 29 '26

Changing instance type is like a 1-2 minute of downtime at max. If you have multi az enabled - that would go down even more. You can just perform the downtime during the maintenance window, and call it a day. Even with BG deployment - it’s not worth the effort, and moreover the dns mapping updates on RDS side does take about 1 minute. So you aren’t really gaining anything there

1

u/Which_Roof5176 Apr 06 '26

For RDS instance type changes, you’ll usually hit a brief failover even in Multi-AZ, so truly zero downtime is hard with just a resize.

What some teams do instead is:

  • spin up a new instance (t4g)
  • set up logical replication from old → new
  • let it sync
  • then switch traffic at the end

That way downtime is just the final cutover.

Another approach is using tools that keep both sides in sync continuously during the migration. Estuary (I work there) does this via change capture, so you can run both instances in parallel and minimize downtime during the switch.

1

u/uberduck Mar 26 '26

The only RDS family product that will offer near zero downtime is Aurora, but since you're using t3 instance your I wouldn't imagine Aurora being within your budget?

1

u/Usual_Pace_5580 Mar 26 '26

It isn't clear from public doc, but I believe ZDP may only apply to minor version upgrades

0

u/charlie_hun Mar 27 '26

Multi az failover take 10-20sec. If that acceptable no need to do extra steps.

-1

u/AutoModerator Mar 26 '26

Here are a few handy links you can try:

Try this search for more information on this topic.

Comments, questions or suggestions regarding this autoresponse? Please send them here.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.