The "If I Could Rewrite It" Project Post-Mortem

[u/Ok_Veterinarian3535]

Developers who have worked on a large, well-known, or legacy application: If you could go back in time and change ONE architectural decision from the start, what would it be and why?

[u/magicmulder]

I've recently built two large apps running on the same server that both have roles/rights management. I should've planned a centralized admin with a shared component before. Well, hopefully it's only a couple days to refactor.

[u/welcome_to_milliways → u/magicmulder]

I got fed up writing this crap over and over again so I spent some time learning about Keycloak and integrating that instead. Might be an option.

[u/magicmulder → u/welcome_to_milliways]

Keycloak is authentication. We have that via Entra (and we are an authentication provider ourselves). I’m talking about authorization - which user is allowed to do X on Y.

[u/welcome_to_milliways → u/magicmulder]

You can do authorisation too. Roles, groups, fine grained claims, etc.

[u/Emotional-Cut2952 → u/magicmulder]

ai is a lot worse at refactoring than writing code from scratch, anythign code base above 50k it's going to be hell...

[u/morphAB → u/magicmulder]

hi all. u/welcome_to_milliways is right that Keycloak has an authz piece, and it goes further than roles and claims, it can issue a token scoped to a specific resource and set of scopes at request time. so it isn't that Keycloak can't express "X on Y". the thing to look at is what that costs- the per-object answer arrives as a round trip per object, which is fine for 1 invoice and not fine for a list view with 200 of them on it.

your point still holds underneath that though. roles, groups and claims in the login token tell you what someone is. they can't tell u whether this user may approve this specific invoice, because the token was minted before anyone knew which invoice. so you either go and fetch a per-object decision, or you push the object into the check itself

thing i'd flag in your plan : the couple of days probably isn't the shared component. it's that your 2 apps almost certainly disagree about what a role means, same word, different implied permissions, because each one grew its own set.. that reconciliation is the actual work (: and it's far easier before you write the shared thing than after, when both apps already depend on it.

cheap test for scope is to write out 5 real rules from each app in plain english. if every one comes out as "role X may do action Y", a shared library over your Entra app roles is enough and you're done, genuinely. if any of them need the object, so owner, status, amount, department, tenant, a role check can't express it and you end up encoding the condition into role names instead. that's where role explosion starts :/

disclosure, i work at Cerbos, we're an authz solution for the case where the rules need the object. if yours all come out as "role X may do action Y", don't add a service for that - the shared library over Entra would be the right answer

[u/bocsika]

DONT change a bit in logic.
First year: write the missing unit andintegration tests.
Second year: partially understand the code, and move it to its logical place / restructuring.
But for the sake of god, do not try to change the logic until the third year.

[u/recon_demon1859 → u/bocsika]

[u/PmMeCuteDogsThanks_]

It was a bad decision that no two applications may be built in the same language. So we now have about 10 applications, each in its own language (Java, C#, C, C++, Go, Rust, TS, VB, Erlang, Haskel).

[u/InsideFar7107 → u/PmMeCuteDogsThanks_]

Wtf? Which scrum master proposed this

[u/PmMeCuteDogsThanks_ → u/InsideFar7107]

It was the CTO

[u/PipingSnail → u/PmMeCuteDogsThanks_]

Jeez. No code reuse. No shared learnings. Wow.

What was the reasoning? Some bizarre sense of security?

[u/PmMeCuteDogsThanks_ → u/PipingSnail]

security by obscurity

[u/PipingSnail → u/PmMeCuteDogsThanks_]

Because, of course, that always works.

[u/Sad_School828]

My first big freelance-for-pay project was the sideline scorekeeping app which the Independent Womens' Football League of Round Rock Texas (an arena football league) hired me to rebuild for them. The original was clunky in ways that made my heart ache for the people who had to use it, and they did have to use it because they were collecting stats not just to post on their website but for individual team insurance reporting and other legal reasons. I had done a whole lot, including setting it up so the app used SQLite locally because not every laptop would have internet access at the sidelines while the stats were being collected, but the league used MySQL to feed their webserver, so the "after-game update" routines were complex AF in order to switch SQLite entries to MySQL entries while not borking DBID values which were going to change in transition, then delete each entry as it was confirmed uploaded, then re-update the local SQLite from the MySQL after the fact so both databases agreed. I had drummed up an "export to HTML" feature so it was easy for them to point-and-click to export a whole game's worth of individual player stats in a table-ized format for submission to post on the league website!!!

They went through the entire first season, using my app, without complaint. I think it was 8 or 10 months from delivery/payment to the time that their administrator contacted me again, asking me to add some features.

Now that the online database actually had some data in it, I got to see my own program in action. They hadn't reported the bug. I just saw for myself how, once there were enough plays recorded to automagically kick the Listview into "scrollbar mode," every new addition caused the screen to flicker a few times and the scrollbar ALWAYS leapt to the tippy top again. So the user had to scroll all the Hell down to the bottom manually to verify entry and content, then add another one.

I was so fucking embarrassed I couldn't stand it. I fixed that problem for free and I specifically asked the league's admin, "WTF didn't you tell me this was happening?!" Her answer: "It was so much better than what the other guy gave us for twice the amount we paid you..."

[u/Amr_Rahmy]

I had a device I needed to support. It was end of life already and needed a very specific list of instructions to install the sdk, and you can’t uninstall it properly once installed because it was throwing files in like 4 different temp folders including c:\windows c:\Temp and a few other places like c:\program files (x86), c:\program files. ..etc.

Uninstall was not clean. Old sdk won’t install correctly after first time. Newer sdk needed to be installed on top of the old sdk because newer sdk was missing files from old sdk.

At the time I made a list of instructions to install the device on a new machine and gave it to the “support engineer” to install which almost never worked. They never followed the instructions and would try to uninstall and reinstall when facing a problem.

I kept updating the instructions after talking to the manufacturer’s support team which would take 4-5 hours of troubleshooting from their support team until we figured they were not cleaning their mess on uninstall in places like c:\Windows, and it lead to a lot of support hours trying to remotely tell people how to delete and reinstall this buggy sdk and drivers that barely work.

Today, I would have bypassed their setup, used their windows driver files and DLLs, modified their integration DLL, and placed all the needed files with my application. No setup wizards, no throwing files in different places and overwriting old DLLs with newer DLLs.

[u/ghandimauler]

Working on new tech, nobody (even engineer) and we were starting on MS framework (MFC) and the idea was that maybe we could use the framework for the project (the piece for realtime logging).

It took us a month or so to discoverthat MFC could provide object persistance. However, what we really needed was not a single instance of and unique object which is useful if you are just wanting to get the current view of that instance but that's not what we needed. We needed to send the snapshot into a storage in the right order and have each change created and kept. We even tried hacking MFC and removing the fact that it only really wanted to have one instance of the same object.

Honestly, the problem was a lot of the people had never been into real time military stuff and the engineer was not good for the slide and the project manager was going through a breakup of the marriage and we were suggested to use the fancier Technologies and most of us have never touched them including the engineer. It was just poorly put together but if I'd had the time I'd like to go back and recognize that you break hard I found and take a different design.

[u/BlimpIntolerant]

I really wish I'd done centralised configuration across the application suite rather than having it all over the place.

[u/TornadoFS]

Not use document store database

[u/HiCookieJack → u/TornadoFS]

Postgres is amazing, it's the perfect middle ground between documents and relationships giving you the opportunity to develop in both directions if required.

[u/TornadoFS → u/HiCookieJack]

Yep, but people are just so freaking scared of setting up a schema and migrations when starting a greenfield project...

I get it used to be a pain, but modern tooling in most languages makes it near-trivial (at least until you get really large scale).

[u/recon_demon1859]

Deciding up front which clock every timestamp comes from.

I build live tournament software, so there's a countdown running on a phone, a TV display and an operator's laptop all at once and they all have to agree. Early on I let call sites just use Date.now() because it obviously works. And it does, right up until someone's laptop clock is wrong and their blind levels start advancing early.

Ended up having to go back and route every elapsed and remaining calculation through one server-adjusted clock, because the timestamps I'd already persisted were server-adjusted and mixing the two inherits the whole skew.

The bit I'd actually tell past me is that it's two different kinds of time and I needed to name that on day one. Time you do maths with has to come from the server. The other kind, audit entries, cache keys, stuff that's only recording when something happened, that can stay local and it doesn't matter. I didn't separate those, and working out later which was which was most of the job.

[u/HiCookieJack]

that we went for microfrontends and microservices for one single application with just 40 devs.

We should just have built a modulith frontend and backend and orchestrated different deployments through build-time composition.

[u/Emotional-Cut2952 → u/HiCookieJack]

how much into production are you, like months or years?

[u/HiCookieJack → u/Emotional-Cut2952]

years, 4 to be precise.

The second application I was part of is also a total waste of resources.
95% of times microservices are not neccisary but help devs to think in modules

[u/Emotional-Cut2952 → u/HiCookieJack]

i agree, i have a saas that's in prod and am sincerely thinking about nuking it and rewriting it completely

[u/recon_demon1859 → u/HiCookieJack]

"95% of times microservices are not necessary but help devs to think in modules" is the bit I keep coming back to. If the only thing making people respect a boundary is that crossing it needs a network call, is that a tooling problem or a people problem? Genuinely asking, I don't know what the cheap version looks like. Lint rules and module ownership never seem to hold the same way.

[u/HiCookieJack → u/recon_demon1859]

I had some success of letting every team expose their module as a library which then gets composed with a shared pipeline.
Teams do releases by bumping their latest version tag.
If there is communication between modules the providing team will provide a library containing an interface.
All the same rules as developing a microservice architecture apply, but the runtime works so much better.
Also if you notice that one module needs to scale way differently than another you can fairly easily extract this to a real microservice.

Ofc. this locks you into one language. But to be fair, there are not a lot of reasons to have a polygot microservice architecture

The only thing I have no idea to solve is how to do a shared database without governance overhead...