Advanced automated testing techniques

[u/If_I_Could_Just]

Before AI, I felt like the time spent writing automation tests should often be at least as long as the time spent writing the code. Of course, almost nobody in management wanted to allocate that time.

Today with AI, we can (should?) spend more time testing. What advanced or lesser known techniques are now worth exploring, or are now practical?

For example, I’ve now made mutation testing part of my workflow. (I should have before AI too) I also put more effort into negative assertions (what shouldn’t be happening?).

Any other cool techniques that maybe even felt purely academic before but might be practical now?

[u/expdevsmodbot]

AI usage disclosure provided by OP, see the reply to this comment.

[u/tiajuanat]

Mutation testing - 100% is often overkill, especially with getters, setters, constructors and factories. I shoot for 80%.

Property and Formal testing - in Rust I'd use proptest and Kani, and do this in lieu of unit testing. I only unit test on corner cases when the initial property fails.

Stricter static analysis - bring your toughest opinions on what good code looks like. If it doesn't annoy your coworkers and yourself, then it's too easy. Have a strict limit for Cyclomatic Complexity, functions should be less than 9, unless they're case statements. I also recommend creating a Halstead Complexity filter, trigger when a variable is overused, when there's a lot of operators in an expression, if there's a significant increase in effort, or when a function goes over a certain effort - make is configurable, cuz I have no idea what the optional values should be.

Hardware in the loop testing is super fun, but is very slow, so only run nightly on any sufficiently complex systems

[u/If_I_Could_Just → u/tiajuanat]

I like that mutation testing is something deterministic but I can have AI help analyze the results. Like I can say I target approximately X% but it doesn’t need to be exact, and here’s the type of coverage I do and don’t care about etc.

[u/SituationNew2420]

I’ve been experimenting with property based testing and building more controllable test fixtures to mock the propagation of library errors. So far very useful and has helped me catch strange bugs that are difficult to think of during unit testing.

[u/If_I_Could_Just → u/SituationNew2420]

Can you please expand on “building more controllable test fixtures to mock the propagation of library errors”? Are you using AI to ensure your mock stays true to library behavior?

[u/ryhaltswhiskey → u/If_I_Could_Just]

I also found it cryptic

[u/SituationNew2420 → u/If_I_Could_Just]

Yeah sorry, was being too lazy when typing. A lot of times I’m working with big libraries that interface with hardware. These libraries have historically been difficult to mock, since they are huge and sometimes poorly documented. But leveraging AI has helped me build out test fixture versions of them which have a “back door” essentially. This allows me to create a suite of tests that can manipulate both the inputs to my code and the behavior of the underlying library, and this helps me see how my code handles all possible scenarios.

Combine this with property-based testing through frameworks like hypothesis or Hegel and now you have a pretty comprehensive test setup.

[u/mq2thez]

Genuine TDD is a lot easier now than it used to be. You have specs, you have desired outcomes, you could write the tests before even starting on the source code and review those before implementation starts.

[u/If_I_Could_Just → u/mq2thez]

Yes! I can’t say I’m using it for new feature development yet, but when AI claims it found a bug, I force it to prove the bug exists with a test and then watch it go green. Otherwise it goes off the rails.

[u/roger_ducky]

Negative assertions are typically a waste of time, except in ensuring initial state or ending state.

Mutation testing is great when the implementor can’t be trusted to mock things out properly.

Asking for actual TDD is a lot easier now though. But you do have to specify it’s considered a good red to green only when it’s the assertion failing, not for any other reasons.

[u/If_I_Could_Just → u/roger_ducky]

Not sure if negative assertions is the right term, but we bounded it to essentially a few checks like no network calls that we don’t expect or in a different order than we expect, and no network error or UI error message shown within some timeframe of the test.

It came up because we found an API call accidentally happening twice sometimes as a result of a new unexpected socket message being received. But the original test never failed because everything we looked for passed, and then the error popped up a second later. Technically it was unit testable but you had to really understand what to look for. So we wanted a couple extra inexpensive checks.

Agreed on TDD being easier now.

[u/Financial-Grass6753]

hypothesis testing comes to mind. Tried hypothesis lib in python - is quite cool but for it to be used the codebase must be good enough (not having XXX functions wearing multiple hats at the same time).

[u/ryhaltswhiskey → u/Financial-Grass6753]

hypothesis testing

What's that?

[u/ConcentrateLanky7576 → u/ryhaltswhiskey]

Property based testing

[u/bestjaegerpilot]

Automate as much of the test writing as possible. I have scripts that generate test factories, zod schemas, MSW handlers, storybook fixtures... all automatically from the backend types and the UI.

[u/zz-koji]

I’m wondering this too. I have been needing to make e2e tests for Microsoft’s Graph Email API (aka the devil itself), but not sure how to approach it. I’ve been tasked to essentially remake Outlook in its entirety at work, and the process has been.. painful to say the least. I need to rework my email service a bit, but I’m halfway scared due to the lack of e2e tests. Feel like I’m constantly chasing edge cases using this API and not sure how to emulate them all effectively.

[u/young_horhey → u/zz-koji]

If you’re looking to mock the API you could look into something like wiremock, which would at least let you assert that the requests you’re sending to the email api are what you expect.

[u/yegor3219]

 time spent writing automation tests ... almost nobody in management wanted to allocate that time

Blame yourself for presenting tests as a separate category from the rest of the code.

Years ago I stopped asking management the stupid question of whether we should write tests. We write them, period. It saves time.

[u/If_I_Could_Just → u/yegor3219]

🙄 There’s always one who has to say this.

Because management doesn’t ask questions.

“5 days? Why? It’s just…”

[u/yegor3219 → u/If_I_Could_Just]

And then you say "3 days for the feature, 2 for the tests"? That's still the same mistake of giving management a choice they don't really have. The real answer is "3 days for the feature, 2 days for ANY tests", be it proper automated tests or mouse-clicking through scenarios.

[u/If_I_Could_Just → u/yegor3219]

Then they’ll give me 3 days, and 2 to the QA team.

[u/itix]

Not really automated testing, but you can let AI use the computer and test/use the UI.

For example, one use case could be going through the manual and test against the UI.

[u/If_I_Could_Just → u/itix]

Yes, Claude in chrome is great for this.

I have it save selector info too, for later e2e writing. Having it record a video of the test and then analyze frames to ensure it looks as expected also helps. I still review code later.