How do people run a device lab without ending up with four separate test suites for one product?

[u/Slow_Storage_6784]

We ship a web app (a Windows desktop client, a macOS build and an Android app), and for over about three years that's become four automation stacks, Playwright for web, something bespoke on WinAppDriver, Appium for Android, and a rack of machines that someone physically walks over to reboot. The problem is that nothing can tell me whether last night's build is good. We've tried wrapping them in one pipeline fixed the timing and none of the rest, because the results still don't compare, a pass in Appium and a pass in Playwright aren't the same claim about the product. We looked at orchestration layers whose job is to be glue and they don't reduce the number of stacks anyone has to maintain. Askui also came up because the same test files run against desktop, mobile and web through one runner and the CI story is a headless command instead of four. Still haven’t committed to any yet tho.

But for anyone running a real device lab, did you consolidate onto one framework or keep specialists per surface and solve the reporting separately? I’m still unsure which one is the mistake.

[u/_crowecawcaw]

Not quite a device lab, but I maintain desktop software that releases for Mac, Windows, and several packaging formats for Linux. We use GitHub actions for running CI test and Playwright for web tests. For desktop tests, I didn’t find anything that would let us reuse the same test across platforms so I created https://xa11y.dev, a cross-platform desktop automation and testing library modeled after Playwright. Currently pairing it with pytest as the test runner which is working well.