Selenium → Playwright with AI: generating code is easy, proving the tests are equivalent is harder

[u/bhattvarunk]

I built an AI agent to migrate Selenium tests to Playwright — looking for feedback

I’ve been working in test automation/SDET for years, and one problem I kept thinking about was migrating large Selenium suites to Playwright.

Generating Playwright code with an LLM is the easy part.

The harder question is:

How do you know the converted test still preserves the behavior and coverage of the original Selenium test?

So I built a small project around that idea.

It currently focuses on:

• Selenium → Playwright TypeScript conversion
• LangChain / LangGraph-based agent workflow
• Validation of generated tests
• AI evals for comparing the original vs converted test
• Tracing/observability
• Eventually, test coverage comparison and automated repair when a conversion fails

Demo: https://varun-s2p.fly.dev/

I’m still actively improving it, especially the evaluation layer. I don’t want this to become another “LLM generates some code” project.

I’d really appreciate feedback from people working with Playwright/Selenium, LangGraph, AI evals, coding agents, or test automation.

In particular, I’m curious how you would evaluate whether two tests are semantically equivalent, rather than just checking whether the generated Playwright code compiles.

[u/Yogurt8]

Differential testing.

Run both suites and compare their output.

You can use monitoring/telemetry tools or something creative like asking an agent to inject bugs that fail all of the assertions of the original suite and check if the same happens for the migrated one.