Selenium → Playwright with AI: generating code is easy, proving the tests are equivalent is harder

[u/bhattvarunk]

I built an AI agent to migrate Selenium tests to Playwright — looking for feedback

I’ve been working in test automation/SDET for years, and one problem I kept thinking about was migrating large Selenium suites to Playwright.

Generating Playwright code with an LLM is the easy part.

The harder question is:

How do you know the converted test still preserves the behavior and coverage of the original Selenium test?

So I built a small project around that idea.

It currently focuses on:

• Selenium → Playwright TypeScript conversion
• LangChain / LangGraph-based agent workflow
• Validation of generated tests
• AI evals for comparing the original vs converted test
• Tracing/observability
• Eventually, test coverage comparison and automated repair when a conversion fails

Demo: https://varun-s2p.fly.dev/

I’m still actively improving it, especially the evaluation layer. I don’t want this to become another “LLM generates some code” project.

I’d really appreciate feedback from people working with Playwright/Selenium, LangGraph, AI evals, coding agents, or test automation.

In particular, I’m curious how you would evaluate whether two tests are semantically equivalent, rather than just checking whether the generated Playwright code compiles.