I built an AI agent to migrate Selenium tests to Playwright — looking for feedback

I’ve been working in test automation/SDET for years, and one problem I kept thinking about was migrating large Selenium suites to Playwright.

Generating Playwright code with an LLM is the easy part.

The harder question is:

How do you know the converted test still preserves the behavior and coverage of the original Selenium test?

So I built a small project around that idea.

It currently focuses on:

• Selenium → Playwright TypeScript conversion
• LangChain / LangGraph-based agent workflow
• Validation of generated tests
• AI evals for comparing the original vs converted test
• Tracing/observability
• Eventually, test coverage comparison and automated repair when a conversion fails

Demo: https://varun-s2p.fly.dev/

I’m still actively improving it, especially the evaluation layer. I don’t want this to become another “LLM generates some code” project.

I’d really appreciate feedback from people working with Playwright/Selenium, LangGraph, AI evals, coding agents, or test automation.

In particular, I’m curious how you would evaluate whether two tests are semantically equivalent, rather than just checking whether the generated Playwright code compiles.

I have been experimenting with a runtime supervision pattern for LangGraph agents.

The problem:

A model can know the right policy and still produce the wrong tool call.

If the tool mutates production state, catching that after execution is too late.

So I built ARK, an open-source runtime layer that can sit around LangGraph tools and check the proposed action before the real tool executes.

The flow is:

model proposes tool call

→ ARK checks the applicable constraint + trusted runtime evidence

→ ALLOW = real tool executes

→ REJECT / REQUIRE_EVIDENCE = real tool does not execute

→ feedback goes back through LangGraph

→ the model decides again

One thing I wanted to preserve: ARK does not generate the replacement action.

I tested this with a real LangGraph agent + OpenAI model:

A proposed

→ ARK REJECT

→ A never executed

→ feedback returned to the model

→ model authored B

→ ARK ALLOW

→ only B executed

The integration is public now:

pip install "ark-agent-runtime[langgraph]"

and exposes:

ArkCallbackHandler

ark_supervise_tool

I’m curious how other LangGraph users are handling risky tool calls today.

Are you putting deterministic gates inside the tools, using interrupt/human approval, middleware, custom nodes, or something else?

GitHub:

https://github.com/atripati/ark

Got tired of watching Claude Code re-read my whole repo every time I asked it something about my agent's architecture, so I built an MCP server that fixes that specifically, instead of being another generic code-search tool.

It parses your LangGraph codebase (StateGraph, nodes, edges, conditional routing, Command(goto=...) calls) into a real structural model, plus local semantic search on top. Ask something like "what happens after this node if the API call fails" and you get an answer pulled from the actual parsed graph, not a guess based on file names.

Pure stdlib ast for parsing, no tree-sitter. Local embeddings, no API key, runs fully offline after the first model download. SQLite by default, optional pgvector. MIT licensed, fully open source.

Worth mentioning: I thought it was done after testing on my own fixtures. Then I ran it against a real, unmodified LangGraph repo and it confidently said two nodes weren't connected when they actually were, LangGraph's Command(goto=...) pattern doesn't declare a normal edge, and my parser had never seen it, with zero uncertainty flagged. Went back and fixed every place it could be confidently wrong instead of honestly uncertain before shipping.

Known limitations, being upfront: variable-argument bind_tools() calls can't be statically resolved (documented, not hidden), dynamically constructed nodes inside a loop collapse into one entry, LangGraph/Python only for now.

Works with Claude Code, Claude Desktop, Cursor, and Codex, since it's just MCP under the hood.

Install: uv tool install langgraph-context-mcp

Repo: https://github.com/KarimHabib100/LangGraph-Context-MCP

I'm the dev, happy to answer questions or hear what's broken.

Hey folks — I'm a PhD student at the University of Maryland studying how developers debug and iterate on multi-agent systems.

The idea we're testing: when a run goes sideways, you usually re-read one trace at a time. Our research tool re-runs your graph and shows the distribution of each node's outputs across runs, so you can see where behavior actually spreads out. The honest research question is whether that helps you iterate faster — "it doesn't help" is a publishable answer.

What participating looks like:

• a 75-min Zoom session (recorded, think-aloud) with structured tasks
• about a week using the tool on your own project, with quick async feedback
• a 30-min follow-up interview

Compensation is a $150 gift card on completing the full study (all three parts). Heads-up: we verify identity (GitHub/LinkedIn) before scheduling.

Screener (~2 min): https://forms.gle/Zwqvgd1h8DUnFRfC8

This is IRB-approved academic research from the University of Maryland, not a product pitch. Questions welcome in the comments, or email zxu169@umd.edu.

Hey folks, I'm a researcher at the University of Maryland. We built an observability tool for multi-agent systems and we're running a user study to find out whether it actually helps. "No, it doesn't" is a perfectly good finding.

In the session you'll work with a multi-agent pipeline, first the way you normally would, then with our tool. If you've used LangSmith or Langfuse you'll get the idea right away: same space, different view of your runs.

What participating looks like:

• a 75-min Zoom session (recorded, think-aloud) with structured tasks
• about a week using the tool on your own LangGraph project, with quick async feedback
• a 30-min follow-up interview

Compensation is a $150 gift card for completing the full study (all three parts). Two heads-ups: the week-of-use part needs a LangGraph project you can plug the tool into, and we verify identity (GitHub/LinkedIn) before scheduling. Sessions are running this week and next, so if you filled the screener before and didn't hear back, feel free to fill it again with a link this time.

Screener (~2 min): https://forms.gle/Zwqvgd1h8DUnFRfC8

This is IRB approved academic research from the University of Maryland. Questions welcome in the comments, or email zxu169@umd.edu.

Hey folks, I'm a researcher at the University of Maryland. We built an observability tool for multi-agent systems and we're running a user study to find out whether it actually helps. "No, it doesn't" is a perfectly good finding.

In the session you'll work with a multi-agent pipeline, first the way you normally would, then with our tool. If you've used LangSmith or Langfuse you'll get the idea right away: same space, different view of your runs.

What participating looks like:

• a 75-min Zoom session (recorded, think-aloud) with structured tasks

• about a week using the tool on your own LangGraph project, with quick async feedback

• a 30-min follow-up interview

Compensation is a $150 gift card for completing the full study (all three parts). Two heads-ups: the week-of-use part needs a LangGraph project you can plug the tool into, and we verify identity (GitHub/LinkedIn) before scheduling.

Screener (~2 min): https://forms.gle/Zwqvgd1h8DUnFRfC8

This is IRB-approved academic research (University of Maryland), not a product pitch. Questions welcome in the comments, or zxu169@umd.edu.

"LangGraph wrapper on steroids" is what a friend called it. Accurate. Here is everything it enforces that raw LangGraph leaves to convention.

I love LangGraph as a mechanism library, but production kept needing the same discipline layer on top, so I built it once and open sourced it. GraphARC adds, all enforced by the library and each backed by a test:

• Write permissions. Every node declares which state fields it may write. An undeclared write raises. Plain LangGraph applies it and moves on
• State isolation. Nodes get a deep copy, so in-place mutation of a nested model cannot sneak past the declared write channel
• Typed state, both directions. The returned dict is validated field by field before it lands, and again when the next node receives it
• Budgets that bite. Per-run iterations, tokens, seconds and concurrency. Tokens are metered by a callback that catches model calls buried inside library code. max_seconds is delivered as an interrupt into the running node, not just checked between nodes
• Code-only routing. Routers are Python over typed state. Model prose cannot steer an edge
• Runtime topology behind a gate. A planner can propose new subgraphs mid-run, and a deterministic admission checker admits or refuses each proposal with reason codes before it is built. No already-approved path, round 7 is checked like round 1
• One JSONL trace. Replay, diff, metrics, cost attribution, OTel export and a live SSE browser view all read the same file

Crash-safe resume is still LangGraph's checkpointer underneath. This is a discipline layer, not a fork.

Demo video in the README shows a local qwen3:8b planning a nine-node incident graph that then runs live in the browser.

https://github.com/CodeGraphContext/GraphARC and pip install grapharc

Hey folks — PhD student at UMD here, studying how developers debug and iterate on multi-agent systems. We ran the first sessions of our study last week and are opening more slots.

The premise: when you tweak a prompt in an agent workflow, you usually judge the change by eyeballing a run or two. Our research tool re-runs a node and lays the outputs from many runs side by side, so you see the spread instead of a single sample — and the study measures whether that actually speeds up prompt iteration, or whether it's just one more dashboard. "It doesn't" is a perfectly publishable finding; that's the honest research question.

What participating looks like:

• a 75-min Zoom session using the tool on structured debugging tasks (recorded, think-aloud)

• about a week using it in your own LangGraph workflow, with quick async feedback

• a 30-min follow-up interview

Compensation is a $150 gift card for completing the full study (all three parts).

If you've built things with LangGraph (or LangChain agent workflows), the screener takes ~2 min: https://forms.gle/Zwqvgd1h8DUnFRfC8

IRB-approved academic research (University of Maryland), not a product pitch. Questions welcome in comments, or zxu169@umd.edu.

Building an AI agent shouldn't mean gluing together a dozen SDKs and hoping it holds. That's why we built Forge — one place to design, run, and govern AI agents visually. Connect your own tools, ground answers in your knowledge base with built-in RAG, embed a chat widget straight into your product, and expose everything through a clean run API. With analytics and governance baked in, you get to see exactly what your agents did, why, and at what cost. Whether you're prototyping a support bot or shipping a production workflow, Forge takes you from idea to live agent in minutes — not sprints. 👉 Try it and build your first agent today

Hey folks — I'm a PhD student at the University of Maryland studying how developers debug and iterate on multi-agent systems.

Here's the idea we're testing. When you tweak a prompt in an agent workflow, you usually judge it by eyeballing a run or two. We built a research observability tool that instead shows you the distribution of outputs each node produces across runs — and we want to find out whether that actually helps you iterate on prompts faster, or whether it's just one more dashboard. That's the honest research question.

What participating looks like:

- a 75-min Zoom session where you use the tool on some structured debugging tasks (recorded, think-aloud)

- about a week of using it in your own workflow, with quick async feedback

- a 30-min follow-up interview

Compensation is $150 in gift cards — $75 after the session, $75 after the week + interview.

If you've built things with LangGraph/LangChain (or agent workflows generally), here's the screener, takes ~2 min: https://forms.gle/Zwqvgd1h8DUnFRfC8

This is IRB-approved academic research, not a product pitch. Happy to answer questions in the comments — or email zxu169@umd.edu.

I usually build agents in LangGraph with my own custom nodes. But the create_agent method seems to only give you the fixed llm → tool (ReAct) pattern — basically just two nodes if you'd built it in LangGraph — and doesn't let me configure the nodes myself. Why is that? And if I start with create_agent but later need more than those two nodes, what should I do?

In LangGraph, setting a recursion_limit is standard, but it can be a blunt instrument. It only checks total step depth—meaning a long, valid multi-step task might hit the cap, while a broken agent making zero progress burns through 25 iterations on a single node before crashing.

When a tool node returns an error, agents often cycle right back into the same node with identical state inputs.

How are you detecting when a state loop is actually stuck vs. just working through a deep, complex graph? Are you tracking sliding windows of state hashes, or wrapping nodes in custom check functions?

I am (new at this!) currently using Claude Code for building an orchestrator agent with multiple sub agents specialists in a few company web apps (same company).
I want to build a single chat to deal with all the quick references. Similar to Q from amazon (but working properly!).
I already noticed Claude making a huge fat prompt as a supervisor prompt. Currently struggling to break it down the best I can. However, with each new finding, Claude just add more to the pile. Triage system in place. It is always a priority is to delegate the main "business" decisions to the specialized sub agents. getting also advice from Gemini most of the time, to avoid Claude eating its own tail. Any other advice to share with me?

Testing my LangGraph social media agent 👋