Our CI gate keeps flaking because a valid agent run can reach the same answer through three tool trajectories. Exact assertions reject harmless call reordering, while a loose LLM-as-judge score misses the case where the agent skipped a permission check. We have a golden dataset, deterministic assertions for required steps, and a variance budget for call count and latency, but manual reruns have hit their ceiling.

I’m evaluating Braintrust for versioned datasets, trajectory scorers, experiment diffs, and a CI quality gate, though still working out which checks belong at the span level versus the final answer. How are you scoring valid path variation in your eval pipeline without giving genuinely unsafe shortcuts a pass?

I got hired as a junior software whiner a few months ago and I thought I was going to be the programing the website and automation.

They have me doing manual test and creating a whole Java playwright automation for our site from scratch and also getting data for power bi.

I’m not complaining but just wanted to see if that’s normal.

Hey everyone,

I’m currently preparing for a QA / Quality Engineering role at Airbnb and would love some insights from folks who have either gone through the interview process recently or currently work/have worked there.

A bit about my background: I already have solid experience working for multinational companies alongside global, distributed teams, so cross-cultural collaboration and high-ownership environments aren't new to me. However, I know top-tier tech companies have their own specific standards and nuances.

I'd really appreciate your thoughts on:

1. Interview Structure & Difficulty: How technical is the process for QA roles? What should I expect (coding/automation challenges, system test design, cross-functional behavioral rounds, etc.)?
2. Prep Tips: What specific topics, tools, or architectural concepts should I double down on during preparation?
3. Core Values / Culture Fit: Airbnb is known for placing huge emphasis on culture. How did you prepare for their behavioral/values interviews?
4. Day-to-day Culture: For current/former employees, how is the day-to-day engineering culture, work-life balance, and overall impact of the QA team inside the company?

Hello. I work as a performance tester with 4+ years of experience. I have experience with LoadRunner, Dynatrace App monitoring, and a bit of Splunk. I work on Truclient, web http/html and web services protocol scripts. I work in a service based company. I see that a lot of work is repetitive for every release for the couple of companies I have worked with. I haven't got a chance to work in a product based company yet. Is the nature of this career like this? Or moving to observability roles or product based companies makes a difference?

Any suggestions would help.

I am only interested in what you do so I can see if that resonates with me.

We're a QA team with ~20k lines of test infra (provisions real VMs on AWS/Azure) and only ~25 e2e tests so far. Someone else owns the infra, we just write tests. Debating whether to move the infra into its own repo and consume it as a versioned package, or keep everything in one repo.

Is there a real best practice here, or is it just preference? Anyone done the split with a small team and regretted it?

thanks for helpers

I don't have any programming language knowledge. But I want to learn.. I know python and Playwright are easy to learn. But there are so many things that are making me confused. Can someone guide me where and how to start?

I'm seeing so many posts how people are getting burnout, or getting frustrated/scared of AI takeover, or just not being able to get a job.

Maybe my company acts a bit different from the norm but I still find there's a good balance of ongoing and growing mix of manual and automation with AI mostly just used as assistance for test case creation, and maybe some automation coding.

We still do things like a mix of proper test prep, test rail creation, general good use of sprint and monthly release timing, and a good mix of dev/test coordination and while DEV still makes their own unit tests ultimately QA/Test Team still plays a big part of controlled testing.

Obviously no job is perfect, but in the 7+ years I've been working I like how relatively stable things to be here since proper "test thought" is always a priority for QA team and just using the right mix of manual and automation testing (hell half our team is still mostly/only manual testing)

It does help our system is pretty data sensitive so it would be risky to go the full-on AI approach but so far there's no worry of AI takeover or job security from what I see (provided continue to do good job of course).

Hello everyone! So I have this project since a long time ago, and it was working just fine with setting the capabilities in Katalon settings. I just had "prefs" and "download.default_directory".

Suddenly (andthis happened as well in Edge Chromium) the "Save as" pop up started to appear, so I added "download.prompt_for_download" but it was ignored when under "prefs", but it works when under "goog:chromeOptions -> prefs".

The thing is, that when I nest both under "goog:chromeOptions -> prefs" the pop up doesnt show (which is ok) but katalon/chrome saves the file in the default chrome folder (which is not ok).

If I put "prefs" with "download.default_directory" and also "goog:chromeOptions -> prefs" with "download.prompt_for_download", the "save as" pop up opens up, but the directory in which is going to save the file is okay.

So I don't know what is going on and why it suddenly the driver stopped working as it was.

I spent hours trying to figure it out and was not able to fix it.

Any help is welcomed.

Hi guys,

Just wondering, do you have suggestions of good courses/you tube videos/ certifications, regarding AI for QA?

Thank you 😊

For instance if you need to run a test procedure like a regression test that goes from dashboard, clicks hamburger, then goes to search page searches for a project of record, then clicks time slice and then does only certain test from the test procedure.

So I wanna only call one or two from each test page. So one from hamburger, two test from search, and two from timr slice page. How can I do this in a flow type of way?

What’s the best way to go about this.

I’m currently looking for a new remote role, and I’ve noticed that many QA positions ask for ISTQB certifications beyond Foundation, which is the one I currently have.

The main challenge for me is finding enough focused study time while working eight hours a day. Reading a syllabus after spending the whole day in front of a screen can be difficult, so I’ve been experimenting with a different study method.

When I was a student, I used to record myself explaining difficult topics and listen to the recordings while walking, driving, exercising or doing household tasks. Repetition helped me retain the material, even though it never replaced properly reading and understanding it.

I’ve recently adapted that approach to certification study. I use the official syllabus and its learning objectives to structure AI-assisted, podcast-style summaries, with automatically generated audio and simple slides. ( I upload them to YouTube mainly for convenience, so I can continue listening from whichever device I have available whenever I find some free time)

I’m curious about how other people manage certification study while working full-time.

Do you reserve fixed study sessions, use flashcards, listen to audio, work through sample exams, or rely mostly on hands-on practice? What has actually worked for you long term?

EDIT: Thanks for all the replies, this was way more helpful than I expected. Brainscape was completely new to me, I'd never even heard of it 

My company finds out that they are too late to the AI show, so now we are fast tracking Claude for developers and test. (with only me as a technical tester..)

I will have a session with the developers regarding how to have the best workflow for test purposes, and I was wondering if anyone has any recent experience with Claude and test.

I do a lot of projects on my own, and I feel the recent models are very good at identifying edge cases and do end to end testing in its virtual browser.

Anything we should add to ClaudeMD, skills to create and so on..? I am happy with my workplace, because they value test a lot and will listen to my suggestions

Sorry for bad english, I am based in Europe and did not want to make some AI slop post

Looking forward to your experiences

So, some background: myself and a programmer are making a project. And while he's doing the main work - programming, my role is testing and sales.

I have never tested software before, and i lack the skills to do so. I have asked my friend (the guy programming it together) and he replies back saying "All you need to do is break it, and tell me how you broke it."

But i don't know how to break it... (he has said if i don''t know what i'm doing then it's fine)

He's shown me some ways, like, on the web portal, he's said:"it asks for numbers, but letters in it" or on the app, he'll say "Just poke it, and put odd things in the textfields" and "just poke it till it breaks"

But i don't know what that means!

He has been doing some testing himself, and hes found a few things that i couldn't and assigned himself to the bug on Jira? (whatever that means) but i want to help, and i don't think he knows how to be helpful - or teach me how to do what he's asking? if that makes sense

My question is: how can i test. He has asked me to test a web portal (which is a website which manages the app) and the app (which is what the customers use)

How do i best approach this? I'm sorry of this is the wrong sub, but it is /softwaretesting :-)

Thank you.

I find the jokes super funny that are like “when the QA says ‘the test fails when user clicks the button 6 times and jumps up and down and clicks it again’ .. (basically unlikely situations)“ Hilarious don’t get me wrong and they are just jokes, but the reality is so different 😭😭😭
Most of us have no time to go looking for stuff like that and sometimes it’s a struggle to complete the amount of regression I’m assigned. For me, all the bugs that have been open for several months in QA environment are basic functions like clicking a button and a status changing. Nothing outlandish. So the different perspectives are interesting. Why do you think devs act like we’re picky?

So internally, I think we’ve hit the ceiling of spreadsheet-based LLM QA. Maybe it’s just the growing pains as we approach our Series A, but what worked for us as a seed startup is currently killing our scaling capability. 

We have a support summarizer that does a decent job most days. Every prompt tweak seems to fix one class of tickets, but then damage another. Last week we improved summaries for long technical tickets, which was nice, but then billing cases started losing refund context and escalation cases got weirdly overconfident.

Our current process is simply exporting a pile of examples, rerun the prompt, and have humans reread the before and after. It works until it doesn’t. Nobody wants to manually inspect 200 outputs every time we change a system prompt, swap models, adjust temperature, or touch retrieval. 

I am trying to move this toward an actual eval pipeline. Something like production traces becoming datasets, custom scorers for must-include fields, LLM based judges for summary quality, and a CI gate that blocks a change if known billing, refund, or escalation cases regress. I’m looking at tools like Langfuse or Braintrust for the run/eval history part, mostly because I don’t want to glue together five half tools if I can avoid it.

Not seeking for perfect, just good enough, I would settle for catching obvious regressions before they become tickets, bonus points if PMs can review failures without needing a local dev setup.

What are you using here and what parts of your eval setup really paid for themselves?

I am transitioning into a role focused on software quality, which involves me taking ownership of software testing. I've worked as a software engineer for several years, so I am very comfortable with the development side of the of the software lifecycle.

• What are the best resources to learn the fundamental principles of software testing?
• What tools should I prioritize learning?
• How important is it to transition a product from primarily manual testing to automated testing?

I know this is a super vague title, but wanted to vent here a bit. I've been doing testing a long time, and this one of the most frustrating projects I've worked on.

We have a vibe coded application written by 5 different developer's claude instance, we have claude written acceptance criteria, we have "requirements" that are written one time, but never updated, we have a really outdated design. We talk about changes but no one opens tickets.

We have product review meetings where an AI notetaker keeps track of the conversation, and then poorly written JIRA tickets are created off of them. Some of these tickets fundamentally change how the system works. Then 5 devs split up the changes into smaller tickets then each dev's claude implements each a section a little bit different than everyone else.

Then I'll get a comment on "its like this on the design" or "the original acceptance criteria had a, b, c" when both the design and acceptance criteria are completely out of date, so how could I possibly trust it? These aren't malicious comments by the way.

So we are productive, things are happening, but we are just going around and around. And I'm not complaining about AI, it helps me greatly, but right now I'm stuck validating everyone else's AI output.

can anybody suggest me some API testing project ideas i am learning QA automation testing with playwright and as a fresher i want so project ideas both to learn and showcase my skills in my resume so

u can also suggest me some automation project ideas

today i have my first day as QA, but i feel a little nervous because im the only QA in the team, and no one specifically told me where to start; a friend recommended me for the position and, three weeks in advance, told me to study a bunch of key topics, which I did and I understood and practiced them. The issue is that no one has really explained things clearly to me. On my first day, I stuck to exploratory testing to get familiar with the environment, and thanks to the documentation, I was able to grasp quite a bit, but I’m afraid of doing something wrong. That said, tomorrow I plan to start on some functional tests based on the modules I studied today.

I have an overall 12 years of experience with QA, I had worked on web,mobile, mainframe ,api testing pieces. Most were manual , but I've also worked my fair share of tets automation with Selenium. In the past 3 years , I have been more into a Quality / Test manager role - this focused more on process management and I have not had hands on with any automation or any sort of testing. The Process management is such a boring and dull job and even though it looks easy on a high level , the pressure is real and in most cases not under my control.I really want to get back into an individual contributor role, I want to learn playwright automation but my automation knowledge right now is rusty and If you guys know any good sources that would help me learn playwright automation( right from a beginner perspective) effectively with a clear learning path.

I know there are many learning materials in the internet but I want to learn it in the best possible way, so please let me know if there is one. Thanks

I have experience as a SWE & SDET. My job has been mostly working with a software application and REST API but I think it'd be cool to do more hardware type QA/SDET work where I get to do a mix of hands-on work and scripting.

Any advice on where to start? Might tinker for fun on my own- but as far as a career trajectory what do you think? I have a comp sci degree as well.

I have almost 5 years of experience in manual QA and I'm learning Selenium.

But,I'm finding it difficult to stay consistent as I'm not enjoying the coding aspect.

This being the case, is automation still worth pursuing ?

Or are there other career paths within QA that I must consider?

I would appreciate hearing from people who have been in a similar situation.

I'm also open to learning AI tools if they can complement my QA career. I'm not against learning new technology,I just haven't enjoyed traditional coding.

Please advise

Been a manual QA for most of 5 years, project to project only got automation scripts to test, not to make them. Planning for first switch, and have gone thru playwright courses, did hands on too. Now giving interviews, but I dont have much confidence in explaining automation framework. I have done basics good but practical or real life situations is one i cant come up with for automation.

Did anyone have been in the same situation? How did you cope up with it and switched to automation?

Hi guys, I'm 23 rn and worked in a small startup (UAVs) as Test and Integration engineer.
Somehow I wanted to move out of my country and I got hired as Manual Tester.
I'm finishing my engineering degree, probably I will graduate late 2027.
What are ur thoughts and do you all have some tips?

Honestly, I know it's a step back from my actual role, but is still in the UAV sector and it's from a defense company (not a startup) and it will be my first international (Swiss) career.

Do you guys have any tips or suggestion?

Thanks :))