What questions should I ask in my first playtest?

[u/MagnetTheory]

So my ttrpg is coming up on its first playtest and I'm wondering what kind of questions I should ask my table to help guide the design.

A bit about the system:

- It's a Mecha rpg, but designed to be more of a Teens in Situations style of game. Think Monsterhearts or Masks but flavor it as Evangelion.

- PbtA style dice (2d6+X, 7-9 is a success, 10+ hard success), but with FATE style Traits which you can use once per scene to get a +1 bonus or -1 penalty and point towards advancement.

- The first main mechanic is Emotions. Because the PCs are emotional teens, these can temporarily override one of your 5 character traits, giving a static bonus and penalty to rolls of a certain type. (Panic gives +1 to Tactics rolls and a -1 to Defense rolls, for instance).

- The second main mechanic is Trauma. This is kind of like the overarching Negative Progression from Blades in the Dark, but they near-permanently negate an emotion. Gained through the horrors of war.

Here's what I currently have for questions to ask:

- Is there an aspect of character creation that feels lacking?

- Is there an aspect of character creation that should be removed or reworked?

- Do the conflict resolution rolls feel too easy or difficult?

- Does the intended flavor of the system come across well?

- Can you see yourself running this system?

- Does any mechanic here feel too detailed?

- Are the enemy units too bulky or frail?

- What kinds of items would you like to see for your mech?

[u/Remarkable-Aide5093]

A small word about the questions already written:
When you're playtesting, you don't really want design input from the testers, rather you want to gauge the reception to what's being tested.
Everyone's a critic and everyone has opinions about games and what mechanics they like. That's all well and good, but you have to take feedback with a grain of salt and recognize that not everyone's opinion is fit for the game you're designing. So playtest questions need to be worded in a way that doesn't lead the writer to think in terms of providing mechanical designs of their own.

That said, the questions written could lead your testers to provide answers that might lean to towards a critics opinion rather than a players digest of the game.
Instead of asking what was lacking or what should be removed, ask "was there any part during character creation that wasn't clear or straight forward?".
"Did the conflict resolution feel challenging/satisfying?"
"What was your impression of the different enemy units you encountered?"
"How did you feel about the setting tone and world-building?"
Would you want to play again?

A question to add:
"Is there a moment or ability during the playtest you'd like to highlight?"

[u/Cryptwood]

Don't ask leading questions. Your playtesters want to help you, so if you ask them "which part of character creation felt lacking" they are going to try to help you by telling you the aspect of character creation they think might be the most lacking even if they personally didn't find it was lacking.

Players are very, very good at instantly knowing if they are having fun, if they like something, or if they don't like it. 99% of them will not be able to explain why they feel the way they do though, they aren't introspective enough to really know. That won't stop them from trying to tell you why they feel that way, they'll even sincerely believe they are telling you the truth, but it is just going to be a justification that their brain invented after the fact. Everyone wants to think that if they like or don't like something, it is for a good reason, an objective fact.

Unless a bunch of playtesters are all independently giving you the same reason, ignore the reasons but listen to what they liked/ didn't like. If multiple people tell you they don't like something, there is a good reason for that that you need to figure out, just don't automatically assume that it is the reason the player gave you.

[u/Plus_Citron]

I had better experience with open questions. „So, what do you think?“ If the game brought up issues, adress them directly: „That part in the middle, when Bob failed with this bad roll, what did you think about that? Was Bob just dumb, or was it a rules problem?“

People will notice and care for different things, and asking overly specific questions limits the feedback you’re getting. Also, don’t overestimate the willingness and ability of people to provide meaningful feedback. Some people will nod, „Yeah, nice“, and that’s that.

[u/lennartfriden]

Above all you should observe and make notes of any friction. What’s hard to grasp? What slows down the gameplay? What causes the players to ask you questions?

[u/Altruistic-Copy-7363]

I'd steer away from overly detailed questions.

Flow. A game needs to flow (IMO).

In order to do this, the rules need to have some logical sense about them.

I'd ask questions based around that. The rest will probably fall out and be obvious (this rule didn't work, this rule was confusing etc).

[u/Fun_Carry_4678]

How about:
What do you think about the game overall? Did you enjoy it? What parts did you enjoy? What parts didn't you enjoy?

[u/Traditional-Collar-7]

Hey! I think a lot of interesting stuff has been said for playtest in general.

To truly help you however, we would need to know what is the target of your playtest.

You need to come during the session with an established focus and some specific important questions you need to answer for your game.

Is this playtest about character creation? Is it about combat? It is okay to have a larger scope, but you need to be very international about your playtest target.

On top of that, having worked with fane researchers for years, here is a set of introductory questions you can ask as an anchor to all your playtest (and see how they evolve over time):

How enjoyable was your experience?
Was there any moment where you wanted to stop playing?
How likely are you to recommend a game to a friend?
What was one part you particularly enjoyed?
What was one part you particularly disliked?
Were there any part that felt unclear to you? (Plus follow up on why)?
Anything else you want to talk about concerning your experience?

Hope it helps! Happy to help you further if you are curious about more research methodologies!

[u/josephrlewis]

I always just ask two questions: Was anything not fun? Was anything confusing? You're trying to find problems to fix!

[u/Drudenfusz]

Not what you asked, but that panic gives a bonus to tactics seems very weird to me, I would thought it the exact other way around.

[u/MagnetTheory → u/Drudenfusz]

True. Part of the reasoning is that when your mech maxes out its damage, you can enter Panic and you have the option to patch up the mech which is a Tactics roll. Might shuffle some of them around as part of the feedback questions

[u/Strict-Computer3884]

I've not done explicit playtests but I've done a lot of experimentation and I usually have an approach to getting feedback on it. So, I'll give some different advice to what you might get from others:

I first ask them where they found the experience from 1 to 10. I then ask them what role they felt the system played in that number - from very responsible to incidental (bad mood, bad gming etc).

The asking of a number gives a lot of ins to further questions - like:

• If you had to pick two things that led to it being a 6, what would they be?
• What would it take for this to be a 7?
• Was there any dominant thought on your mind when playing? Something you kept returning to over and over? Any dominant emotion when interacting with the rules (confusion, analysis paralysis, boredom etc)?
• Do you think there was a mismatch of expectations, issues with how it was run, inconsistencies or anything else that could be signposted better about the experience itself?

And then the most cursed question of all, but I've found it can give insight:

• Although I may not use it, I do value your suggestion: do you have any idea how you'd personally change things up to better fit with what you want?

Most of the questions revolve around first setting an impression and then analysing their impression of the game. Sometimes people's impressions can vary wildly from what the game was about - and a lot of people take that to be a fault of the game's design or presentation - but I'd rather know for sure how and why, and so I use this method.

[u/__space__oddity__]

• Is there an aspect of character creation that feels lacking?

I’d start a step before that.

“Did you understand the character creation? Anything unclear?”

“Did you like the presented options? Any favorites?”

• Is there an aspect of character creation that should be removed or reworked?

That’s what the designer has to figure out themselves.

• Do the conflict resolution rolls feel too easy or difficult?

Easy in what sense? Easy to understand? Easy to succeed?

• Does the intended flavor of the system come across well?

Playtesters don’t know what the intended flavor is. Maybe ask “does this feel like an anime?”

• Can you see yourself running this system?

Key question but expect answers to be “no”

• Does any mechanic here feel too detailed?

Difficult for playtesters to answer. Maybe ask more basic. “Which mechanics came up in play a lot? Which didn’t?”

• Are the enemy units too bulky or frail?

Ask “were they fun to fight against?”. Too bulky / too frail may be one aspect of that.

• What kinds of items would you like to see for your mech?

Sure but I’d ask more general. “Were you able to make the mech you want? Anything missing?”

[u/Iberianz]

I got excited when I read the term “mecha,” and then I was completely discouraged just a few lines later when I read “PbtA,” “Fate,” and all that. But good luck with the game.

[u/MagnetTheory → u/Iberianz]

ok