Although I've been thoroughly entertained by the events, and outrage of the last 20 or so hours, it's been increasingly evident for a while now that there are major issues with the playtesting for our favourite hobby, and it's become quite troubling. Other, more known and respected members of the community have raised these concerns in the past (MajorJuggler, Sozin, the Carolina Krayts podcast, as well as others), so I won't re-hash all the gory details here, but wanted to give my observations.

The current method of playtesting is completely broken. Then, yesterday someone looked at the broken pieces on the ground, and bashed them with a hammer. Too many clearly over-powered elements have been introduced to the game in the last few waves. The entire jumpmaster expansion being the biggest offender - until wave 11... This has happened for one of three four reasons:

1) the playtesters are missing very obvious broken combos

2) the playtesters are finding these combos, and not reporting them

3) the playtesters are finding and reporting these combos, but FFG is not taking their advice

4) the combos are spotted and FFG responds, but changes are made at the last minute and not playtested before publication. - brought up by @thespaceinvader

It really doesn't matter which scenario is happening, because they all point to a system that is/was broken. Now, FFG is likely not very trusting of their current pool of playtesters, and quite frankly, they shouldn't be. I've seen dozens of people express outrage about this leak for a variety of reasons, but to me, the most troubling are the humble brags.

If you're not sure what I mean, it looks like this: "I'm so mad this guy broke the trust of FFG! I knew about this a while ago, but didn't go spreading it around." -guy who is impressed with himself for being in-the-know

There have been so many of these that it's clearly obvious there is a lot more than just one guy breaking his NDA. There are multiple playtesters spreading this stuff around. We should be happy that this only happened for an FAQ rather than a ship with episode 8 spoilers.

Something needs to change. I love what Sozin suggested, making playtesting an open, community driven project, where you would need to provide data to back up your findings. The only other option is hiring people to do the job, but I don't get the impression that FFG is willing to pony up any extra cash to have people do a proper job of it.

Anyways, these are my early morning, can't get back to sleep thoughts on the matter. I'm hoping we can have an insightful conversation about this.

Lord of the Rings TMG had the same idea of an open community of forum play testing for their second expansion. The one of the results was one of the play testers introduced a power creep into one of the pieces. Having an open community of play testers can be a peculiar situation for a variety of factors due to a play testers gaming ability/understanding of the game, gaming community/environment in their area, etc.

I think it's a pretty tricky situation with play testers whether it's an open community or a controlled group of players.

@Crit Happens

Playtesters and designers are distinct groups. A playtester should not be designing any cards - unless you win the right to do so...

Just now, Crit Happens said:

Playtesters and designers are distinct groups. A playtesters should not be designing any cards - unless you win the right to do so...

You'd be shocked at how much this does happen, at least if sources are to be believed.


Another thing I'll bring up is a quote I saw the other day. Playtesters know the new ships and FAQ early. They inevitably leak and share with their friends. "The only people that don't know the playtest material are the ones without friends." That mentality has got to go.

4 minutes ago, Crit Happens said:

Playtesters and designers are distinct groups. A playtester should not be designing any cards - unless you win the right to do so...

Play testers provide feedback to designers on what works and doesn't. The designers makes changes based on their feedback whether they take their suggestions or not.

13 minutes ago, hey_yu said:

Play testers provide feedback to designers on what works and doesn't. The designers makes changes based on their feedback whether they take their suggestions or not.

I get that, but the "many eyes find more bugs" scenario with an open system should help the playtesters keep each other in check.

16 minutes ago, Kelvan said:

Another thing I'll bring up is a quote I saw the other day. Playtesters know the new ships and FAQ early. They inevitably leak and share with their friends. "The only people that don't know the playtest material are the ones without friends." That mentality has got to go.

This is the one that bothers me the most in the aftermath of these leaks...there seems to be an 'underground' network of people 'in the know'.

1 minute ago, Crit Happens said:

I get that, but the "many eyes find more bugs" scenario with an open system should help the playtesters keep each other in check.

I understand that too. There are trade-offs from having a controlled group of play testers versus an open community. One of the draw backs with an open community of play testers you really don't know the person's background and gaming environment for their feedback. In addition, the designers need to peruse thru a boot load of feedback with potential differing opinions. There's draw backs to a controlled group as well. I'm just saying an open community of play testers is not necessarily the solution.

3 minutes ago, LordBlades said:

This is the one that bothers me the most in the aftermath of these leaks...there seems to be an 'underground' network of people 'in the know'.

It's always interesting to note who suddenly drops a list that's dominating to play something unusual.

"you're pairing Miranda with that, right?"

"no"

"oh right any reason? Surely Miranda is the best thing there

"...nope, no reason... 'wink'..."

I never knew until the last 24 hours just how many people "knew" what was coming or had friends who told them that...

Man, I must be an island onto myself.

Personally, I like the idea of open source based community play testing. Codename the secret stuff we can't know about per the Mouse, and just let us fly proxy of the new things. Don't allow it in organized play events just in standard xwing games where we report back on the jank we caused to be broken, etc.

It's not like we don't already do that with things that they spoil in their preview articles...

8 minutes ago, Stay On The Leader said:

It's always interesting to note who suddenly drops a list that's dominating to play something unusual.

"you're pairing Miranda with that, right?"

"no"

"oh right any reason? Surely Miranda is the best thing there

"...nope, no reason... 'wink'..."

The same can be said about developing a highly synergistic list on release day. Doubly so if it includes unspoiled/freshly spoiled content. Maybe you had a stroke of genius overnight. Or maybe you had a few extra months to think about it.

Yeah, this issue has popped up a few times, like when they dropped the deadeye nerf FAQ a couple weeks before worlds. Brutal timing for those not in the know, and had been practicing with/against deadeye scouts for months. For the playtesters... how long did they know that they didn't have to worry about it? That's a huge disadvantage for people that are spending a ton of money in travel, hotel, and vacation days to go play in an event that should be an equal playing field.

How can we get them to stop this "playtesting"?

I wonder if some well thought out letters to the game company can convince anyone to stop this nonsense of " playtesting".

There is no "playtesting" going on. There is " me and my buddies now have the inside scoop on the future" going on.

The first clue that the entire playtesting situation is a scam is when you have the playtesters telling you how important and necessary it is.

You missed option 4: the combos are spooted and FFG responds, but changes are made at the last minute and not playtested before publication. Rumour has it that's how the Scout came by its EPT.

2 minutes ago, thespaceinvader said:

You missed option 4: the combos are spooted and FFG responds, but changes are made at the last minute and not playtested before publication. Rumour has it that's how the Scout came by its EPT.

yes, thank you! I will add to the post.

I feel for you. When I brought something like this up about Nym and powercreep. People went nuts. God Bless you sir.

4 minutes ago, Velvetelvis said:

How can we get them to stop this "playtesting"?

I wonder if some well thought out letters to the game company can convince anyone to stop this nonsense of " playtesting".

There is no "playtesting" going on. There is " me and my buddies now have the inside scoop on the future" going on.

The first clue that the entire playtesting situation is a scam is when you have the playtesters telling you how important and necessary it is.

Certainly seems that way. Perhaps it is time for FFG to clean house with playtesters and get new people? Obviously there are a bunch of playtesters that don't understand the whole NDA thing going on.

I agree with a lot of those points, but I can't imagine a scenario where this event makes it better though. Further restricting play-testers or the number of playtesters is the likely outcome of this kerfuffle. And it should be pretty obvious that'll only make balance issues more likely to slip through.

1 minute ago, Braxton said:

Certainly seems that way. Perhaps it is time for FFG to clean house with playtesters and get new people? Obviously there are a bunch of playtesters that don't understand the whole NDA thing going on.

Cleaning house, and re-hashing the same system does not get rid of the problem. There will be others who violate their NDA. The system itself needs to be changed.

9 minutes ago, TasteTheRainbow said:

I agree with a lot of those points, but I can't imagine a scenario where this event makes it better though. Further restricting play-testers or the number of playtesters is the likely outcome of this kerfuffle. And it should be pretty obvious that'll only make balance issues more likely to slip through.

Then let them slip through....to everyone at the same time.instead of just the buddy buddy crew.

8 minutes ago, Crit Happens said:

Cleaning house, and re-hashing the same system does not get rid of the problem. There will be others who violate their NDA. The system itself needs to be changed.

Won't happen. This is how FFG do all their games and tbh they don't care enough about most of these issues to overhaul their whole company design ethos to suit a few people who take one of their many games a bit too seriously.

20 minutes ago, TasteTheRainbow said:

I agree with a lot of those points, but I can't imagine a scenario where this event makes it better though. Further restricting play-testers or the number of playtesters is the likely outcome of this kerfuffle. And it should be pretty obvious that'll only make balance issues more likely to slip through.

I agree. There is a stronger possibility of a negative result of all of this than a positive. Some seem to be worried that the house of mouse may even get their gloved fingers involved in this mess...

Just now, Stay On The Leader said:

Won't happen. This is how FFG do all their games and tbh they don't care enough about most of these issues to overhaul their whole company design ethos to suit a few people who take one of their many games a bit too seriously.

I completely agree, but wish it wasn't the case... It's also the reason they will never release a set of printed errata cards, and why they wont send out new CROC bases that are properly printed. It's all about money, which is understandable because they are in the business of making money. They SHOULD do what makes them the most profit.

That doesn't mean the problem shouldn't be addressed though.

17 minutes ago, Velvetelvis said:

Then let them slip through....to everyone at the same time.instead of just the buddy buddy crew.

This sort of paranoid nonsense isn't useful. Very, very few games are getting decided because a playtester knew the rough idea of a mechanic a few months in advance.