How My D&D Surveys are Wrong and Why I Use Them Anyway

by Mike Shea on 10 November 2017

Special thanks to Pete Watson-Wailes for helping me work through the logic and science in this post.

Recently, I've posted a number of articles that attempt to take a more data-centric look at the D&D hobby. Examples include the following:

I use these polls to step back from personal anecdotes and individual opinions and get a better understanding of how D&D DMs and players think about the game in aggregate. It's easy for us to assume everyone else sees and plays the game we do (a cognitive bias known as the false consensus effect). It's easy to think we know how everyone else thinks about the game. These polls help us get closer to figuring out if that's true or not.

Often, I receive feedback pointing out the flaws of these analyses including selection bias and potentially leading questions. These flaws are completely valid. The respondents to these polls are self-selected (they choose to take the poll) and are not chosen randomly (they come from the Facebook 5e D&D group and my own twitter followers). I'm also not a professional pollster so the questions could mislead the answers we seek. Though flawed, I still choose to conduct these polls and quote the results for three reasons:

  1. They're still better than guessing.
  2. The risk is low.
  3. They're the best I can get.

They're still better than guessing. Though biased, these surveys are probably still better than just trusting our gut or taking anecdotal evidence from the handful of tweets or comments as truth. We need not ignore our own experiences. Thinking like a Bayesian helps us use our experiences as priors and then modify this prior with the results of the surveys to get us closer to the truth.

The risk is low. I'm not forming national policy or selecting a new cancer drug. I'm just trying to better understand a piece of the Dungeons & Dragons hobby. If I'm wrong (and I'm almost certainly wrong at least some of the time), it won't create a global catastrophe. Hopefully I can find out that I'm wrong with new evidence.

They're the best I can get. I can't run a 1,000 person randomly selected study of all D&D players and DMs. I only have the tools I have and access to the D&D players and DMs who choose to take the survey. That doesn't mean these surveys are any likelier to be true, but when taken as new evidence on a topic, even as flawed evidence, it can still give us a better view of whats going on.

These polls aren't perfect models of how all D&D DMs think about and play this game. To quote George Box, "all models are wrong. Some are useful". If you see flaws in my logic or in the polls I run let me know. I am always happy to be proven wrong if it gets us closer to understanding this wonderful hobby of ours.

If you enjoyed this article, you might enjoy the Lazy Dungeon Master and Sly Flourish's Fantastic Locations. You can also support this site by using these links to purchase the D&D Starter Set, Players Handbook, Monster Manual, or Dungeon Master's Guide.

Need some dice? Check out this pack of 105 dice by Easy Roller Dice! Or how about some Flat Plastic Miniatures?

Send feedback to @slyflourish on Twitter or email