New to Sly Flourish? Start Here or subscribe to the newsletter.
How My D&D Surveys are Wrong and Why I Use Them Anyway
by Mike on 10 November 2017
Special thanks to Pete Watson-Wailes for helping me work through the logic and science in this post.
Recently, I've posted a number of articles that attempt to take a more data-centric look at the D&D hobby. Examples include the following:
- 2016 D&D DM Survey
- D&D Facebook Surveys
- Analysis of the #dnd Twitter Community
- Top Tips from a Sample of 4,000 Dungeon Masters
- The Top Traits of a Good Dungeon Master
I use these polls to step back from personal anecdotes and individual opinions and get a better understanding of how D&D DMs and players think about the game in aggregate. It's easy for us to assume everyone else sees and plays the game we do (a cognitive bias known as the false consensus effect). It's easy to think we know how everyone else thinks about the game. These polls help us get closer to figuring out if that's true or not.
Often, I receive feedback pointing out the flaws of these analyses including selection bias and potentially leading questions. These flaws are completely valid. The respondents to these polls are self-selected (they choose to take the poll) and are not chosen randomly (they come from the Facebook 5e D&D group and my own twitter followers). I'm also not a professional pollster so the questions could mislead the answers we seek. Though flawed, I still choose to conduct these polls and quote the results for three reasons:
- They're still better than guessing.
- The risk is low.
- They're the best I can get.
They're still better than guessing. Though biased, these surveys are probably still better than just trusting our gut or taking anecdotal evidence from the handful of tweets or comments as truth. We need not ignore our own experiences. Thinking like a Bayesian helps us use our experiences as priors and then modify this prior with the results of the surveys to get us closer to the truth.
The risk is low. I'm not forming national policy or selecting a new cancer drug. I'm just trying to better understand a piece of the Dungeons & Dragons hobby. If I'm wrong (and I'm almost certainly wrong at least some of the time), it won't create a global catastrophe. Hopefully I can find out that I'm wrong with new evidence.
They're the best I can get. I can't run a 1,000 person randomly selected study of all D&D players and DMs. I only have the tools I have and access to the D&D players and DMs who choose to take the survey. That doesn't mean these surveys are any likelier to be true, but when taken as new evidence on a topic, even as flawed evidence, it can still give us a better view of whats going on.
These polls aren't perfect models of how all D&D DMs think about and play this game. To quote George Box, "all models are wrong. Some are useful". If you see flaws in my logic or in the polls I run let me know. I am always happy to be proven wrong if it gets us closer to understanding this wonderful hobby of ours.
Related Articles
- Dungeons & Dragons Twitter, YouTube, and Facebook Poll Results
- 2016 Dungeons & Dragons Dungeon Master Survey Results
- Top Tips from a Sample of 4,000 Dungeon Masters
Subscribe to the Newsletter
Subscribe to the weekly Sly Flourish newsletter and receive a free adventure generator PDF!
