New to Sly Flourish? Start Here or subscribe to the newsletter.

How My D&D Surveys are Wrong and Why I Use Them Anyway

by Mike on 10 November 2017

Special thanks to Pete Watson-Wailes for helping me work through the logic and science in this post.

Recently, I've posted a number of articles that attempt to take a more data-centric look at the D&D hobby. Examples include the following:

I use these polls to step back from personal anecdotes and individual opinions and get a better understanding of how D&D DMs and players think about the game in aggregate. It's easy for us to assume everyone else sees and plays the game we do (a cognitive bias known as the false consensus effect). It's easy to think we know how everyone else thinks about the game. These polls help us get closer to figuring out if that's true or not.

Often, I receive feedback pointing out the flaws of these analyses including selection bias and potentially leading questions. These flaws are completely valid. The respondents to these polls are self-selected (they choose to take the poll) and are not chosen randomly (they come from the Facebook 5e D&D group and my own twitter followers). I'm also not a professional pollster so the questions could mislead the answers we seek. Though flawed, I still choose to conduct these polls and quote the results for three reasons:

They're still better than guessing.
The risk is low.
They're the best I can get.

They're still better than guessing. Though biased, these surveys are probably still better than just trusting our gut or taking anecdotal evidence from the handful of tweets or comments as truth. We need not ignore our own experiences. Thinking like a Bayesian helps us use our experiences as priors and then modify this prior with the results of the surveys to get us closer to the truth.

The risk is low. I'm not forming national policy or selecting a new cancer drug. I'm just trying to better understand a piece of the Dungeons & Dragons hobby. If I'm wrong (and I'm almost certainly wrong at least some of the time), it won't create a global catastrophe. Hopefully I can find out that I'm wrong with new evidence.

They're the best I can get. I can't run a 1,000 person randomly selected study of all D&D players and DMs. I only have the tools I have and access to the D&D players and DMs who choose to take the survey. That doesn't mean these surveys are any likelier to be true, but when taken as new evidence on a topic, even as flawed evidence, it can still give us a better view of whats going on.

These polls aren't perfect models of how all D&D DMs think about and play this game. To quote George Box, "all models are wrong. Some are useful". If you see flaws in my logic or in the polls I run let me know. I am always happy to be proven wrong if it gets us closer to understanding this wonderful hobby of ours.

Subscribe to Sly Flourish

Subscribe to the weekly Sly Flourish newsletter and receive a free adventure generator PDF!

Sly Flourish's Books

Share this article by copying this link: https://slyflourish.com/errors_in_dnd_data_analyses.html

Have a question or want to contact me? Check out Sly Flourish's Frequently Asked Questions.

This work is released under a Creative Commons Attribution-NonCommercial 4.0 International license. It allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, for noncommercial purposes only by including the following statement in the new work:

This work includes material taken from SlyFlourish.com by Michael E. Shea available under a Creative Commons Attribution-NonCommercial 4.0 International license.

This site uses affiliate links to Amazon and DriveThruRPG. Thanks for your support!

How My D&D Surveys are Wrong and Why I Use Them Anyway

Related Articles

Subscribe to Sly Flourish

More from Sly Flourish

Sly Flourish's Books