New to Sly Flourish? Start Here or subscribe to the newsletter.

2016 Dungeons & Dragons Dungeon Master Survey Results

by Mike on 19 December 2016

On 28 October 2016 I released a survey for 5th edition Dungeons & Dragons Dungeon Masters to help us all get a better understanding of how DMs prepare for and run their D&D games.

This article analyzes the results of the survey and discusses the methods I used to conduct and analyze the results. I analyzed the results in a bunch of different ways including standard result counts for each question and more advanced (and not necessarily accurate) methods such as result correlations and text processing of the questions regarding tools and DM tips.

I'm going to avoid jumping to any conclusions from these results. When we look at these results, it is easy for us to want to leap to answer the question of "why". We're not going to do that here. This is just the facts, ma'am.

If you want to explore the data yourself, you can download the original survey results or a cleaned up version of the results both in CSV format. You can also examine the code used to conduct the analysis and all of the various ways I sliced the data at the 2016 D&D DM Survey Github site.

This is a huge article, weighing in at about 5,000 words, so here's a table of contents to get you to the stuff you're most interested in:

Primary Results
The Locations People Play
Survey Response Correlations
Top Tools DMs Use
Favorite Tips
Favorite Tips Text Clusters
Favorite Tips Word Graph
How I Analyzed the Results
Potential Survey Flaws
Explore the Data Yourself
A Data-Driven View of Our Hobby

The Primary Results

Once I removed duplicate entries and trimmed the data to the pre-determined date range (more on this in our methods), there were a total of 6,600 results from 28 October to 28 November 2016.

The three charts below show the results of the survey for frequency of sessions, length of sessions, and overall preparation time.

Below are the results for other categorical questions of the survey:

Of 6,600 respondents on primary locations played, 55% answered home, 16% answered Roll20, 14% answered another private location, 5% answered local game shop, 4% answered another public location, 2% answered another online tool, 2% answered equal mix, and 1% answered Fantasy Grounds.

Of 6,600 respondents on primary campaign worlds used, 55% answered personal setting, 38% answered Forgotten Realms, 5% answered another D&D campaign world, and 2% answered non-D&D campaign world.

Of 6,600 respondents on primary adventures used, 64% answered personal adventures and 36% answered published adventures.

Of 6,600 respondents on preferred combat type, 63% answered 5' gridded combat, 19% answered abstract maps, and 18% answered theater of the mind.

The chart below shows the results of the 6,600 responses for various preparation activities. They are generally ordered from the activities most often using the most time to the activities most often using the least time.

The Locations People Play

One of the survey questions let respondents select all of the locations they play along with the question on their primary location, noted above.

Of 9,711 total locations selected, 48% of the selections were for home, 19% were for another private location, 17% were for Roll20, 6% were for a local game shop, 5% were for another public location, 4% were for another online tool, and 1% were for fantasy grounds.

We can also examine at the results of this question by looking at the combinations of locations selected by respondents. The table below shows the top 10 combinations of locations selected.

Location Combination	Percentage of 6,600 results
Home	36%
Home, another private location	14%
Online using Roll20	10%
Home, online using Roll20	8%
Another private location	7%
Home, a local game shop	3%
A local game shop	3%
Another public location	2%
Home, online using another tool	2%
Home, another public location	2%
Home, another private location, online using Roll20	2%

This table covers the top 87% of responses to this question. The remaining 13% were spread across the remaining combinations with less than 2% for any single combination of results so I didn't bother adding them to the table.

Survey Response Correlations

Beyond raw results, we can also examine how the responses of some questions correlate to the responses of other questions. Are DMs who run games less often more likely to take longer preparing for their games? Are DMs who primarily run games on Roll20 more likely to run shorter games? We can investigate these correlations using a technique called categorical chi-squared testing, which helps us mathematically figure out potential correlations by matching all of the actual results for any pair of questions with the expected result based on a typical distribution.

The result is a series of graphs that show when some results are more or less likely than the expected distribution. Here are a few example charts.

Note that large dark red dots indicate a high negative correlation between those two responses (respondents were less likely to select one of those responses if they had selected the other). A large blue dot represents a high positive correlation between those two responses (respondents were more likely to select one response if they had selected the other. The size and intensity of the dots indicates the difference from the expected norm. The larger and darker the dot, the greater the residual difference from zero and the stronger the correlation.

In the above chart you can see a strong positive correlation between running games at a local game shop and spending 15 minutes preparing for a session. We have a slightly smaller negative correlation between spending about 15 minutes on preparation and running games on Roll20.

In the above plot, you can see a strong positive correlation between preparing for four hours or more and running games monthly or less than monthly.

In the above plot we can see strong positive correlations with running published adventures in published campaign worlds and the same for personal adventures and personal campaign worlds. We see strong negative correlations for the opposite. This chart is a good benchmark of what we would expect to see. People who play in the Forgotten Realms are more likely to play published adventures while those who run homebrew campaign worlds are more likely to play their own adventures. Makes sense.

Below is a full list of correlation plots for each combination of question results. I removed combinations that had a p-value lower than .02, essentially filtering out results had a high chance for inaccuracy.

It is also important to note that most of these strong correlations appear for results with fewer answers. This indicates that answers with a greater number of responses, such as running 4 hour sessions, running sessions weekly, or running games at home, rarely have a strong correlation to any other response since those answers were selected by the most people regardless of the rest of their other answers.

Top Tools Used

This survey included two open text fields that respondents could fill in with their top tools and their favorite tip for running a great game. Because these were open fields, our count of top tools isn't as accurate as the survey results above. Take the numbers below as an estimate instead of an exact figure.

Here are the results for the top 30 tools chosen by 4,164 survey respondents.

Tool	Respondents
Kobold Fight Club	569
Dungeon Master's Guide	548
Monster Manual	471
Donjon	449
Roll20	374
Onenote	268
Reddit	209
Players Handbook	169
Google Docs	135
paper	104
Photoshop	82
Google Drive	80
books	78
notebook	76
Google	71
Evernote	70
core rulebooks	67
pencil	67
pen	66
dice	62
Excel	59
pen and paper	56
index cards	48
graph paper	47
Fantasy Grounds	43
laptop	39
DM screen	38
miniatures	37
Inkarnate	36
maps	36

Favorite Tips

Grouping up top tips is much harder than grouping up tools, which were much easier to normalize around commonly recommended tools. I used a bunch of different natural language processing techniques to attempt to summarize the 4,046 tips people submitted.

First, we can do a simple word aggregation. We start by processing the text by removing common words (called "stopwords"), lower-casing the text, and removing punctuation. Next, we "lemmatize" words to group them into their root form (turning "players" into "player" and "stories" into "story"). Finally, we can count up all of the words used in all of the tips and take a look at the top 50 words with counts in parens:

player (2044), story (522), character (461), game (425), npc (410), dont (350), fun (278), thing (271), session (254), keep (244), time (233), encounter (217), will (205), pc (199), combat (198), play (175), idea (173), good (163), sure (153), adventure (148), rule (142), feel (138), roll (137), lot (135), party (131), making (130), action (129), plan (127), plot (127), dm (123), work (120), campaign (119), great (112), improv (110), going (101), create (99), group (93), improvise (93), choice (90), help (89), trick (86), monster (85), prep (84), build (83), find (83), happen (80), music (80), letting (80), table (79), prepare (79).

That's not particular useful, obviously. We see common words but we don't know the context at all. We can step this up by taking a look at top "bi-grams" or pairs of words once we removed stopwords. Here are the top fifty bi-grams for the 4,046 tips:

player character (54), player will (51), letting player (49), listen player (42), player story (39), making sure (38), player feel (32), player game (32), player dont (31), sure player (30), keep player (29), player action (29), player idea (28), game player (27), dont afraid (27), player fun (27), player choice (26), allow player (26), story player (25), keep thing (25), combat encounter (25), role play (22), character story (21), knowing player (19), player agency (19), player decide (19), random encounter (18), player play (18), player find (18), fun player (17), player thing (17), thing player (17), rule cool (17), listening player (17), drive story (16), ahead time (16), magic item (16), player going (15), plot point (15), npc player (15), player describe (15), plot twist (15), role playing (14), side quest (14), memorable npc (14), player control (14), player engaged (14), plot hook (14), spend time (14), player decision (13).

This is somewhat more interesting, we can start to get an idea what we're talking about. Note that the number of results drops significantly compared to single words, limiting the potential significance. Still, we can go one more level with "tri-grams" which, as you might have guessed, is three words together. Here are the top 50 tri-grams:

keep thing moving (13), making sure player (11), player drive story (11), sure player fun (6), pay attention player (6), making sure fun (6), player character story (5), allow player freedom (5), player guide story (5), spend lot time (5), player write story (5), random encounter table (4), keep flow going (4), player develop story (4), letting player story (4), listen player idea (4), keep thing interesting (4), fudge dice roll (4), dont sweat small (4), great dd game (4), sweat small stuff (4), keep player engaged (4), player feel character (4), player feel awesome (4), keep action moving (4), game player play (4), player role play (4), dm screen initiative (3), idea session will (3), solution letting player (3), flying seat pant (3), sure player engaged (3), letting player decide (3), pop culture reference (3), player heavy lifting (3), player decision matter (3), list name npc (3), music help set (3), change thing fly (3), dont afraid change (3), dont rule lawyer (3), great game player (3), character moment shine (3), open communication player (3), side quest ready (3), skill check player (3), list npc name (3), character dont afraid (3), player moment shine (3), consequence player action (3)

The phrases get more meaningful but the counts go way down making them less useful for understand at a high-level what we're looking at.

Favorite Tips Text Clustering

There's another natural language technique for analyzing big piles of text called "text clustering". You do this when you have a lot of text and don't really know what's in it. This technique, called K-means clustering, attempts to group up tips into a number of clusters and then tells you what the top words were that defined each cluster.

I'll warn you, this is a bit like reading tea leaves. We have to sort of squint to see the results and the results are still a bit fuzzy. Still, it brings up some interesting terms.

Here's the result when I attempted to bin all 4,153 tips into eight clusters and list out the 100 significant terms per cluster:

Cluster 0 (427 tips): game, session, use, player, like, rule, thing, time, card, initiative, combat, make, cool, world, npc, run, story, dont, dm, idea, start, encounter, know, way, think, help, prep, pc, try, rule cool, best, play, lot, want, note, monster, people, adventure, create, cliffhanger, feel, trick, end, possible, just, campaign, track, work, need, great, inspiration, build, roll, use player, music, good, little, map, book, ha, really, player game, table, ive, come, plan, allow, game player, ask, outline, character, running, end session, set, moment, point, im, stuff, background, let, having, prepared, want game, write, event, important, great game, previous, remember, prepare, scene, using, small, flow, going, run game, start session, make game, different, player session

Cluster 1 (742 tips): player, want, story, player want, listen, world, idea, letting, listen player, letting player, action, know, play, try, choice, work, know player, roll, thing, listening, game, npc, listening player, dont, ask, think, time, build, player idea, come, adventure, freedom, like, possible, open, knowing, way, campaign, knowing player, allow, making, ask player, building, good, player action, encounter, tell, going, giving, great, player choice, engage, setting, run, plan, moment, just, invested, agency, scene, dm, expect, getting, better, consequence, work player, question, path, interact, enjoy, need, player agency, having, allowing, improvising, create, flow, allow player, really, giving player, getting player, using, player come, twist, lot, develop, good player, roll player, react, learn, plot, option, player story, player try, player think, mind, allowing player, control, prep, remember

Cluster 2 (316 tips): make, sure, make sure, player, make player, player make, npc, feel, world, game, story, sure player, like, dont, make world, thing, try, making, combat, make sure player, try make, pc, time, just, make game, great, think, making sure, npc make, choice, want, ha, fly, interesting, real, know, party, good, campaign, decision, player feel, going, matter, engaged, plan, make feel, cool, action, just make, make interesting, enemy, idea, character, enjoy, really, moment, fun, session, work, make story, make pc, feel like, situation, way, make sense, make combat, hard, sure ha, theyre, make npc, encounter, having, making sure player, prepared, sense, setting, make player feel, better, change, fight, play, kill, solution, consequence, awesome, motivation, scene, shine, help, note, alive, let, happen, voice, run, group, playing, people, roll, important

Cluster 3 (295 tips): let, let player, player, story, want, world, dont, try, just, let player want, thing, adventure, idea, game, player want, rule, fun, let player run, run, let player try, player run, decide, think, guide, player try, dont let, dictate, world let, control, build, say, action, drive, play, let player decide, make, create, let story, plot, let player drive, player decide, player drive, tell, lead, let player guide, drive story, talk, let player drive story, player guide, player drive story, player let, flow, plan, player tell, fun let player, know, let player tell, let player play, npc, fun let, let player dictate, work, follow, player dictate, let player tell story, player play, possible, tell story, run game, player control, let player talk, player tell story, let player lead, succeed, develop, let player make, let try, player lead, world let player, roll, let player think, let player run game, improvise, player run game, character, like, player talk, explore, happen, player create, let player control, good, pace, bogged, build world, use, solution, write, story just, campaign

Cluster 4 (307 tips): character, player, player character, story, npc, world, make, game, plot, play, like, background, character story, getting, making, encounter, voice, think, feel, backstory, way, ask, personal, know, create, character background, combat, help, character backstory, session, invested, let, group, choice, thing, want, build, campaign, just, lot, goal, make character, using, try, include, hook, specific, giving, having, shine, develop, people, dont, character feel, information, getting character, npc character, good, dm, action, adventure, situation, scene, encourage, interaction, really, know character, moment, sheet, character dont, point, pc, main, question, interesting, use, letting character, let character, possible, event, tell, tie, ask player, involved, party, story character, letting, character voice, space, arc, character npc, motivation, development, care, ha, character make, like character, doing, important, stay

Cluster 5 (1689 tips): npc, dont, pc, music, just, improvise, improv, combat, good, encounter, story, voice, prepare, time, thing, party, know, lot, prepared, improvisation, roll, world, random, trick, great, preparation, note, monster, player, try, using, plan, flexible, plot, adventure, having, making, way, like, description, table, ready, group, prep, flow, play, dm, going, idea, set, fly, steal, map, really, playing, memorable, moving, campaign, unexpected, wing, work, interesting, keeping, twist, enemy, humor, setting, situation, happen, planned, say, mind, skill, rule, mood, dont know, list, alcohol, create, choice, background, improvising, im, effect, scene, look, friend, open, people, need, dungeon, moment, long, dice, funny, think, location, possible, hand, d

Cluster 6 (113 tips): yes, say, say yes, yes player, say yes player, saying yes, player, saying, yes yes, improv, improvisation, try say yes, try, reason, try say, possible, player want, rule, phrase, want, time, dm, npc, consequence, working, way, work, instead, storytelling, lot, thing, let, play, doesnt, happens, let player, creative, sense, player choice, set, flow, pc, accent, having, improvisational, generally, best, choice, relax, prepare, difficulty, problem, adaptable, stupid, listen, style, make, happen, wing, story, remembering, id, realistic, audience, fun, appropriate, thing player, think, aka, thats, game play, imagination, stay, matter, make work, answer, reward, let player want, let player dictate, exciting, environment player, energy, player dictate, bad, knowing, player action, player control, shape, game, drama, old, plan, flesh, fast, just try, amazing, heavy, rule cool, imagine, dictate

Cluster 7 (157 tips): fun, having fun, having, rule, player, game, sure, just, making, making sure, make, fun player, player fun, player having fun, dont, make fun, player having, let, dm, sure having fun, sure having, try, focus, group, story, remember, want, fun dont, just fun, npc, youre, improv, loose, letting, like, play, end, know player, sure player, make sure, plan, everybody, flexible, providing, pc, doing, matter, just try, fun npc, encounter, important, make sure having, run, cool, know, rule cool, isnt, lot, ha, way, lead, stuff, plot, say, dont let, making sure player, interesting, ask, adventure, forget, outcome, thing, meant, inspiration, bend, roll, consequence, happen, working, job, try make, asking, adapt, dd, choose, long, roleplay, come, improvisation, having player, time, game fun, great, gm, want play, think, dont sweat, preparation, friend, thats

If we squint and swirl around these words in our teacup, we can start to see a few general trends.

Cluster 0 is a big mix of stuff. No clear trend there that I can see.

Cluster 1 we might call the focus on the player cluster.

Cluster 2 is all about the words make things happen; "make", "making", and "make sure".

Cluster 3 is sort of the opposite, it's the let things happen cluster; let things happen. Let the players do things. Let things go as they are. "Make" is highly active. "Let" is much more passive. Interesting.

Cluster 4 is the focus on character background and story cluster.

Cluster 5 are the party theater people. Lots about improvise, voice, stealing, and alcohol.

Cluster 6 is the say yes cluster.

Cluster 7 is our have fun cluster.

So, given our tea-leaf reading, here are the big tips we might coalesce around:

Focus on the players.
Make things happen.
Let things happen.
Focus on characters, backgrounds, and stories.
Improvise.
Say yes.
Have fun.

Not a bad set of tips if you ask me.

Favorite Tips in a Word Graph

We have one final way we can look at submitted tips, in what is called a "word graph". In this case, we take those bi-grams we created up above and use them to map relationships between every pair of words. This is known as a graph in which you have two nodes (the words) connected by an "edge" (the fact that those two words showed up as a pair). We can then visualize all of the relationships of the words. Showing every pair of words would make the visualization too dense, so instead we only show word pairs that have occurred at least five times. Here is the resulting graph. Click on it to load a much larger image and explore it a bit.

This gives us a lot of interesting information and it beats "word clouds" all to hell. Now we can see what words connect to the word "dont" and all of the words related to "player". The little disconnected word pairs around the outer edge are also interesting.

We'll look at another graph in which I removed the word "player" since it so heavily dominates the graph, and focus in on the central cluster, getting rid of those extra word pairs around the edge. Take a look.

This gives us other interesting insights into the tips that people submitted.

Of course, if we want to read over 4,000 tips, we can just go look at all the tips and see for ourselves what's in them.

When Did People Respond?

I announced the survey on 28 October and each response included a datestamp for the time the survey was submitted. I wanted to analyze this datestamp in order to see the overall trends of posting. The chart below shows every survey response by date and by time. The bands have been "jittered" randomly so that they don't all pile up on one another. Here's the result.

This chart shows the the high number of responses after I advertised and re-advertised the survey on five social media platforms: Enworld, Reddit, Facebook, Google Plus, and Twitter.

Note that these responses in this chart haven't been de-duplicated yet. Overall I identified about 120 duplicate responses in which every field in the response, including non-blank text fields, were identical to another. I left in 44 potential duplicate responses in which every result was the same but had blank text fields since it was possible multiple people submitted the exact same results and chose not to fill out the text field. Given the small number (0.6%) of potential duplicates with blank text fields, it was unlikely to affect the results one way or the other.

How I Analyzed the Results

For this survey I used a host of scripts to run the survey and analyze the data. I hosted the survey itself using the "form" from Google Drive. It scaled well for the number of responses and dumped out the results into a handy comma separated file (known as a CSV file). The survey form was fast to load, easy to fill out, and worked well on mobile devices.

During the survey I wrote a big pile of scripts in both Python and R to analyze the data. I also used good old Microsoft Excel (the only Microsoft product I like) to double-check the results to ensure I didn't screw something up in the Python and R code.

The R and Python scripts run through a bunch of different jobs including the following:

Cleaning up the data by converting column names and response values into values more manageable by the software and visualizations. For example, I changed "How often do you run 5th Edition Dungeons and Dragons games?" to "Frequency of Games" and "I prefer to run combat with maps, miniatures, and five foot per square grids." to "5' gridded combat".
General data cleanup (turning all results into strings, formatting dates, that sort of stuff).
Eliminating results outside of the survey range (7pm on the 28th of October to 7pm on the 28th of November)
Sorting and prepping the data to build plots.
Building the actual plots.
Generating the statistical sentences of the primary results.
Analyzing the multiple locations selection question using Excel.
Running a series of Chi-squared tests on the survey results to identify correlations on results for pairs of questions and graphed them in R.
Generating a plot of dates and times of the survey responses to check for weird anomalies.
Removing duplicate entries where every field matched more than once including non-blank text fields. 44 (0.6%) potential duplicate entries that had blank text fields were retained.
Building a different output for the "locations played" multiple potential choices per answer. I used Excel for this.
Running a bunch of Python scripts for natural language processing including general text processing, lemmatization, word counts, bi-gram counts, and tri-gram counts.
Running a Python script to build word-vectors of the tips, build a tf-idf of the word vectors, and run a k-means clustering algorithm to generate clusters of tips.
Running an R script to generate a word-graph from a big bi-gram list of tips.

All of the scripts and data are hosted on the 2016 DM Survey Github site.

Potential Survey Flaws

Survey data like this is almost always inherently flawed. Instead of using it to build what we think is a perfect model of a typical D&D DM, we can use these results to get a general feeling for how DMs prepare and run their games.

In particular, I chose to use categorical groups for answers rather than a wider variety of choices (such as number of hours, whatever they be). This was to aid in the overall aggregation of results, avoid wild variances in potential responses, and to make the survey easier to answer. I also chose to ask mostly questions that required a single answer. I suspected that, had I offered a lot of "all of the above" answers, I wouldn't get a good gauge of the differences between styles and activities.

I posted the request to take the survey to a number of different online communities including the Dungeons & Dragons Google Plus community, the Reddit D&D Next community, on the ENWorld forum for D&D, the Facebook D&D community, and on Twitter. From the links above you can see the comments and criticism people left about the survey, all of which is worth considering when we look at these results.

Many DMs who commented on these forums stated that they often front-load their prep time and don't have a specific amount of time between sessions. They might spend many hours preparing for a big campaign and then spend only a little bit of time between sessions on preparation. The survey does not account for this and it's not clear how such a DM would respond to the question of time per week.

Though I requested participation in this survey across many online communities, there is still a high likelihood of a sample bias which is true for just about any online survey. Given the avenues of the survey's reach, the respondents only make up a small and particular portion of DMs who actually run.

We should consider all of these flaws when we look at the results. That said, I, for one, think that these flaws do not negate the results of the survey. Given the high number of respondents, I'm willing to bet it is a good general model of how DMs prepare and run their games.

Explore the Data Yourself

As part of this analysis I built a web-based DM Survey Explorer tool that lets us all explore the data and filter it by various responses. Give it a shot. Beware of p-hacking, however, by filtering the data until you get whatever response you want to see. As data visualization sage Edward Tufte states, if you beat your data hard enough, it will tell you whatever you want. Further, the more you filter the data, the fewer results you'll be looking at and the less statistically significant the results will be.

If you want to explore the raw data yourself, you can download the original 2016 D&D DM Survey CSV file or the cleaned up 2016 D&D DM Survey response CSV file and explore it with a tool like Excel. The cleaned-up file removes the timestamps, gets rid of duplicates, and renames columns and answers to make it a bit more manageable.

All of the code used to explore the data and build the plots is available on Github.

I am releasing the data and software under a Creative Commons Attribution Non-Commercial license. You are free to download, analyze, explore, and post your results as long as you link back to this article, reference Mike Shea at Sly Flourish, and do so without commercializing the results, the data, or the code.

A Data-Driven View of Our Hobby

My hope in conducting this survey and exploring the results was to gain a more accurate view of how D&D DMs prepare for and run their D&D games. I'm glad to see the results, even if they are somewhat unsurprising. The wider the view we can get of our hobby, the more we can understand each other and ourselves. Look for more articles diving deeper into these results in the future. I hope you've enjoyed looking through the results as much as I enjoyed working with them.

I want to give special thanks to Emily Dresner, James Introcaso, Jacky Leung, Benjamin Reinhart, and Mike Schiller for their great help in reviewing and editing this article before it's release.