GWS 2022: On the reliability of data

By Jon Freitag

The Great Wargaming Survey (GWS) accepts voluntary responses from any participant willing to complete the survey. No random sampling mechanism is in place. Sampling is thus by convenience. Since a respondent must have knowledge of the survey, access to the questionnaire to participate, and an interest in wargaming, the final dataset is naturally constrained to this population. For each annual survey, all respondents submitting a survey (excluding those having data quality issues) are included in a cross-sectional study. Cross-sectional studies collect data from many subjects at a single point in time.

The relaxation of random sampling methods may introduce bias. Having a sample not been drawn randomly, a sample may not fully represent the population as a whole. Inferences may therefore be misleading. Understandably, the possibility of sampling bias has been a concern raised by more than one reader over the years of publishing aggregated survey results.

Is the concern for sampling bias justified? Does a sample size hovering around 10,000 annually mitigate bias? Can survey analysis provide reasonable, and useful, results in the face of potential sampling bias?

What if the same target population is sampled at regular intervals? Does repeated sampling improve confidence in the underlying method? At each interval (for the GWS the interval is a year), the survey collects a different sample of the target population. This is a repeated cross-sectional study. Repeated cross-sectional studies can be used for analyzing population changes over time even though the samples, by themselves, are not identical.

To gain confidence in the methods used and results presented, each survey ought to be drawing similar samples from the target population. To assess this hypothesis, a few of the demographic attributes are selected for the years 2016-2022.

Every year, respondents are asked if they have taken part in the GWS before. In a repeated survey, the earliest years ought to see fewer repeat respondents than in later years as the probability of having previously completed the survey increases. What is interesting is that the percentage of repeat respondents stabilizes at about 50% since 2020. About half of each sample is generated by respondents who say they have not participated in the survey before. Surprising result?

When capturing the number of years spent wargaming, besides 2016 and 2017, the percentage of respondents in each duration bin is relatively stable. Wargamers gaming 31 years or more remains locked in the upper 30% range. 

When examining respondent location over time, again, the time series remains stable.

Examining the time series by age shows an interesting, but not unexpected, result. Given that prior analyses suggest that the population of wargamers is getting older in the aggregate, samples drawn from a similar population ought to produce this tendency. The chart below exhibits the tendency of older cohorts to increase over time.

Finally, consider a respondent's primary interest in wargaming. Over time, we see a shift from Fantasy/Sci-Fi to both Mixed, and Historicals. Given the tendency for non-historical wargamers to gravitate away from preferring purely non-historical wargaming as they age, the underlying samples seem drawn from similar populations.

While GWS samples may not be drawn randomly, the hypothesis that these data are drawn from similar populations across time should not be rejected outright. Can results from several years of the GWS be relied upon for useful inferences on the population of wargamers? The answer is up to the individual. I know my position.