Fred StutzmanThoughts about information, social networks, and privacy

In the past few months, we’ve seen a number of studies of dubious methodology make sweeping generalization about Twitter.  Examples include the Twitter gender study; another study asserted that because only one out of every five teens tweet, that teens don’t use the service (wouldn’t you like one out of every five teens to use your product?).   Nielsen joins this discussion by stating that “Teens Don’t Tweet”, based on  findings from their online panel.  They assert that “In June 2009, only 16 percent of Twitter.com website users were under the age of 25. Bear in mind persons under 25 make up nearly one quarter of the active US Internet universe, which means that Twitter.com effectively under-indexes on the youth market by 36 percent.”  Oh noes!

As these data will undoubtedly be reported breathlessly elsewhere, I thought it might be useful to step back and explore some of the issues with the methodology and conclusions.  So first, a note about the methodology.  The Nielsen NetView panel contains an impressive 250,000 users.  Metering software located on client machines records the websites visited by panel members.  A vast majority of the panel is recruited online; the panel is “calibrated” (weighted) against gold-standard sampling methods (Random Digit Dialing, etc.).

Survey weighting is a standard, fairly uncontroversial process.  It is commonly used and is thought of as preferable to census-type approaches that may systematically under-represent some populations.  However, reliable survey weighting gets tricky when the population is small.  Since teens are a notoriously hard-to-reach population, we generally see inflated standard errors around weighted teen respondents in a population survey.  Nielsen does not report standard errors, and the makeup of their panel is confidential, so therefore it is impossible to know how much error there is around the estimate of use.  If the panel is like other panels, though, there may be more error in young people than a high-response population, such as adults.  We’re very familiar with margins of error (the things you see in political polls, where error is reported as plus or minus 3 percent, etc).  An inflated error means the margin is larger, meaning that the estimate may vary by a larger amount.

This is not to put down Nielsen.  With 250,000 members, the panel likely has good coverage of young people.  Since my purpose is to use this example to critique web panels, we must point out two other issues.  First, bigger is not necessarily better if the sample is convenience driven.  Nielsen’s panel is very large, but simply because it is large doesn’t mean it is representative.  In fact, Nielsen is likely more interested in the larger size for better sparse-market coverage, as opposed to statistical reliability.  Second, the particular nature of recruitment into the main panel introduces selection bias.  If people aren’t selected randomly, then there may be characteristics of the population that covary with the variables of interest.  This is an omni-present issue with polling, but it must be noted.

So when we read a web poll of this particular nature, what are the critical questions we should be asking?  First, we should be concerned about cell size (the number of respondents) for a hard-to-reach population.  If young users are underrepresented, the standard errors on the estimates can be quite large (which may push an estimate around by +/- 10 percent).  We should also question the method of recruitment; if the majority of the panel comes in via the web, then who gets left out?  Since this poll is designed to represent online users, it is seems likely that heavy web users are participants (my guess).  But what if Twitter users actually aren’t like heavy web users?  There are a whole host of other questions we should ask regarding polls (response rate, sampling frame, etc) that generally aren’t answered in online polls.

It is important to understand the potential methodological issues when reading research.  Nielsen’s methods are standard for the industry, and they acknowledge the drawbacks and limitations.  In my opinion, the major problem isn’t the methods component, it is Nielsen’s spinning/presentation of its results.  In the Nielsen study (and the previous Participatory Media Network study), the findings focus on lack of teen use of Twitter.  However, the findings reported by Nielsen cover the following age ranges: 2-24, 25-54 and 55+.  The critical category, 2-24, covers a wide range of users – incredibly young children, adolescents, teens and adults.  The grand mean reported by Nielsen is affected by variation inside the different age categories.  Using census data, we can look at age breakdown over the ranges 2-24.  According to census, there are 80MM Americans under age 24 (0-24).  There are approximately 15-16MM Americans in the age ranges 0-4, 5-9, 10-14, 15-19, and 20-24.  Therefore, each category is pretty much weighed equally.  So lets (hypothetically) assume that no one age 0-9 uses Twitter, 5% of people age 10-14, 35% of people age 15-19, and 40% of people age 20-24 use Twitter.  To calculate the grand mean we would weight the percentages and then sum.  Such a formulation would give us 16% use for the demographic age 0-24 (0+0+.01+.07+.08).

The second problem with Nielsen’s presentation is the comparison range.  Comparing the age ranges 2-24 and 25-54 is not fair on a number of levels.  The first category can really only meaningfully cover age 13-24, while the 25-54 age range meaningfully covers 30 years.  If we weigh the estimates (16%/64%) by volume coverage (1:3 ratio), then the category volume for older users would be ~21% (I didn’t bother to weigh by census, just an estimate).  And what if we compared just teens against an adult category – we might even find that teens Twitter more than adults.  Keep in mind, with all the advantages afforded to older users (no Internet restrictions, etc) there are major differences between older users and teen/young users in their capacity to partake in online community.

My analysis is simplistic and speculative, but in certain configurations, “young people” could plausibly use Twitter at higher rates than “adults.”  I don’t have a guess regarding what is right, but my gut tells me that if Nielsen was more upfront regarding their sampling, and less misleading with their infographics, we’d have a different story.  And that story would not be as catchy and headline-grabbing as “Teens Don’t Tweet.”