Google’s Counts: Worth Every Penny You Pay For Them

statistician 400x310

The Internet theoretically is a statistician’s dream. Let’s hope it’s not an nightmare. In our March 10, 2014 post about the irreproducible results of an Ngram search we warned that nothing prevents Google from changing their definitions or conventions … and not telling us about them. But since they tell us precious little, it seems wise not to base important conclusions or critical decisions solely on any relatively lengthy history of the counts data. And that “relatively lengthy” may be even as short as a month or a quarter, because it is easy for Google to change their mind and their software. This was brought to our attention in the December 21 New York Times by economist Seth Stephens-Davidowitz, who apparently makes a career analyzing counts produced by Google searches of certain key words or survey data collected by other surveyers. Overall, the New York Times article showed mostly upbeat behavior during the holiday season, which one would hope for. Whether the annual trends are accurate or not, likely only Google knows for sure. And we are not opining that Google is doing anything malicious in making their changes; they may all be done with the goal of improved accuracy and usability. But without more transparency we will never know.

Pew Research’s Web IQ Quiz Suggests Need for Higher Questionnaire Design IQ

Pew Research iPhone Release Date 600x300

What makes a survey good? It needs to be correct (i.e., to have correct answers that are representative of the populations being surveyed) and actionable (i.e., to have conclusions that are useful and can be implemented). Unfortunately the recent Pew Research Center’s Web IQ Quiz fails on the second criterion. Most of its questions would be more appropriate to a game of Trivial Pursuit than to a proof that U.S. adults are clueless about using the Web. Being able to identify Bill Gates or Sheryl Sandberg from their pictures is meaningless. Ditto the name of the first browser or the date of introduction of the iPhone. Moore’s Law is important as a predictor of computer speed and storage size, but has nothing to do with how people should use the Web. Etc. Interestingly, two of the most useful questions—on privacy policy and net neutrality—were answered as well (or even better) by older (50+) people than younger ones.