It’s amazing, in the world of ratings, how many different conclusions can be drawn from the same set of numbers. Someone can be winning a nightly battle in “total viewers,” but lose in the 25-54 year-old demo. An anchor may be up a certain percentage from a year ago, but be down compared to the previous month. There are so many opportunities for twisted language and clever turns of phrases that when networks come out with their year-end ratings press releases, there’s room enough for everyone to just say wonderful things about their own shows, and not have to bring up those on rival networks.
But MSNBC’s release, sent out this week, had a little bullet point that seemed out-of-step with the rest of the information (“A25-54” is the estimated number of viewers in the demo; “P2+” is the estimated number of total viewers, all numbers courtesy of Nielsen):
“Hardball with Chris Matthews” at 7 p.m. ranked #2 with A25-54 (188,000 vs. CNN’s 173,000) and P2+ (734,000 vs. CNN’s 537,000). This is “Hardball’s” best A25-54 delivery since 2009. CNN’s “Erin Burnett Outfront” is performing 23% below what “John King USA” did in the hour with Adults 25-54.”
The Erin Burnett mention pits her show against another on her own network, as opposed to solely contrasting it against MSNBC’s Hardball. Furthermore, the numbers MSNBC used for comparison are not the same as the ones CNN used.
MSNBC looked at John King‘s numbers at 7 p.m. from the start of the year to Sept. 30, 2011. It then compared them to Burnett’s numbers from Oct. 3, 2011 (OutFront‘s debut), until Dec. 15, 2011.
In CNN’s release, they compare Burnett’s numbers from Oct. 3, 2011, to Dec. 15, 2011, with John King’s numbers in the same time period last year. Their data finds Burnett to be up 9% in total viewers and 1% in the demo.
In other words, MSNBC measured Burnett directly against her predecessor in the same year, while CNN measured her against the same predecessor in the same time period a year before. And, after comparing the two, it ends up with CNN holding the advantage in methodology. There were plenty of huge events taking place in the first nine months of the year that drew viewers to all the cable news networks (the Arab Spring, bin Laden’s assassination, among others), and there’s no way of knowing what Burnett’s numbers would have been had she been the anchor, instead of King, for those events. More likely than not, OutFront would have gotten a boost, possibly up to King’s levels, from viewers tuning in to (as cliched as it sounds) watch history unfold.
Even holding her 2011 ratings against the 2010 numbers are somewhat unfair, as the Chilean miner drama was unfolding in October 2010, and resulted in ratings boosts across the board. But using that time frame is really the best — and only — window to measure against. So to grow 9% in total viewers, and even 1% in the demo, is an impressive feat.
In baseball (just stay with me for a moment), there are groups of numbers known as “splits.” They are, basically, statistics split into different categories. A hitter’s splits, for example, could include his batting average in the month of September. They can be represented in almost any category: home runs in day games, runs scored on turf in April, ERA in June on the road. The specificity of the splits can sometimes be maddening, but they can be used both to make a player look good or bad. And this, I think, is where MSNBC got tripped up.
MSNBC is looking at one set of Burnett’s splits, while CNN is looking at another. What wins out, however, is the relevance of the splits one looks at, as well as what lies beneath the numbers. While MSNBC could be looking at something like Burnett’s “ERA” and King’s “ERA” in different months of the same season, they’re brushing off the fact that she (figuratively) had a sore elbow for one game and was facing the Yankees twice, while he faced the Mets and Dodgers without DHs in their lineups. King would, obviously, have better numbers in that scenario. Not to take away from King’s ratings, but Burnett didn’t have the luxury of an Arab Spring to report on. She had a handful of debates and Halloween.
CNN, on the other hand, is looking at a more appropriate predictive statistical measure, like Burnett and King’s WHIPs against the same team (say, the Orioles).
In short, MSNBC may be ignoring some extenuating circumstances in stating that Burnett is down 23% against King. It is the truth, of course, and they aren’t making numbers up, but their methodology may be hiding some significant elements buried beneath the numbers. CNN’s comparison — looking at the same time periods in essentially (or as close as possible to) the same circumstances — results in a more accurate measure of Burnett’s ratings compared to King’s.
Both networks, however, are ignoring another baseball statistical term: small sample size. No matter how good or bad Burnett looks right now, she’s been on CNN’s air for under 90 days. Any good statistician knows that she needs at least a few more months in the majors before we can make any kind of real judgment.
Have a tip we should know? firstname.lastname@example.org