A Brief Note on Bad Faith
A Media Matters report has gone viral with a deeply misleading analysis of violent crime coverage on Fox News
Media Matters for America is an influential “web-based, not-for-profit, 501 (c)(3) progressive research and information center dedicated to comprehensively monitoring, analyzing, and correcting conservative misinformation in the U.S. media.” According to their website, Media Matters has the “means to systematically monitor a cross-section of print, broadcast, cable, radio, and Internet media outlets for conservative misinformation - news or commentary that is not accurate, reliable, or credible and that forwards the conservative agenda - every day, in real-time.”
Last week, Media Matters published a report entitled, “Fox’s coverage of violent crime dropped after the midterms: Weekday violent crime segments decreased by 50% in the week of the election compared to the weekly average since Labor Day.” According to the report, Fox News was “using violent crime as a political cudgel against Democrats throughout the midterms.” Comparing this issue to the 2018 “migrant caravan,” the report implied that Fox News was engaging in an excessive amount of violent crime coverage for no other reason than because of the perception that it would help Republicans win elections.
The evidence contained in this report (along with its claim that Fox News was responding to “perceived political benefits”) was tweeted out by Matthew Gertz (a senior research fellow at Media Matters) on November 17:
The tweet went viral, receiving over 40,000 likes in its first 24 hours (including a retweet from Alexandria Ocasio-Cortez). Mainstream news outlets quickly picked up the story as well. Yahoo News’ The Wrap, for example, ran the following story based on the Media Matters analysis:
The Daily Beast followed with a similar story accusing Fox News of “fearmongering”:
The implication in all of these reports was clear: Fox News was strategically emphasizing violent crime as a way of advancing the Republican Party’s electoral interests in the midterm congressional elections. Unfortunately for Media Matters, AOC, Yahoo, and the Daily Beast, the empirical evidence is woefully insufficient for drawing this conclusion.
In this post, I’m going to show that Media Matters’ claim about Fox News falls apart once we examine coverage of other campaign issues and on other cable television news networks. Specifically, I present evidence that Fox News did not focus on violent crime more than the other cable television news networks, they were not unique among cable television news networks in turning away from violent crime coverage after November 8, and they talked about other campaign issues at levels similar to that of other cable television news networks. In other words, contrary to Media Matters’ argument, we can’t conclude that Fox News was motivated by the “perceived political benefits” of covering violent crime.
The analysis I present below should have been the first step for any organization interested in a good-faith examination of media coverage during the 2022 election campaign. What’s more, the data I rely on was not difficult to obtain. Given that Media Matters chose to ignore very simple and very obvious pieces of contextual data in favor of presenting a single graph from Fox News, I have to conclude that they are operating in bad faith. Sadly, by employing this bad-faith approach (i.e. strategically reporting or ignoring information based on its “perceived political benefits”), Media Matters is doing exactly what they accuse Fox News of doing.
As always, I hope that this post provides a model for how to critically examine the empirical claims that are increasingly populating our political discourse.
Measuring Coverage of Violent Crime
How did Media Matters produce the graph contained in the viral tweet? In the report summarized in the tweet, Media Matters describes their methodology in the following way:
Media Matters does not make their data publicly available and it’s not possible to replicate their “internal database of all original, weekday programming on Fox News.” It is also not entirely clear how they operationally defined a news “segment” or what a “violent crime in general” refers to in their coding scheme (e.g. would the category “violent crime” include carjacking, rape, physical assault, robbery, etc.?).
The difficulty of replicating Media Matters’ analysis often works to their advantage. Without the resources to track news coverage on a daily basis, it can be difficult to counter their empirical arguments about what the media is covering (and what it is ignoring). The asymmetry between the data available to well-funded groups like Media Matters and the data available to casual observers interested in critically assessing the validity of popular empirical claims makes fact-checking the well-funded groups difficult if not impossible.
Fortunately in this case, however, we can construct an alternative measure of the Fox News agenda that closely approximates the one employed in the Media Matters analysis by using the Stanford Cable TV News Analyzer. The Analyzer” searches the Internet Archive's TV News Archive of MSNBC, CNN, and Fox News coverage since January 1, 2010. Researchers can “write queries that compute the amount of time words are heard in cable TV news.”
Using the “Analyzer,” I collected data on three aspects of cable television news coverage between September 5 and November 14 (the time period covered in the Media Matters tweet and report): (1) the amount of violent crime coverage on Fox News, MSNBC and CNN; (2) the amount of election coverage on Fox News, MSNBC, and CNN; and (3) the amount of non-crime, non-election coverage on Fox News, MSNBC, and CNN. Collectively, these data allow us to assess the Media Matters claim that Fox News excessively focused on violent crime in order to advantage Republicans in the midterm congressional elections.
I built my measure of violent crime coverage by using the “Analyzer” to search the Fox News, MSNBC, and CNN captions for the words “murder(s),” “violent,” “violence,” “assault,” and “crime.” While this is an incomplete and imperfect list of terms for fully measuring the concept “violent crime,” it should capture the essence of what Media Matters was attempting to measure in their study. Even if the terms underestimate the overall amount of time devoted to “violent crime” discussions, they will provide a consistent measure across the time frame of the study (September 5 through November 14) and across the three major cable news networks.
In order to replicate the analysis presented by Media Matters, I aggregated the data from the “Analyzer” into weeks. The graph below displays the number of minutes each week Fox News devoted to “violent crime” during the last two months of the 2022 campaign (placed next to the graph from Media Matters for the purposes of comparison):
Graph #1: Measuring Violent Crime Coverage
As you can see, the data from the “Analyzer” closely matches the data from Media Matters, suggesting that the former is an exceptionally good proxy for the latter. Given that we have a good proxy measure of coverage, we can now begin to explore the quantity and dynamics of violent crime discussion on Fox News (relative to the dynamics of violent crime discussion on other cable news channels).
Did Fox News Cover Violent Crime More than the Other Networks?
If the Media Matters thesis that Fox News covered violent crime an excessive amount in order to help Republicans win the midterm congressional elections is correct, we should expect to find (1) that Fox News covered violent crime far more than MSNBC and CNN and (2) that violent crime was the most covered issue during the period of study on Fox News but not on MSNBC and CNN.
The graph below shows how much the three cable television networks covered violent crime during the last two months of the 2022 campaign:
Graph #2: Violent Crime Coverage on Fox News, MSNBC and CNN
The graph clearly shows that there were relatively small differences in the amount of coverage devoted to violent crime on Fox News and MSNBC between September and November. Indeed, during any given week, MSNBC only covered crime about 6 minutes less than Fox News (or, put differently, about a minute less each day). While CNN covered violent crime less than Fox News and MSNBC, it still spent a significant amount of time on the issue. For example, 25 minutes of CNN’s coverage each week during late October were focused on violent crime (compared to 35 minutes on MSNBC and 39 minutes on Fox). In short, violent crime was an important issue for all of the cable TV channels, not just Fox.
To show just how absurd the Media Matters claim is, we could easily create a graph of coverage on MSNBC and advance the exact same argument about the motivations behind their coverage (i.e. that they were focusing on violent crime in order to help Republicans do better in the election). Notice how this MSNBC graph could be easily substituted from the Media Matters graph:
Graph #3: Violent Crime Coverage on MSNBC
The Media Matters claim that Fox News was cynically focusing on violent crime to help Republicans might carry more weight if Fox News covered violent crime to the exclusion of other important political issues. In order to see how much attention Fox News paid to other issues, I used the “Analyzer” to track the time spent on four other major campaign issues: abortion, inflation/the economy, immigration, and threats to democracy. The “Analyzer” results for Fox News are presented below (with crime included as a reference):
Graph #4: Total Issue Coverage on Fox News
Fox News covered violent crime less than immigration between the weeks beginning September 12 and September 25. Fox News also covered inflation and the economy more than violent crime at numerous points during the two months of the Media Matters study (e.g. the weeks beginning September 12, September 19, and October 17). Coverage of violent crime spiked in late September and late October (reaching its peak with approximately 40 minutes of coverage a week immediately prior to the election). Overall, between September 5 and November 14, violent crime was discussed a total of 262.2 minutes on Fox News, abortion was discussed 78.2 minutes, inflation and the economy were discussed 270.9 minutes, immigration was discussed 111.3 minutes and threats to democracy were discussed 51.5 minutes. In short, violent crime was not the most discussed issue on Fox News overall or during significant periods of the 2022 campaign.
It is hard, however, to know what to make of these numbers in the absence of a broader context. One thing that would be helpful to consider, for example, is whether coverage reflects reality (e.g. is there a significant focus on violent crime because there is a significant problem with violent crime?). We should expect that rising violent crime rates will lead to rising news coverage rates.
While collecting and reporting violent crime data takes time, there is strong evidence that violent crime increased in the months before the 2022 election. As Axios summarizes:
Given that violent crime jumped 4.2% in the months prior to the election campaign and robberies “skyrocketed” by 12%, it is completely unsurprising that media attention to these issues also increased. In fact, if news coverage ignored these changes, we would probably be right to worry about media bias running in the direction of suppressing important stories for political purposes. Of course, it is difficult to know what the “right” amount of attention to social problems is but big increases in crime probably should be met with big increases in news coverage. In other words, there is an explanation for Fox News’ (and the other networks) coverage of violent crime that does not involve their “perceived political benefits.”
Alternatively, we might want to know whether the candidates running for office are making violent crime an issue. If news organizations are covering candidates running for office, their coverage will naturally (and appropriately) reflect what these candidates are talking about.
In order to determine what candidates were talking about during the 2022 campaign, I used data from the Wesleyan Media Project. The Wesleyan Media Project “provides real-time tracking and analysis of all political television advertising in an effort to increase transparency in elections.” In particular, it codes every ad run by Democratic and Republican candidates and groups for the issues discussed.
The data from the Wesleyan Media Project on House and Gubernatorial races are below (with “public safety” highlighted):
As the data show, both Democrats and Republicans were highly focused on crime during the final weeks of 2022 campaign (receiving attention in more than 25% of Democratic ads and 33% of Republican ads). Given this, it would have been extremely strange (and would represent bias of an entirely different kind) if Fox News and the other cable television news networks did not emphasize violent crime in their coverage during the last two months of the campaign. To state the matter more plainly, there are (again) good explanations for the pattern of coverage that do not involve attempts at electoral manipulation by Fox News.
A final way of contextualizing Fox News’ coverage of violent crime is to consider what other cable television news networks were covering. As shown above, Fox News was not the only network talking about violent crime. Still, it would be helpful to know other cable news networks were covering violent crime significantly less than the other main campaign issues.
The “Analyzer” results for MSNBC are below:
Graph #5: Total Issue Coverage on MSNBC
As the data show, MSNBC discussed violent crime more than any other issue during the period of study. Specifically, MSNBC discussed violent crime for a total of 198.5 minutes (compared with 145.5 minutes for abortion, 130.6 minutes for inflation and the economy, 50.4 minutes for immigration, and 122.7 minutes for threats to democracy). Put another way, both Fox News and MSNBC devoted roughly the same proportion of their issue coverage to violent crime between September 5 and November 14 (33% vs. 31%).
The “Analyzer” data for CNN are here:
Graph #6: Total Issue Coverage on CNN
The data for CNN tell basically the same story as the data from Fox News and MSNBC. Violent crime was a major issue during the last two months of the 2022 campaign and its salience increased rapidly during the final weeks of October. Overall, inflation and the economy were discussed more on CNN than violent crime (138.9 minutes to 124.2 minutes) but the differences were relatively small. In short, the other networks made decisions about what to cover that were very similar to those of Fox News. It is difficult, therefore, to believe that Fox News was uniquely motivated by “political benefits.”
Did Fox News Stop Covering Crime After the Election?
Media Matters accuses Fox News of abandoning its violent crime coverage after November 8 because the election “eliminated the perceived political benefit of talking about crime all the time.”
As Graph #3 clearly shows, violent crime coverage on Fox News did fall after the election. As Graph #3 also shows, however, it also fell in exactly the same way (approximately to 0) on MSNBC and CNN. Additionally, coverage of all issues (abortion, inflation and the economy, immigration, threats to democracy) on Fox News (Graph #4) fell after the election. This is also what happened on MSNBC and CNN for every issue except abortion (Graphs #5 and #6). Rather than cynically shifting their coverage in response to shifting “political benefits,” Fox News (and the other networks) merely replaced their election campaign coverage with election result coverage. The result was a declining focus on every kind of political issue.
Another way of conceptualizing the way that election results displaced issue coverage (including violent crime) is to track discussion of voting and elections over time. The “Analyzer” data from Fox News is here:
Graph #7: Election and Violent Crime Coverage on Fox News
The data are unambiguous: there was less coverage of violent crime after the election and more coverage of voting and election results after the election. This, of course, is completely unsurprising. Any major event (e.g. election, war, terrorist attack) would have meant less coverage of violent crime. Just because there was less coverage of violent crime on Fox News after the election does not necessarily mean (as Media Matters argues) that Fox News was covering crime only to shape the outcome of the election.
Below are the graphs detailing coverage of violent crime on MSNBC and CNN:
Graph #7: Election and Violent Crime Coverage on MSNBC and CNN
As the graphs show, there was almost no difference between the pattern of violent crime and election coverage on CNN (right) or MSNBC (left). It is hard, therefore, to believe the argument that Fox was motivated by a fundamentally different set of incentives than either of these two networks.
Conclusion
The data from Media Matters is consistent with the claim that Fox News was attempting to focus on violent crime in order to advance the Republican Party’s electoral interests. As I have shown above, however, the data was not sufficient to draw this conclusion. Examining coverage of other issues and other networks reveals that Fox News behaved in ways that were not surprising and not unique. In short, the Media Matters argument can only be sustained by ignoring incredibly important pieces of contextual information.
I should point out here that there is a way of explaining this overall pattern of results that is consistent with Fox News behaving in accordance with a set of politically-motivated incentives. If Fox News chose to cover violent crime in order to help Republicans and MSNBC’s and CNN’s coverage primarily follows the lead of Fox News’ coverage (responding to the issues raised on Fox News), we would have the same pattern of results that are presented in the graphs above. This explanation, of course, requires that (1) other networks respond to Fox News; (2) Fox News does not respond to the other networks; and (3) MSNBC and CNN are also responding to political incentives (i.e. they want Democrats to do well so they counter Fox News information that is beneficial to Republicans).
Assessing each of these claims is beyond the scope of this post. I merely want to point out here that litigating these more complicated and nuanced issues would require a much different analytic approach than the one presented by Media Matters. In the future, I hope they do better.
Heh, nice work. But I am not sure it was bad faith. It is consistent with bad faith, but there is another alternative explanation. Confirmation bias. I'd guess it never occurred to the Media Matters analyst to even check other outlets to do a comparison. I'd guess he found a pattern that fit his cynical view of Fox, and ran with it, without considering alternatives. I think it was Heinlein who said something like never attribute to malice that which is equally explained by incompetence. It does not make it any better, or the story any less pernicious. Of course, I do not have evidence that it is confirmation bias rather than bad faith. And I am not sure how one might tease them out in this type of situation.