The State of Evidence Evaluation In Debate
The recent discussions of evidence quality in high school policy debate have highlighted the need for debaters, coaches, and judges to revisit the prevailing assumptions about the proper role of cited material in our activity. While a drastic shift in the community’s approach to the evaluation of evidence remains exceedingly unlikely, there is an emerging consensus among debate educators that improving this facet of our pedagogy is both possible and necessary.
What is the problem? In short, the explosion of content enabled by new media has shattered traditional constraints on what constitutes “published” scholarship. While debaters in past decades were limited in their research to published books, journals/magazines, and newspapers, the debaters of today have access to a nearly limitless stream of information—all at their fingertips, and searchable in ways never before thought possible. As Gordon Mitchell describes in “Debate and authority 3.0,” the resulting information abundance has created a need for new ways of separating the good from the bad.
Publication, previously a one-to-many transaction, has become a many-to-many enterprise unfolding across a complex latticework of internetworked digital nodes. Now weblogs, e-books, online journals, and print-on-demand book production and delivery systems make it possible for a whole new population of prospective authors to publish material in what Michael Jensen (2008), National Academy of Sciences Director of Strategic Web Communications, calls an “era of content democracy and abundance.”
In content abundance, the key challenge for readers and referees has less to do with finding scarce information, and more to do with sorting wheat from the proverbial chaff (the ever-burgeoning surplus of digital material available online). The pressing nature of this information-overload challenge has spurred invention of what Jensen (2007) calls “new metrics of scholarly authority” – essentially, new ways of measuring the credibility and gravitas of knowledge producers in a digital world of content abundance.
Policy debate’s “metrics of scholarly authority” have developed slowly—changes in dominant assumptions about what constitutes “good evidence” have occurred over decades based on the organic back-and-forth of the contest round. At the high school level, the influence of summer debate institutes and the trickle-down from intercollegiate competition have played a major part in this evolution. While regional differences remain, the vast majority of those that participate in policy debate on the “national circuit” hold remarkably similar views about what makes a piece of evidence “good”. Indeed, the dominant conception of “good evidence” has become so normalized that it is often framed as self-evident: good evidence “speaks for itself”.
But the evidence can’t “speak for itself”—judges must inevitably make sense of the evidence cited by debaters in order to render a decision in the face of competing arguments. In a thought-provoking 2007 thread on the e-Debate listserv, Michael Antonucci revealed the impossibility of a neutral evaluation of evidence.
Judges enforce “evidence quality-control” all the time.
It’s called reading evidence?
The standards that we apply when reading evidence aren’t neutral, natural valuations. The community’s evolved a fairly homogenized set of techniques, but this set isn’t logically inevitable. … Judges could apply a radically different set of standards to evidence, if they so chose. … Tweaking the standards for good evidence would really be no more or less interventionist than current practice. Ultimately, you have to decide what a “good” card is, and it’s difficult for a debater to discuss every decision criterion you might employ in the course of [five] minutes. At some point, when you’re reading that card, you’re on your own.
Antonucci argues that most judges approach the evaluation of evidence in more-or-less the same way.
When reading evidence, judges generally apply a series of tests. How current is the evidence? Does it make a strong claim? Does it have a warrant? Does it appear to respond to the opponent’s claim? Does it have totally sweet violent imagery? Does it have supersweet 24 point font? Is that font Copperplate, which is clearly the best font ever?
The reading techniques that we currently consider “neutral” are anything but. They are artificial, stylized, and frankly a little bit weird. We value claims, or predictions, over warrants. We favor specific phrases and tropes, even — clean binaries, simple spatial analogies, and some fairly violent imagery generate a warm reception, for pretty subjective reasons.
This generalized agreement provides an important benefit: because most judges evaluate evidence in the same way, students can enter a contest round with a predictable understanding of what kinds of research will be rewarded. If the basic benchmarks for separating good evidence from bad were to change from round-to-round, students would be justifiably frustrated by inconsistent standards.
At the same time, the dominant assumptions about evidence in debate should be subjected to steadfast scrutiny. If the qualities that currently characterize “good evidence” are antiquated or simply wrong, they should be changed—slowly, perhaps—through an ongoing process that acknowledges the authoritative role played by judges in the creation and maintenance of community norms. In other words, judges need to accept that their assessment of evidence is not—and can never be—neutral. This acknowledgment is both constraining and emancipating: while it removes the facade of objectivity protecting judges’ decisions, it also encourages students to experiment with different metrics for the evaluation of evidence.
The danger, of course, is that deviating from existing community norms will result in frustratingly inconsistent decisions. But improvements in the way that high school policy debate approaches evidence do not require a descent into judicial anarchy. Minor changes prompted by small shifts in community norms can have immensely positive results without jeopardizing the predictability upon which the legitimacy of contest round decisions relies.
The Debate Judge As “Change Architect”
In their 2008 book Nudge, noted University of Chicago professors Richard Thaler and Cass Sunstein argue that human beings—when left to our own devices—tend to make bad choices. In order to encourage better choices, public policymakers should use “choice architecture” to encourage good choices and discourage bad ones. In the context of high school debate, judges can be appropriately conceived of as “choice architects”: they have the responsibility “for organizing the context in which people [in this case, debaters] make decisions (Thaler and Sunstein, p. 3).” While performing as choice architects, judges have enormous influence over the behavior of the students who are competing for their approval. As the only decision-maker tasked with assigning a win or a loss, even seemingly insignificant changes in the way the judge approaches the debate can have a profound impact on the choices of individual debaters.
[S]mall and apparently insignificant details can have minor impacts on people’s behavior. A good rule of thumb is to assume that “everything matters.” In many cases, the power of these small details comes from focusing the attention of users in a particular direction. (Thaler and Sunstein, p. 3)
In this article, I will argue that judges should exercise their discretion as “change architects” to encourage students to read the qualifications of their evidence aloud in debates. This change, while minor, will have a profoundly positive effect on the quality of evidentiary analysis in high school policy debate. Implementing this norm is simple: judges should insert the following text (or something like it) into their published judging philosophies.
Debaters are expected to verbally cite the last names of the author(s), their qualifications, and the year/date of publication for each piece of evidence presented in the round. Evidence that is presented without citation of the author(s)’ qualifications will be assumed to be unqualified (“no quals available”). Debaters are encouraged to advance arguments about the qualifications of the authors/sources cited in the debate.
These expectations are not mandates: judges need not levy draconian punishments against debaters that do not follow them. But debaters are easily influenced by the expectations of their judges, and change will occur quickly if teams begin to lose debates because their opponents exploit this opportunity to align themselves with a judge’s expectations. The best—and perhaps only—way to improve the quality of the evidence read in debates and the discussion of that evidence is for judges to nudge debaters in the desired direction. If judges are unwilling to alter their expectations—whether out of fear or an ideological commitment to “let the debaters be”—no change will occur. If we care about improving the quality of evidentiary analysis in high school policy debate, however, this is an unacceptable outcome.
The Case For Verbal Presentation Of Author Qualifications
Debates about the qualifications of a given piece of evidence are important and desirable. By verbalizing these qualifications during a contest round, their importance is amplified—even if students choose not to explicitly discuss them during a given round. The current norm—in which only the last name of the author and the year of publication are read aloud—serves to de-emphasize the importance of qualifications in two important ways:
1. Excluding qualifications from verbal presentation implicitly de-values their importance when considering the quality of a piece of evidence. If the author(s)’ qualifications are not important enough to read aloud, after all, how important can they really be? The current community norm emphasizes consideration of only the recency and content of a piece of evidence—the author’s last name is just a reference point. Qualifications are just one of many things that are not deemed important enough to be read aloud, lumped in with the title of the book or article being cited and the page number or URL from which the cited material is taken.
This does not mean that the debate community values the qualifications of an author equally with the page number from which a card was cut, of course. But the message sent to debaters is clear: as long as it has an author and a date, a piece of evidence is worthy of consideration. And not just consideration: equal consideration, at least until the other team challenges the quality of the source/author being cited. Given this norm, debaters rightly prioritize evidence with outlandish claims and strong language over evidence written by qualified experts.
2. Requiring students to locate the qualifications of a given piece of evidence “privately”—during speech or prep time—prevents the judge from considering qualifications as part of their initial understanding of the evidence as it is being presented. The judge is not merely an information processing machine: s/he experiences the presentation of arguments and evidence by debaters and actively participates in the process through which this rhetoric becomes meaningful. When only a last name and date are verbally cited, the judge cannot consider the qualifications of a piece of evidence as they initially make sense of it—while judges may ask themselves questions like “does this evidence contain a strong conclusion?” or “is this evidence recent enough to account for relevant changes regarding this issue?” while they listen to the initial presentation of a piece of evidence, they are prevented from considering questions like “is this evidence from a qualified author?” or “how might this author’s qualifications affect the argument they are advancing?”.
Some critics of “interventionism” will undoubtedly contend that this exclusion is good: judges should not think about these things, they will argue, because it is the job of the debaters to make these arguments. But this relies on the naive assumption that the judge can remove themselves from the process through which meaning is created from the back-and-forth of a contest round. In his seminal defense of the “critic of argument” approach to debate judging, V. William Balthrop explained the crucial role played by the judge in the hermeneutical process.
[A]t some point in time—either at the end of the debate or, more likely, during the presentation of arguments themselves—the judge is involved in interpretation, placing these positions into a scheme such that the importance of some evidence and some arguments can be identified relative to others. It is here, at the point of interpretation, that the judge engages in the hermeneutical process. It is here, that an awareness of the “rules” of debate as that judge understands them, of the content area of the debate as that judge understands it, of various judging paradigms as that judge understands them, all combine to influence the interpretation by that judge as to which team did the better job of debating. The judge cannot withdraw from participating with the debaters in the creation of meaning, though it is a different kind of participation; and rather than admitting one’s biases and seeking to keep them from affecting one’s decision, a better perspective seems to be that which tests one’s biases against the debate itself and seeks to understand how they affect the decision. (“The Debate Judge As ‘Critic of Argument’,” originally published in the Journal of the American Forensic Association (1983), republished in Advanced Debate: Readings In Theory Practice & Teaching, 3rd Edition, 1987, p. 170-171.)
When author qualifications are not read aloud, they do not enter the hermeneutical process of the contest round unless/until an argument is advanced about them. But this leaves these arguments weaker and less meaningful than arguments about the content of a given piece of evidence because its content and not its qualifications were factored in to the judge’s initial understanding of it. Even if the judge enters the contest round intending to emphasize the importance of author qualifications in their assessment of the evidence presented by the debaters, they will be unable to do so during the course of the debate itself unless these qualifications are read aloud. While the post-round review process provides an opportunity for the judge to unearth the qualifications of the evidence that has been read, the consideration of these qualifications is at this point artificial and removed from the debate itself.
This is not to say that the verbalization of author qualifications will give judges a green light to consider them in their decisions regardless of the arguments made by debaters during the round. The onus remains on the student participants to discuss the importance of qualifications within their speeches. But by enforcing an expectation that qualifications be verbalized as part of the initial presentation of a piece of evidence, judges can “nudge” students toward doing so more frequently and more productively while ensuring that the hermeneutical process through which a given debate is rendered meaningful includes author qualifications.
Answering Objections
Critics of my position will undoubtedly advance a variety of objections. In what follows, I will attempt to respond to some of these objections while further developing the larger case for a change in the community’s evidence citation norms.
Debaters can make arguments about author qualifications regardless of whether they have been read aloud.
This is true, of course: even when only a last name and date are cited aloud, debaters can advance arguments about the qualifications of a given piece of evidence. But these arguments are necessarily separated from the evidence itself as it was initially understood. Verbalization at the time of initial presentation of a piece of evidence emphasizes the importance of qualifications while inviting judges and debaters to take them into consideration as they first make sense of the arguments being advanced.
In addition, time pressures should not be discounted. Reviewing each piece of evidence read by one’s opponents in order to discern the authors’ qualifications requires debaters to invest valuable—and scarce—time that could be spent on other forms of preparation. The dearth of qualifications debates in the status quo seems to support that students will most often ignore qualifications in order to preserve valuable preparation time.
A piece of evidence bestowing the benefits of clean coal from “Lucas ‘9” will be understood differently than the same piece of evidence from “Lucas, spokesman for the American Coalition for Clean Coal Electricity, ‘9”—and for good reason. By normalizing the verbalization of author qualifications as part of the presentation of evidence in a contest round, judges can encourage students to take qualifications more seriously. At the same time, this norm will ensure that an author’s qualifications are understood as part of the evidence itself.
Reading qualifications aloud requires scarce speech time: it will tradeoff with the reading of additional evidence.
This, too, is true. But it relies on an unhealthy obsession with quantity over quality and with hyper-technicality over intelligence. If the verbal presentation of author qualifications results in three or four fewer cards being read in each constructive speech, that seems like an acceptable cost when compared with the benefits of improving the quality of evidentiary analysis in debate.
Because discussions of author qualifications are difficult, students will inevitably choose not to engage in them.
So long as arguments about author qualifications are not perceived as “winners”, debaters will choose not to make them. This is obvious. But in their role as “choice architects,” judges have immense power to alter the behavior of students. If judges expect students to read qualifications aloud, they will. And if judges take qualifications arguments seriously when evaluating debates, students will make them more frequently. Even in debates where they were not discussed, judges can play a positive role in their post-round commentaries by highlighting opportunities that students missed for advancing arguments about author qualifications.
The norm in favor of reading qualifications will be exploited by debaters—they will “highlight down” quals and “massage” them to make their authors seem more qualified than they really are.
This is a legitimate concern, for sure, but it is not damning to my position. First, any manipulation of the norm in favor of reading qualifications aloud will still take place within a context in which more emphasis is placed on qualifications. This provides a natural remedy for manipulative practices: because qualifications will be considered more important, those that attempt to disguise the shortcomings of their evidence will be subjected to heightened scrutiny. Even in a worst case scenario, this is an improvement over the status quo in which qualifications are largely ignored.
Second, the ethical norms that guide debate research will remain a strong firewall against abusive manipulation. The fabrication of an author’s qualifications should be treated in the same way as the fabrication of the text of a piece of evidence: as an unacceptable breach of the community’s trust. In short, there is no unique risk of unethical behavior associated with the verbalization of author qualifications.
Conclusion
High school policy debate is a research-driven activity: we rely on the citation of published material to support our arguments about issues of public policy controversy. In order to make sense of this “evidence” as it is presented in contest rounds, debaters and judges have developed a generalized agreement about the “metrics of scholarly authority” that determine what separates good evidence from bad. But recent developments in new media have called into question the veracity of these metrics. With a functionally limitless stream of information available at our fingertips, separating the good from the bad has become even more important than ever—and, unfortunately, more difficult.
In order to improve the quality of evidentiary analysis in debate, this article has advocated a shift in the community norm about the verbal citation of evidence. Instead of expecting students to read aloud only the author’s last name and the date of publication for each piece of evidence, judges should exercise their discretion as “change architects” to enforce an expectation that the author’s qualifications be verbalized as well. By adopting this minor change in expectations, judges can emphasize the importance of author qualifications to students while encouraging them to make arguments about qualifications more frequently and more seriously in contest rounds.
This “nudge” is no panacea. Many issues will remain regarding the community’s approach to evidence even if reading qualifications aloud becomes normalized. But this small step can have a profoundly positive impact. By elevating the importance of qualifications and forcing debaters and coaches to take them more seriously before and during debates, this move by judges will contribute to the much-needed development of new “metrics of scholarly authority” in high school policy debate.
This is the second in a series of full-length essays published by The3NR.com. If you are interested in syndicating or republishing these essays, please contact the author.
15 Comments.