Noise

Noise Plot Summary

Introduction

Have you ever wondered why different judges hand down vastly different sentences for similar crimes? Or why experienced doctors sometimes disagree on diagnoses when looking at the same patient data? These variations represent a pervasive but largely unrecognized problem in human judgment called "noise." While we often focus on bias—systematic deviations in judgment—we rarely notice noise, which is random variability in judgments that should be identical. This oversight is costly: noise creates unfairness in the judicial system, leads to misdiagnoses in healthcare, causes financial losses in business, and undermines decision quality across all domains of life. The authors present a groundbreaking theoretical framework that distinguishes between different types of noise—system noise, level noise, pattern noise, and occasion noise—and explains how these interact with bias to produce error. They demonstrate that wherever human judgment exists, noise exists too, typically in greater amounts than we realize. By understanding the psychology behind noise and implementing practical strategies like decision hygiene, we can significantly improve the quality of judgments in professional settings, organizations, and our personal lives. This framework not only helps us recognize a previously invisible problem but also provides concrete solutions for addressing it.

Chapter 1: Distinguishing Noise from Bias in Decision Making

Noise and bias represent two fundamentally different components of error in human judgment. To understand this distinction, imagine four teams at a shooting range. Team A hits close to the bull's-eye with tightly clustered shots—they demonstrate accuracy with minimal error. Team B's shots are tightly clustered but systematically off-target—they show bias. Team C's shots are scattered widely around the bull's-eye—they exhibit noise. Team D's shots are both widely scattered and systematically off-target—they display both bias and noise. Bias is a systematic deviation that pushes judgments in a consistent direction. When judges are biased against certain types of defendants or insurance underwriters consistently overvalue certain risks, their judgments predictably err in one direction. Noise, on the other hand, is unwanted variability among judgments that should be identical. It occurs when different judges assign vastly different sentences to similar cases or when the same doctor makes different diagnoses when examining the same patient data on different occasions. The critical difference between bias and noise becomes apparent when we look at the "back of the target"—where we can see the pattern of shots without knowing where the bull's-eye is. Even without seeing the target, we can immediately recognize noise by the scatter of the shots. This metaphor illustrates why noise can be measured even when we don't know the "correct" answer to a judgment problem. We don't need to know the "right" sentence for a crime to measure how much judges disagree with each other. In real-world settings, noise creates serious problems. A study of federal judges revealed that when presented with identical case files, some judges would sentence a defendant to several years in prison while others would recommend probation. In the insurance industry, claims adjusters evaluating identical claims varied by an average of 43% in their assessments, and underwriters pricing identical risks showed 55% variability. This inconsistency represents not just unfairness but also significant economic costs—one insurance executive estimated that noise in underwriting decisions cost his company hundreds of millions of dollars annually. Unlike bias, which often receives attention in discussions about improving decision making, noise remains largely invisible. This invisibility stems from our tendency to evaluate judgments one at a time rather than comparing multiple judgments of the same case. Organizations maintain an "illusion of agreement" because they rarely test whether their professionals would reach the same conclusions when faced with identical cases. When noise audits reveal the extent of the problem, the results are typically shocking to the organizations involved.

Chapter 2: Components and Measurement of Judgment Variability

System noise—the unwanted variability in judgments made by different professionals within an organization—can be broken down into distinct components. Level noise refers to consistent differences between judges in their average judgments. Some judges are simply more lenient or severe than others across all cases, just as some insurance underwriters are consistently more cautious or aggressive in their pricing. This component is relatively straightforward to understand and measure. Pattern noise, the second major component, is more complex and typically larger than level noise. It reflects how different judges respond differently to specific features of cases. For instance, one judge might be particularly harsh on drug offenses but lenient on white-collar crimes, while another shows the opposite pattern. Pattern noise itself has two subcomponents: stable pattern noise, which reflects consistent individual patterns in how people respond to different cases, and occasion noise, which captures how the same person might judge the same case differently on different occasions. Occasion noise stems from various transient factors that shouldn't affect professional judgment but do. Studies have shown that judges give more lenient rulings after lunch breaks and harsher sentences after their local football team loses. Weather, time of day, recent experiences, and even random fluctuations in mood all contribute to occasion noise. In one striking experiment, fingerprint experts who had previously identified a match were shown the same prints months later and sometimes reached different conclusions. The measurement of noise requires special techniques, particularly the noise audit. In a noise audit, multiple professionals independently evaluate the same cases, allowing researchers to quantify the overall variability and decompose it into its components. When organizations conduct such audits, they typically discover much more noise than they expected. In the insurance company study mentioned earlier, executives predicted about 10% variability in premium assessments but found 55%—more than five times their expectation. Understanding the mathematical relationship between error, bias, and noise is crucial. The error equation (Mean Squared Error = Bias² + Noise²) reveals that bias and noise contribute equally to overall error. This means that reducing noise by a given amount improves accuracy just as much as reducing bias by the same amount. This insight is counterintuitive but essential—organizations should care just as much about reducing noise as they do about reducing bias, yet they typically focus almost exclusively on the latter.

Chapter 3: Psychological Mechanisms Behind Noisy Judgments

The human mind employs several cognitive processes that contribute to noise in judgment. One fundamental mechanism is substitution, where people unconsciously replace a difficult question with an easier one. Instead of directly assessing a job candidate's likelihood of success (a complex prediction), interviewers might evaluate how articulate the candidate is (a simpler observation) and then substitute this impression for the more difficult judgment. When different judges substitute different questions, pattern noise results. Another key mechanism is matching, where people translate their impressions onto judgment scales without consistent calibration. What one person calls a "7 out of 10" risk might be another person's "5." This inconsistent mapping between impressions and numerical ratings creates level noise. The problem is exacerbated when scales lack clear anchors or when judges have different reference points—for instance, doctors from different hospitals might have different thresholds for what constitutes "severe" pain. Excessive coherence describes our tendency to form impressions quickly and then interpret new information in ways that confirm these initial impressions. Once we start seeing a case in a particular light, we unconsciously emphasize evidence that fits our emerging narrative and discount contradictory information. This coherence-seeking tendency explains why the order in which information is presented can dramatically affect judgment. Two judges who encounter the same information in different sequences may reach different conclusions because their initial impressions shape how they interpret subsequent details. Group decision making can amplify noise rather than reduce it. When people influence each other through information cascades and group polarization, the final judgment may depend heavily on who speaks first or most confidently. This explains why similar groups can reach dramatically different conclusions when evaluating the same information. Experiments with jury deliberations have shown that real juries produce more variable judgments than "statistical juries" created by simply averaging individual jurors' pre-deliberation opinions. Our judgments are also affected by irrelevant factors like mood, fatigue, and recent experiences. A judge might be more lenient after lunch or more severe after hearing about a particularly heinous crime. A doctor might prescribe antibiotics more readily late in the day when decision fatigue sets in. These transient influences create occasion noise—variability in how the same person judges the same case on different occasions. Perhaps most importantly, we are generally unaware of these sources of variability in our own judgments. We experience our judgments as arising from a careful consideration of the evidence, not as the product of psychological mechanisms that introduce noise. This "illusion of validity" makes it difficult for professionals to recognize noise in their own judgments and explains why many are skeptical when confronted with evidence of variability.

Chapter 4: Decision Hygiene: Practical Strategies for Noise Reduction

Decision hygiene represents a systematic approach to reducing noise in judgment without necessarily knowing which specific errors are being prevented. Just as washing hands prevents the spread of various germs without targeting any specific pathogen, decision hygiene practices prevent various judgment errors without targeting specific biases. These practices are particularly valuable because they address the invisible problem of noise, which organizations typically underestimate or overlook entirely. The first key strategy is sequencing information to prevent premature intuitions. When judges receive all case information at once, they tend to form quick impressions that bias their interpretation of subsequent details. Forensic science laboratories have implemented "linear sequential unmasking" procedures where examiners first analyze evidence in isolation before being exposed to potentially biasing contextual information. Similarly, medical diagnosticians can reduce noise by considering test results before learning about a patient's demographic characteristics or previous diagnoses. Aggregating multiple independent judgments represents another powerful strategy. The wisdom of crowds principle demonstrates that averaging several estimates typically produces more accurate results than even the best individual estimate. This occurs because random errors tend to cancel out in the aggregate, while valid insights accumulate. Organizations can implement this principle through prediction markets, Delphi methods, or simply by collecting independent judgments before group discussion. Research on forecasting shows that aggregation can reduce error by 12.5% on average across diverse domains. Guidelines provide structure and consistency to judgment processes. In medicine, diagnostic guidelines have significantly reduced variability in areas ranging from psychiatric diagnosis to cancer staging. The key to effective guidelines is finding the right balance—too rigid, and they prevent professionals from exercising necessary discretion; too loose, and they fail to reduce noise. Well-designed guidelines incorporate decision trees, checklists, or scoring systems that ensure all judges consider the same factors in a similar sequence. Structuring complex judgments involves breaking them into distinct components that can be assessed separately before being combined. In personnel selection, for example, interviewers who rate candidates on specific dimensions (like technical knowledge, communication skills, and leadership potential) before making an overall evaluation show less noise than those who make holistic judgments. This approach reduces the halo effect—the tendency for a general impression to influence all specific ratings. Implementing decision hygiene requires organizational commitment and cultural change. The benefits of these practices are often invisible—you never know which specific errors you've prevented—making them less intuitively rewarding than bias-reduction efforts that target visible problems. However, the cumulative impact of reducing noise can be substantial, improving both the quality and fairness of judgments across an organization.

Chapter 5: Information Sequencing and Judgment Aggregation

Two particularly powerful decision hygiene strategies deserve special attention: controlling the sequence of information and aggregating multiple independent judgments. Both strategies directly address fundamental cognitive tendencies that produce noise. Sequencing information works by preventing premature intuitions from distorting judgment. When people receive all information simultaneously, they quickly form coherent impressions that influence how they interpret subsequent details. Once these impressions form, confirmation bias leads people to emphasize information that supports their initial view and discount contradictory evidence. By controlling when judges receive different pieces of information, organizations can reduce this source of noise. In forensic science, this principle has transformed fingerprint analysis. After a high-profile error in the 2004 Madrid bombing case, where FBI experts mistakenly identified an innocent American as the source of a crime scene fingerprint, researchers discovered that contextual information dramatically influenced examiners' conclusions. When examiners knew that a suspect had confessed or that other evidence pointed to guilt, they were more likely to find a fingerprint match. The solution was "linear sequential unmasking"—examiners now analyze the crime scene print in isolation, document their observations, and only then compare it to a suspect's print, all before learning any contextual case information. Aggregating multiple judgments represents perhaps the most universally applicable noise-reduction strategy. Mathematical principles guarantee that averaging independent judgments reduces noise—specifically, noise decreases by the square root of the number of judgments averaged. Averaging four judgments cuts noise in half; averaging nine judgments reduces it by two-thirds. This principle works across domains from economic forecasting to medical diagnosis to performance evaluation. The Good Judgment Project, which studied forecasting accuracy across thousands of participants, demonstrated the power of aggregation combined with selection. The researchers identified "superforecasters" who consistently outperformed others in predicting geopolitical events. When these superforecasters worked in teams, their collective judgments were more accurate than intelligence analysts with access to classified information. The project found that aggregation worked primarily by reducing noise rather than bias. For maximum benefit, organizations should combine these strategies with others. For instance, having judges first make independent assessments (using structured evaluation forms) before coming together for discussion prevents groupthink while preserving the benefits of deliberation. Similarly, sequencing information works best when combined with documentation requirements that create accountability for changes in judgment after new information is received. These strategies are particularly valuable because they address noise without requiring judges to recognize their own biases—something humans are notoriously poor at doing. By changing the process rather than trying to change how people think, organizations can achieve substantial improvements in judgment quality.

Chapter 6: The Mediating Assessments Protocol

The Mediating Assessments Protocol (MAP) represents a comprehensive approach to reducing noise in complex judgments by integrating multiple decision hygiene principles. It was developed specifically for high-stakes decisions where multiple factors must be considered and where both bias and noise pose significant risks. The core insight behind MAP is that complex judgments should be decomposed into separate assessments of distinct aspects, which are evaluated independently before being combined into a final decision. This approach prevents the "halo effect," where an overall impression influences judgments of specific attributes, and reduces the tendency to form premature conclusions based on limited information. The protocol begins with identifying the key factors that should influence the decision—called "mediating assessments." For a hiring decision, these might include technical skills, leadership ability, cultural fit, and relevant experience. For a medical treatment decision, they might include efficacy, side effects, patient preferences, and cost. These assessments should be comprehensive (covering all relevant aspects) and independent (with minimal overlap). Once the mediating assessments are defined, information relevant to each assessment is gathered and evaluated separately. Ideally, different team members are assigned to different assessments to prevent one person's overall impression from contaminating multiple evaluations. Each assessment is completed and documented before moving to others, reducing the tendency to form premature conclusions. In the decision meeting, each assessment is discussed separately before any holistic evaluation occurs. The group follows a structured process: first discussing one assessment thoroughly, reaching a conclusion about it, documenting that conclusion, and only then moving to the next assessment. Only after all assessments have been thoroughly discussed does the group turn to the final decision, which is informed by but not mechanically determined by the assessments. MAP incorporates several other decision hygiene principles. It uses relative rather than absolute judgments where possible, comparing the current case to reference cases rather than placing it on an abstract scale. It encourages taking the outside view by considering base rates and statistical patterns rather than treating each case as entirely unique. And it creates accountability through documentation of assessments before the final decision is made. Organizations can adapt MAP for recurring decisions by creating standardized assessment frameworks. For example, a venture capital firm might evaluate all potential investments using the same set of mediating assessments, creating a consistent structure while still allowing for judgment on each specific case. Over time, this approach builds an organizational memory that improves decision quality. The MAP approach has been successfully implemented in various contexts, from strategic business decisions to medical diagnoses. While it requires more upfront effort than unstructured deliberation, it typically saves time overall by focusing discussion on relevant factors and preventing premature closure. More importantly, it significantly reduces both bias and noise, leading to consistently better decisions.

Chapter 7: When to Embrace or Reduce Variability

While noise generally represents unwanted variability in judgment, there are situations where some variability serves important values. Understanding when to reduce noise and when to preserve some degree of variability requires balancing competing considerations. Complete elimination of noise would often require replacing human judgment with rigid rules or algorithms. While this approach works well for many predictive tasks, it raises concerns in domains where human discretion serves important purposes. In criminal sentencing, for example, some degree of individualization allows judges to consider unique circumstances that rigid guidelines might miss. Similarly, in medical treatment decisions, physicians need flexibility to tailor approaches to individual patients' needs and preferences. The optimal level of noise is rarely zero for several practical reasons. First, the cost of noise reduction must be weighed against its benefits. Implementing comprehensive decision hygiene practices requires investment in training, process redesign, and sometimes additional personnel. Second, excessive standardization can undermine professional morale and autonomy, potentially leading to resistance or workarounds that introduce new forms of error. Third, in rapidly changing environments, some variability in judgment allows for innovation and adaptation that rigid systems might prevent. When algorithms are part of the solution, they raise additional considerations. Algorithms trained on historical data may perpetuate or even amplify existing biases. For instance, if an algorithm for predicting recidivism is trained on data from a biased criminal justice system, it may reproduce those biases in its recommendations. However, research shows that well-designed algorithms can actually reduce both noise and bias simultaneously. In bail decisions, for example, algorithms have been shown to reduce racial disparities while improving overall accuracy. The appropriate balance between noise reduction and discretion varies across domains. In high-stakes domains with clear objectives—like medical diagnosis or financial forecasting—aggressive noise reduction is typically warranted. In domains involving complex value judgments—like child custody decisions or ethical dilemmas—preserving some room for discretion may be more appropriate, though still with structured processes to limit unnecessary variability. Organizations should approach noise reduction as a continuous improvement process rather than a one-time effort. By conducting periodic noise audits, they can identify areas where variability remains problematic and refine their decision processes accordingly. The goal is not perfect consistency but rather a level of variability that balances accuracy, fairness, efficiency, and the legitimate need for professional discretion.

Summary

Noise—unwanted variability in judgments that should be identical—represents a pervasive but largely invisible flaw in human judgment. The theoretical framework presented in this book reveals that noise exists wherever judgment exists, typically in greater amounts than we realize, and contributes as much to error as bias does. By understanding the distinct components of noise—level noise, pattern noise, and occasion noise—and implementing decision hygiene strategies like information sequencing, judgment aggregation, and structured evaluation processes, organizations can significantly improve the quality and fairness of their judgments. The implications of this framework extend far beyond the specific domains examined. Whether in healthcare, criminal justice, business, government, or our personal lives, reducing noise offers a powerful opportunity to improve decisions without requiring perfect knowledge or eliminating necessary discretion. By making the invisible visible, the noise framework transforms how we understand human judgment and provides practical tools for enhancing it. In a world increasingly concerned with fairness, accuracy, and consistency in decision making, addressing noise represents not just a technical improvement but a moral imperative—one that can help create systems that better serve human values and needs.

About Author

Loading...

Daniel Kahneman

From Wikipedia:Daniel Kahneman (Hebrew: דניאל כהנמן‎; born 5 March 1934 - died 27 March 2024), was an Israeli-American psychologist and winner of the 2002 Nobel Memorial Prize in Economic Sciences, notable for his work on behavioral finance and hedonic psychology.With Amos Tversky and others, Kahneman established a cognitive basis for common human errors using heuristics and biases (Kahneman & Tversky, 1973, Kahneman, Slovic & Tversky, 1982), and developed Prospect theory (Kahneman & Tversky, 1979). He was awarded the 2002 Nobel Prize in Economics for his work in Prospect theory. Currently, he is professor emeritus of psychology at Princeton University's Department of Psychology.http://us.macmillan.com/author/daniel...

Read more

Related Books by Daniel Kahneman

Loading...

Thinking, Fast and Slow

Daniel Kahneman

Intuition or deliberation? Where you can (and can't) trust your brain

★★★★☆4.4

Loading...

Nudge

Richard H. Thaler

Improving Decisions about Health, Wealth and Happiness

★★★★☆4.1