Analysis of Competing Hypotheses
How Compass evaluates which explanation best fits the evidence.
In 1999, Richards Heuer, a veteran CIA analyst, published a methodology that would reshape intelligence analysis. His insight was deceptively simple: analysts fail not because they lack information, but because they anchor on a favored hypothesis and unconsciously filter evidence to confirm it. The fix was structural — force every hypothesis to confront every piece of evidence, systematically, in a matrix.
Compass implements an upgraded version of Heuer's Analysis of Competing Hypotheses (ACH). The original ACH used qualitative ratings: an analyst would judge whether each piece of evidence was consistent or inconsistent with each hypothesis, then look for the hypothesis with the fewest inconsistencies. This was a breakthrough in structured thinking, but it left three problems unsolved: ratings were subjective, evidence was often correlated (violating the implicit independence assumption), and there was no way to quantify confidence in the result. The Compass ACH solves all three.
Three pathways to a rating
Each cell in the ACH matrix — the intersection of one hypothesis and one piece of evidence — receives a consistency rating through one of three pathways, color-coded in the interface:
- Simulation (blue): The gold standard. Compass translates the hypothesis into a formal scenario specification, runs it through the Monte Carlo engine, and asks: given this hypothesis, how likely is the observed evidence value? If the hypothesis predicts VIX at 15 and it's actually at 32, that's strong inconsistency. The rating is derived from the percentile rank of the observed value within the simulated distribution.
- Market-implied (green): When a prediction market prices the probability of an outcome, that price embeds the collective intelligence of all market participants. Compass extracts distributional information from prediction-market prices and converts it to a consistency rating.
- AI-estimated (amber): When neither simulation nor market data is available, the AI estimates consistency by reasoning about the relationship between hypothesis and evidence. Amber carries lower confidence; it's flagged so you know when you're relying on model judgment.
The platform always prefers simulation > market > AI. When you see a blue cell, the rating was computed from ten thousand Monte Carlo trials. When you see amber, the system fell back to AI estimation. This transparency is intentional: you should weight blue cells more heavily, and the system does the same in its posterior calculation.
Handling correlated evidence
A subtle failure mode in traditional ACH: two pieces of evidence that appear independent but are actually driven by the same underlying cause. If VIX spikes and gold prices rise, those might feel like two separate data points favoring the “risk-off” hypothesis. But if both are driven by the same flight-to-safety response, counting them separately double-counts the evidence.
Compass handles this through d-separation analysis on the causal DAG. When you build a causal graph connecting your evidence items (either manually or with AI assistance), the system identifies which evidence items are conditionally independent given the hypothesis and which share common causes. Evidence that shares a common cause receives reduced effective weight — it still counts, but it counts less.
Reading the posterior
After all cells are computed and d-separation weights applied, Compass produces a posterior probability for each hypothesis. In the distributional upgrade this is not a single number but a probability distribution:
Hypothesis A: 68% (95% CI: 52–81%) Hypothesis B: 32% (95% CI: 19–48%)
When confidence intervals overlap, the analysis is telling you something important: the evidence does not clearly discriminate between hypotheses. You need more or better evidence. The overlap itself is the most honest thing the system can tell you.
The sensitivity ranking in each cell tells you which cells have the most influence on the posterior. A cell with rank 1 is the single piece of evidence that, if its rating changed, would shift the posterior the most. This directs your attention to where it matters.
Technical: Bayesian posterior computation
The distributional posterior uses Monte Carlo integration:
For each of 10,000 MC trials: 1. Draw rating_i from each cell's rating_distribution (metalog) 2. Draw weight_i from each cell's weight_distribution (metalog) 3. Compute log P(H|E) = Σ rating_i · weight_i The collection of P(H|E) draws IS the posterior distribution. Normalize across hypotheses per trial. Fit metalog to posterior.
Diagnosticity uses Kolmogorov–Smirnov distance between rating distributions under each hypothesis pair. Low KS distance = non-diagnostic evidence (tells you the same thing regardless of which hypothesis is true).
From the book: Beyond Synthesis, Chapter 15–18: “The Operators Panel.”