Judgment & Teams

As recent high-profile security breaches have shown, cybersecurity is an increasing issue in the face of industrial-scale hacking by nation-state actors. We explore the idea that the efficiency of hackers may be reduced by exploiting cognitive biases, focussing hackers on goals that are relatively unrewarding or unlikely to succeed. There is a wealth of knowledge in the cognitive science literature about cognitive biases and how to reduce them, but we investigate two new elements: how to observe these biases in cyber scenarios and how to increase (worsen) the biases. We developed a three-tiered, 'Gold-Silver-Bronze' experimental paradigm, in which participants complete cyber activities (Gold), computer-based behavioural tasks (Silver), and established methods of measuring cognitive biases (Bronze). Data collection was first conducted at a wider scale using undergraduates and online participants, then at a smaller scale using cyber experts, recruited for their experience with hacking style behaviours. Gold-tier data collection utilised a 'capture the flag' reward structure whereby participants earned rewards based on successfully completing objectives in a simulated hacking task. Silver-tier data collection utilised a custom-built 'fun fair' experiment, in which tasks were presented as games of skill and chance. Bronze-tier tasks were deployed as online surveys. In each tier, cognitive vulnerabilities were measured and methods for magnifying the hackers' biases were tested. We present some examples of the manipulation and measurement of cognitive biases, such as Representativeness and Loss Aversion, in each tier. Data from each tier was analysed to assess the impact of our manipulations of these biases, and compared across tiers to establish the validity of the behavioural tasks. Preliminary findings suggest that targeted manipulations of cognitive biases can potentially disrupt hackers' efficiency. Future analysis will include cognitive modelling of data from each task to investigate the latent properties of each cognitive vulnerability.

AI tools have made it easier than ever to create believable synthetic media. In the identification of AI-generated and authentic images of human faces, novices appear to have, on average, near-chance accuracy overall and worse than chance accuracy for AI-generated images (Nightingale and Farid, 2022) and poor metacognitive insight into their (lack of) ability, as ascertained via confidence judgments (Miller et al., 2023). Despite individuals’ shortcomings, collective performance may be salvageable by harnessing the Wisdom of Crowds. Through analysis of prior published data (Miller et al, 2023) and a new experiment with more diverse stimuli, we investigate the performance of individuals and crowd aggregation strategies in the difficult perceptual judgment task of classifying faces as authentic or AI-generated. In particular, we investigate a model-based method of recalibrating probability judgments with the goal of correcting individuals’ systematic errors prior to aggregation, building on work by Turner et al., 2014. Recalibration is highly effective, producing crowds that outperform simple aggregation and common performance-weighting algorithms (e.g., accuracy weighting, subcrowd selection). This effectiveness appears to be robust to differences in crowd size and the prevalence of wicked (misleading) stimuli. We additionally compare different recalibration models in order to evaluate performance sensitivity to specific model qualities and constraints. Our results have implications for crowd aggregation strategies, highlighting the benefit of using modeling approaches to correct individuals’ errors, rather than simply using errors as indicators of ability. In a rapidly evolving digital landscape, robust crowd-sourced methods of identifying AI-generated media may be of particular value.

Subjective probabilities (SPs) often violate description invariance where different descriptions of the same event can result in different SPs. This violation is often explained by the hypothesis that SPs are based on the support people accumulate from an event’s description rather than the event itself. As a result, different descriptions can elicit vastly different SPs. However, extant models of support are static, failing to capture how beliefs dynamically evolve. We address this limitation by adapting Decision Field Theory to model SPs. Our proposed Judgment Field Theory (JFT) leverages a similar evidence accumulation framework, modeling how forecasters iteratively construct beliefs about a hypothesis. A key insight of JFT is that different belief states correspond to different SPs, allowing forecasters to report an SP at any moment or continue accumulating support. We tested JFT using two rich datasets in which basketball enthusiasts (N=125, N=364) provided probabilistic forecasts of NCAA men’s basketball end-of-season rankings. The model successfully predicted both SPs and response times. Notably, the data revealed that participants exhibited classic context effects (attraction, similarity, and compromise), patterns that challenge many prevailing SP theories. At the same time, the results exposed a key challenge for JFT (and many other evidence accumulation models): accurately modeling SPs required accounting for the subjective representations of teams, which shaped the support accumulated for each team. This work advances the modeling of probability judgments by introducing a dynamic framework that accommodates belief evolution, offering a novel approach for understanding how SPs develop over time.

Traditional models of team decision making rely on classical probability frameworks that assume commutative operations of information processing. However, real-world decision making in distributed heterogeneous teams—such as those in military task environments—often exhibits non-commutative, context-dependent effects. Effective team communication—crucial for sharing goals, information, and perspectives—is inevitably and frequently disrupted by asynchronous, partial, and sequential information. These challenges intensify under cognitive strain, fatigue, stress, and time pressure, especially in cross-domain joint force missions where team members interpret data from diverse perspectives and contexts. This study extends quantum probability theory, the “prototypic” non-commutative probability theory, to model human decision making in team communication networks, where information is shared asynchronously across team members with different roles and cognitive constraints. Using a series of human experiments with distributed team networks, we compare quantum probability models with traditional Bayesian and Markov models in predicting how information flow structures, message framing, and decision sequences influence team decision outcomes. Our initial experiments examine a simple three-agent network (A-B-C) in a military Battle Damage Assessment task, empirically testing quantum-like context effects such as non-commutativity and sequential interference. Experiment 1 evaluates how the order of information from agents A and B influences agent C’s decision, testing the violation of the classical law of total probability. Experiment 2 manipulates message framing to assess how perspective alignment affects shared cognition and decision accuracy. Our findings will inform the development of computational models and software for evaluating and training team communication in complex, high-stakes decision environments.