Explore-Exploit Symposium I

There are enormous incentives in all of science to exploit current knowledge and theory rather than explore new territory and develop new theory. Publications are the lifeblood of scientists and are easier to obtain by research exploiting what is already published, and by confirming current theoretical beliefs. Scientists’ training is extensive and narrowly focused in certain domains using specific techniques and testing a small range of theories, making it difficult to carry out research in any different way or with a different focus. Funding is easier to obtain and submissions to journals easier to succeed for research consonant with the beliefs of the reviewers, editors and granting agencies. Scientists never want their results to be challenged and their theories to be replaced. Scientists carry out their research with students and postdocs and need to see that the trainees publish, making exploration projects risky for them. Recent trends in Cognitive Science seem designed to further exacerbate these existing trends to exploit rather than explore: Scientists are already using Large Language Models (like Chat GPT) in every aspect of their profession, something sure to increase, and likely to homogenize research. The so-called reproducibility crisis and certain aspects of the open science movement demand replicability and foster ‘safe’ exploitation rather than ‘dangerous’ exploration. These trends are unfortunately likely to increase over the next twenty years. Countering these trends are scientists’ strong curiosity, something that seems to be found in young children (e.g. Alison Gopnik’s depiction of children are scientists). A bias to explore is likely produced by scientists’ motivation to obtain new measurement methods using new equipment, knowing that results not obtainable previously are a sure oath to success. That is, most of the important advances in science occur when new and unexpected results are found and existing theory is replaced, a factor motivating at least some scientists. There are some examples of foundations and private organizations funding basic research, probably more on the exploration than exploitation side of the ledger. Researchers seeking patents and researchers funded in business and industry, largely based on a profit motive, are probably mostly exploiting than exploring, but the distinction is rather fuzzy if one considers drug and biotech companies (and one cannot forget the considerable funding Bell Labs once put into basic research). This summary appears to suggest a strong exploitation bias, but if so, is this optimal? A proper balance of exploration and exploitation is needed to make maximum progress toward any of the main goals of science, and the optimal balance point likely differs for different goals. This symposium is aimed to discuss these issues and possible changes in practice that might enhance scientific progress toward any of its goals. The problem is one faced by all of science, including the fields represented by members of the Society for Mathematical Psychology.

This talk examines the tradeoff between exploration and exploitation in the behavioral science. I argue that academic incentives—particularly around innovation and idea ownership—encourage exploration. Two key psychological mechanisms drive this tendency: collaboration and competition. Collaborations foster exploration by promoting a systems perspective, in which scientists share the rewards of publishable findings while distributing the costs of failed pursuits (Winet et al., 2022). Competition, in turn, spurs uniqueness-seeking in crowded intellectual areas. When scientists underexplore, it is often due to constrained imagination or, conversely, fear of deviating too far from mainstream theories and findings. This results in a tendency to exploit well-established topics (e.g., group perception and discrimination) while neglecting urgent but less charted societal issues such as climate change or public health crises. Notably, within a given topic, over exploration can lead to instability in findings—as seen in the rise (and fall) of priming and embodied cognition research in the early 2000s. Ultimately, social dynamics and institutional norms shape the trajectory of scientific inquiry, contributing to both under-exploration and under-exploitation.

The exploitation-exploration dilemma is typically treated as essentially a decision-making issue, where the goal is to understand the optimal balance between exploitative and exploratory behaviors. As an issue that crosses disciplines, it has been considered from a variety of fields, such as developmental psychology, machine learning, and neuroscience. Framed in that way, neither the history of science nor the philosophy of science have been as present among the widespread disciplinary interest on this issue. Thus, the current talk has two goals: First, to approach the exploitation-exploration dilemma as fundamentally an issue of scientific progress. Second, and following from the first, to motivate the idea that it is uniquely fruitful to understand the exploitation-exploration dilemma as a history and philosophy of science issue. Both goals begin by recognizing that a core set of issues in the history and philosophy of science center on questions of what is meant by progress in science and how to recognize such developments. From this context, the exploitation-exploration dilemma manifests in the form of deciding when a scientific framework should be maintained or abandoned. Ideas will be leveraged from historians and philosophers of science such as Heather Douglas, Imre Lakatos, Thomas Kuhn, Nancy Nersessian, and Karl Popper.

Issues of Exploitation/Exploration discussed as they operate in decision research. The discussion focuses on the cost of exploring working assumptions that differ from the reviewers' favorite assumptions. I propose that this cost slows progress, and this problem can be mitigated by the organization of choice prediction competitions. Recent studies demonstrate the potential of this approach to (1) accelerate the development of cognitive models with high predictive validity and (2) clarify the conditions under which cognitive models can enhance the predictive power of machine learning algorithms.

How the explore-exploit dilemma plays out in scientific research is a complex question that spans a dizzying array of domains, from studies of individual thought and behavior to those of institutions and societies. What can a neuroscientist like me possibly have to say regarding this question? The answer is, unfortunately, not a lot – but the interesting part is that this shortcoming does not reflect my own limitation (as vast as that is) but the traditional paradigm of our field. Our field’s foundational paradigm, I will argue, analyzes behavior in “small world”, bounded environments, in which individuals learn the best action to take while relying on the benefit of previous knowledge about the goal they seek to achieve and the information that is relevant to that goal. To be sure, this paradigm has had much success, and our more sophisticated explore-exploit models can tackle difficult questions such as how we learn to drive, read, or ski. And yet, these models miss the key point that biological brains evolved to cope with large-world, open-ended environments -- in which not only the optimal actions but also the context, possible states, relevant information, outcomes and goals are unknown and must be discovered or created by the agent itself. This is the realm in which scientific research operates and it is also the frontier of cognitive science research. The major question facing our field, therefore, is how cognitive agents generate discoveries and inventions in large open-ended environments. How do we self-organize our learning curricula (decide what to learn or become curious about) – and how do we manage to discover useful, exploitable regularities despite the practically infinite complexity of the environment and our own stark limitations of energy, time and resources? I will touch briefly on answers that begin to emerge from recent studies of curiosity in neuroscience and artificial intelligence (AI). I will emphasize that we are at the very beginning of probing this question – the sweet spot of ignorance and confidence which, in the words of Mark Twain, is the springboard of future success.