Reliable Measurement Symposium

Psychological researchers apply quantitative models to discover the structure of their construct of interest, typically relying on the fit of the model to the data to reach their conclusions. It has been argued, however, that fit of the model is not sufficient to shield one from model misspecification, that is from inaccurately representing the underlying psychological process of interest. Yet it is unclear to which extent misspecified models can be estimated reliably, and whether the reliability with which parameters can be estimated may signal such misspecification. In this work, we simulate and estimate a whole range of correctly specified and misspecified models, going from the inclusion/exclusion of interaction effects to the aggregation across heterogeneous populations to a linear/nonlinear structure of the model. For each of these models, we then compute the reliability of the parameter estimation for the two types of models and compared the results. We predict that misspecified models may still yield parameter estimates with acceptable reliability metrics when evaluated in isolation. Additionally, we predict that when compared directly to correctly specified models, these same misspecified models will exhibit distinctive reliability degradation patterns reflecting compensatory mechanisms. This dual perspective highlights that while misspecified models might appear adequate when judged solely on standard reliability metrics, comparative analysis against well-specified alternatives can reveal systematic differences that signal model-process misalignment.