How many trials make a reaction time test reliable?

The historical counts come from Hamsher and Benton, whose abstract explicitly limits its conclusion to the study conditions. They should not be imported into a modern browser game as a certification rule. Reliability concerns the reproducibility of a defined score; power concerns the chance of detecting a defined effect. They are related but different questions. de S. Hamsher et al., 1977 Miller, 2023

Why repeating the same task helps

If trials were independent draws with unchanged variability, the standard error of their mean would equal the trial standard deviation divided by the square root of the number of trials. The fourfold example follows from that formula. Real trials can violate those assumptions because people learn, fatigue or change strategy. A trial-count calculation is therefore a model, not a guarantee of a stable personal score. Miller, 2023

The modern trial-count analysis also shows that effects can themselves change with practice. More trials do not simply create more copies of an unchanging first trial. A research design must choose an outcome and account for the task's behaviour across the session. Miller, 2023

What kind of reliability do you want?

Consistency within one run is different from consistency across days or consistency in ranking people. The reliability paradox literature documents how tasks can show robust group effects while producing less reliable individual differences. A large average Stroop or reaction effect does not automatically make a short score suitable for assigning a person's percentile. Hedge et al., 2017

Means, medians and minimums answer different questions

The fastest response estimates a different feature of performance from the mean. It is especially sensitive to how many chances you give yourself. Slow outliers can pull a mean upward, while deleting them without a prespecified rule can bias the result. Research on reaction-time outliers is a reason to keep scoring rules fixed before comparing runs. Ratcliff, 1993

The sleep literature illustrates why outcome choice matters: psychomotor vigilance researchers distinguish slow responses and lapses from central reaction-time summaries. A brief best-of-several game cannot be assumed to capture the same vulnerability to sleep loss as a sustained vigilance task. Basner et al., 2011

What changes the apparent number of trials needed?

Device jitter adds variability, and fixed device delay adds bias. More trials can help with the former but do not calibrate away the latter. Both timing evaluations and laboratory calibration results support distinguishing those sources of error. Anwyl-Irvine et al., 2020 Bridges et al., 2020 Woods et al., 2015

For informal tracking, choose a manageable protocol, use the same number of attempts each time, and record all completed runs rather than only personal records. Treat this as a comparison method, not a validated prescription for a specific minimum. If the goal is a scientific percentile, the task needs a representative norm sample and measured reliability as well as enough trials. Hedge et al., 2017 Miller, 2023

Age, preparation and sleep can also change performance during or between sessions. Holding conditions reasonably steady makes the comparison easier to interpret, but no amount of standardisation turns a browser result into a clinical assessment. Woods et al., 2015 Basner et al., 2011