Hou mony trials mak a reaction-time test trustable?

The historical nummers come fae Hamsher and Benton, whose abstract says plainly that its conclusion is limited tae the study conditions. They shouldnae be uised as a certification rule for a modern browser gemme. Reliability is aboot whether a defined score can be got again; pouer is aboot the chance o findin a defined effect. They are sib but different questions. de S. Hamsher et al., 1977 Miller, 2023

Why repeatin the same task helps

Gin trials were independent draws wi variability that didnae change, the standard error o their mean wad be the trial standard deviation divided by the square root o the nummer o trials. The fourfold example comes fae that formula. Real trials can break thae assumptions because fowk learn, get tired or change their strategy. Sae a trial-count calculation is a model, no a guarantee o a steady personal score. Miller, 2023

The modern trial-count analysis shaws an aw that effects can change wi practice. Mair trials dinnae juist mak mair copies o an unchynin first trial. A research design maun choose an ootcome and tak account o hou the task behaves throughoot the session. Miller, 2023

Whit kind o reliability dae ye want?

Consistency in ae run is different fae consistency atween days, or consistency in the rankin o fowk. Research on the reliability paradox records hou tasks can shaw strong group effects but produce less reliable individual differences. A muckle average Stroop or reaction effect disnae automatically mak a short score fit for giein somebody a percentile. Hedge et al., 2017

Means, medians and minimums answer different questions

The fastest response estimates a different pairt o performance fae the mean. It is especially sensitive tae hou mony chances ye gie yersel. Slow ootliers can pull a mean up, while deletin them wi nae rule set aforehand can skew the result. Research on reaction-time ootliers is a reason tae keep the scorin rules fixed afore ye compare runs. Ratcliff, 1993

Research on sleep shaws why the choice o ootcome matters: psychomotor vigilance researchers tell slow responses and lapses apairt fae central reaction-time summaries. Ye cannae assume a brief best-o-several gemme catches the same vulnerability tae loss o sleep as a sustained vigilance task. Basner et al., 2011

Whit changes the apparent nummer o trials needit?

Device jitter adds variability, and a fixed device delay adds bias. Mair trials can help wi the first yin but cannae calibrate oot the second. Baith timing evaluations and laboratory calibration results back up the need tae tell thae sources o error apairt. Anwyl-Irvine et al., 2020 Bridges et al., 2020 Woods et al., 2015

For informal trackin, choose a manageable protocol, uise the same nummer o tries ilka time, and record aw the runs ye complete, no juist yer ain records. Treat this as a wey tae compare, no a validated prescription for a particular minimum. Gin ye want a scientific percentile, the task needs a representative norm sample and measured reliability, as weel as enough trials. Hedge et al., 2017 Miller, 2023

Age, preparation and sleep can change performance in or atween sessions an aw. Keepin conditions reasonably steady maks the comparison easier tae mak sense o, but nae amount o standardisation turns a browser result intae a clinical assessment. Woods et al., 2015 Basner et al., 2011