Meta-Science

Alpha (Type I Error)

The 'Courtroom' Problem

Think of a statistical test like a trial. We assume the Null Hypothesis is 'Innocent' until proven 'Guilty'.

Type I Error ( $\alpha$ ): Convicting an innocent person. (False Discovery)
Type II Error ( $\beta$ ): Letting a guilty person go free. (Missed Discovery)

The standard $\alpha = 0.05$ implies that convicting an innocent person is 4 times worse than letting a criminal walk free. But is this ratio true for every experiment?

The Mathematical Fix

Daniel Lakens argues that we should not use a 'magic number' like 0.05. Instead, we should minimize the Total Cost of Error based on our specific situation.

Loss = (C_1 \times \alpha) + (C_2 \times \beta)

Where C1, C2 are the costs of each error type.

Use the tool below to find your Optimal Alpha by balancing these costs.

alpha.tool.title

alpha.tool.desc

alpha.tool.result_label

0.0070

alpha.status.strict

alpha.control.ratio1:1

alpha.control.ratio_tag
alpha.control.ratio_desc

alpha.control.sample100

alpha.control.sample_tag
alpha.control.sample_desc

alpha.control.effect0.5

alpha.control.effect_tag
alpha.control.effect_desc

Case A: Particle Physics

Claiming a new particle exists when it doesn't is catastrophic. Cost Ratio: 100:1
Result: Extreme rigor ( $\alpha \ll 0.001$ ).

Case B: The A/B Test

A False Positive just means we change a button color unnecessarily. A Missed Discovery means we lose revenue. Cost Ratio: 1:1
Result: Relaxed Evidence ( $\alpha \approx 0.10$ ).

Case C: The Standard

Fisher's legacy convention. Cost Ratio: 4:1
Result: $\alpha = 0.05$ .

header.brand

Laboratory Index

Read the Paper