P-value
A p-value is the probability, under a specified null model, of a test result at least as extreme as the observed one according to the chosen test.
A p-value does not assign a probability to the null hypothesis or measure the size or practical importance of an effect. Report the test, assumptions and analysis choices alongside an effect estimate and appropriate uncertainty. [1]
For two independent Poisson counts at fixed equal exposures, conditioning on their total under an equal-rate null gives a binomial count with probability 1/2. The same observed rates can yield very different p-values at different exposures. This comparison is distinct from a survival-distribution test. [2] [3]
A small p-value can also reflect failure of the model assumptions; it cannot identify which assumption failed or establish a causal hardware difference. [4]
Relations
Limitations
- Selecting comparisons after seeing results or stopping at a favorable value changes the inferential problem. A test threshold alone is not a scientific or engineering decision rule. [1]
Citations
- ASA (2016): Statement on Statistical Significance and P-Values, six principles
- R stats: Exact Poisson tests, conditioning on the total event count
- Hesela: Equal-exposure rate comparison, exact synthetic example and method limits
- Wasserstein and Lazar (2016): The ASA's Statement on p-Values: Context, Process, and Purpose; principles 1 and 6