Fact-checking methods
How to assess the credibility of assertions about statistical significance using p values, power analysis, and effect sizes.
A practical guide to evaluating claims about p values, statistical power, and effect sizes with steps for critical reading, replication checks, and transparent reporting practices.
Published by
Henry Brooks
August 10, 2025 - 3 min Read
When evaluating a scientific claim, the first step is to identify what is being claimed about statistical significance. Readers should look for clear statements about p values, confidence intervals, and the assumed statistical test. A credible assertion distinguishes between statistical significance and practical importance, and it avoids equating a p value with the probability that the hypothesis is true. Context matters: sample size, study design, and data collection methods all influence the meaning of significance. Red flags include selective reporting, post hoc analyses presented as confirmatory, and overly dramatic language about a single study’s result. A careful reader seeks consistency across methodological details and reported statistics.
Beyond inspecting the wording, assess whether the statistical framework is appropriate for the question. Check if the test aligns with the data type and study design, whether assumptions are plausible, and whether multiple comparisons are accounted for. The credibility of p values rests on transparent modeling choices, such as pre-specifying hypotheses or clarifying exploratory aims. Researchers should disclose how missing data were handled and whether sensitivity analyses were performed. Power analysis, while not deciding significance by itself, provides a lens into whether the study was capable of detecting meaningful effects. When power is low, non-significant findings may reflect insufficient information rather than absence of effect.
Examine whether power analysis informs study design and interpretation of results.
A thorough assessment of p values requires knowing the exact test used and the threshold for significance. A p value by itself is not a measure of effect size or real-world impact. Look for confidence intervals that describe precision and for demonstrations of how results would vary under reasonable alternative models. Check whether p values were adjusted for multiple testing, which can inflate apparent significance if ignored. Additional context comes from preregistration statements, which indicate whether the analysis plan was declared before data were examined. When studies present p values without accompanying assumptions or methodology details, skepticism should increase.
Effect sizes reveal whether a statistically significant result is meaningfully large or small. Standardized measures, such as Cohen’s d or odds ratios with confidence intervals, help compare findings across studies. A credible report discusses practical significance in terms of real-world impact, not solely statistical thresholds. Readers should examine the magnitude, direction, and consistency of effects across related outcomes. Corroborating evidence from meta-analyses or replication attempts strengthens credibility more than a single positive study. When effect sizes are absent or poorly described, interpretive confidence diminishes, especially if the sample is unrepresentative or measurement error is high.
Replication, consistency, and methodological clarity strengthen interpretability.
Power analysis answers how likely a study was to detect an effect of a given size under specified assumptions. Read sections describing expected versus observed effects, and whether the study reported a priori power calculations. If power is low, non-significant results may be inconclusive rather than evidence of no effect. Conversely, very large samples can produce significant p values for trivial differences, underscoring the need to weigh practical relevance. A robust report clarifies the minimum detectable effect and discusses the implications of deviations from planned sample size. When researchers omit power considerations, readers should question the robustness of conclusions drawn.
In practical terms, transparency about design choices enhances credibility. Look for explicit statements about sampling methods, inclusion criteria, and data preprocessing. Researchers should provide downloadable data or accessible code to enable replication or reanalysis. The presence of preregistered protocols reduces the risk of p-hacking and cherry-picked results. When deviations occur, the authors should justify them and show how they affect conclusions. Evaluating power and effect sizes together helps separate genuine signals from noise. A credible study presents a coherent narrative linking hypotheses, statistical methods, and observed outcomes.
Contextual judgment matters: limitations, biases, and practical relevance.
Replication status matters. A single significant result does not establish a phenomenon; consistent findings across independent samples and settings bolster credibility. Readers should probe whether the same effect has been observed by others and whether effect directions align with theoretical expectations. Consistency across related measures also matters; when one outcome shows significance but others do not, researchers should explain possible reasons such as measurement sensitivity or sample heterogeneity. Transparency about unreported or null results provides a more accurate scientific picture. When replication is lacking, conclusions should be guarded and framed as provisional.
Methodological clarity makes the distinction between credible and suspect claims sharper. Examine whether researchers preregister their hypotheses, provide a detailed analysis plan, and disclose any deviations from planned methods. Clear reporting includes the exact statistical tests, software versions, and assumptions tested. Sensitivity analyses illuminate how robust findings are to reasonable changes in parameters. If a paper relies on complex models, look for model diagnostics, fit indices, and rationale for selected specifications. A well-documented study invites scrutiny rather than defensiveness and invites others to reassess with new data.
Synthesis: a cautious, methodical approach to statistical claims.
All studies have limitations, and credible work openly discusses them. Note the boundaries of generalizability: population, setting, and time frame influence whether results apply elsewhere. Biases—such as selection effects, measurement error, or conflicts of interest—should be acknowledged and mitigated where possible. Readers benefit from understanding how missing data were handled and whether imputation or weighting might influence conclusions. The interplay between p values and prior evidence matters; a small p value does not guarantee a strong theory without converging data from diverse sources. Critical readers weigh limitations against purported implications to avoid overreach.
Finally, assess how findings are framed in communicative practice. Overstated claims, sensational phrasing, or omitted caveats accompany many publications, especially in fast-moving fields. Responsible reporting situates statistical results within a broader evidentiary base, highlighting replication status and practical significance. When media coverage amplifies p values as proofs, readers should revert to the original study details to evaluate legitimacy. A disciplined approach combines numerical evidence with theoretical justification, aligns conclusions with effect sizes, and remains cautious about extrapolations beyond the studied context.
A disciplined evaluation begins with parsing the core claim and identifying the statistics cited. Readers should extract the exact p value, the test used, the reported effect size, and any confidence intervals. Then, consider the study’s design: sample size, randomization, and handling of missing data. Power analysis adds a prospective sense of study capability, while effect sizes translate significance into meaningful impact. Cross-checking with related literature helps situate the result within a broader pattern. If inconsistencies arise, seek supplementary analyses or replication studies before forming a firm judgment. The ultimate goal is to distinguish credible, reproducible conclusions from preliminary or biased interpretations.
In sum, credible assertions about statistical significance are built on transparent methods, appropriate analyses, and coherent interpretation. Effective evaluation combines p values with effect sizes, confidence intervals, and power considerations. It also requires attention to study design, reporting quality, and reproducibility. A prudent reader remains skeptical of extraordinary claims lacking methodological detail and seeks corroboration across independent work. By practicing these checks, students and researchers alike can discern when results reflect true effects and when they reflect selective reporting or overinterpretation. The habit of critical, evidence-based reasoning strengthens scientific literacy and informs wiser decision-making.
