Intro Statistics 9 Dance of the p Values

What people said

Strong rec Isabella Ghement · 2019-02 · Cross Validated Stack Exchange · on “Why does a t-test on similar random samples often come out with a small p-value?”

I used R with your setup and generated 300 p-values from a t-test under the same setting you used. Here is the histogram of the p-values: [LINK] Here is the quantile-quantile plot which depicts the quantiles of the p-value distribution against uniform quantiles: [LINK] As expected, the distribution of the p-values looks uniform. Finally, here are the p-values plotted against their index: [LINK] As you can see in this last plot, the p-values can be pretty much anything in the range 0 to 1. In fact, as explained by Geoff Cumming in his fabulous video Dance of the p-values , when you replicate a study under similar conditions many times, you can't really use the p-value from the current replica

Recommends amoeba · 2016-12 · Cross Validated Stack Exchange · on “High variance of the distribution of p-values (an argument in Taleb 2016)”

+1. One related -- and fun -- analysis that comes to my mind is what Geoff Cumming calls "A dance of p-values": see [this video] (the "dance" happens at around 9 minutes mark). This whole little presentation basically emphasizes how variable the p-values are even for relatively high power. I don't quite agree with Cumming's main point that confidence intervals are so much better than p-values (and I *hate* that he calls it "new statistics"), but I do think that this amount variability is surprising for many people and the "dance" is a cute way to demonstrate it.

Recommends Stephan Kolassa · 2017-04 · Cross Validated Stack Exchange · on “Generating i.i.d samples in MATLAB: large p-values”

Recall the definition of the p value: it is the probability of observing a test statistic as extreme as the one you observed or more extreme under the null hypothesis. Or, looking at this under a slightly different angle: If your null hypothesis holds, then your p value is a random variable that is uniformly distributed in $[0,1]$ . Try this. Re-run your code and note the p values you get. Make a histogram. You should get a flat histogram with a uniform distribution. One consequence is that "small" and "large" p values will occur equally often under the null hypothesis. For instance, you should see a $p<0.05$ and a $p>0.95$ each about in one in twenty cases. Another consequence is that you s

Recommends Stephan Kolassa · 2017-06 · Cross Validated Stack Exchange · on “Exactness of p-value in testing by simulation”

This effect is not limited to p values obtained by simulation. P values are a statistic, i.e., a function of the data you have observed (and your distributional assumptions and model). As such, they have an expectation and a variance. The variance is due to sampling variability. Suppose you run an experiment, observe data and calculate a p value using some parametric approach. Now suppose you re-run the experiment in the exact same way, so the data generating process of both experiments is precisely identical. You will still get different p values, simply because of the variation in your experimental observations. P values are a statistic. They have a variance. Suggested reading: Boos & Stef

Recommends Stephan Kolassa · 2024-07 · Cross Validated Stack Exchange · on “Why is my pvalue for chisquared test so high?”

+1. A tiny little bit more precisely: under the null hypothesis (which is true here, as you write), we don't necessarily anticipate a large p value. Rather, under the null, p is uniformly distributed on [0,1], so it's equally likely to be small as to be large. In contrast, under the alternative hypothesis, p is more likely to be small, which is why we don't accept the null if p is large. Look at "The dance of the p values" on YouTube [this video] .

Cites Nick Stauner · 2014-04 · Cross Validated Stack Exchange · on “Can a narrow confidence interval around a non-significant effect provide evidence for the null?”

Null hypotheses exemplify the meaning of "All models are wrong, but some are useful." [LINK] They're probably most useful if not taken literally and out of context – that is, it's important to remember the epistemic purpose of the null. If it can be falsified, which is the intended objective, then the alternative becomes more useful by comparison, albeit still rather uninformative. If you reject the null, you're saying the effect is probably not zero (or whatever – null hypotheses can specify other values for falsification too)...so what is it then? The effect size you calculate is your best point estimate of the population parameter. Generally, chances should be equally good that it's an ov

Cites Stephan Kolassa · 2018-03 · Cross Validated Stack Exchange · on “What is the correct way to interpret a $95\%$ confidence interval from the t.test function in R?”

Yes, this is one of the problems with NHST. Note that this is just a reformulation of calculating a single p value, checking whether this is less than 0.05 and then declaring something as "statistically significant". What this disregards is that p values (and CI endpoints) are random variables in themselves [this video] . What NHST does is to hope that we are right 95% of the time. (Assuming preconditions are fulfilled.)

Cites Stephan Kolassa · 2020-05 · Cross Validated Stack Exchange · on “Histogram residual not lining up with normality test”

You have a lot of bins (10) for few data points (100). Even perfectly normal distributions can yield "non-normal-looking" histograms in such cases. ( Compare this. [LINK] ) Simulations are always good to get an idea of what we can reasonably expect under specific hypotheses - here, under the assumption of a normal distribution. For instance, let's simulate 100 normally distributed observations and plot them in a histogram with 10 bins, as in your case. Let's do this six times, just to get a feeling: [LINK] As you see, these histograms can look pretty non-normal indeed - all but one of them are even multimodal! The important point is that these are all equally normal. After all, they were all

Cites Stephan Kolassa · 2022-04 · Cross Validated Stack Exchange · on “Clarification on Danger of Inference on Predictors for Time Series Regression”

This is absolutely not specific to time series analysis (so I took the liberty of editing your tags). Any model selection procedure will try to select predictors that make the model "better". But the mathematical theorems used in calculating p-values - specifically, theorems about the distribution of certain statistics under the null hypothesis do not make allowance for this kind of selection. Essentially, the null hypothesis never included a selection step. You can run a quick simulation of this effect yourself. Simulate $n$ observations data points, and a $n\times k$ matrix of predictors that are completely unrelated to the observations. Run a regression, and record the $p$ values of your