Here are some things you can find and report as alternatives
to the pvalue: confidence intervals, Bayes factors, and magnitudebased interference. These are used mostly the same situations, but all of them are
more informative especially in combination with the P value.
The Problems with PValue
First what is the pvalue, and why do people hate it? Pvalue refers to the probability of obtaining at least as extreme evidence against your current null hypothesis if that null hypothesis is actually true.
First what is the pvalue, and why do people hate it? Pvalue refers to the probability of obtaining at least as extreme evidence against your current null hypothesis if that null hypothesis is actually true.
There are some complications with the definition. First, “as
extreme” needs to be further clarified with a onesided or twosided alternative
hypothesis. Another issue comes from the fact that you're dealing with a
hypothesis as if it’s already true. If the parameter comes from a continuous distribution,
the chance of it being any given value is zero, so we’re assuming something
that is impossible by definition. If we are hypothesizing about a continuous variable
parameter, the hypothesis could be false by some trivial amount that would take
an extremely large sample to find.
Pvalues also convey little information on their own. When
used to describe effects or differences, they can only really reveal if some effect
can be detected. We use terms like statistically significant to describe this
detectability, which makes the problem more confusing. The word ‘significant’
sounds like the effect it should be meaningful in real world terms; it isn’t.
Pvalue is sometimes used as an automatic tool to decide if
something is publication worthy (this is not as pervasive as it was even ten
years ago, but it still happens). There’s
also undue reverence from the threshold of 0.05. If a pvalue is less than
0.05, even by a little, then it the effect or difference it describes is
(sometimes) seen as much more important than if the pvalue were even a little
greater than 0.05. There is no meaningful difference between pvalues of 0.049 and
0.051, but using default methods, the smaller pvalue leads to a conclusion
where an effect is ‘significant’, where the larger pvalue does not. Adapting
to this reverence to the 0.05, some researchers make small adjustments to their
analysis when a pvalue is slightly above 0.05 in order to try and push it
below that threshold artificially. This practice is called phacking.
So, we have an unintuitive, but very general, statistical
method that gets overused by one group and reviled by another. These two groups
aren't necessarily mutually exclusive.
The generalpurpose feature is pvalues is fantastic though,
it’s hard to beat a pvalue for appropriateness in varied situations. pvalues
aren’t bad, they’re just misunderstood. They’re also not alone.
Confidence intervals.
Confidence intervals are ranges that are assumed to contain
the true parameter value somewhere within them with a fixed probability. In
many cases confidence intervals are computed alongside pvalue by default. A
hypothesis test can be conducted by checking if the confidence interval
includes the null hypothesis value for the parameter. If we were looking for a
difference between two means the null hypothesis would be that the mean is 0
and we would check if the confidence interval includes 0. If we were looking
for a difference in odds we could get a confidence interval of the odds ratio
and see if that includes one.
There are two big advantages to confidence intervals over pvalues.
First, they explicitly state the parameter being estimated. If we're estimating
a difference of means, the confidence interval will also be measured in terms
of a difference. If we're estimating a slope effect in linear regression model,
the confidence interval will give the probable bounds of that slope effect.
The other, related, advantage is that confidence intervals
imply the magnitude of the effect. Not only can we see if a given slope or difference
is plausibly zero given the data, but we can get a sense of how far from zero
the plausible values reach.
Furthermore, confidence intervals expand nicely into
twodimensional situations with confidence bands, and into multidimensional
situations with confidence regions. There are Bayesian analogues called
credible intervals and credible regions, which have a similar end results to
confidence intervals / regions, but different mathematical interpretations.
Bayes factors.
Bayes factors are used to compare pairs of hypotheses.
For simplicity let’s call these the alternative and null
respectively. If the Bayes factor of an alternative hypothesis is 3, that
implies that the alternative is three times as likely as the null hypothesis
given the data.
The simplest implementation of Bayes factor is between two
hypotheses that are both at some fixed value, like a difference of means of 5 versus
a difference of 0, or a slope coefficient of 3 versus a slope of 0. However, we
can also the alternative hypothesis value to our best (e.g. maximum likelihood,
or least squares) estimate of that value. In this case the Bayes factor is
never less than 1, and it increases but naturally as we move further away from
the null hypothesis value. For these situations we typically use the log Bayes
Factor instead.
As with pvalues, we can set thresholds for rejecting a null
hypothesis. For example, we may use the informal definition of a Bayes factor
of 10 as strong evidence towards the alternative hypothesis, and reject any
null hypotheses for tests that produce a Bayes factor of 10 or greater. This
has the advantage over pvalues of giving a more concrete interpretation of one
thing as more likely than another, instead of relying on the assumption that
the null is true. Furthermore, greater evidence of significance produces a
larger Bayes factor, which makes it more intuitive for people expecting a large
number for strong evidence. In programming languages like R, computing Bayes factor
is nearly as simple as pvalues, albeit more computationally intense.
Magnitude based
inference
Magnitude based inference (MBI) operates a lot like
confidence intervals except that it also incorporates information about biologically
significant effects. Magnitude based inference requires a confidence interval (generated
in the usual ways) and two researcherdefined thresholds: one above and one
below the null hypothesis value. MBI was developed for physiology and medicine,
so these thresholds are usually referred to as the beneficial and detrimental thresholds,
respectively.
If we only had a null hypothesis value and a confidence
interval we could make one of three inferences based on this information: The
parameter being estimated is less than the null hypothesis value,: Is more than
the null hypothesis value, or it is uncertain. These correspond to the
confidence interval being entirely below the null hypothesis value, entirely
above the null hypothesis value, and straddling the value respectively.
With these two additional thresholds, we can make a greater
range of inferences. For example,

If a confidence interval is entirely beyond the
beneficial threshold, then we can say with some confidence is beneficial.

If the confidence interval is entirely above the
null hypothesis value, but includes the beneficial threshold, we can say with
confidence that the effect is real and nondetrimental, and that it may be
beneficial.

If a confidence interval includes the null
hypothesis value but no other threshold, we can say with some confidence that
the effect is trivial. In other words, we don't know what the value is but
we're reasonably sure that it isn't large enough to matter.
MBI offers much greater Insight than a pvalue or a
confidence interval alone, but it does require some additional expertise from
outside of statistics in order to determine what is a minimum beneficial effect
or a minimum detrimental effect. Sometimes thresholds involve guesswork, and
often involve research discretion, so it also opens up a new avenue for phacking.
However, as long as the thresholds are transparent, it’s easy to readers to
check work for themselves.
very nice, clear succinct explanation.
ReplyDelete