P-values, False Discovery Rate (FDR) and q-values: A Practical Guide

A p-value tells you how surprising one result would be if nothing were really going on. When you run one test, that is enough. When you run hundreds or thousands of tests at once, as you do on every spot of a 2D gel or every gene on an array, the p-value on its own will hand you a long list of false positives. The false discovery rate (FDR) is the fix: it controls the proportion of your significant results that are expected to be false, and the q-value is the FDR expressed for each individual test. This guide explains all three, walks through the Benjamini-Hochberg procedure with a worked example, and shows how to choose a threshold you can defend.

What is a p-value?

A p-value is the probability of seeing a difference at least as large as the one you observed if the null hypothesis were true, that is, if there were no real difference between the groups. It is calculated from the size of the difference relative to the variability in the data, which is why a large difference with high variance can be non-significant while a small difference with low variance can be highly significant. In a 2D gel experiment the test is usually a t-test for two groups or a one-way ANOVA for more, run separately on every spot.

By convention, a p-value below 0.05 is called significant. There is nothing special about 0.05. It means that, if the null hypothesis were true, you would see a result this extreme about one time in twenty.

A p-value is not the probability that the null hypothesis is true, and it is not the probability that your result is a false positive. Those are different quantities, and the difference is exactly what the rest of this page is about.

The multiple testing problem

A positive is a significant result. A false positive is a significant result where no real difference exists. At a threshold of 0.05, each test that is truly null has a 5% chance of producing a false positive.

That is acceptable for one test. It is not acceptable for a thousand. If you test 200 spots on a gel and none of them really differ, you expect 200 × 0.05 = 10 false positives. Test 839 spots at p < 0.01 and you expect 839 × 0.01 = 8.39 false positives among spots where nothing is happening. Those false positives look exactly like real results. They have small p-values, they pass the threshold, and they will send you to the mass spectrometer to identify a protein that did not change.

This is the multiple testing problem, and it is why the p-value alone is the wrong tool for any experiment that tests many things at once.

Bonferroni and the family-wise error rate

The oldest fix is to make each individual test stricter. The Bonferroni correction divides your threshold by the number of tests: with 839 spots and a target of 0.05, a spot must reach p < 0.05 / 839 = 0.0000596 to be called significant. This controls the family-wise error rate (FWER), the probability of making even one false positive across the whole set.

Bonferroni is simple and safe, and for a handful of planned comparisons it is the right choice. For a discovery experiment it is usually far too conservative. Demanding near-certainty that not a single false positive gets through means throwing away most of the true positives as well. In a proteomics experiment where you expect dozens of proteins to change, Bonferroni can return none.

What is the false discovery rate?

The false discovery rate reframes the question. Instead of asking “what is the chance of any false positive at all?”, it asks “of the results I am calling significant, what proportion are expected to be false?”

Formally, if R is the number of tests you call significant and V is the number of those that are actually null, the FDR is the expected value of V / R. Controlling the FDR at 5% means that, on average, 5% of your significant results will be false positives and 95% will be real. For a discovery experiment that is exactly the trade-off you want: a short, mostly reliable list to follow up, rather than either a long list full of noise or a near-empty list.

The concept and the standard procedure for controlling it were introduced by Yoav Benjamini and Yosef Hochberg in 1995. It has since become the default correction in genomics, proteomics and most other high-throughput fields.

The Benjamini-Hochberg procedure, step by step

The Benjamini-Hochberg (BH) procedure controls the FDR at a level you choose, written Q. It works like this.

1. Run your test on every feature and collect the m p-values.
2. Sort them from smallest to largest, so p(1) ≤ p(2) ≤ … ≤ p(m). The rank of each p-value is its position i in this list.
3. For each rank i, calculate the BH critical value (i / m) × Q.
4. Find the largest rank k for which p(k) ≤ (k / m) × Q.
5. Call every p-value from rank 1 to rank k significant, even if some of them individually sit above their own critical value.

Step 5 is what people get wrong. BH is a step-up procedure. You find the highest rank that passes, and everything below it passes with it.

The same procedure can be expressed as an adjusted p-value for each test, which is what most software reports. The BH-adjusted p-value for rank i is the smallest value of (m / j) × p(j) for any rank j ≥ i, capped at 1. A test is significant at FDR Q if its adjusted p-value is ≤ Q. This is what R’s p.adjust(method = “BH”) returns and what many packages label as an FDR-adjusted p-value.

Worked example: ten tests

The largest rank that passes is 4, so spots 1 to 4 are significant at an FDR of 5%. Read the adjusted column and you get the same answer: four adjusted p-values are at or below 0.05.

Compare that with the raw p-values. Six of the ten are below 0.05, so an uncorrected analysis would have reported six hits, and on the evidence in the table two of those are likely to be noise. Compare it with Bonferroni, where the threshold would be 0.05 / 10 = 0.005 and only spot 1 would survive. The FDR sits between the two: it keeps the well-supported results and drops the marginal ones, without demanding certainty.

Notice also that the adjusted p-values for ranks 2 and 3 are identical. That is a feature of the step-up rule, not an error, and you will see repeated values in real output too.

What is a q-value?

A q-value is the FDR analogue of a p-value. Where a p-value is the smallest significance level at which a test would be called significant under the false positive rate, the q-value is the smallest FDR at which the test would be called significant. A spot with a q-value of 0.02 is significant at any FDR threshold of 2% or higher.

In practice the term is used in two slightly different ways, and it helps to know which one your software means.

The BH-adjusted p-value described above is often labeled a q-value, and for most purposes it can be read as one.

The q-value proper, introduced by John Storey in 2002 and applied to genome-wide studies by Storey and Tibshirani in 2003, goes one step further. It estimates the proportion of tests that are truly null, written π0, from the shape of the p-value distribution, and scales the adjustment by that proportion. When many features are really changing, π0 is well below 1 and Storey’s q-values are smaller, and more powerful, than BH-adjusted p-values. When nothing is changing, π0 is close to 1 and the two agree.

Reading a p-value histogram

The distribution of all your p-values tells you more than any single one of them. If no spot were really changing, the p-values would be spread evenly between 0 and 1: a flat histogram. Real differences show up as a spike of small p-values on the left, sitting on top of that flat background.

Both histograms flatten off to the right. The height of that flat region is the density of null tests, and it is how Storey’s method estimates π0: the flat part is scaled up across the whole range to estimate how many tests are null, and the spike above it is the excess that represents real change.

A histogram with no spike means the experiment found nothing, whatever the individual p-values say. A histogram that rises toward 1 rather than being flat usually means the statistical test’s assumptions are being violated and the p-values are not trustworthy in either direction.

Interpreting q-values in a real experiment

Here is a real SameSpots experiment with 839 spots, sorted by p-value:

Take spot 52 in the sorted list. Its p-value is 0.01 and its q-value is 0.0141. Reading the p-value the way most people do, 0.01 sounds excellent. But 52 spots have p-values at or below 0.01, and if every spot in the experiment were null you would expect 839 × 0.01 = 8.39 false positives to reach that threshold by chance. Some of those 52 are very likely noise, and the p-value cannot tell you which.

Now read the q-value. A q-value of 0.0141 means that, among all the spots with a q-value this small or smaller, about 1.41% are expected to be false positives. There are 52 such spots, so you expect 52 × 0.0141 = 0.73 false positives, fewer than one. The q-value has taken the whole list into account and told you what the p-value could not: this set of 52 is almost entirely real.

The general rule is to read q-values as a list, not one at a time. Pick the point in the sorted list where the expected number of false positives (number of spots above the line × q-value at the line) is one you can live with, and draw the line there. Two features of the output are worth knowing in advance. q-values always lie between 0 and 1, and repeated q-values in the list are normal.

Choosing a threshold

There is no correct threshold, only a correct way to choose one. Decide how many false positives you are prepared to follow up, and set the line accordingly.

If each hit costs a week of spot picking and mass spectrometry, an FDR of 1% on a list of 50 means you expect to waste one week. If hits are cheap to screen, an FDR of 10% on a list of 200 means twenty wasted screens in exchange for many more true positives than a stricter line would give you. Both are defensible. Picking 0.05 because it is 0.05 is not a decision; it is the absence of one.

Whatever you choose, state it in the methods section together with the procedure (BH or Storey), the number of tests, and the number of features called significant. That is what a reviewer will ask for.

Bonferroni vs FDR: which to use

Use Bonferroni, or another family-wise error rate control such as Holm’s method, when you have a small number of pre-planned comparisons and a single false positive would matter: a confirmatory study, a clinical endpoint, a regulatory submission.

Use the FDR when you are screening many features to find candidates for follow-up: differential expression on a 2D gel, a microarray, an RNA-seq experiment, a metabolomics panel. A few false positives in a candidate list are the acceptable price of finding the true ones, and the follow-up experiment will remove them.

Do not correct at all when you have one pre-specified hypothesis, or when the analysis is explicitly exploratory and you are ranking features rather than declaring any of them significant. Say which you are doing.

Assumptions and caveats

The FDR procedures assume that the p-values going in are valid: that the test you ran is appropriate for the data and its assumptions hold. Correcting a set of p-values from a test whose assumptions are violated does not make them valid.

The BH procedure controls the FDR when the tests are independent or positively dependent, which covers most proteomics and genomics data. Under arbitrary dependence the Benjamini-Yekutieli procedure, a more conservative variant, is guaranteed to control it. Spots on a 2D gel are often correlated (isoforms of the same protein, spots in the same train), which BH tolerates but which is worth being aware of.

Storey’s q-value depends on a good estimate of π0, which in turn depends on having enough tests and a p-value histogram that is flat on the right. With very few tests, or a distorted histogram, BH-adjusted p-values are the safer choice.

Finally, the FDR is controlled on average across repeated experiments. In any single experiment the actual proportion of false discoveries can be higher or lower than the target.

Frequently asked questions

Q: What is the false discovery rate?
A: The expected proportion of results called significant that are actually false positives. Controlling the FDR at 5% means that, on average, 5% of your significant results are false and 95% are real.

Q: What is a q-value in statistics?
A: The smallest false discovery rate at which a test would be called significant. A q-value of 0.02 means the test is significant at any FDR threshold of 2% or above. It is the FDR analogue of a p-value.

Q: What is the difference between a p-value and a q-value?
A: A p-value is about the false positive rate: the chance of a result this extreme if this one test were null. A q-value is about the false discovery rate: the proportion of results as significant as this one, across all your tests, that are expected to be false. A p-value of 0.05 means 5% of null tests produce a result this extreme; a q-value of 0.05 means 5% of your significant results are expected to be false.

Q: What is the difference between FDR and p-value?
A: The p-value is calculated for one test in isolation. The FDR is a property of a whole set of tests and describes how many of the ones you call significant are expected to be wrong. FDR correction adjusts each p-value to account for the number of tests run.

Q: What is an FDR-adjusted p-value?
A: A p-value transformed by the Benjamini-Hochberg procedure so that comparing it against a threshold Q controls the false discovery rate at Q. It is often labeled a q-value in software output.

Q: How do you calculate the FDR?
A: Sort the m p-values, compare each p-value at rank i against (i / m) × Q, find the largest rank that passes, and call everything up to that rank significant. Equivalently, compute the BH-adjusted p-value for each test as the minimum of (m / j) × p(j) over ranks j at or above i, and compare to Q. The worked example on this page shows both.

Q: What is the Benjamini-Hochberg procedure?
A: The standard step-up procedure for controlling the false discovery rate, published by Benjamini and Hochberg in 1995. It ranks p-values and compares each against a critical value that grows with rank, so that the threshold is stricter for the smallest p-values and looser further down the list.

Q: Bonferroni or FDR: which should I use?
A: Bonferroni (or Holm) for a small number of planned, confirmatory comparisons where any false positive matters. FDR for screening many features to find candidates, where a small proportion of false positives is acceptable.

Q: Why are some of my q-values identical?
A: The step-up rule assigns each test the minimum adjusted value from its rank upward, so several consecutive tests can share the same q-value. It is expected behavior, not a calculation error.

Q: What does a flat p-value histogram mean?
A: That the experiment found no real differences. Under the null hypothesis p-values are uniformly distributed, so a flat histogram with no spike at zero means the data look like noise, whatever the individual p-values say.

Q: Does SameSpots calculate q-values?
A: Yes. SameSpots reports a p-value and a q-value for every spot in the experiment and lets you sort the spot list by either, so you can set an FDR threshold directly from the list.

References

1. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B. 1995;57(1):289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

2. Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Annals of Statistics. 2001;29(4):1165-1188. https://doi.org/10.1214/aos/1013699998

3. Storey JD. A direct approach to false discovery rates. Journal of the Royal Statistical Society, Series B. 2002;64(3):479-498. https://doi.org/10.1111/1467-9868.00346

4. Storey JD, Tibshirani R. Statistical significance for genomewide studies. Proceedings of the National Academy of Sciences. 2003;100(16):9440-9445. https://doi.org/10.1073/pnas.1530509100

Statistics built into your 2D gel analysis

SameSpots runs one-way, two-way and repeated measures ANOVA on every spot, reports p-values and q-values side by side, and lets you sort, filter and tag spots straight from the results table. Because every spot is matched on every gel, the tests run on a complete dataset with no missing values.