Epidemiological studies that fail to follow established principles can lead to or promote false assumptions. Attention to the principles of epidemiological studies and avoidance of extrapolation beyond the data can remove much of the confusion that presently exists among the health professions and general public. This article offers guidelines to evaluating epidemiological studies.

"In science, as in everything else, people should treat every pronouncement of human beings as fallible in the first place and tentative in the second place." -- Alex Michalos

"Science is organized skepticism." -- Anonymous



Professionals and the lay public alike are besieged by reports and claims that a given observation or treatment supports or proves that two health care events are linked; and, therefore, an association or causation is involved. At times, this "association/causation" amounts to no more than the simplistic "before it, therefore because of it." Almost universally, such claims are later disproved; but they are rarely so reported in the media or scientific publications, leading to inappropriate behavior or outright disillusionment that science has misled us again.

It is then appropriate that the guidelines for the proper establishment of epidemiological studies and their interpretation be set forth in a manner that can be readily understood and applied to any of these claims. Hopefully, this may help to avoid future misinterpretations and improve the quality of epidemiological studies.

Criteria for Epidemiological Studies

Since data from experiments in humans to prove causation are generally unavailable due to ethical reasons, determination of association/causation relationships in human disease rely to a great extent on epidemiological findings. Table 1 lists the principal criteria necessary for the establishment of such relationships.^1-3

Virtually all these criteria apply to formulating an association and not causation. Causation can only be proven epidemiologically with prospective interventional studies that eliminate or alter the course of the disease.^4,5 Purely observational studies cannot prove or disprove causality.^4,5

Clinical vs. Statistical Significance

All too often, clinical studies synonymously equate statistical significance with clinical significance, leading to probable misinterpretations of the data presented.⁶ The statistical significance of a study is the result of a statistical test that yields a sufficiently small "P" value (the probability that the observed difference is due to chance) and leads to a rejection of the null hypothesis of no difference between treatments.⁶ Clinical significance is the smallest change in a measurement between treatment groups that would result in a decision to modify treatment.⁶ A clinical result could easily be statistically significant without being clinically significant and, due to the methodology of the study, the reverse might also occur: that the difference in treatment groups was medically but not statistically significant. If these differences are not clearly stated in the study or attempts are made to equate statistical with clinical significance, then serious difficulties exist with the data and conclusions.

P values are arbitrary and commonly considered significant at the 0.05 level (a 5 percent probability that the results were due to chance). If the P value were 0.01 percent (a 1 percent probability that the results were due to chance) or of "high statistical significance" then there would be greater confidence in the rejection of the null hypothesis. Nonsignificant P values would support the probability that either there was no difference between the treatment and control groups, no significant differences between treatments, or that the sample size was too small.⁷

The Null Hypothesis

It is commonly stated that a study is proposed to "prove" that a particular treatment or effect does or does not occur. This is a complete misuse of the null hypothesis (that no differences exist between treatment groups) and implies an automatic bias in the study toward "proving" one result or another. It is imperative in epidemiological studies that the null hypothesis be strictly adhered to and that every effort be made to disprove that differences exist between treatment groups.⁸ If differences are then found and the null hypothesis is rejected, sound science has likely occurred; and some degree of confidence can be placed in the conclusions.

Meta-Analyses

A meta-analytical study is a combination of the research results from several studies^9,10 and is commonly used to assess weak risk factors that have potentially large public impact (passive smoking, microorganisms and cardiovascular disease, low-level radiation.)¹¹ When done properly, a meta-analysis can provide a more objective appraisal of evidence than traditional narrative reviews, offer a more precise estimation of treatment effects, and explain apparent difference between studies.¹⁰ However, meta-analyses can be misleading or erroneous depending on any biases toward including or excluding given studies, the database used to search for the studies, data pooling, failure to consider all variables, and the sometimes serious disagreement in results with large, controlled randomized studies that are unlikely to be wrong.^9-11

Odds Ratios and Risk Ratios

Odds or risk ratios are often employed to present the relative medical significance of a particular association. The risk ratio is the number of people who experience an event divided by the total number of people at risk for the event.¹² It is expressed as a proportion (percentage): risk ratio of 0.1 = 10 percent; risk ratio of 0.5 = 50 percent. An odds ratio is the number of people who experience the event divided by those that do not.¹² It is expressed as a number from zero (will never happen) to infinity (certain to happen): an odds ratio of 6.0 (6:1) means that six will experience the event for every one that does not; an odds ratio of 1.5 means that 1 1/2 people will experience the event for every one that does not.¹² An odds ratio of less than one implies a reduction in risk and odds ratios of 1.5 to 2.0 are weak associations that commonly are later found to be associated with confounding variables not controlled for or detected in the study.¹³

Confidence Intervals

Increasingly, clinical trial results are expressed with confidence intervals: the limits within which the "real differences" between the treatments is likely to lie and, therefore, the strength of the inferences that can be drawn from the results.⁷ For example, an association may be expressed as an odds ration of 3.0 (95 percent confidence interval, 1.5-6.0) or an odds ration of 3.0 with 95 percent probability that the "real effect" lies between 1.5 and 6.0.⁷ The narrower the confidence interval, the more likely the result is to be definitive; the larger the confidence interval, the weaker the association.⁷ If the confidence interval overlaps zero (95 percent confidence interval, -2.0-4.0), then this is a negative result (trial) or a very weak association.

Conclusions

Epidemiological studies can be very well-performed leading to reasonable conclusions or, as with many, fail to follow established principles and lead to or promote false assumptions. Epidemiological studies can only prove causation with prospective interventional studies, which document that elimination or modification of the proposed cause of the disease decreases or eliminates the disease. Attention to the principles of epidemiological studies and avoidance of extrapolation beyond the data can remove much of the confusion that presently exists among the health professions and general public.

Author

Thomas J. Pallasch, DDS, MS, is a professor of pharmacology and periodontology at the University of Southern California School of Dentistry.

References

1. Evans AS, Causation and disease: The Henle-Koch postulates revisited. Yale J Biol Med 46:175-95, 1976.

2. Slots J, Casual or causal relationship between periodontal infection and non-oral disease? J Dent Res 77(10):1764-5, 1998.

3. Hill AB, The environment and disease: Association or causation? Proc Roy Soc Med 58:295-300, 1965.

4. Petitti DB, Associations are not effects. Am J Epidemiol 133:101-2, 1991.

5. Sutter MC, Assigning causation in disease: Beyond Koch’s postulates. Perspect Biol Med 39(4):581-92, 1996.

6. Lindgren BR, Wielinski CL, et al, Contrasting clinical and statistical significance with the research setting. Pediat Pharmacol 16:336-40, 1993.

7. Greenhalgh T, Statistics for the non-statistician. II. "Significant" relations and their pitfalls. Br Med J 315:422-5, 1997.

8. Scheutz F, Poulsen S, Determining causation in epidemiology. Comm Dent Oral Epidemiol 27(3):161-70, 1999.

9. Bailar JC III, Passive smoking, coronary heart disease, and meta-analysis. New Eng J Med 340(12):958-9, 1999.

10. Egger M, Smith GD, Phillips AN, Meta-analyses: Principles and procedures. Br Med J 315:1533-7, 1997.

11. Blettner M, Sauerbrei W, et al, Traditional reviews, meta-analyses and pooled analyses in epidemiology. Int J Epidemiol 28:1-9, 1999.

12. Davies HTO, Crombie IK, Tavakoli M, When can odds ratios mislead? Br Med J 316 (7136):989-91, 1998.

13. Friedman GD, Kaltsky AL, Is alcohol good for your health? New Engl J Med 329:1882-3, 1993.

To request a printed copy of this article, please contact/Thomas J. Pallasch, DDS, MS, USC School of Dentistry, University Park MC-0641, Los Angeles, CA 90089-0641.

Table 1. Principal Criteria for Epidemiological Studies.^1-3

* The prevalence of the disease should be significantly higher in those exposed to the putative (proposed) cause than in those not exposed.

* The exposure to the putative cause should be more commonly present in those with the disease than those without the disease when all risk factors are held constant.

* The incidence of the disease should be higher in those exposed to the putative cause than in those not so exposed as documented in prospective studies.

* The disease should follow the exposure to the putative cause.

* There must be a certain strength of association (dose-response relationship).

* The cause must be related in time and place to the effect.

* A consistency of association must exist: agreement among observers in different places by different researchers using different techniques.

* Elimination or modification of the cause should decrease the incidence of the disease.

* A coherence of association should exist: the cause and effect interpretation should not conflict with the known pathology of the disease.

* The entire concept of the relationship must make epidemiological and biological sense.