Cochrane Answers Critics About Its Fluoridation Review

From the Cochrane Oral Health Group's Blog   (which has been removed due to pressure from fluoridationists 

Our response to the feedback on the Cochrane fluoridation review

We welcome the ongoing debate around the Cochrane review on water fluoridation. Whilst we cannot respond to all issues raised individually, there are several themes that have arisen which we feel it would be useful to clarify at this stage.

Firstly, we are sorry that some readers feel that the blog has not accurately represented the review as a whole. We make every effort to ensure that our reporting is transparent. We also want the blog to be accessible to a wide audience and one of the ways we do this is by limiting its length. As a result, the blog is much less detailed than the review itself. We would encourage interested parties to read the full review. The review should be read in full before application of the findings is considered. We should stress that the review is for an international audience. It is not the purpose of a Cochrane review to make recommendations; we aim to present the research evidence in a robust, reproducible, transparent way to inform health care decisions. Those decisions are likely to vary according to setting.

Exclusion of studies

One, recurring criticism of our review is that “97% of the fluoridation research [was] ignored”(Perrott, 2015).   This criticism is based on the PRISMA flow diagram which shows our search strategy identified >4500 records (after duplicates were removed) but included only 107 studies in our analyses. Those involved in searching electronic bibliographic databases will be aware that the first search often returns articles that are unrelated to the topic of the search, it is usual in systematic reviewing to discard over 90% of the references initially retrieved (Rathbone, 2014). In order to be as comprehensive as possible in our search, and ensure high sensitivity, we often have to cast our net very wide. Certainly, with searches to identify observational studies (which are often poorly indexed), rather than RCTs, we have to trade off specificity with sensitivity.

Inclusion criteria

The protocol for the Cochrane review, published in December 2013, set out very clear inclusion criteria for the studies to be reviewed with regard to caries. Studies eligible for inclusion needed to have a concurrent control group and they needed to have collected data from at least two points in time, i.e. they could not be single time point, cross sectional studies. We acknowledge that there may be concerns regarding the exclusion of cross-sectional studies from the current review. We gave much thought to the inclusion criteria we used, and, as in the York review, felt cross-sectional studies do not adequately evaluate the effects of water fluoridation on the prevention of dental caries.

Within the discussion of the Cochrane review we consider the role of cross-sectional studies, and highlight previous reviews that have included such studies. For example, the comprehensive review by Griffin et al (2007) that evaluated the effectiveness of fluoride in preventing caries in adults. The review included nine studies that examined the effectiveness of water fluoridation, all of which fell outside the scope of the Cochrane review. Of the nine studies they included, eight were cross-sectional studies, with single time-point data. The review by Griffin et al (2007) demonstrated a caries prevented fraction of 34.6% (95% CI 12.6% to 51.0%), when pooling data from seven studies of lifelong residents of control or fluoridated-water communities. This effect size was still present, but reduced, when the analysis was limited to studies published after 1979 (prevented fraction 27.2% (95% CI 19.4% to 34.3%; 5 studies). The most recent of the post-1979 papers included in the review was published in 1992 and only one study reported evaluating a fluoride concentration that could be considered applicable today (two studies did not report fluoride concentration and two evaluated fluoride concentrations above 1.6 ppm).

It has been suggested that the recent review by Anglemyer et al (2014) strengthens the argument to include cross-sectional studies in our review. The review summarizes the results of methodological reviews that compare the results of randomized controlled trials with the results of from observational studies addressing the same question. Their analyses are predominantly from comparisons of RCTs with cohort or case-control studies. No comparisons were made between RCTs and cross-sectional studies. We acknowledge that there are more recent cross-sectional studies evaluating water fluoridation and caries levels however, we are yet to be convinced of the additional benefit of including single time point, cross sectional studies in our review.

Several readers have raised the issue that two studies, published 15 years ago by a member of the review team (Professor Helen Worthington), have been excluded from our review. These studies evaluated disparities in caries levels across different social groups and, again, were single time-point, cross-sectional studies that did not meet the inclusion criteria for our review. The cross-sectional studies, whilst able to provide information on whether water fluoridation is associated with a reduction in disparities, are not able to address the question of whether water fluoridation results in a reduction in disparities in caries levels. We would not alter our inclusion criteria simply to allow inclusion of our own primary research papers.

It has also been suggested that our inclusion criteria for cessation studies were too restrictive. However, as with the evaluation of the initiation of water fluoridation, the review team felt an appropriate concurrent control was necessary for cessation studies. Our inclusion criteria were set to allow the inclusion of studies where groups under comparison were as comparable as possible at baseline with regard to caries levels and confounding factors.

Inequalities in dental health

Over the past 15 years there has been misinterpretation of the evidence in the York review, McDonagh et al (2000). It is often stated that the York review found some evidence that water fluoridation reduces inequalities between 5 and 12 year olds from different socio-economic groups in their average levels of decayed, missing and filled teeth. This statement does not accurately reflect the evidence in the review. The review did find some evidence that water fluoridation reduces the inequalities in dental health across social classes in five year olds, using the dmft/DMFT measure. Similarly, there was very limited evidence that water fluoridation reduces the inequalities in dental health across social classes in 12 year olds, when measuring DMFT. However, no effect was seen in any of the other age groups evaluated (i.e there is no evidence that water fluoridation reduces inequalities between 5 and 12 year olds from different socio-economic groups). When evaluating the proportion of caries free children there was no evidence from the included studies to suggest that fluoridation reduces disparities across social class.

The authors of the York review have previously raised concerns about the misinterpretation of their findings and, in 2003, issued a statement in which they explicitly state “The evidence about reducing inequalities in dental health was of poor quality, contradictory and unreliable.”

Within the Cochrane review, we felt there were insufficient data to determine whether initiation of a water fluoridation programme results in a change in disparities in caries levels across different groups of people.

Fluorosis

There has been discussion around the terminology used within the review with regard to ‘water fluoridation’. We accept that in the fluorosis section of the review it is more appropriate to refer ‘the effects of fluoride in the water’ rather than ‘water fluoridation’ throughout.

Whilst we do not make direct comparison between areas with and without fluoride in the water, we do present the marginal probabilities of dental fluorosis of aesthetic concern and all levels of fluorosis for a range of fluoride concentrations. The data should not be used as ‘proof’ that community water fluoridation causes dental fluorosis in 40% of the population, or dental fluorosis of aesthetic concern in 12% of the population. However, as the review concludes, the evidence does show that there is a significant association between dental fluorosis (of aesthetic concern or all levels of dental fluorosis) and fluoride level. The evidence is limited due to high risk of bias within the studies and substantial between-study variation.

We are grateful to readers of the review for highlighting our omission of the paper by McGrady et al (2012). Despite our comprehensive search and extensive peer review, we apologise that this study was not picked up sooner for inclusion in the review. Whilst it does not meet the inclusion criteria for caries studies, it does contain relevant fluorosis data. We have reanalysed the fluorosis data to include this study, and will update the review in due course. The study does not make a substantial difference to the data, as shown below. The values for fluorosis at 0.7 ppm remain unchanged.

Aesthetic fluorosis, concentrations 5ppm or less

Fluoride exposure	Probability of fluorosis aesthetic concern (95% CI)
0.1	8 (5 to 13)
0.2	9 (5 to 13)
0.4	10 (6 to 14)
0.7	12 (8 to 17)
1	15 (10 to 21)
1.2	18 (12 to 24)
2	31 (23 to 40)
4	60 (47 to 72)

All fluorosis, concentrations 5ppm or less

Fluoride exposure	Probability of fluorosis (95% CI)
0.1	28 (23 to 33)
0.2	30 (25 to 34)
0.4	33 (28 to 38)
0.7	40 (35 to 44)
1	47 (42 to 51)
1.2	52 (47 to 56)
2	68 (63 to 73)
4	83 (77 to 89)

Assessment of Risk of Bias

A bias, in an epidemiological context is a systematic error, or deviation from the truth. The term is widely used with this meaning. While it is appreciated that there are other uses of the word bias, what is meant by bias in the context of the review is detailed in the methods section and appropriate references are given. 

We agree that many of the gold standard conditions necessary to completely avoid bias would be very difficult, if not impossible, to achieve for a study looking at water fluoridation. However, just because these biases are inevitable, it doesn’t mean their potential impact on the results of a study can be ignored. Findings must always be assessed critically against how close to the truth they can reasonably be assumed to be. The truth does not make allowances for studies that are practically difficult to conduct.

Assessing the risk of bias of included studies is not about labelling a study as good or bad, it is about objectively assessing how true the findings of a study can be considered to be based on the way in which the study was conducted and reported. The criteria by which a study may be assessed vary according to study type. The risk of bias assessment undertaken within the Cochrane review used criteria relevant to the assessment of non-randomized studies.

Assessing the overall quality of the evidence

The review has received some criticism for its use of GRADE in assessing the overall quality of the evidence. This is an area that we, again, gave much thought to and we detail our thoughts in the review’s discussion. GRADE has developed over recent years as an internationally recognised framework for systematically evaluating the quality of evidence within both systematic reviews and guidelines. As a review team, we feel GRADE is an appropriate method for assessing the overall body of evidence. However, we acknowledge that the terminology used in GRADE relating to ‘quality’ may appear too judgmental. We also acknowledge that studies on water fluoridation, as for many public health interventions, are complex to undertake and that researchers are often constrained in their study design by practical considerations. In order to overcome these concerns, a decision was made to omit the GRADE terminology of relating to ‘quality’ and discuss the review’s findings in terms of our confidence in the results. This is reflected in the abstract, summary of findings and plain language summary (PLS). Unfortunately, due to word limits for both the abstract and PLS we were unable to include all relevant qualifications. However, we would again stress that those involved in decision making should read the full review and not rely on information presented in abstracts, PLS, blogs, or any summaries of the review.

We thank all those contributing to the discussion around this review. We apologise for not being able to respond to all, individual contributions but will continue to read postings with interest. 

                                                                             END