Virology blog post, by David Tuller, 11 September 2017: Trial By Error: The NICE “Topic Expert” Reports
Freedom of Information requests to NICE
My first recent freedom of information request to the National Institute for Health and Care Excellence (NICE) was for information about the experts consulted in the current process of reviewing CG53, the 2007 guidance for the illness the agency calls chronic fatigue syndrome/myalgic encephalomyelitis. In its response, the agency explained that seven topic experts had been consulted in the process of preparing the surveillance document, which recommended leaving the guidance as is.
(I have previously written about the NICE review process on CG53 here, here and here. My e-mail exchange with Sir Andrew Dillon, chief executive of the NICE guidance executive, is described here.)
Who are the experts?
NICE, of course, declined to reveal the names of the seven topic experts, although it reported that four of them were members of the original 2007 guidance development committee. NICE also told me, as it told the ME Association, that the panel included three psychiatrists, two neurologists, a pediatrician, and a patient. NICE did not explain why the panel included more psychiatrists than other specialists, but the selection of topic experts certainly provides insight into NICE’s perspective on the illness.
The experts’ reports
My next FOI request was for any reports prepared by these topic experts–properly anonymized, of course. I recently received them and have asked for them to be posted on Phoenix Rising. If this is the full extent of the input provided by the topic experts, it seems like NICE made a remarkably anemic effort to seek outside advice. And it sought that advice from a relatively narrow spectrum of professional opinion, especially given the enormous complexity of the disease.
In its e-mail to me, NICE explained that the topic experts provided “three types of contributions.” Each person was asked to fill out two questionnaires. The first questionnaire sought suggestions on new studies or research that could impact diagnostics, interventions and other aspects of the guidance. The second sought input about two specific studies and whether the surveillance report should contain in-depth commentaries on them. These questionnaire responses were due last December. The questionnaires were sent to me as separate documents.*
(The previous sentence has been corrected. The sentence originally read: “The questionnaires, which were often only partially filled out, were sent to me as separate documents.”)
In March of this year, NICE surveyed the same experts about three 2015 U.S. reports—from the Agency for Healthcare Research and Quality, the Institute of Medicine (now the National Academy of Medicine), and the CFS/ME Advisory Committee to the Department of Health and Human Services. NICE wanted to know about the possible impact of these reports on diagnostic criteria and other aspects of CG53. Not all of the topic experts answered. The responses were all sent to me as part of a single document.
Strangely, the first questionnaire sent to all seven topic experts included this sentence in the first paragraph: “As a member of the committee that developed this guideline we welcome your views on any areas that need updating.” That suggests that all seven topic experts consulted were, in fact, members of the committee that developed the 2007 guidance–even though NICE had already informed me that only four of the seven were on the 2007 committee.
I have no idea whether NICE misinformed me that only four of the seven were on the original committee, or whether the wrong form was sent three topic experts who were not actually on the 2007 guidance committee. Whatever. NICE obviously has glitches in its fact-checking department. I hope that problem gets worked out soon.
Statements from the experts
The documents I received include some interesting and irritating statements from the experts, but nothing particularly surprising. Two of the six medical experts (#4 and #7) expressed modest concerns that recent events had rendered the current guidance somewhat out-of-date. For example, #4 cited the “considerable on-going public and media interest in this disorder” and advised NICE to have “a current and up to date view on this controversial area.” The other four (#1, #2, #3, #6) found that the research supported the current recommendations.
The patient in the group (#5) expressed personal opinions about the difficulties patients face as well as concerns about case definition–but did not fill out the questionnaires provided by NICE and did not make recommendations related to the guidance. #5 attached two additional statements from other ME patients to his/her/their submission to NICE; together they included many references to the emerging science and to critiques of the PACE trial.*
(The previous sentence has been corrected–originally I wrote that NICE did not send me these two additional attachments as part of the FOI response, but it turned out I didn’t notice them in the set of documents.)
Expert #6, an apparent CBT/GET hardliner, had particularly firm opinions about what was what: “This guidance has stood the test of time well, and has been instrumental in shifting thinking about the common problem of CFS/ME and medically unexplained symptoms (functional) symptoms in general. I hope that a new guidance on management of functional disorders will extend this process further.”
In response to the CFSAC suggestion that it was important to focus on biomarkers and diagnostic tests, #6 wrote this: “In the UK I think we would see what the HHS CFSAC see as a failure to undertake rigorous research, more as a failure of the biological model to explain the condition adequately. However, I can understand that an alternative reaction to the failure of biological models to explain the condition is to try and define a subset of patients with the condition who appear to share a common biomarker. This approach has failed so far.”
Of the six medical professionals, #7 adopted the most moderate view, suggesting that “some aspects of practice have moved on considerably so this guideline needs at least partial review in relation to interventions, diagnostics and prognostics.”
Moreover, #7 made some key suggestions about the guideline development group that should undertake these critical revisions: “It is important that the chair is someone who has no previous connection to CFS/ME practice or research. That person should not be a mental health professional if the guideline is going to be accepted by NHS clinics and patient organizations. However, I think there should be a mental health professional on the guideline development group.”
#7 was also the only one to offer specific concerns about PACE. Here’s what he/she/they wrote in discussing the research since the guidance came out in 2007: “There are some larger and more definitive UK RCTs of treatments recommended for CFS/ME by NICE in 2007 but these are controversial and one in particular PACE in over 600 patients, published in the Lancet, has been subject to legal challenge on the grounds that after the study was started the primary outcome was changed.”
That comment isn’t exactly accurate and it does not indicate a full awareness of the facts or of the widespread rejection of the PACE trial by the larger scientific community. But it’s more skepticism and concern than shown by his/her/their colleagues, who mostly toed the standard line and responded as you’d predict.
In response to safety questions, #3 provided more evidence—if any were needed—that the Cochrane reviews of GET and CBT continue to be major obstacles. Here’s what #3 wrote: “Whilst the patient community remain concerned about safety, recent Cochrane reviews suggest exercise therapy is safe and patients are more likely to improve with treatment.”
“recent Cochrane reviews suggest exercise therapy is safe “
So Cochrane says it’s safe, and that means the patients are wrong. And yet the reviews include the studies featuring the methodological flaws that patients—and the worldwide scientific community—are challenging in the first place. Reviews can’t provide good evidence if they’re based on bad studies, of course, but #3 did not point this out.
In the second questionnaire, the Dutch FITNET trial was one of two being considered as the subject of an in-depth commentary in the surveillance report. The questionnaire briefly described the study and asked for input. Here’s what NICE included in its description of FITNET in the questionnaire:
“CBT is already recommended by CG53 and this study shows it can be delivered via the internet, indicating an alternative mode of delivery for this treatment for this age group. Although the study was from the Netherlands, a UK trial is underway. There were some issues with the study (blinding was not possible due to nature of interventions, and usual care varied because the quality and quantity of CBT differed according to local availability and was often combined with other treatments such as GET). These issues would be discussed in full in the commentary.”
It must be pointed out that lack of blinding is not just some small problem with the Dutch FITNET study. The lack of blinding combined with the reliance on subjective outcomes renders FITNET, and the bulk of studies from the CBT/GET ideological brigades, incapable of providing reliable or valid evidence of treatment efficacy. Yet the topic experts generally agreed with NICE’s positive assessment of FITNET and largely failed to address the study’s serious flaws in their comments.
Among those problems was that the “usual care” group did very poorly. Unlike in PACE, usual care was not just a few meetings with a clinician. In FITNET, the usual care group mostly received CBT or GET, or both—in other words, the treatments already recommended for the illness in the Netherlands as well as in CG53. And yet apparently these treatments largely failed to work. That raises this critical question: If CBT and GET are performing so poorly, why are they the standard of care in the Netherlands, not to mention the U.K.?
In its comments about the study in the questionnaire, NICE fudged this enormous problem by suggesting that the usual care in the study “varied” in quality—suggesting that the treatments performed poorly because they were not optimally delivered. That’s a very convenient and self-serving explanation, when the obvious and simpler answer is that the therapies simply do not work as promised.
Citing this study, Professor Esther Crawley of Bristol University has received approval to inflict the FITNET approach on British children. But NICE, Professor Crawley, the topic experts and the Dutch investigators themselves have not actually considered one obvious reason why the patients assigned to the FITNET intervention might have reported better subjective outcomes. For sick patients, it would involve much less exertion to have online CBT at home than to run around town attending multiple in-person sessions. It seems logical that patients treated while in front of their laptops and not forced to travel weekly to clinic might do better than those in “usual care” for that reason alone.
In the document compiling the responses whether the 2015 U.S. reports should impact the diagnostic criteria used to define the illness in the NICE guidance, this comment from #1 jumped out as particularly uninformed: “I am unaware of any concerns about the inclusion criteria of trials in CFS.” Wow. Given the enormous international controversy over the range of case definitions that have been developed for the illness, it is hard to know how to respond to such a statement.
Not surprisingly, this person expressed satisfaction with the criteria outlined in CG53: “I do not see any need to change the diagnostic criteria at present. From a clinical perspective, CG53 are pragmatic and useful criteria. In my clinical experience in a CFS/ME clinic over the 10 years since the guidelines were published, no concerns have ever been expressed by patients attending the clinic about the diagnostic criteria used in CG53.”
And here’s what #6, the CBT/GET hardliner, had to say about the U.S.-based reports: “I am suspect [sic] that there is a tendency in the USA to push towards an entirely biological explanation for the condition, whereas in the UK there is an increasing acceptance amongst patients and clinicians alike of a model that includes CFS/ME in the umbrella of functional neurological disorders, i.e. that it is an emotionally driven disorder.”
CFS/ME – “an emotionally driven disorder”
Ok, then. The notion that CFS/ME is “an emotionally driven disorder” is certainly a popular concept in specific academic and medical circles in the U.K. But #6 does not cite any data to support the claim that there is an “increasing acceptance” of such an idea among British patients and clinicians. That this pet theoretical project of a powerful clique of psychiatrists (and some others) is now presented to NICE as having popular appeal and clinical legitimacy is distressing.
And that this set of comments is the best NICE could drum up in seeking external guidance for this critical review process is deeply worrying.
