Virology blog post, by David Tuller, DrPH, 28 August 2018: Trial By Error: Professor Edwards’ Letter to MRC’s Fiona Watt

Jonathan Edwards, an emeritus professor of medicine at University College London, recently sent a letter to Professor Fiona Watt, executive chairwoman of the UK Medical Research Council. The MRC was the main funder of PACE and has continued to defend the trial and its conduct. In recent years, Professor Edwards has been very involved in the effort to promote biomedical research and blunt the impact of PACE and related research. In his letter, he urged the MRC to publicly acknowledge that it was a mistake to fund the trial in the first place. Not surprisingly, Professor Watt declined to take such a step.

This week, The Times published a letter from Professor Watt about the PACE trial. She wrote the letter in response to last week’s Times article about the open letter to The Lancet, which was signed by more than 100 experts, ten members of Parliament, and 70 patient and advocacy organizations. The Times article was followed by an article in The BMJ on the issue.

Professor Watt’s letter reaffirms the MRC’s support for PACE. In the letter, she appears to imply that critics–perhaps including the signers of the open letter to The Lancet–are operating out of “hostility” toward the PACE authors. This is nonsense. As the main organizer of the open letter, I can say with confidence that signatories were motivated by dismay at PACE’s methodological and ethical lapses and the journal’s refusal to address them.

Professor Watt apparently thinks it’s fine for 13 % of a trial’s participants to already be “recovered” on a key measure at baseline–even as they were considered disabled enough on the same measure to enter the study. She apparently believes it’s fine for investigators to promise in their protocol to inform participants about any possible conflicts of interest–and then to fail to disclose serious conflicts. She also apparently believes that subjective outcomes in open-label studies are capable of providing reliable information, even though other fields of medicine have abandoned such evidence. Why Professor Watt believes any of this is unclear.

Professor Edwards signed the open letter to The Lancet. He originally wrote his own letter to Professor Watt in confidence. Since she has now strongly endorsed PACE in her letter to The Times, he has given me permission to post what he wrote. (I have edited the letter slightly to remove references to others.)

**********

20th July 2018

Dear Professor Watt,

[Name deleted] asked me to comment on your letter of 9th July to his MP, Jeremy Quin, regarding funding for ME/CFS research and concerns about the PACE trial. As it happens I was about to contact you anyway with regard to the MRC position on PACE…

I think that collectively we have to face up to just how big an error setting up PACE was, because it continues to have severe adverse effects on both clinical care and research that will go on until we do.

In your letter to Mr Quin you say of PACE ‘I do believe that the trial was designed, conducted and overseen in accordance with expected standards at the time…’ I am hoping that you are not fully familiar with the design of the trial and its problems and wonder if you have been poorly briefed. You mention the problems of trials where it is impossible to anonymise either the patient or the clinician involved. The actual problem is ‘anonymising’, or blinding, treatments – i.e. not labelling them as ‘the good new treatment’ or ‘the usual old nothing much’. Not only did the PACE authors not try to conceal which was which, but they emphasised it in quite unusual ways – including both information sheets and a newsletter during the trial.

This problem has nothing to do with ‘expected standards at that time’. It has to do with something we as scientists had drummed into us as students. If your assessment of results, whether in a clinical trial or (in my case) scoring cells in a tissue section, is open to subjective bias, then you have to blind yourself to whether you are scoring ‘test’ or ‘control’. If you cannot blind yourself to that then you have to make use of objective measurements. Otherwise your data are valueless.

It might be argued that a basic truth about how human nature colours scientific observation was not known to clinical triallists in 2004, but it was. At the time PACE was being planned I was publishing my proof of concept trial for rituximab in rheumatoid arthritis in NEJM and had been involved in trials for over a decade. That trial included cyclophosphamide, which cannot be blinded. Everyone involved was aware of the problem and the potential solutions – solutions that the PACE team made no attempt to make use of. With a trial the size of PACE you only need slight systematic bias to get statistically significant differences. Multicentre trials are a big problem for bias because peripheral centre staff think they are ‘helping’ by feeding in ‘positive’ results. These basic realities were all too familiar to those of us doing trials.

On this basis alone my expectation is that if you asked anyone else heavily involved in trials around that time, such as Ravinder Maini or Bob Souhami, they would agree that PACE was nowhere near expected standards for 2004, or even 1984, in terms of being capable of producing usable positive evidence of efficacy. No drug trial using this format would have been publishable in a quality journal – so why a trial of therapist-delivered treatment, where bias problems were known to be worse? Perhaps psychiatrists were ignorant of trial standards, but surely, ignorance of established rules of practice is no defence. The argument that if you do not know how to get interpretable results you get uninterpretable results and treat them as interpretable clearly does not wash.

And that is just the first layer of the problem. The second layer is that the PACE trial suffers from the problems of subjective bias in worst-case scenario terms. The two treatments that purportedly came out better involved training the patient to take on a mindset of being better. The primary outcome measure was a questionnaire and it is hardly surprising that patients in those two groups said they were better. People do what they are told. In the other ‘test’ arm patients were trained to accept their condition and cope with it by pacing. The comparator arm was not a meaningful control because it was explicitly ‘nothing more than usual’.

In other words, the subjective biasing that in most experiments we try to minimise by recognising our tendency to cherry pick was, in PACE, the intended mechanism of the treatment. This should have been obvious to psychologists!

I think it is significant that the recent ‘SMILE’ trial of an alternative therapy, which also trains patients to think they are better, produced a similar result to CBT and GET. We do not know whether any of these treatments have a specific effect. (If PACE shows anything, it is that there is no useful effectl: no return to work, no reduction in benefit claims, no increased activity and not even a subjective difference at two and a half years.) There is nothing like a dose response curve. We are left knowing pretty much nothing, as was predictable.

There are all sorts of other issues about the trial that make it very difficult to maintain that ‘the authors made every effort to ensure that research was conducted to a high standard’. For instance, an objective measure of activity could have been built into the primary outcome in the way the American College of Rheumatology measure for arthritis trials combined objective and subjective components using multiple thresholds. But we have minutes from a PACE committee meeting where it was decided to abandon such a measure for PACE because previous studies had not shown a positive change in response to treatment. And so on…

I do not see how we can escape the conclusion that the supervision of PACE was not competent. Some people may have to eat humble pie but too much is at stake for that to be a consideration. The lapse seems surprising but may be explained by too much focus being put on statistical and structural issues and not enough on practical psychological realities…

In the longer term, PACE continues to have a disastrous effect on clinical care, equally relevant to research. It seems likely that treatments are being provided that do not work and cause distress. PACE is a major prop for the £1B expansion of so-called evidence-based therapies proposed now not just for ME but for any unexplained symptoms. The more I see the more I suspect none of this ‘evidence’ means much. Even Simon Wessely, who helped set PACE in train, is looking on, like the Sourcerer’s Apprentice, as PACE is used to underpin subcontracting care to providers whose staff are not even formally trained in CBT, let alone have useful knowledge of the illness. Commissioning groups are dispensing with physician contact. Whereas in the past physicians like Stephen and I could gain experience with the clinical picture and ponder possible causes we are faced with a future in which nobody even knows what the problem is that requires scientific input.

NICE are forming a new committee to reassess guidelines for ME/CFS. The outcome of that re-assessment will depend on whether or not evidence quality is put foremost. Cochrane are now realising that all has not been well with systematic reviewing for ME/CFS but re-commissioning reviews will take time. Despite the obvious failings of PACE opinions are still heavily influenced by the stance of establishment bodies like Lancet and MRC. PACE was not competent science and I do not think it is ethical to continue to defend it as such.

Above all, we need that trust and respect. Both patients and scientists need to feel that there is some form of quality assurance in the science. And the only way I see it coming is if the MRC makes a public statement acknowledging that by any reasonable view of scientific standards the sponsoring of PACE was a serious misjudgement that should have been foreseen. I would like to make a formal, but private, request that such a statement be made. Trust and respect from patients is paramount, but trust and respect within the scientific community is also critically important. It could be achieved very simply.

Yours sincerely,

Jonathan Edwards
Professor Emeritus
Department of Medicine
University College London