Profs Vincent Racaniello, James Coyne and Klaas van Dijk comment on the need for the release of PACE trial data to check the claims for cost-effectiveness.
Revisiting the PLoS One economics analysis of PACE by Vincent Racaniello
22 December 2015
On October 23rd, virology blog published the third installment of David Tuller’s investigative report about the PACE study of treatments for ME/CFS. In the post, Dr. Tuller demonstrated that the key finding of an economic analysis of the PACE trial, published in PLoS One in 2012, was almost certainly false. The finding–that cognitive behavior therapy and graded exercise therapy were cost-effective treatments–relied on an inaccuracy in the paper about whether the results of sensitivity analyses were “robust.”
Since the publication of the virology blog series, the PACE study has come under sustained and blistering public criticism for its many flaws. The PLoS One paper is currently the center of attention as a result of the researchers’ insulting response to Dr. James Coyne, a well-known psychologist and PLoS blogger. Dr. Coyne requested data to verify the results from the PLoS One paper, and was told that his request was “vexatious.” The researchers have called patients “vexatious” for years, of course, but the effort to use this strategy against a respected researcher has caused an uproar. Several colleagues and I, including Dr. Tuller, cited this rejection recently in our own request for a different set of PACE-related data.
Because of the open data policies of the PLoS journals, requesting data on that basis was a smart move by Dr. Coyne, and he has done a brilliant job of rousing support for the larger issue of access to data in scientific research. The PACE authors must recognize by now that at some point they will have to release all of their data.
The PLoS One study reported that cognitive behavior therapy and graded exercise therapy, the two treatments long favored by the main investigators, were more cost-effective than other approaches. The investigators have routinely cited these findings in promoting use of the two treatments. The truth or falseness of these claims from the PLoS One study are at the heart of the current controversy
In fact, it is already clear that the claim is highly unlikely to withstand serious scrutiny, based on the public record. In the October 23rd post, Dr. Tuller demonstrated that subsequent public comments of the lead author contradicted a critical statement in the paper about the PLoS One study’s sensitivity analyses..
The relevant excerpt from virology blog is below:
“In another finding, the PLoS One paper argued that the graded exercise and cognitive behavior therapies were the most cost-effective treatments from a societal perspective. In reaching this conclusion, the investigators valued so-called “informal” care—unpaid care provided by family and friends–at the replacement cost of a homecare worker. The PACE statistical analysis plan (approved in 2010 but not published until 2013) had included two additional, lower-cost assumptions. The first valued informal care at minimum wage, the second at zero compensation.
The PLoS One paper itself did not provide these additional findings, noting only that “sensitivity analyses revealed that the results were robust for alternative assumptions.” Commenters on the PLoS One website, including Tom Kindlon, challenged the claim that the findings would be “robust” under the alternative assumptions for informal care.
In fact, they pointed out, the lower-cost conditions would reduce or fully eliminate the reported societal cost-benefit advantages of the cognitive behavior and graded exercise therapies.
In a posted response, the paper’s lead author, Paul McCrone, conceded that the commenters were right about the impact that the lower-cost, alternative assumptions would have on the findings. However, McCrone did not explain or even mention the apparently erroneous sensitivity analyses he had cited in the paper, which had found the societal cost-benefit advantages for graded exercise therapy and cognitive behavior therapy to be “robust” under all assumptions. Instead, he argued that the two lower-cost approaches were unfair to caregivers because families deserved more economic consideration for their labor.
“In our opinion, the time spent by families caring for people with CFS/ME has a real value and so to give it a zero cost is controversial,” McCrone wrote. “Likewise, to assume it only has the value of the minimum wage is also very restrictive.”
In a subsequent comment, Kindlon chided McCrone, pointing out that he had still not explained the paper’s claim that the sensitivity analyses showed the findings were “robust” for all assumptions. Kindlon also noted that the alternative, lower-cost assumptions were included in PACE’s own statistical plan.
“Remember it was the investigators themselves that chose the alternative assumptions,” wrote Kindlon. “If it’s ‘controversial’ now to value informal care at zero value, it was similarly ‘controversial’ when they decided before the data was looked at, to analyse the data in this way. There is not much point in publishing a statistical plan if inconvenient results are not reported on and/or findings for them misrepresented.””
Formal request to PLOS One to issue an Expression of Concern for PACE cost-effectiveness study, by James C Coyne, December 12, 2015
Kudos to Dutch research biologist Klaas van Dijk. for his letter to PLOS One:
Dear Editors of PLOS One,
Attached is a formal letter from King’s College in London (UK) in which it is stated that King’s College in London is unwilling to release to Professor James Coyne (in cc) the primary research data of http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0040808
Such a refusal to give other researchers access to the raw research data of a published paper is a clear violation of your own rules in regard to free access to all primary research data of papers published in PLOS ONE for other researchers (so including Professor Coyne of the University of Groningen, The Netherlands [RUG]).
I therefore suggest you immediately issue an Expression of Concern attached to the paper in which it is clearly articulated why you have concerns about this paper. I also suggest the paper be retracted if Professor Coyne has not received full access to all raw research data of http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0040808 within one month.
See http://www.nrc.nl/nieuws/2015/07/01/universiteit-integriteit-in-geding-bij-taalfoutonderzoek for the consequences in the Netherlands when researchers at RUG are unwilling to share raw research data, published in a PhD thesis, to others.
Thanks in advance for a response in which you indicate how you have decided to respond.
Best wishes,
, Groningen, The Netherlands
Why I don’t know how PLOS will respond to authors’ refusal to release data Dec 13 2015
Update on my formal request for release of the PACE trial data Dec 4 2015
Recognizing when “protecting patient privacy” is mere excuse for not sharing data Dec 22 2015
5 thoughts on “Formal request to PLOS One to issue an Expression of Concern for PACE cost-effectiveness study”
Many thanks for this. It’s great to see other scientists supporting you in your attempts to help patients protect themselves from bad science. I hope other scientists will add their voices.
Science needs to get its house in order. It would be great if PLOS One was the first scientific institution to uphold principles of good science in relation to PACE. The Lancet, Psychological Medicine, Lancet Psychiatry, the MRC, Queen Mary University of London and Kings College London have all let themselves down badly.
The PACE authors are dragging the reputation of this £5 million, taxpayer-funded trial through the dirt and it’s time they were stopped.
Step up, PLOS One! Protect science and patients!
From: Formal request to PLOS one to issue an expression of concern for PACE cost-effectiveness study
2010 Jan 29;340(8):c181.
The BMJ article notes
“Many peer reviewed journals now require authors to be prepared to share their raw, unprocessed data with other scientists or state the availability of raw data in published articles, but little information on how such data should be prepared for sharing has emerged. Iain Hrynaszkiewicz and colleagues propose a minimum standard for de-identifying datasets to ensure patient privacy when sharing clinical research data”
Among others, Ivan Oransky, Co-Founder, Retraction Watch emailed me and queried whether King’s College had requested that I sign a confidentiality agreement. Of course they had not asked me to sign any agreement.
After I gave the same answer to another inquiry, the other person replied: “They should have at least offered you to sign a confidentiality agreement, not turn it into a FOIA request and then reject it with insults.”
I am not sure why a confidentiality agreement would be needed to provide the data from article from the PLOS article to me or anyone else who requests it.
In this blog post I present my analysis of why raising the issue of protecting patients as it as been raised in the press release is simply a distraction. When you finish the brief analysis, you may be tempted to consider King’s College raising the issue of protecting patients in this context as hypocritical and sanctimonious.
The “PACE investigators” group needs to be unpacked as the specific professionals who will are doing unprofessional things. The seven authors of the PLOS One article are not living up to their promise to share data are Paul McCrone, Michael Sharpe, Trudie Chalder, Martin Knapp, Anthony Johnson, Kimberley Goldsmith, Peter White.
To summarize my analysis of the PACE investigators citing “protecting patient privacy” as a reason for continuing to withhold the data:
– The larger PACE investigator group recklessly endangered patient privacy with careless practices, that at least in the United States, could subject them to harsh penalties and legal action by participants in the PACE trial.
– The claim that PACE investigators have already shared data with “independent” investigators refers to a situation in which they share data with close cronies in a manner in which they could control presentation and interpretation of the data consistent with their previously stated objective of protecting their reputations from scrutiny of their work.
Unlocked drawers and a serious data breach: Patients are already potentially harmed by carelessness of PACE investigators
The press release from King’s College suggests that due care has been taken to ensure the confidentiality of patient data.
“We are however also concerned for the rights and welfare of trial participants. Participants did not give consent to the public release of their data when they entered the trial. In particular we are concerned to ensure that there is no risk of misuse of the data such as through inadvertent personal identification. The scientists who have already received data have all signed a formal confidentiality agreement, approved by the independent PACE Trial Steering Committee, which required that they respect the confidential nature of the data, and keep them secure, as agreed with trial participants when they consented to take part.”
However, a shocking instance of data being stolen from an unlocked drawer reveals both the bad research practices of the PACE trial and a casual attitude towards patient data being leaked to unknown persons.
Margaret Williams, whom I do not know, left the following information as a comment on my previous blog post. The official documentation of the incident is available here.
“Professor Peter White is claiming that he promised to keep patients’
data securely; has he forgotten that some of this data was not held securely? The following extract comes from “Magical Medicine: How to make a disease disappear” (p256):
Click to access magical-medicine.pdf
On 31st March 2006 Peter White wrote to the West Midlands Multi‐centre Research Ethics Committee to inform them of the theft of a digital audio recording (DAR) of GET sessions from Centre 03 (which is King’s College, ie. Trudie Chalder’s Centre). This confidential information was stolen from an unlocked drawer in the therapists’ office. Peter White informed West Midland MREC that: “There are no lockable cabinets in any of the therapists’ rooms so the drawer was not locked” (cf SSMC Participant Information Sheet). His letter continued: “The burglary was reported to Southwark police on the day that it happened, which was Wednesday 22nd March 2006. The crime number is 3010018‐06. The therapist was away on leave 22nd‐24th March and therefore the DAR was not found to be missing until Monday 27th March 2006”. It was only after the theft that Professor Trudie Chalder sought advice on how to secure the data properly.
The letter also said: “The Principal Investigator for this centre, Professor Trudie Chalder, is awaiting advice from the Trust R&D as to whether the affected participants should be made aware of the theft”.
The same letter stated that recordings were being downloaded to CD only on a monthly basis, a working methodology that is not compatible with the promises of confidentiality set out in the “Invitation to join the PACE trial” leaflet.
The letter carries a handwritten annotation dated 13th April 2006:
“Noted. Sad! No action needed”.
It seems that the patients involved were not warned that confidential information about them had been stolen.
………………
Additional note: The data stolen were not backed up. The same letter from Peter White to the MREC makes this clear: The DAR “contained six sessions recorded with five participants which had not been downloaded at this time, and therefore is not available for review.””
Cronyism: Yes, the data are available – to family and friends
The press release from King’s college states “We have supplied requested, anonymised data to independent scientists […] [T]hese include a Cochrane review group who have independently validated the trial findings on the primary outcomes, and who are now undertaking an individual patient data meta-analysis using these data.”
Let’s take a look at the proposed “individual patient data
meta-analysis”- Larun L, Odgaard-Jensen J, Brurberg KG, Chalder T, Dybwad M, Moss-Morris RE, Sharpe M, Wallman K, Wearden A, White PD, Glasziou PP. Exercise therapy for chronic fatigue syndrome (individual patient data) (Protocol). Cochrane Database of Systematic Reviews 2014, Issue 4. Art. No.: CD011040.
Note that a number of the PACE investigators are included as authors on this protocol, as well as authors of the sister trial to PACE, the Fatigue Intervention by Nurses Evaluation (FINE) study.
As authors on the resulting systematic review article, the PACE investigators have veto power over what is said and they are among friends and family.
Hmm, comfortably among “friends and family” in a way that invokes Tony Soprano’s uses of the term.
I’ll have to take up the issue in another blog post, but this Cochrane review group seems to be a serious breach of what has been a long established policy of the Cochrane Collaboration. But more basically, a systematic review by design should attempt to overcome the risks of bias in the individual studies that are included. The Cochrane Collaboration previously gave my colleagues and me the Bill Silverman Award for an article that cause them to tighten their risk of bias associated with investigator conflict of interest. And we know that systematic reviews and meta-analyses conducted by those with an investment in a particular treatment produce exaggerated and less valid estimates of effect sizes of the investigators.
