Comparative Clinical Effectiveness: Some Thoughts

The book, Comparative Effectiveness Research: Evidence, Medicine, and Policy by Ashton and Wray is an exceptionally well written presentation of the issues and politics behind CER through the current time. As the authors state, there are many definitions of what CER or its implementation as Comparative Clinical Effectiveness (“CCE”) is. A simple but reasonable definition is that CER is the process of examining various modalities of medical treatment for some specific disease state and the determination of what one may be the most effective according to some metric (See pp 122-123 for various definitions).

CCE is then the dissemination of the results of the CER and its use in day to day clinical practice. An additional condition on CER is that unlike FDA randomized clinical trials the CER is conducted in “real life” clinical environments. In the current ACA the CER has been embodied in a non-Governmental entity called PCORI. This book ultimately is a justification of this entity, its purpose and goals and the very nature of its embodiment. Fundamentally the authors make presentations which covers the need for CER/CCE and then provide an exceptionally detailed and well written expose of how it was ultimately implemented in the ACA and why.

Let me first summarize the material presented and then I will provide a critique on it and the approach taken by the authors.

Part I of the book covers the principles of evidence based research and practice. Part II presents the political story of the development of PCORI and the institutionalization of CER/CCE under the ACA. Part III is the authors’ presentation of their views on the politics and the implementation and usefulness of CER/CCE.

On p 23 the authors use the prostate cancer (“PCa”) studies as a vehicle to discuss CER. This relates to procedures which are used to treat PCa. PCa can be treated in a variety of means, ranging from surgery, radiation therapy, nuclear implants, proton beam, and the cheapest, called watchful waiting. The problem with PCa is that about 90% of it is indolent, namely the patient will never die from it and the treatment is worse than the disease. However some 10% or so, we really do not know the number is aggressive. The typical case goes from a PSA of 4 to 40 in two years and 40 to dead in another two. The challenge is how do we save the 10% while not making the 90% have a higher morbidity? The therapies all too often depend on who the patient see and not based upon any clinical evidence of efficacy or patient satisfaction, the Patient Reported Outcomes (“PRO”). The problem the authors have here is that the problem is more complex than how they present it and it is that complexity that the physician does or should intermediate with.

On p 51 the authors again come back to PCa. Now they address the issue of robotic surgery. It is costly, is advertised, and results in less than clear advantages. Now for clarity purposes the robot is not really some autonomous entity. It merely allows for better controlled movements under the total control of the surgeon. Prostatectomies are difficult because of all the nerve involvement and one prostate is not necessarily the same as all others. The intent is to remove all the malignancy while retaining nerve functions. Frankly a superb surgeon can do this without any “robot” and usually do. The problem is that the patients may not have a clue and somehow want the “robot”. Thus the authors make a good argument for CER/CCE being targeted at not only the physicians but the patients.

Chapter 3 is an excellent expose of FDA assessment of therapeutics. The FDA has over the last half century developed a set of scientifically based procedures to ascertain the safety and efficacy of drugs. This is the randomized placebo based clinical Trials. There generally are three phases before approval and hopefully by the end of Phase III we can rely on the therapeutic doing what is says and not having any morbidity or mortality effects. Sometime that is not the case since the Trials are small and may not be large enough for some of the negative effects. The authors explain this in superb detail.

Chapter 4 will make all surgeons happy. Not really. It is a discussion on the fact that all too often surgeons perform procedures for which there may be little if any clinical evidence. To take the authors back even further, in the 40s and 50 tonsillectomies were almost a child’s rite of passage. The Table on p 82 is a good example of the many surgeries performed where there is little if any evidence of efficacy.

P 103 discusses the problems with imaging. I remember in 1972 when the second CAT system was introduced to MGH. It allowed a neurologist to ascertain a block of bleed stroke. It was quick and saved lives. The problem now is that when a CAT is performed, along with massive amounts of radiation, one can find “incidental” observations which then demand follow up despite no clinical evidence of necessity other that shadows.

On p 113 the authors commence discussing the creation of clinical guidelines.  The problem, however, is that all too often medical knowledge changes so quickly that the physician is incorporating this knowledge in the way they treat a patient. On the other hand the compilation of CCE procedure such as that proposed by the USPTF is based upon oftentimes decade’s old data which was predicated on now outdated medical practice. This is the fundamental flaw of CCE guidelines. They are outdated and often too static. If then developed by Governmental entities they may also often end up becoming the least common denominator.

On p 130-131 the authors make the attempt in discussing the concept of “producing a public good” as the basis for arguing that only the Government can be the creator and manager of CER/CCE. Frankly this is the weakest part of the book and would best have been left aside. However it does provide a vehicle to understand from whence the authors come at this issue.

The Part II discussions are quite interesting. The authors clearly present the Washington drama behind the development of the PCORI side of the ACA, pre current Administration and in the current Administration. For anyone interested in Washington politics this compares well with Woodward and his presentations. The influence of the various stakeholders is discussed and for those not familiar with Washington the key role of Congressional Staff is portrayed effectively.

On p 193 as I comment on later, the authors go off on Rush Limbaugh. Clearly the ACA was a political issue and received not a single Republican vote. Yet the Limbaugh discussion has a question of relevancy. They continue on p 195 regarding the death panel issues etc. Inflamed words are all too often used in politics and one must ask; what was the basis of this phrase? To some degree, being part of the discussion at the time, it related directly to CCE, and the NHS use of QALY measures in the UK. Namely if one examines the PCa problem, then using a USPTF result and the recommendation to no use PSA on any many over 75, and also mammograms in women over 75, despite what health they may be in and despite any family predispositions, was sending a chilling effect to those affected.

Chapter 10, p 208 and on, is an excellent presentation of PCORI, the non-Governmental entity set up to do CER and assist it to be converted into CCE recommendations. Any reader interested should go to their web site as well. It is too early to determine what they will accomplish but with a $500 million a year budget paid for by the taxpayers they should have no excuses. However when one looks at the FDA, it took then a good half a century to reach the point where they are today. The FDA still has some issues but it is the sine qua non entity in the world. It has developed methodologies and procedures which albeit costly had led to better safety and efficacy in devices and pharmaceuticals. PCORI lacks many of the elements of the FDA and furthermore it is not clear that it can draw on the competencies that exist in the field. The author’s speak of what it can do but fail to discuss just how it will go about it.

As the authors reach their concluding comments they seem to go into niches that may or may not be warranted. On pp 248-249 there is a discussion of the public and scientific evidence. They return to their attack on the “political Right” and its “anti-science” philosophy. The problem they are discussing is that of educating patients as well as physicians. Again take the PSA issue. What should a general physician know and what should a patient know? Here we have the problem that PSA management is recording test data over significant time spans. Assume starting at 40 years of age baselines should be take and then data analyzed periodically. It is thus not any single reading, as is the case with BP or fasting blood glucose, but the temporal behavior. However physicians do not have the data and patients do not understand it. It is not because patients are politically influenced to not believe it, the Press oftentimes has a stronger confusing influence, but because they do not get the information. Thus the discussions on these pages warrant attention but not the way the authors provide it.

On pp 250-251 the authors sink into the Medicare swamp. First to make some statements of fact, more than 45% of those on Medicare have or will have paid in more than they will ever receive in benefits. 55% of participants will have paid in substantially less and will have received a dramatically greater share. Medicare is an insurance plan and not a Government benefit, albeit a bit poorly managed. On p 251 the authors say, “Medicare is a federal health insurance program, tax-supported, that was established in 1965” It is tax supported only for that portion of subscribers who have not made enough in their lifetimes. It is subscriber supported by those who have made enough and who continue to make money after 65. This is a distinction and a difference that many fail to understand. It also colors the perception of Medicare of many.

One topic that seems not to have been discussed is that of Patient Reported Outcomes (“PRO”), as an integral part of CER. PROs are complicated testing methodologies to ascertain what the patient felt was the result of a procedure. It should be an integral part of any of the CER efforts. I was disappointed that they failed to focus on this area.

To examine CER/CCE in some specific detail, let us consider the Prostate Cancer debate. The key observation that one should make is that what we understand in Medicine is changing almost daily so what we think today is common practice will be unacceptable in short order. In 2009 there were two papers published in NEJM, one American and one European, which stated that there was no significant different in death rates between men who were PSA screened and those who were not. Now one can examine the data and see some fundamental flaws, after the fact. The study regarded a PSA level of 4.0 as the point at which one should take action. That was acceptable in 1995 but by 2009 we knew that velocity of PSA, % Free PSA, volume adjusted PSA, as well as age adjusted PSA were more critical. Thus, the reports using a 1995 standard presented results that were devoid of 20 years of information.

More importantly, the question posed was: "Using a PSA of 4.0, and using a PSA testing methodology on a biennial basis, did such a testing procedure have an effect upon mortality?" The question which should have been asked was: "Is there a PSA screening methodology which when adjusted for patient classes results in a material change in patient PCa mortality?" The first question was good for 1995 with the limited knowledge available then. The second question opens the door for expanded insight into temporal PSA evaluation. The problem with many CCE trials is that they fixate on the wrong question and thus produce potentially deleterious rules of treatment.

Furthermore as the reports were published a plethora of genetic markers were being developed. Disregarding all of this, the USPTF then graded PSA testing as a D, a level which would prohibit Medicare patients from receiving its benefit. This type of CER/CCE demonstrates the fundamental flaw in CER/CCE in toto. It uses clinically collected data which may be flawed based upon the significant changes in medical knowledge over the time frame of the study and thus potentially inflicting a lethal sequella upon patients. This may not be a death panel but it is flawed medicine.

Overall the authors present a clear picture of CER and its positive benefits. However there are two basic flaws in their presentation. First they fail to fully discuss and disclose the fundamental flaws that may be part of CER/CCE. Second, they fill their presentation with comments as regards to conservative opponents which may add some color but on the other hand actually colors their own attempt at a professional presentation. Frankly the book would have been much better served without references to Rush Limbaugh and the negative comments regarding conservatives as a whole.

Finally one should more carefully examine PCORI. It is a non-Governmental entity like Ginnie Mae and Fannie Mae, albeit not a financial entity, but it has a $500 million annual fund taken from taxes on insurance plans. Recently PCORI awarded NIH a $5 million contract to study their PRO, patient reported outcome questionnaire, which has been around for a while. What is especially strange is why the Government did not fund NIH directly. In this case the taxpayer pays a tax to the insurance company, which hands it to the IRS, which gives it to The Comptroller of the Currency, which hands it over to PCORI who awards it to NIH who now has to manage the contract as external funding. One estimate is that the $5 million funding may result in $4 million of work at a costs of $12.5 million! Only a Congressionally mandated schema like this would produce such a result. That perhaps is a dark side of the PCORI effort.