In Cancer – If It Seems Too Good to Be True, It Probably Is

The panoply of genomic tests that have become available for the selection of chemotherapy drugs and targeted agents continues to grow. Laboratories across the United States are using gene platforms to assess what they believe to be driver mutations and then identify potential treatments.

Among the earliest entrants into the field and one of the largest groups, offers a service that examines patient’s tumors for both traditional chemotherapy and targeted agents. This lab service was aggressively marketed under the claim that it was “evidence-based.” A closer examination of the “evidence” however, revealed tangential references and cell-line data but little if any prospective clinical outcomes and positive and negative predictive accuracies.

I have observed this group over the last several years and have been underwhelmed by the predictive validity of their methodologies. Dazzled by the science however, clinical oncologists began sending samples in droves, incurring high costs for these laboratory services of questionable utility.

In an earlier blog, I had described some of the problems associated with these broad brush genomic analyses. Among the greatest shortcomings are Type 1 errors.  These are the identification of the signals (or analytes) that may not predict a given outcome. They occur as signal-to-noise ratios become increasingly unfavorable when large unsupervised data sets are distilled down to recommendations, without anyone taking the time to prospectively correlate those predictions with patient outcomes.

Few of these companies have actually conducted trials to prove their predictive values. This did not prevent these laboratories from offering their “evidence-based” results.

In April of 2013, the federal government indicted the largest purveyor of these techniques.  While the court case goes forward, it is not surprising that aggressively marketed, yet clinically unsubstantiated methodologies ran afoul of legal standards.

A friend and former professor at Harvard Business School once told me that there are two reasons why start-ups fail.  The first are those companies that “can do it, but can’t sell it.”  The other types are companies that “can sell it, but can’t do it.”  It seems that in the field of cancer molecular biology, companies that can sell it, but can’t do it, are on the march.

Type I Error

Scientific proof is rarely proof, but instead our best approximation. Beyond death and taxes, there are few certainties in life. That is why investigators rely so heavily on statistics.

Statistical analyses enable researchers to establish “levels” of certainty. Reported as “p-values,” these metrics offer the reader levels of statistical significance indicating that a given finding is not simply the result of chance. To wit, a p-value equal to 0.1 (1 in 10) means that the findings are 90 percent likely to be true with a 10 percent error. A p-value of 0.05 (1 in 20) tells the reader that the findings are 95 percent likely to be true. While a p-value equal to 0.01 (1 in 100) tells the reader that the results are 99 percent likely to be true. For an example in real time, we are just reporting a paper in the lung cancer literature that doubled the response rate for metastatic disease compared with the national standard. The results achieved statistical significance where p = 0.00015.  That is to say, that there is only 15 chances out of 100,000 that this finding is the result of chance.

Today, many laboratories offer tests that claim to select candidates for treatment. Almost all of these laboratories are conducting gene-based analysis. While there are no good prospective studies that prove that these genomic analyses accurately predict response, this has not prevented these companies from marketing their tests aggressively. Indeed, many insurers are covering these services despite the lack of proof.

So let’s examine why these tests may encounter difficulties now and in the future. The answer to put it succinctly is Type I errors. In the statistical literature, a Type I error occurs when a premise cannot be rejected.  The statistical term for this is to reject the “null” hypothesis. Type II errors occur when the null hypothesis is falsely rejected.

Example: The scientific community is asked to test the hypothesis that Up is Down. Dedicated investigators conduct exhaustive analyses to test this provocative hypothesis but cannot refute the premise that Up is Down. They are left with no alternative but to report according to their carefully conducted studies that Up is Down.

The unsuspecting recipient of this report takes it to their physician and demands to be treated based on the finding. The physician explains that, to his best recollection, Up is not Down.  Unfazed the patient, armed with this august laboratory’s result, demands to be treated accordingly. What is wrong with this scenario? Type I error.

The human genome is comprised of more than 23,000 genes: Splice variants, duplications, mutations, SNPs, non-coding DNA, small interfering RNAs and a wealth of downstream events, which make the interpretation of genomic data highly problematic. The fact that a laboratory can identify a gene does not confer a certainty that the gene or mutation or splice variant will confer an outcome. To put it simply, the input of possibilities overwhelms the capacity of the test to rule in or out, the answer.

Yes, we can measure the gene finding, and yes we have found some interesting mutations. But no we can’t reject the null hypothesis. Thus, other than a small number of discreet events for which the performance characteristics of these genomic analyses have been established and rigorously tested, Type I errors undermine and corrupt the predictions of even the best laboratories. You would think with all of the brainpower dedicated to contemporary genomic analyses that these smart guys would remember some basic statistics.