New developments in scientific thought and hypotheses testing are changing the ways we think about truth and certainty.

Not only do scientific decisions rely heavily on tools and procedures of quantitative analysis, but also on social life and daily decisions. I want to start with a story from the judiciary system about the misuse of an important quantitative tool of the scientific community, Hypothesis Testing. It’s about the overturned Amanda Knox case in Italy. In 2007, she was accused of killing her roommate and sent to jail. The odds of DNA matching in the case of accidental death were reported as incredibly rare by a forensic data analyst, who examined data with regard to incidents of crime, concluding that the accident is not statistically an accident. The judges then made their decision based on this rareness, a single value, named p-value. Later, this was overturned (New York Times, March 27, 2013), because of a misinterpretation by the judges and lawyers, of the rare probability. What factors forced the judiciary system to make this decision based on a single value alone? Honestly, this is the story of the century and a story of Hypothesis Testing. This article introduces the logic behind their decision, draws attention to the misuse of data, and mentions alternative logics to this traditional approach on the matter (in March 2013, Italy’s Supreme Court ordered a retrial and found the two suspects guilty on January 30, 2014).

What is Hypothesis Testing?

We live in a data-driven world. Statistical inference is the process of drawing conclusions or decisions from data. The Hypothesis Testing procedure in the Neyman-Pearson paradigm is one of the procedures of this process, widely used in the scientific community for over a century. In this mindset, two complementary hypotheses are first defined: one is the conventional thesis (the null hypothesis) accepted to be true by default; the other is the alternative thesis (the alternative hypothesis) that needs evidence from data to falsify the null hypothesis. It results in a single value, called p-value, which is calculated from the data using theoretical model assumptions. A p-value is a measure – in probability sense, ranging from 0% to 100% – of how much evidence you have against the null hypothesis. The smaller the p-value, the more evidence you have against the null hypothesis. One may combine the p-value with the significance level to make decisions on a given hypothesis. This is also called significance testing. It has an accept-reject mindset (dichotomous decision or binary logic); in such a case, if the p-value is less than some threshold (usually .05) then the null hypothesis is rejected. Basically, it is a black or white decision, without considering the contrasts between them. This interpretation has been widely accepted in the twentieth century of scientific research, and many scientific journals routinely publish papers using this interpretation for the results of hypothesis tests, even though there are current tendencies not to use this. Let’s take a look at the history of this black-white logic.

History

The history of the process of hypothesis testing starts with Fisher (1890-1962), around the early twentieth century. Later, the contributions of Pearson (1857-1936) and Neyman (1894-1981), who were early fathers of statistics, about the interpretation of the hypothesis tests were integrated into present-day applications. Fisher’s approach focuses on inductive inference about a single hypothesis, whereas the Neyman-Pearson approach informs future behavior based on a test using two complementary hypotheses (Newman 2007).

These approaches are strongly influenced by Popper’s logic of falsification. Falsification can be defined as the act of disproving a proposition, hypothesis, or theory. This logic asserts that sufficiently improbable events can be considered impossible. A p-value suggests whether a null hypothesis is sufficiently improbable to be considered practically falsified in the sense of a logical refutation (Newman 2007). When we come back to the Amanda Knox case, the decision the judges made and justified is followed from this falsification logic in hypothesis testing. Another criticism with this mindset is: how fair it is to rely on a single value that yields a rare result? Are we going to generalize one lucky drawing of raffle tickets to believe all tickets are winners?

Logics and proofs

We use logical flaws and biases in our daily life. Finding the logical flaw in scientific papers is something most readers aren’t interested in. Instead, the results are believed as a fact. One of the widely used flaws/biases is in seeking or interpreting evidence in ways that are partial to existing beliefs or a hypothesis in hand (Mercie and Sperber, 2011). We tend to prove or show the accessibility or superiority of arguments by bringing evidence. Failure to prove that a treatment – say, a low fat diet – is effective is not the same as proving it is ineffective.

Let me give another example to clarify this: The New York Times editorial news reported on February 9, 2006, “Millions of Americans have tried to reduce the fat in their diets, and the food industry has obligingly served up low-fat products. Yet now comes strong evidence that the war against all fats was mostly in vain.” Actually, in the hypothesis testing mindset, the evidence collected from research should either reject the null hypothesis which is “diet is ineffective,” or fail to reject it. The mindset in testing is not about finding evidence to support the null statement. It is also not about proving the alternative hypothesis, “diet is effective.” Rather, it is basically looking for evidences to falsify the null statement. In the report, the “evidence” the reporter meant should be against the null hypothesis, not against the alternative hypothesis. Data is collected to falsify the null hypothesis, not to prove its truthiness, because the null is already accepted to be true unless convincing evidence is collected against it.

Let me give another example of this paradigm. As Newfoundland (2013) described, a person is innocent until proven guilty by bringing convincing evidence. The jury can’t say “he is innocent,” instead, the jury uses evidence that produces reasonable doubt to reject his innocence.

Developments in inferential paradigms

The accept-reject paradigm in inference represents a conventional wisdom. There are other, or currently developing, paradigms in inference, too. One of the dominating paradigms is the Bayesian-likelihood approach. After enjoying much wider acceptance in social and natural sciences, the Bayesian method suggests different views of hypothesis testing. The Bayesian approach is a method of data analysis in which subjectivity, conditionality, or past information is used to update the hypothesis as additional evidence is acquired. This approach in hypothesis testing offers a dynamic structure in probability calculation so hypotheses, and decisions, are not seen as a static truth; instead, they are updated with current data and the decision is stated in the sense of likelihood.

In our court room example, the Bayesian inference is applied to all evidence presented, with the past information being combined with the current evidence. The benefit of a Bayesian approach is that it gives all historical information so the decision is unbiased all along as the past data (prior knowledge) is used correctly. This paradigm changes the way statistics are calculated and how the result and inference are interpreted. Currently it is widely appreciated in quantitative data analysis, especially after convenient software exists for its implementation. However, the criticism to this approach, made by many, is found in its subjectivity. Objectivity and handling prior knowledge is a concern here so that different people, having different opinions, may arrive at different results.

Another developing paradigm: Fuzzy Logic

Fuzzy logic is another method in quantitative decision making. In contrast to binary logic (yes-no, or accept-reject), fuzzy logic can be thought of as gray logic, which allows a way to express in-between data values. It emerged in the development of the theory of fuzzy sets, by Lotfi Zadeh (1965). Fuzzy logic is mostly seen as a branch of artificial intelligence that deals with reasoning algorithms used to emulate human thinking and decision making. It handles the concept of partial truth using linguistic variables, where the truth value may range between completely true and completely false. The concept of partial truth would be subjective and would depend on the observer. For example, for the temperature of the weather, we want to determine when to say cold, warm, and hot. The meaning of each of these concepts can be represented by a certain membership (called a fuzzy set). The concept of each would be subjective. One might consider cold for all values up to 40 0F. In Figure 1, the meanings of the expressions cold, warm, and hot are represented by functions mapping a temperature scale to “truth values” ranging from 0 to 1. A point on that scale has three truth degrees, one for each of the three functions (expressions). The vertical line in the figure represents a particular temperature that the truth values (see three arrows) are measured. Since the red arrow points to zero, this temperature may be interpreted as “not hot.” The orange arrow (pointing at 0.2) may describe it as “slightly warm,” and the blue arrow (pointing at 0.8) “fairly cold” (Fuzzy Logic, 2013, para. 6). A rule could then be adopted, like for example, if “slightly warm” then stop the fan.

Picture was obtained from

Fuzzy logic uses truth degrees as a mathematical model of the vagueness phenomenon, and it summarizes data analysis in facts with truth degrees, and leaves the decision to the observer. Its advantage is its ability to deal with vague situations with respect to linguistic variables. While the significance testing for a hypothesis declares one accept or reject, fuzzy logic allows for degrees of acceptance or rejection. However, in fuzzy logic, the notion of truth doesn’t fall by the wayside, but it is expressed in degrees and offers possibilities for different situations. Regarding inference, fuzzy logic uses the mindset ‘everything is a matter of degree and open to interpretation,’ and this mindset is also adopted to machine learning, which is considered a more suitable mindset with human reasoning instead of binary logic. The constraints in fuzzy logic are found in its tools and interpretations when complex inputs are considered.

The developments of paradigms in statistical inference have similar fates as in the developments of mathematics, geometry, and physics. According to the Euclidean parallel postulate, in space, there exist no two parallel lines that intersect each other. However, after Non-Euclidean geometry was developed in the nineteenth century, a wider geometrical and mathematical reasoning stemmed from it; accordingly, the geometry of the physical universe and particles came to be understood better. The development of Riemannian geometry, offering that distance properties might vary, resulted in the synthesis of diverse results concerning the geometry of higher dimensional surfaces and the behavior of geodesics on them. It also made Einstein’s theory of general relativity justifiable. Likewise, as time passes, we witness wider paradigms or logics that abandon or correct former approaches in scientific decision making tools. This is a good reason why teachers and professors should update their current teachings as to be consistent with convincing trends, as well as to train students to be ready for wider paradigms in the future.

Conclusion

In today’s scientific community, the way decisions are made and fallacies proved, are changing as new paradigms emerge. In order to make better decisions or to validate claims, many aspects and methodologies should be considered. One way to avoid mistakes as much as we can would be to expect fallacy points to exist in human mental processing during decision making, and improving and seeking better alternatives with well-established wisdoms.

(Thanks Ugur Sahin for reviewing the preliminary copy of this article.)

References

• Fuzzy Logic. In Wikipedia. Retrieved August 1, 2013, from http://en.wikipedia.org/wiki/Fuzzy_logic
• Kass, Robert E. 2011. Statistical Inference: The Big Picture 1. Statistical Science, 2011, Vol. 26, No. 1, 1-9, DOI: 10.1214/10-STS337, Institute of Mathematical Statistics.
• Mercier, Hugo, Dan Sperber. 2011. “Why do humans reason? Arguments for an argumentative theory.” Behavioral and Brain Sciences. 34, 57-111.
• Newfoundland, Jason. 2013. “The Cell Phone-Brain Cancer Controversy.” The Fountain Magazine, Jan-Feb, Issue 91.
• Newman, Michael C. 2008. “‘What exactly are you inferring?’ A closer look at hypothesis testing.” Environmental Toxicology and Chemistry, Vol. 27, No. 5, pp. 1013-1019, 2008.

In recent years, people have been divided by conflicting studies about the risk of cancer posed by cell phone radiation. Does the current conflicting research eradicate or support the cell phone-cancer controversy?

Cell phone use has shown a dramatic increase in the world during the 1990s. The heated controversy today is about whether there is a relationship between cell phone use and the risk of developing malignant and benign brain tumors. This controversy did not exist—at least in the eyes of the public—until accumulating anecdotal evidence began suggesting a link between cell phone use and cancer. Since the first pieces of anecdotal evidence, numerous studies have investigated the cell phone-brain cancer link and a general summary of the findings is, at best, confusing. The substantial room for improvement in the experimental designs of these studies, the appearance of brain cancer after a long period of exposure, and some conflicts of interest among researchers prevented the results from being conclusive. More recent studies provide growing evidence for the link, suggesting there is reason to be suspicious about the studies that refute the relationship between cell phone use and brain cancer.

Possible health issues regarding exposure to radio frequency (RF) energy were described in a previous Fountain article (Tombak, 2002). This article also stated that proving or disproving the existence of RF exposure’s biological hazards remains an issue for epidemiology due to relatively low exposure levels, relatively small populations, and a lack of reliable dose estimates. The National Cancer Institute (NCI) is now maintaining an up-to-date web page to inform the public on key points that can be drawn from epidemiological studies investigating the cell phone-brain cancer link. It would be advisable that individuals concerned about this potential link become familiar with this web page and visit frequently to check the updates. A regular visitor will notice that the language on this page is evolving in every update in a way that the most recent version is less likely than the previous one to discredit the link as “out of the question”. This is because we are starting to see—albeit still opposed as weak—stronger signs of the alleged link as there is an increase in both the sheer number and the experimental design quality of relevant studies.

The first two key points that NCI draws on concern the type of electromagnetic energy emitted by cell phones, and the factors that determine a users energy exposure level. It would not be surprising if this train of thought followed with a third key point that stated that the cancer risk depended on the amount of energy that each individual was exposed to. However, the third key point quickly draws the conclusion that “studies thus far have not shown a consistent link between cell phone use and cancers of the brain, nerves, or other tissues of the head and the neck.” It further states “more research is needed because cell phone technology and how people use cell phones have been changing rapidly”.

Regardless of what type of general message one gets from these three key points, I want to emphasize that there is benefit in avoiding a lump-sum conclusion, and in making oneself aware of the results of individual studies. Before I move on to individual studies, however, I will point out how scientifically sound interpretations of statistical power and significance may lead to categorizations as ‘non-existent’ or ‘weak’, but how this scientific reasoning can potentially be misleading for the population at large. In this manner, one would better be able to make sense of why cell phone companies are, on one hand, highlighting the studies that have failed to find a causal link between cell phone use and brain cancer, but on the other hand, are taking all legal precautions necessary to prevent a future litigation by inserting a warning slip in fine print that cautions users not to hold the phone closer than a certain distance against one’s head or body.

The null vs. the alternative

A statistical hypothesis test involves two hypotheses: the null and the alternative hypothesis. The null hypothesis represents the status quo; it assumes that there is no real difference between the two groups under study and the observed difference can be attributed to random chance. Drawing a parallel with legal systems, the presumption that a defendant is innocent until proven guilty can be interpreted as saying that his or her innocence is the null hypothesis. There has to be sufficient evidence on the contrary, i.e showing the guilt, in order to be able to convict the defendant. In a similar fashion, an epidemiological study investigating the presence of a link between cell phone use and brain cancer would have a null hypothesis that states the absence of such a link. The cell phone technology is assumed to be innocent unless the data prove otherwise.

The alternative hypothesis, the latter of the two, represents the claim that there actually is a statistically significant link between cell phone use and brain cancer, and the observed link cannot be attributed to random chance. In our legal analogy, the alternative hypothesis is laying the charges against cell phone technology, thus as the Latin maxim “semper necessitas probandi incumbit ei qui agit” states, the burden of proof lies with the alternative hypothesis.

As a consequence of this construction, a hypothesis test can have only one of two conclusions. If the data shows results that are beyond some predetermined significance level, the null hypothesis is rejected and the researchers believe that there is sufficient evidence to say that the alternative is true. Such a conclusion would establish a link between cell phone use and brain tumors. On the other hand, if the results do not reach the desired significance level, the conclusion is not the confirmation of the null hypothesis, but a failure to conclude that the alternative is true. In other words, when the desired significance level is not reached, the only outcome is a lack of conclusion; the test would not falsify the null hypothesis but would not declare it to be true either. The accused would be vindicated on the basis of insufficient evidence.

News on the innocence of cell phone technology, or any technology for that matter, should be read primarily with this perspective in mind. The conclusions of research studies are reported on the basis of whether the results “reach or fail to reach significance”. However, a more meaningful statistic to report from the study would be the deviation of the results from significance, if they were not significant. Results that are close to statistical significance can still be “meaningful”. After all, the significance level chosen for most studies relies more on traditional scientific habits than anything else. In this perspective, it can even be called arbitrary. In reality, we may not have expertise in today’s world to determine whether a 5% significance level is more meaningful than a 10% when it comes to studying the link between cell phone use and brain cancer. So when the NCI officials mention “lack of a consistent link,” all of what they mean is that the desired significance level has not been reached in most credible and up-to-date studies. Yet, the public is not informed about how significant the results were. Furthermore, as we see in the much-acclaimed Interphone study, failure to reach significance in the entire study may be overshadowing the fact that significance was attained for a subgroup of people, for instance, the top 10% of the population with highest cell phone use (The Interphone study group, 2010).

Perspectives on brain cancer risk

News regarding cell phone tumor risks is plainly confusing because a battle continues among different international panels and interest groups over how to analyze and interpret cell phone tumor data. An article (July 6, 2011) on Microwave News explains why there is no overlap in the conclusions made by the International Commission for Non-Ionizing Radiation Protection (ICNIRP) and the International Agency for Research on Cancer (IARC), the two panels that are supposed to work together, but fell into deep disagreement as the data started showing some link between cell phone use and brain cancer. This piece is a highly suggested read for anyone who would like to be able to make more sense of the past and potentially future news on cell phone tumor risks.

The Interphone study

Much of the current debate on cell phone tumor risks actually revolve around the Interphone study, which was conducted by a consortium of researchers from 13 countries, and is the largest health related case control study of the use of cell phones and head and neck tumors. According to NCI’s summary, “most published analyses from this study have shown no statistically significant increases in brain or central nervous system cancers related to higher amounts of cell phone use. One recent analysis showed a statistically significant, albeit modest, increase in the risk of glioma among the small proportion of study participants who spent the most total time on cell phone calls. However, the researchers considered this finding inconclusive because they felt that the amount of use reported by some respondents was unlikely and because the participants who reported lower levels of use appeared to have a reduced risk of brain cancer.”¹

The general message that comes across in NCI’s summary of Interphone results is that we do not have enough reason to believe that cell phones are dangerous. However, it is very important to remember that one can never accept the null hypothesis that “cell phones are safe.” The only conclusion that can be drawn is on the basis of insufficient evidence, which is to say that cell phones are dangerous since the desired significance level has not been reached.

As the Interphone study is the largest of its kind, it has drawn substantial attention from concerned parties, and a significant part of this attention has been in the form of harsh criticisms for the experimental design, data analysis, and stated conclusions. For instance, one of Interphone’s biggest critics, the International Electromagnetic Field (EMF) Collaborative, published a paper in May 2010 detailing the flaws of the study. Among other things, these flaws included using data from 2004 and before when cell phone use was much less common, categorizing subjects who used cordless phones (which emit the same microwave radiation as cell phones,) as ‘unexposed’; exclusion of many types of brain tumors; exclusion of people who had died, or were too ill to be interviewed, as a consequence of their brain tumor; and exclusion of children and young adults who are more vulnerable.

In August 2009, more than forty leading independent scientists, physicians and other experts from fourteen countries endorsed the white paper “Cell-phones and Brain Tumors: 15 Reasons for Concern, Science, Spin and the Truth Behind Interphone” by US researcher Lloyd Morgan. Investigating the research on cell phone tumor risks including the Interphone study, this paper concluded that “there is a risk of brain tumors from cell phone use; telecom funded studies underestimate the risk of brain tumors; and children have larger risks than adults for brain tumors”. Unlike the Interphone study, some industry funded research also accepts the risks associated with cell phone use. In 1999, Dr. George Carlo, head of a $25m research body funded by the mobile phone industry in the US, said his study showed an increased risk of getting a type of rare brain tumor from using mobile phones. This early in the debate, he was probably one of the first researchers with links to industry who stopped ruling out the tumor risks of cell phones.

A review of other main studies

Hardell et al.

Dr. Lennart Hardell, from Örebro University in Sweden, is one of the most adamant leaders in cautioning the world about cell phone tumor risks. In 2007, he and his team reported that cell phone users were at an increased risk of malignant glioma, and that a daily one-hour exposure significantly increased the risk for developing a brain tumor after 10 years (Hardell et al., 2007). In a more recent study also cited by NCI, they found statistically significant trends of increasing brain cancer risk for the total amount of cell phone use and the years of use among people who began using cell phones before the age of 20 (Hardell et al., 2011). They also published a number of other papers in epidemiological journals pointing to the risks associated with cell phone and cordless phone usage.

The Danish cohort study

A 2011 cohort study in Denmark linked billing information from more than 420,000 cell phone subscribers with brain tumor incidence data from the Danish Cancer Registry (Frei et al., 2011). This study was an update on the 2006-update of a 2001 cohort study (Schüz et al., 2006; Johansen et al., 2001) that has been dogged by controversy and political suspicions since the first results were published ten years ago. NCI cites the study and states that the analyses found no association between cell phone use and the incidence of glioma, meningioma, or acoustic neuroma, even among people who had been cell phone subscribers for 10 or more years. However, there is no mention of the published or vocal criticisms of the study.

The main criticism for the study is that more than 200,000 corporate mobile subscribers were excluded from the cohort as cell phone bills were not in users’ names. Microwave News states, “In the time period covered in the Danish project—from 1987 through 1995—cell phones were expensive and it’s no stretch to assume that those who did not have to pay their own bills racked up the most talk time.”² Thus, the study designers effectively removed one-third of the population with the heaviest cell phone use. Dr. Lennart Hardell had also criticized the original 2001 paper by publishing on the shortcomings that make the conclusions premature (Hardell and Mild, 2001). Concerning the 2011 update, the Microwave News bluntly suggests, “Don’t believe a word of it”.

It is also interesting to note that the results came just five months after a panel of experts from the World Health Organization’s International Agency for Research on Cancer (IARC) deemed cell phones a possible cause of cancer—a statement that sparked fear in many of the world’s 5 billion cell phone users.

Conclusion

While there is still no established causal link between cell phone use and cancer, we know as a fact that different research groups have found an increased risk of a rare type of brain cancer among heavy users. Even though children are known to be at a greater risk because of being in earlier stages of neural development, it is unfortunate that data from children were not included in studies until very recently. Concerned citizens of the world need to raise awareness about the behind-the-scenes battle taking place between different international panels and interest groups. This will make a reliable interpretation of conflicting news more possible. Further corroboration for both statistical and anecdotal evidence on the relationship between cell phone use and brain cancer may be necessary to “prove” a link, but this should, by no means, be interpreted as a vindication of cell phones. In the meantime, it is only safe to take precautions oneself, and encourage loved ones to reduce exposure to electromagnetic energy from cell phones by using a hands-free device and by reserving the use of cell phones for shorter conversations.

Notes

1 http://www.cancer.gov/cancertopics/factsheet/Risk/cellphones

2 http://www.microwavenews.com/DanishCohort.html#Continued

References

• Frei P, Poulsen AH, Johansen C, et al. 2011. “Use of mobile phones and risk of brain tumours: update of Danish cohort study.” British Medical Journal; DOI: 10.1136/bmj.d6387.
• Hardell, L., Walker, M. J., Walhjalt, B., Friedman, L. S. and Richter, E. D. 2007. “Secret ties to industry and conflicting interests in cancer research.” American Journal of Industrial Medicine,; 50: 227–233. doi: 10.1002/ajim.20357.
• Hardell L, Carlberg M, Soderqvist F, Hansson-Mild K, Morgan LL. 2007. “Long-term use of cellular phones and brain tumours: Increased risk associated with use for > or = 10 years.” Occup Environ Med. 64:626–632.
• Hardell L, Carlberg M, Hansson Mild K. 2011. “Pooled analysis of case-control studies on malignant brain tumours and the use of mobile and cordless phones including living and deceased subjects.” International Journal of Oncology; 38(5):1465–1474.
• Hardell L, Mild KH. 2001. “Re: Cellular Telephones and Cancer—a Nationwide Cohort Study in Denmark”; JNCI J Natl Cancer Inst. 93(12): 952.
• Johansen C, Boice Jr. JD, McLaughlin JK, Olsen JH. 2001. “Cellular telephones and cancer: a nationwide cohort study in Denmark.” Journal of the National Cancer Institute; 93(3):203–207.
• Schüz J, Jacobsen R, Olsen JH, et al. 2006. “Cellular telephone use and cancer risk: update of a nationwide Danish cohort.” Journal of the National Cancer Institute; 98(23):1707–1713.
• Tombak, Ali. 2002. “Biological Effects of Cellular Phones.” The Fountain, 37 (1).
• The Interphone Study Group. 2010. “Brain tumour risk in relation to mobile telephone use: results of the Interphone international case-control study.” International Journal of Epidemiology; 39(3):675–694.