New developments in scientific thought and hypotheses testing are changing the ways we think about truth and certainty.

Not only do scientific decisions rely heavily on tools and procedures of quantitative analysis, but also on social life and daily decisions. I want to start with a story from the judiciary system about the misuse of an important quantitative tool of the scientific community, Hypothesis Testing. It’s about the overturned Amanda Knox case in Italy. In 2007, she was accused of killing her roommate and sent to jail. The odds of DNA matching in the case of accidental death were reported as incredibly rare by a forensic data analyst, who examined data with regard to incidents of crime, concluding that the accident is not statistically an accident. The judges then made their decision based on this rareness, a single value, named p-value. Later, this was overturned (New York Times, March 27, 2013), because of a misinterpretation by the judges and lawyers, of the rare probability. What factors forced the judiciary system to make this decision based on a single value alone? Honestly, this is the story of the century and a story of Hypothesis Testing. This article introduces the logic behind their decision, draws attention to the misuse of data, and mentions alternative logics to this traditional approach on the matter (in March 2013, Italy’s Supreme Court ordered a retrial and found the two suspects guilty on January 30, 2014).

What is Hypothesis Testing?

We live in a data-driven world. Statistical inference is the process of drawing conclusions or decisions from data. The Hypothesis Testing procedure in the Neyman-Pearson paradigm is one of the procedures of this process, widely used in the scientific community for over a century. In this mindset, two complementary hypotheses are first defined: one is the conventional thesis (the null hypothesis) accepted to be true by default; the other is the alternative thesis (the alternative hypothesis) that needs evidence from data to falsify the null hypothesis. It results in a single value, called p-value, which is calculated from the data using theoretical model assumptions. A p-value is a measure – in probability sense, ranging from 0% to 100% – of how much evidence you have against the null hypothesis. The smaller the p-value, the more evidence you have against the null hypothesis. One may combine the p-value with the significance level to make decisions on a given hypothesis. This is also called significance testing. It has an accept-reject mindset (dichotomous decision or binary logic); in such a case, if the p-value is less than some threshold (usually .05) then the null hypothesis is rejected. Basically, it is a black or white decision, without considering the contrasts between them. This interpretation has been widely accepted in the twentieth century of scientific research, and many scientific journals routinely publish papers using this interpretation for the results of hypothesis tests, even though there are current tendencies not to use this. Let’s take a look at the history of this black-white logic.

History

The history of the process of hypothesis testing starts with Fisher (1890-1962), around the early twentieth century. Later, the contributions of Pearson (1857-1936) and Neyman (1894-1981), who were early fathers of statistics, about the interpretation of the hypothesis tests were integrated into present-day applications. Fisher’s approach focuses on inductive inference about a single hypothesis, whereas the Neyman-Pearson approach informs future behavior based on a test using two complementary hypotheses (Newman 2007).

These approaches are strongly influenced by Popper’s logic of falsification. Falsification can be defined as the act of disproving a proposition, hypothesis, or theory. This logic asserts that sufficiently improbable events can be considered impossible. A p-value suggests whether a null hypothesis is sufficiently improbable to be considered practically falsified in the sense of a logical refutation (Newman 2007). When we come back to the Amanda Knox case, the decision the judges made and justified is followed from this falsification logic in hypothesis testing. Another criticism with this mindset is: how fair it is to rely on a single value that yields a rare result? Are we going to generalize one lucky drawing of raffle tickets to believe all tickets are winners?

Logics and proofs

We use logical flaws and biases in our daily life. Finding the logical flaw in scientific papers is something most readers aren’t interested in. Instead, the results are believed as a fact. One of the widely used flaws/biases is in seeking or interpreting evidence in ways that are partial to existing beliefs or a hypothesis in hand (Mercie and Sperber, 2011). We tend to prove or show the accessibility or superiority of arguments by bringing evidence. Failure to prove that a treatment – say, a low fat diet – is effective is not the same as proving it is ineffective.

Let me give another example to clarify this: The New York Times editorial news reported on February 9, 2006, “Millions of Americans have tried to reduce the fat in their diets, and the food industry has obligingly served up low-fat products. Yet now comes strong evidence that the war against all fats was mostly in vain.” Actually, in the hypothesis testing mindset, the evidence collected from research should either reject the null hypothesis which is “diet is ineffective,” or fail to reject it. The mindset in testing is not about finding evidence to support the null statement. It is also not about proving the alternative hypothesis, “diet is effective.” Rather, it is basically looking for evidences to falsify the null statement. In the report, the “evidence” the reporter meant should be against the null hypothesis, not against the alternative hypothesis. Data is collected to falsify the null hypothesis, not to prove its truthiness, because the null is already accepted to be true unless convincing evidence is collected against it.

Let me give another example of this paradigm. As Newfoundland (2013) described, a person is innocent until proven guilty by bringing convincing evidence. The jury can’t say “he is innocent,” instead, the jury uses evidence that produces reasonable doubt to reject his innocence.

Developments in inferential paradigms

The accept-reject paradigm in inference represents a conventional wisdom. There are other, or currently developing, paradigms in inference, too. One of the dominating paradigms is the Bayesian-likelihood approach. After enjoying much wider acceptance in social and natural sciences, the Bayesian method suggests different views of hypothesis testing. The Bayesian approach is a method of data analysis in which subjectivity, conditionality, or past information is used to update the hypothesis as additional evidence is acquired. This approach in hypothesis testing offers a dynamic structure in probability calculation so hypotheses, and decisions, are not seen as a static truth; instead, they are updated with current data and the decision is stated in the sense of likelihood.

In our court room example, the Bayesian inference is applied to all evidence presented, with the past information being combined with the current evidence. The benefit of a Bayesian approach is that it gives all historical information so the decision is unbiased all along as the past data (prior knowledge) is used correctly. This paradigm changes the way statistics are calculated and how the result and inference are interpreted. Currently it is widely appreciated in quantitative data analysis, especially after convenient software exists for its implementation. However, the criticism to this approach, made by many, is found in its subjectivity. Objectivity and handling prior knowledge is a concern here so that different people, having different opinions, may arrive at different results.

Another developing paradigm: Fuzzy Logic

Fuzzy logic is another method in quantitative decision making. In contrast to binary logic (yes-no, or accept-reject), fuzzy logic can be thought of as gray logic, which allows a way to express in-between data values. It emerged in the development of the theory of fuzzy sets, by Lotfi Zadeh (1965). Fuzzy logic is mostly seen as a branch of artificial intelligence that deals with reasoning algorithms used to emulate human thinking and decision making. It handles the concept of partial truth using linguistic variables, where the truth value may range between completely true and completely false. The concept of partial truth would be subjective and would depend on the observer. For example, for the temperature of the weather, we want to determine when to say cold, warm, and hot. The meaning of each of these concepts can be represented by a certain membership (called a fuzzy set). The concept of each would be subjective. One might consider cold for all values up to 40 0F. In Figure 1, the meanings of the expressions cold, warm, and hot are represented by functions mapping a temperature scale to “truth values” ranging from 0 to 1. A point on that scale has three truth degrees, one for each of the three functions (expressions). The vertical line in the figure represents a particular temperature that the truth values (see three arrows) are measured. Since the red arrow points to zero, this temperature may be interpreted as “not hot.” The orange arrow (pointing at 0.2) may describe it as “slightly warm,” and the blue arrow (pointing at 0.8) “fairly cold” (Fuzzy Logic, 2013, para. 6). A rule could then be adopted, like for example, if “slightly warm” then stop the fan.

Picture was obtained from

Fuzzy logic uses truth degrees as a mathematical model of the vagueness phenomenon, and it summarizes data analysis in facts with truth degrees, and leaves the decision to the observer. Its advantage is its ability to deal with vague situations with respect to linguistic variables. While the significance testing for a hypothesis declares one accept or reject, fuzzy logic allows for degrees of acceptance or rejection. However, in fuzzy logic, the notion of truth doesn’t fall by the wayside, but it is expressed in degrees and offers possibilities for different situations. Regarding inference, fuzzy logic uses the mindset ‘everything is a matter of degree and open to interpretation,’ and this mindset is also adopted to machine learning, which is considered a more suitable mindset with human reasoning instead of binary logic. The constraints in fuzzy logic are found in its tools and interpretations when complex inputs are considered.

The developments of paradigms in statistical inference have similar fates as in the developments of mathematics, geometry, and physics. According to the Euclidean parallel postulate, in space, there exist no two parallel lines that intersect each other. However, after Non-Euclidean geometry was developed in the nineteenth century, a wider geometrical and mathematical reasoning stemmed from it; accordingly, the geometry of the physical universe and particles came to be understood better. The development of Riemannian geometry, offering that distance properties might vary, resulted in the synthesis of diverse results concerning the geometry of higher dimensional surfaces and the behavior of geodesics on them. It also made Einstein’s theory of general relativity justifiable. Likewise, as time passes, we witness wider paradigms or logics that abandon or correct former approaches in scientific decision making tools. This is a good reason why teachers and professors should update their current teachings as to be consistent with convincing trends, as well as to train students to be ready for wider paradigms in the future.

Conclusion

In today’s scientific community, the way decisions are made and fallacies proved, are changing as new paradigms emerge. In order to make better decisions or to validate claims, many aspects and methodologies should be considered. One way to avoid mistakes as much as we can would be to expect fallacy points to exist in human mental processing during decision making, and improving and seeking better alternatives with well-established wisdoms.

(Thanks Ugur Sahin for reviewing the preliminary copy of this article.)

References

• Fuzzy Logic. In Wikipedia. Retrieved August 1, 2013, from http://en.wikipedia.org/wiki/Fuzzy_logic
• Kass, Robert E. 2011. Statistical Inference: The Big Picture 1. Statistical Science, 2011, Vol. 26, No. 1, 1-9, DOI: 10.1214/10-STS337, Institute of Mathematical Statistics.
• Mercier, Hugo, Dan Sperber. 2011. “Why do humans reason? Arguments for an argumentative theory.” Behavioral and Brain Sciences. 34, 57-111.
• Newfoundland, Jason. 2013. “The Cell Phone-Brain Cancer Controversy.” The Fountain Magazine, Jan-Feb, Issue 91.
• Newman, Michael C. 2008. “‘What exactly are you inferring?’ A closer look at hypothesis testing.” Environmental Toxicology and Chemistry, Vol. 27, No. 5, pp. 1013-1019, 2008.

We sit for tests of all sorts throughout our lives, tests in varying scopes and levels of difficulty. Tests to enroll a school, to pass a class, to graduate, the test a son happens to endure with his father or the other way around, and many more tests. An exceptionally difficult and humiliating test is being disqualified, pushed aside, and stripped of civil rights due to one’s humanistic thoughts and lofty ideals.

The most unbearable test, however, is the one that comes with the hands of disloyal friends, particularly when one is already suffering from the words and actions of relentless and merciless foes. Although, regardless of however inhumane and short of dignity a foe may be, his or her enmity can still be reconciled by an antagonistic logic. One may rationalize—using the same logic—that the more differences in thought, worldview, and values increase, so does enmity. Nevertheless, it is unreasonable and irrational when those striving on the same line of fate, those who cherish the same feelings and ideas, betray one another because of jealousy and competition. This is no way reconcilable with human dignity.

Indeed, it hurts—as much as it is worrying—when one ends up with betrayal and pain while expecting faithfulness. What else one can do other than enduring it, as tests of this nature are rampant in a world in which deceit is considered equal to being smart, obsession is equal to to loyalty, and bigotry means conservatism. Yes, as individuals, families, and community as a whole, we have no other option than to endure and say:

A reprimand from Your Majesty
or a favor from Your Grace;
Both are a pleasure for my soul;
How pleasant is Your grace
How pleasant is Your wrath

A community or a nation may have suffered from the greatest and most painful of all tribulations and betrayals from foes, some of who were in the disguise of friends. The entire world might have advanced upon them and brought them under siege from all corners, leading some to a vain belief of their complete annihilation. Yet, they were able to survive all these tests of life-or-death, nullifying all the destructive plans. It is highly probable for similar tests to come up along the way through hills of flames and oceans of blood, all of which will in fact help them with renewal and with metaphysical regeneration. Each of these tests will be a means of awareness for them to identify who is who, for self-empowerment through learning how to recollect and get back on their feet after being knocked down by them. A human being is purified through tests and gets hold of his or her inner essence, thus their life is saved from stagnancy and becomes colorful. Our spirit matures in the face of tests and becomes prepared for bigger challenges, as the bigger the tests and harder the questions are, the more entitled the human is to pass the class and to rise higher in the school of human values and perfection.

Refinement of the individual and empowerment of the society cannot take place without tests. Only souls who have been burdened and pressured by tests can coil like a spring, thrust like an arrow, and reach their target with the blink of an eye. Worries that haunt them all day, hunger and thirst, harm and damage, troubles of all kinds that befell their properties or lives, and being trodden under the metal tracks of unexpected events will all make them as strong as steel and prepare them for the future.

Deadened hearts and raw spirits that have not been tested whatsoever cannot rise to their human potential, nor are they of any use to the community. Distinguishing the diamond-like souls from coal-like ones is only possible by way of testing, at the lack of which gold cannot be distinguished from soil, or diamond from coal. If it was not for testing, the most inauspicious ones would be confused with the highest soaring souls, for it is through testing that angel-like ones can be told apart from the evil souls and ascend to their predestined zeniths.

For those who are appreciative of this truth, each test is one added feather to a wing that will enable him or her to take off to the realms beyond the heavens, and each difficulty faced during a test is an elixir of strength and vigor. For these lofty spirits, being cast into fire is the greatest leap into a friendship with the Creator, and being nailed on the cross is an exalted means of ascension to Him.

For a heart directed towards the highest of ideals, a test is a whip lashed on his or her perseverance, a spell that makes their willpower rear up, and a ray of light that shines on the dial of heart. After each test they crystallize even more, tighten like a bow, and soar high up to the Gardens they already built in their hearts.

Let not the muffled hearts with no refinement to welcome both His reprimand and favor fathom even a grain of this phenomenon; the idealist souls that have devoted themselves to this truth will never recognize anything more pleasant than the pains suffered in this cause. They will never lament and complain to others, even though they burn like a furnace inside. Unfaithfulness of friends and ruthlessness of foes will never take them aback from serving people and their country, for they have given oath as in the words of the poet:

Let the world come at me with all possible reasons for trouble
I am a traitor if I give up any conviction in the service of my nation. (Namık Kemal)