Based on the article, Editor’s Comments: Avoiding Type III Errors:  Formulating IS Research Problems that Matter (Rai, 2017), formulate a research  problem. Describe the steps y

EDITOR’S COMMENTS

Avoiding Type III Errors: Formulating IS Research Problems that Matter

By: Arun Rai Editor-in-Chief, MIS Quarterly Regents’ Professor of the University System of Georgia Robinson Chair of IT-Enabled Supply Chains and Process Innovation Harkins Chair of Information Systems Robinson College of Business Georgia State University [email protected]

Type III errors occur when a researcher answers the wrong question using the right methods (Mitroff and Silvers 2009; Raiffa 1968). A lot of effort may be expended, a great deal of rigor may be applied, but coming up with the right answer to the wrong question does not create value. Understandably, this can be frustrating for authors of such work when their peers do not judge the work favorably. It is also a missed opportunity for the field when scare resources of scholars in the community are directed at the wrong research questions.

In this editorial, I discuss a key aspect of the research process—problem formulation— that dramatically influences the research question that is addressed, the value that is created by a research study, and the suitability of the research for MISQ. I focus on this aspect of the research process, as it is where less attention tends to be placed by scholars even though it is a process riddled with misconceptions, risks, and common errors that can lead to Type III errors (Table 1).

Formulate the Research Problem So the Answer to the Question Will Matter

A research problem is “any problematic situation, phenomenon, issue, or topic that is chosen as the subject of an investigation” (Van de Ven 2007, p. 73). It is through the process of problem formulation that a researcher decides on the research question that “merits scientific investigation to better understand the problem and its resolution” (Van de Ven 2007, p. 87).

For IS researchers, a problem of interest may originate in different ways. A researcher may see the genesis of a problem in the practical world (e.g., a major security breach), a theoretical domain (e.g., assumptions of economic or behavioral theories that are in conflict with the behaviors of hackers), or a combination (e.g., the breakdown of network and behavioral theories in accounting for the failure of a system to prevent a security breach).

But a problem that is of interest to a researcher may not be necessarily important to take on to significantly advance IS knowledge. The problem may be readily structured and solved by applying current knowledge. It may also be one-off and idiosyncratic, but not prevalent or unlikely to be prevalent.

Calling for scholars to take on problems where the answers will matter in important ways, Medawar cautions against conflating a problem being interesting with it being important:

Any scientist of any age who wants to make important discoveries must study important problems. Dull or piffling prob- lems yield dull or piffling answers. It is not enough that a problem should be interesting—almost any problem is interesting if it is studied in sufficient depth … the problem must be such that it matters what the answer is—whether to science gener- ally or to mankind. (P. B. Medawar, Nobel Laureate in Medicine and Physiology, 1979; cited in Van de Ven 2007, p. 71.)

IS research problems vary significantly in structure and in maturity of knowledge to address them. We have developed significant cumulative understanding in a variety of domains such as IS acceptance and use; IS development; task–technology fit; online collaboration; mobile commerce; and design of recommendation agents, market mechanisms, and trust-building mechanisms in online exchanges. As a result, some problems can be readily structured and solved by applying extant knowledge from IS or from other disciplines. Other problems may be such that our existing understanding exhibits systematic breakdowns and limitations, prompting the need to challenge conventional formulation of the problem and thinking about how to solve it. And, there are other areas where the very nature of the problem and plausible solutions are unclear to start with, a messy state of affairs that call for extensive and creative muddling to decipher how to formulate the problem.

MIS Quarterly Vol. 41 No. 2 pp. iii-vii/June 2017 iii

Editor’s Comments

Table 1. Formulating IS Research Problems to Avoid Type III Errors

Formulate the Research Problem So the Answer to the Question Will Matter

• Differentiate between interesting and important problems • Differentiate between three types of value that the answer to the

research question can create: aesthetic, scholarly, practical utility • MISQ expects work to create significant scholarly value; practical

utility resulting in a broader impact on business and society is highly desired; creating all three types of value is ideal

Risks to Safeguard Against:

Streetlight Effect • Ease of research (e.g., easy to access datasets and easy to use tools) rather than need for research, drives the problem formulation

Being Solution-Driven Rather than Problem Minded

• Unclear or pseudo problems are formulated, with the idea to advance a theory, method, or solution

Gap-Spotting and Gap-Patching • Areas not addressed in past work drive the problem formulation—will patching the gaps make a substantial change in knowledge about IS?

Overlooking the Generic Archetypal Problem in Problem Formulation

• Immediate practical need or a limited manifestation of a phenomenon inform the formulation—relation to generic problems or broader phenomenon is overlooked

• Contribution may be more ephemeral than if the problem was formulated by relating the immediate practical problem to the accumulated knowledge about the generic problem and its solution

The Answer to the Question Is Derivative to Current Understanding

• Problem is formulated to reevaluate a well-established theory, model, or IS solution in a different setting with expectation of affirmation, which is confirmed through the research

• Straight-up applications of theories and models from other disciplines

The Goldilocks Principle: Excessive or Marginal Scope

• Is the problem formulation too narrow seeking to know “everything about nothing” or too diffuse and lacking depth seeking to know “nothing about everything”

Being Disciplinary While Being Interdisciplinary

• Very light in the treatment of IS and IT, problem thinly veneered as an IS problem

• Terms such as digital, online, and mobile technologies are sprinkled through the manuscript without a substantive engagement with one or more aspects of IS in the problem formulation

Chance Favors the Prepared Mind and the Connected Mind

• Having expectations based on a good grasp of prior knowledge provide a comparative advantage in spotting anomalies that refute the expectations

• Creating favorable conditions to connect (and rebut) diverse ideas, insights, and perspectives can provide a comparative advantage in how a problem is seen and solved

The goal of problem formulation is deciding on the research question or objective. This is likely the most important decision that a researcher makes, as it is a decision by the researcher on what merits scientific investigation. The quality of this decision depends on the effectiveness of problem formulation, where critical choices need to be made with respect to the stakeholder perspectives that will be considered, elements that will be in the foreground versus those that will be in the background, and creating the “knowledge puzzle” relative to conventional understanding. Einstein succinctly captures the essential nature of problem formulation to the success of the overall research endeavor in this quote:

The formulation of a problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill. (Albert Einstein quoted in Csikszentmihalyi 1988, p. 160.)

iv MIS Quarterly Vol. 41No. 2/June 2017

Editor’s Comments

Regardless of whether an IS researcher’s attention to a problem originates in the practical world or a theoretical domain or some combination, the researcher needs to surface why answering the question will matter. A defining characteristic of a good research question is that an answer, if found, needs to be regarded as valuable. The value created by the answer can be (1) aesthetic, arising from “powerful simplicity,” (2) scholarly, by advancing the area under study in fundamental ways that influence future progress, and (3) of practical utility (Simon 1991, p. 2). Answers to questions can create value on all three of these dimensions.

Given MISQ’s editorial objectives, the answer to a research question needs to make a significant scholarly contribution to the IS discipline. With the dramatic potential of IS to redefine the functioning of business and society, the answer can have utility by making, or having the potential to make, the practical world of affairs in business and society better in important ways. Beauty, arising from the powerful simplicity of the answer—be it mathematical or qualitative, can be an important differentiator of the work. As such, an ideal contribution to MISQ is one where the answer to the question is valuable on all three dimensions.

Safeguard Against Key Risks in Formulating Research Problems

Although there is much that has been written about the importance of problem formulation and it may even be taken by some to be well- understood in the IS scholarly community, editors and reviewers routinely observe that papers they review lack a compelling research question. In many instances, editors and reviewers appreciate the significant time and effort that authors have invested in the design and execution of a study but note that the study’s contributions are not compelling because the research question was not important—in other words, a Type III error. Receiving this feedback can be frustrating for authors who may have spent years working on a project.

Below, I enumerate six reasons that I have found to arise routinely in review processes as to why the formulation of the problem can lead to the wrong research question.

Streetlight effect, driven by ease of research rather than need for research: Editors and reviewers sometimes lament that the work is motivated by easily available datasets and easy-to-use tools and not by the need for research arising from a well-formulated IS problem.

In an earlier editorial, I talked about the need to safeguard against this risk, which is likened to the streetlight effect where a drunkard decides to search for his/her lost wallet under a street lamp because the bright illumination makes it easier to search at that location relative to a darker location where the wallet was likely dropped (Rai 2016).

Being solution-driven rather than problem-minded: Another observation that comes up in review processes is that the research advances a new theory or method for an unclear or pseudo problem. This situation—where a researcher overlooks or short-changes problem formulation but advances a theory or a method—arises when the researcher is “solution-driven” rather than “problem-minded” (Van de Ven 2007, p. 72). This mindset makes it more likely that a researcher will solve the wrong problem by applying the right methods, creating Type III errors (Van de Ven 2007).

Gap-spotting and gap-patching, but does the gap matter: Motivating the research question with the rationale that past work has not examined something, say relationships among a set of constructs, without addressing why the gap is important invokes the reaction that the gap may be inconsequential. In domains where significant knowledge has accumulated through past work, gap-spotting and gap-patching problem formula- tions can be deemed incremental in relation to the expectation for scholarly contribution at premier journals such as MISQ. This is not to say that gaps may not be important to address but spotting them does not make the case for the value that the research will generate. A gap may merely exist because it is not worth pursuing.

Myopic problem formulation, while overlooking the generic, archetypal problem: Sometimes authors formulate the problem with a sole focus on an immediate practical problem of interest to them, but do not evaluate how the problem relates to a more generic, archetypal problem (Weber 2003). For instance, a problem may be formulated to evaluate how intelligent wearable devices can persuade diabetic patients to make necessary behavioral changes to comply with therapy. However, looking at the problem more generically, this problem may be considered as an instance of the archetypal problem of how information systems (along with other means) can persuade patients with chronic diseases to make necessary behavioral modification to comply with therapy. Moving up an additional layer, there is the generic problem of how infor- mation systems (along with other means) can persuade individuals to modify behaviors to comply with a new set of behavioral norms that is necessary to attain goals. By relating the immediate practical problem to a generic, archetypal problem, authors can formulate the study to not only address the immediate practical need but also to make a broader and long-lasting scholarly contribution at the level of the generic problem.

Grounding a problem from up-close (the perspective of those that are experiencing the particular problem) and afar (generalizable characteristics of the problem domain) can be effective strategies to link an immediate problem to a generic class, and enable problem

MIS Quarterly Vol. 41 No. 2/June 2017 v

Editor’s Comments

formulation to achieve fundamental contributions (Simon 1991; Van de Ven 2007). Authors can dramatically bolster the scholarly value that is created by their research by formulating the problem to be less transient and idiosyncratic and by connecting with the base of IS knowledge that corresponds to the generic, archetypal problem.

The answer to the question is derivative to current understanding: Papers do not fare well in the MISQ review process when editors and reviewers see a paper as taking what is well known and reiterating it in a different context. Some of our editors refer to this type of paper as “affirming that gravity works in my kitchen.”

Papers are candidates for desk rejects at MISQ when the answer to the research question is derivative to existing knowledge and available through inference and without the research. Reevaluating well-established theories or models in different contexts and presenting evidence that further affirms the validity of the theory or model do not represent scholarly contributions at the level expected at MISQ. Similarly, design science papers reevaluating the utility of an IS solution in a different setting and concluding that the solution also has utility in the different setting does not make a scholarly contribution at the level expected at MISQ. Straight-up applications of theories and models from another discipline (e.g., an economic theory that is applied to an IS context or is applied with marginal tweaks) do not push our understanding forward about IS in a substantive manner. These papers can be characterized as “have-theory-will-travel” (Weber 2003, p. vi).

As Simon (1991, p. 3) observes, “Novelty is an essential component of contributions to science. No prizes are awarded for being second to discover a scientific law.” Of course, a researcher can make an important contribution by building on others’ work, and generating something novel that is valuable relative to the current knowledge.

For MISQ, novelty necessarily needs to be with respect to the scholarly contribution to IS. Showcasing that existing knowledge operates as expected in a new problem context (e.g., a different country, application domain, technology platform, user demographic base) may have practical utility, potentially make an incremental contribution to external validity, and aid in the diffusion of established knowledge—but these types of papers do not make a significant IS scholarly contribution for MISQ publication.

Of course, a novel context does not necessarily need to be a background empirical consideration in which existing theories, models, and solutions are reified. Aspects of context can be moved to the foreground and leveraged in reformulating the problem; challenging assumptions underlying theories, models, and IS solutions; and uncovering how the meaning of constructs and the relationships among constructs change (Johns 2006). Through such a process, the problem may be formulated to leverage a novel context to make a significant scholarly contribution to our knowledge about IS.

The Goldilocks principle—excessive or marginal scope: Researchers need to make essential scoping decisions in problem formulation. They need to safeguard against seeking to know “everything about nothing” (i.e., being excessively narrow) or “nothing about everything” (i.e., being excessively broad and lacking in depth) (McGrath et al. 1982). Deciding on the appropriate problem scope—what to place in the foreground and what in the background—has been referred to in various disciplines as the Goldilocks principle.1

Be Disciplinary While Being Interdisciplinary

How should interdisciplinary work be formulated and positioned for MISQ? This is a question that comes up routinely.

The boundaries of IS-related phenomena and problems continue to shift and now span an increasingly wide variety of business and societal domains. Given the pervasiveness of digital phenomena that are transforming economic and social systems, scholars in a number of disciplines are actively studying these phenomena. For example, cybersecurity is of interest to criminologists, psychologists, computer scientists, and IS scholars. Privacy is of interest to scholars in marketing, legal studies, computer science, and information systems. Although journals in several fields are interested in publishing work in interdisciplinary domains such as cybersecurity and privacy, the types of problem formulations that are expected are different.

Interdisciplinary papers are welcomed at MISQ, with the expectation that these papers place salience on the role of IS in the formulation of the problem and consequently in the contribution. It is important that we do not conflate the need to be adaptive as IS scholars to the shifting boundaries of IS problems and phenomena with the enduring need to make disciplinary IS contributions while engaging in interdisciplinary work.

1Derived from the children’s story “Goldilocks and the Three Bears” by Robert Southy.

vi MIS Quarterly Vol. 41No. 2/June 2017

Editor’s Comments

Sprinkling terms such as online, digital, IT, and mobile to describe the IS phenomenon without engaging deeply with one or more aspects of IS in formulating the problem means that the study’s key objective is not to generate significant contributions regarding IS. These submissions tend to receive comments from editors and reviewers such as “very light in the treatment of IS and IT,” “problem thinly veneered as an IS problem,” and “lack of contribution to IS.”

Scholars who are engaged in interdisciplinary work can generate contributions to our understanding about IS by reformulating problems to move IS from a background to a foreground role. They can surface how and why, at the deep structure of the problem or phenomenon, certain IS characteristics are important. By placing primacy on IS in interdisciplinary research, the work is not a mirror image of work in another discipline but rather contributes to the accretion of IS knowledge.

Chance Favors the Prepared Mind and the Connected Mind

Louis Pasteur’s aphorism, “In the fields of observation, chance favors only the prepared mind,” effectively captures why a researcher can have a comparative advantage in formulating problems and avoiding Type III errors. A researcher can develop this comparative advantage by having expectations based on a good grasp of prior knowledge to spot anomalies that refute the expectations.

While the role of an individual researcher in having a prepared mind to formulate a problem can be important, an idea is often not a standalone eureka moment of an isolated researcher. Rather breakthrough ideas often arise from establishing connections among ideas and hunches that may be spread across people, suggesting that “chance favors the connected mind” (Johnson 2010, p. 174). The engaged scholarship approach which advocates interactions among practitioners and academics at different stages of the research process, research teams that bring together complementary expertise, and fluid innovation spaces that companies have established for employee interaction have one thing in common—they all emphasize creating the conditions to connect potentially synergistic perspectives to see and solve complex problems and create value.

Concluding Thoughts

Problem formulation determines the research question that will be answered. As Mitroff and Silvers (2010) observe, an incomplete or imprecise answer to the right question can be a significant advance, while a complete and precise answer to the wrong question does not create value. By safeguarding against common risks and errors in problem formulation, IS researchers can position themselves to address the right question, the answer to which will advance IS scholarly knowledge. It is not about getting the answer right but about making progress in answering the right questions that will accelerate the progress of our field and the contributions we collectively make in areas of broader impact.

References

Csikszentmihalyi, M. 1988. “Motivation and Creativity: Toward a Synthesis of Structural and Energistic Approaches to Cognition,” New Ideas in Psychology (6:2), pp. 159-176.

Johns, G. 2006. “The Essential Impact of Context on Organizational Behavior,” Academy of Management Review (31:2), pp. 386-408. Johnson, S. 2010. Where Good Ideas Come From: The Natural History of Innovation, New York: Riverhead Books, Penguin Group. McGrath, J. E., Martin, J. M., and Kulka, R. A. 1982. Judgment Calls in Research (Vol. 2), Beverley Hills, CA: Sage Publications. Mitroff, I. I., and Silvers, A. 2009. Dirty Rotten Strategies: How We Trick Ourselves and Others into Solving the Wrong Problems Precisely,

Stanford, CA: Stanford University Press Rai, A. 2016. “Editor’s Comment: Synergies Between Big Data and Theory,” MIS Quarterly (40:2), pp. iii-ix. Raiffa, H. 1968. Decision Analysis, Reading, MA: Addison-Wesley. Simon, H. A. 1991. “Random Thoughts on Methods of Research,” Unpublished Manuscript, Carnegie Mellon University, Pittsburgh, PA. Van de Ven, A. H. 2007. Engaged Scholarship: A Guide for Organizational and Social Research, New York: Oxford University Press. Weber, R. 2003. “Editor’s Comment: The Problem of the Problem,” MIS Quarterly (27:1), pp. iii-ix.

MIS Quarterly Vol. 41 No. 2/June 2017 vii

viii MIS Quarterly Vol. 41No. 2/June 2017