Research

Design Research Methodology (by Example)

In this section of the page the general method underlying design research in its multiplicity of as-practiced variants is described, followed by a discussion of the method as used in a published example of IS design research.

The astute reader will recognize Figure 5, the general methodology for all design research, as a variant on Figure 3, reasoning in the design cycle. This is a logical and inevitable result of the fact that in design research knowing (Figure 3) is making (Figure 5). To better focus on the process as a research method, a column labeled Outputs has been substituted for the Logical Formalism column.

Note: there are many excellent descriptions (and diagrams) of the process of design research in IS, cf. Hevner, et al., 2004; Purao, 2002; Gregg, et al., 2001;. March and Smith, 1995; Nunamaker, et al., 1991. We chose this diagram because it emphasizes the knowledge generation inherent in the method and because it originated in an analysis of the processes inherent in any design effort.

With reference to Figure 5 a typical design research effort proceeds as follows.

Awareness of Problem: An awareness of an interesting problem may come from multiple sources: new developments in industry or in a reference discipline. Reading in an allied discipline may also provide the opportunity for application of new findings to the researcher’s field. The output of this phase is a Proposal, formal or informal, for a new research effort.

Suggestion: The Suggestion phase follows immediately behind the proposal and is intimately connected with it as the dotted line around Proposal and Tentative Design (the output of the Suggestion phase) indicates. Indeed, in any formal proposal for design research such as one to be made to the NSF or an industry sponsor, a Tentative Design and likely the performance of a prototype based on that design would be an integral part of the Proposal. Moreover, if after consideration of an interesting problem a Tentative Design does not present itself to the researcher, the idea (Proposal) will be set aside. Suggestion is an essentially creative step wherein new functionality is envisioned based on a novel configuration of either existing or new and existing elements. The step has been criticized as introducing non-repeatability into the design research method; human creativity is still a poorly understood cognitive process. However the step has necessary analogues in all research methods; for example, in positivist research creativity is inherent in the leap from curiosity about an organizational phenomena to the development of appropriate constructs that operationalize the phenomena and an appropriate research design for their measurement.

Development: The Tentative Design is implemented in this phase. The techniques for implementation will of course vary depending on the artifact to be constructed. An algorithm may require construction of a formal proof. An expert system embodying novel assumptions about human cognition in an area of interest will require software development, probably using a high-level package or tool. The implementation itself can be very pedestrian and need not involve novelty beyond the state-of-practice for the given artifact; the novelty is primarily in the design, not the construction of the artifact.

Evaluation: Once constructed, the artifact is evaluated according to criteria that are always implicit and frequently made explicit in the Proposal (Awareness of Problem phase). Deviations from expectations, both quantitative and qualitative are carefully noted and must be tentatively explained. That is, the evaluation phase contains an analytic sub-phase in which hypotheses are made about the behavior of the artifact. This phase exposes an epistemic fluidity that is in stark contrast to a strict interpretation of the positivist stance. At an equivalent point in positivist research, analysis either confirms or contradicts a hypothesis. Essentially, save for some consideration of future work as may be indicated by experimental results, the research effort is over. For the design researcher, by contrast, things are just getting interesting! Rarely, in design research, are initial hypothesis concerning behavior completely borne out. Instead, the evaluation phase results and additional information gained in the construction and running of the artifact are brought together and fed back to another round of Suggestion (cf. the circumscription arrows of Figures 3 and 5). The explanatory hypotheses, which are quite broad, are rarely discarded, but rather are modified to be in accord with the new observations. This suggests a new design, frequently preceded by new library research in directions suggested by deviations from theoretical performance. (Design researchers seem to share Allen Newell’s conception (from Cognitive Science) of theories as complex, robust nomological networks. This conception has been observed by philosophers of science in many communities (Lakatos, 1978), and working from it Newell suggests that theories are not like clay pigeons, to be blasted to bits with the Popperian shotgun of falsification. Rather they should be treated like doctoral students. One corrects them when they err, and is hopeful they can emend their flawed behavior and go on to be ever more useful and productive (Newell, 1990).)

Conclusion: This phase is the finale of a specific research effort. Typically, it is the result of satisficing, that is, though there are still deviations in the behavior of the artifact from the (multiply) revised hypothetical predictions, the results are adjudged "good enough.” Not only are the results of the effort consolidated and "written up” at this phase, but the knowledge gained in the effort is frequently categorized as either "firm” - facts that have been learned and can be repeatably applied or behavior that can be repeatably invoked - or as "loose ends” – anomalous behavior that defies explanation and may well serve as the subject of further research.

An Example of IS Design Research

The example we have chosen to add detail and concreteness to the discussion of design research philosophy and method in Information Systems is one from the joint experience of the design research page authors. We make only two claims for this research: (1) it is a reasonable example as it comfortably encompasses all the points of the preceding discussion (2) since it is our research we are privy to and able to present a multitude of details that are rarely written up and available in journal publications. We describe the research, from conception to the first publication to be drawn from it, in phases corresponding to those in diagrams 3 and 5.

Smart Objects: A Design Research Project

Awareness of Problem

In the mid-1980’s one of the senior project participants, Vijay, began actively seeking to extend his research from designing efficient data and file structures (a primarily computer science topic) to software engineering (an area with a significant IS component). In the course of a discussion with one of his colleagues at Georgia State University (GSU) he became aware of a situation that showed research promise: development of a computerized decision support system for nuclear reactors. Three Mile Island had brought national awareness to the problems associated with safe operation of a nuclear power plant, rule based decision support systems were a current area of general IS interest, and the director of the research reactor at Georgia Tech was interested in developing a system to support its operations.

A doctoral student (Gary) was brought into the project to begin a preliminary support system development in the rule-based language Prolog. Within a few weeks it became apparent that a system to support the several thousand procedures found in a typical commercial power plant would be nearly impossible to develop in Prolog and if developed would be literally impossible to maintain. The higher-level expert system development packages available at the time (and currently) were more capable but still obviously inadequate. The difficulty of constructing and maintaining large expert systems was widely known at the time however, the Prolog pilot project gave the research group significant insights they would not otherwise have had into the root causes of the problem: continuously changing requirements and the complexity inherent in several thousand rule-based interlocking procedures. Out of detailed analysis of the failed pilot system emerged the first awareness of the problem on which the research would focus: how to construct and continuously maintain a support system for the operation of a complex, hierarchical, procedure driven environment.

Suggestion

There are many approaches to the problems of software system complexity and the research group discussed them over a period of months. Some of the alternatives that were discarded were: development of a new software development methodology specifically focused on operation support systems, automation of the maintenance function, and development of a high-level programming environment. New insights into the problem continued to emerge even as (and precisely because) potential solutions to the problem were considered. One key insight was that the system complexity resided primarily in control of the system, that is, although the individual procedures could be modeled straightforwardly, the procedure which should take precedence (control) over the others and where the results of that procedure should be routed depended in a highly complex fashion on past and present states of multiple procedures. Essential to the development of the system was the effective modeling of this complex control structure.

By this point Gary had decided to adopt the problem as his dissertation topic and under Vijay’s direction began extensive research into various mechanisms for modeling (describing in a precise, formal way) control. As the realization grew that they were in effect seeking to describe the semantics of the system, his reading began to focus especially on some of the techniques to emerge from the area of semantic modeling.

During the alternating cycles of discussion, reading and individual cogitation that characterize many design research efforts, several software engineering concepts were brought together with a final key insight to yield the ultimately successful direction for the development. During one discussion Vijay realized that the control information for the system was knowledge, identical in form to the domain knowledge in the procedures and could be modeled with rules, in the same way. However, since the execution of the individual procedures was independent of the control knowledge, the two types of rules could execute in different cycles, partitioning and greatly reducing the complexity of the overall system. Finally, the then relatively new concept of object orientation seemed the ideal approach to partitioning the total system knowledge into individual procedures. And if each object were further partitioned into a domain knowledge and an control knowledge component, and if the rules were stated in a high level English like syntax that was both executable and readable by domain experts . . .

Awareness of Problem Redoux

As noted in the general discussion of the design research method, any of its phases may be spontaneously revisited from any of the other phases. Especially in the early stages of a project, this results in a conceptual fluidity that can be disconcerting to practitioners of less dynamic paradigms. Though it is difficult in retrospect to pinpoint exactly where in the process the change occurred, by the inception of the development phase the problem statement had changed to a sub-goal implicit in the original problem statement: how to effectively model operations support systems for complex, hierarchical, procedure driven environments. [This sort of "drilling down” into the problem or re-scoping the research at a more basic level occurs frequently in all research, but is effectively part of the method in design research.]

Development

Although development of a design research artifact can be straightforward, that was not the case for smart objects. The construction was completely conceptual and involved the "discovery” through multiple thought and paper trials of the details of the novel entity that had been conceptualized at a high level in the Suggestion phase, the "smart object.”

For example: what (exactly) would the syntax be for the two types of rules, domain and control? How (exactly) should the two rule evaluation cycles for each type of knowledge interleave? Should the two types of knowledge be permitted to interact? If so, how? Should control rules have the ability to "write” or "rescind” domain rules, a la Lisp? Or vice versa?

In a conceptual development such as this, the suggestion and construction phases blur because a successful design decision is an output product. The final deliverable (from this initial development) was a conceptual model consisting of: (1) a set of meta-level rules for implementing domain knowledge and control knowledge separately, but within a single structure, the "smart object” and (2) another set of meta-rules that described how the domain and control knowledge, once "modeled” as smart objects, would be interpreted (a virtual machine for executing the smart objects.)

Evaluation

In a sense evaluation takes place continuously in a design process (research or otherwise) since a large number of "micro-evaluations” take place at every design detail decision. Each decision is followed by a "thought experiment” in which that part of the design is mentally exercised by the designer. However for the remainder of this section we will describe the "formal” evaluation that occurred after the design had stabilized.

In order to test the conceptual design, various operating environments were modeled and "hand-stepped” through the execution rules to determine that logically correct system behavior occurred at appropriate times in the simulation. The simulation that appeared in Gary’s dissertation, the first publication to result from the research, was a grocery bagging "robot.” This example had been popularized in a best-selling artificial intelligence textbook of the time and had the advantage of being a familiar logic test bed to many external evaluators of the artifact. Exponents of other IS research paradigms may find the evaluation criteria simplistic, and wonder why, for example, modeling of the nuclear power plant operating environment was not the obvious choice. The answer is: resources; the modeling and hand testing of even the grocery-bagging example occupied several man-months. During the evaluation minor redesign of the artifact (the smart object conceptual model) occurred on several occasions, a common occurrence in design research. By the end of the evaluation phase the smart object model had successfully completed simulation of numerous bagging exercises and was adjudged a success by the design team.

Conclusion

The finale for the first research effort involving Smart Objects was the codification of the problem development, design basis in prior work, the design itself, and the results of the evaluation effort in Gary’s dissertation (Buchanan, 1991) The successful defense of the dissertation at GSU required careful consideration and judgment of the artifact and its performance by a committee made up primarily of other design researchers. The core concepts were considered to have substantial merit, and Gary and Vijay produced several conference papers based on smart objects.

Epilogue

After Gary’s graduation Vijay and Gary collaborated on a paper based on the research project and submitted it to IEEE Transactions on Data and Knowledge Engineering (TDKE). The paper was returned for substantial revisions. At this point Gary’s interest in the project waned, however a recently admitted GSU CIS doctoral student (Bill) found the concepts interesting enough to enter into the research group and continue the development effort. After four years, four conference papers on smart objects and related topics and three major revisions the TKDE paper was finally published as "A Data/Knowledge Paradigm for the Modeling and Design of Operations Support Systems.” (Vaishnavi, Buchanan and Kuechler, 1997) By the time of acceptance, smart objects had been through several additional design research cycles, each focusing on the refinement of a different aspect of the original design, or a critical support function for its use-in-practice such as the methodology developed for partitioning workflow information into smart objects.

Design Research vs. Design

A significant and valid question posed frequently to design researchers is: How is your research different from a design effort; what makes your work research and not simply state-of-practice design?

We propose that design research is distinguished from design by the production of interesting (to a community) new knowledge. In a typical industry design effort a new product (artifact) is produced, but in most cases, the more successful the project is considered to be, the less is learned. That is, it is generally desirable to produce a new product using state-of-practice application of state-of-practice techniques and readily available components. In fact, most product design efforts in industry are preceded by many meetings designed to "engineer the risk out of” the design effort. The risks that are identified in such meetings are the "we don’t know how to do this yet” areas that are precisely the targets of design research efforts. This is in no way meant to diminish the creativity that is essential to any design effort. We merely wish to point out that design is readily distinguished from design research (within its community of interest) by the intellectual risk, the number of unknowns in the proposed design which when successfully surmounted provide the new information that makes the effort research and assures its value.

Citations on Design Research

Resources for Design Researchers

Communities of Practice

  • Design Research Society (DRS) is a multi-disciplinary international learned society founded in 1967. Members of DRS are drawn from diverse backgrounds ranging from fine art to engineering to computing. The aims of DRS include advancing the theory and practice of design and understanding design research and its relationship with its education and practice. DRS is involved with such activities as organizing and sponsoring conferences, sponsoring e-mail discussion groups and a monthly e-mailed newsletter.
  • Design-Based Research Collective: The design research paradigm appears to have been pursued as well in education research (especially educational software and systems design). Although many of the early core readings are the same (Schon, Simon, Alexander) the later traditions overlap a lot less. In education it has traditionally been called design experiments, although this term is falling out of favor. The Design-Based Research Collective has been helping define the theory and practice of this research paradigm.
  • AIS Systems Analysis and Design Special Interest Group (SIGSAND) "AIS SIGSAND provides a forum for AIS members: To discuss, debate, collaborate, develop, and promote issues pertaining to the history, reference disciplines, theories, ontologies, methodologies and techniques, principles, new developments, practice, evaluation, quality control, management and pedagogy of systems requirements, analysis, design, and implementation tasks and technologies In the business and organizational contexts. "
  • AIS Special Interest Group on Philosophy and Epistomology in IS (SIGPhilosophy) "Currently we can observe a growing methodological debate in IS research. This debate appears to focus on epistemic issues, especially research methods and techniques without relating to the broader issues of the philosophy of science, epistemology and theory of knowledge. To overcome too narrow focus, it will be necessary to link the debates in IS research to questions about the very nature of research and science and their societal role in general."
  • Association for Information Systems (AIS) is "the premier global organization for academics specializing in Information Systems."
  • Association for Computing Machinery (ACM), founded in 1947, is "a major force in advancing the skills of information technology professionals and students worldwide."
  • Association for Information Systems (AIS) is "the premier global organization for academics specializing in Information Systems."
  • IEEE Computer Society (IEEECS) is the "world's leading organization of computer professionals with nearly 100,000 members.
  • IEEE Systems, Man, and Cybernetics Society (IEEESMC)
  • American Society for Information Science and Technology
  • INFORMS (Institute for Operations Research and the Management Sciences)
  • The Information Institute"is an academe-industry consortium founded to further understanding of intricate relationships between information science and technology."

Design Research Centers / Labs

  • Center for Design Research Center for Design Research at Stanford University, established in 1984, focuses on engineering design, design tool development, and design process research and promotes collaboration between industry and academia.
  • MIT Media Lab The Media Laboratory at MIT, established in 1985, emphasizes interdisciplinary research that combines disciplines such as cognition, electronic music, graphic design, holography with research in computation and human-machine interfaces.
  • The Palo Alto Research Center The Palo Alto Research Center (PARC), a subsidiary of Xerox Corporation, conducts interdisciplinary design research in physical, computational, and social sciences.
  • Carnegie Mellon Software Engineering Institute (SEI) SEI is a federally funded research and development center sponsored by the US Department of Defense devoted to making measured improvements in software engineering capabilities.
  • Information Systems Integration and Evaluation [ISEing] is a center located at Brunel University, West London (U.K.) that is devoted to the conducting of basic and applied research on designing organizations and systems for actual operational conditions.
  • AI in Design Webliography is a a collection of informaiton at Worcester Polytechnic Institute about AI Design, Knowledge Based Design, Intelligent CAD, Computational Approaches to Design, and Design Theory and Methodology.
  • IIT Institute of Design The Institute of Design (ID) at Illinois Institute of Technology is an international leader in teaching systemic, human-centered design. [I was especially intrigued by the section on design [research] methods. wlk]

Other Research Centers / Labs

Journals

  • Journal of Design Research an electronic journal established in 2001, is a general design research journal that is devoted to integrated studies of social sciences and design disciplines emphasizing human aspects as a central issue of design

The following journals tend to be receptive to design research in information systems:

ACM Transactions on:

IEEE Magazines

IEEE Transactions on:

Other Journals

Conferences

Help Build This Page

You are invited to contribute links to design research material. Additionally, we are soliciting critiques of the page and short abstracts of items in the references pages (maximum 100 words). Please contact the Section Editors by email Vijay Vaishnavi or Bill Kuechler to see how you can help

Citation Information

Vaishnavi, V. and Kuechler, W. (2004). "Design Research in Information Systems” January 20, 2004, last updated August 16, 2009.

This page was last updated on August 20, 2009

 

Connect With Us :