Survey Instruments in IS by Peter Newsted, Sid Huff Malcolm Munro, Andrew Schwarz, & Tung Cu
A Tutorial on Survey Instruments
For students, and those less experienced, what follows is intended to explain what surveys do and to help one understand when and how to use them. It is not a comprehensive tutorial, but rather an introduction with pointers to more detailed sources. It also includes a bibliography on materials about surveys, their construction, and their use.
Epistemologically surveys provide one way of obtaining and validating knowledge. Consequently, it is important that the context for this type of IS research be considered. Hence, this section also provides a summary that explains when survey research is appropriate and the steps required to use the survey approach correctly.
This tutorial will provide answers to the following questions:
- What does survey research try to achieve?
- When is survey research appropriate?
- How are surveys developed and used?
- Are there issues in automating surveys?
What does Survey Research try to Achieve?
First, one should understand what a survey is actually used for. A survey is a way of going from observations to theory validation. As part of a panel discussion (Newsted, Chin, Ngwenyama, and Lee, 1996), Chin presented an exhibit that nicely outlines how this is done:
- Observed responses become data on single questions.
- These questions are aggregated into scales.
- Appropriate numerical formulas are applied to these numbers.
- The results of these formulas lead to conceptual representations of what has been measured.
These concepts are the constructs of interest to researchers. The usual objective for Information Systems researchers using this approach is to determine the relationship of these constructs as a way of making sense of behavior surrounding and involving IS.
The methodology used to study these relationships is also an important aspect of survey research. Techniques range from the reporting of simple scale means, the use of analysis of variance of results in different conditions, through regression analysis and the analysis of paths between constructs, to the use of "second generation" techniques such as LISREL and PLS, which analyze measurement models and structural models simultaneously.
Chin's (1995) website provides pointers toward more advanced approaches using structural equation modeling (SEM). Among other things it discusses:
- The need for a covariance matrix to show the relations among constructs and variables.
- Model specification and graphical ways of representing a model.
- Steps involved in using structural equation models.
- A link describing the Partial Least Squares Approach (PLS) to SEM.
- An extensive set of references using the PLS approach.
When is Survey Research Appropriate?
In a panel discussion at the 1996 International Conference on Information Systems, Newsted, Chin, Ngwenyama, and Lee (1996) raised the question whether surveys had outlived their usefulness. The panelists concluded that surveys are appropriate in certain conditions and less useful in others. But this is more complicated than just saying surveys should be used only in objective or positivist research and not in more subjective or interpretivist research. Burrel and Morgan (1979) provide a detailed distinction between these approaches.
In the '96 panel, Allen Lee indicated that in positivist research, surveys are particularly useful in determining the actual values of variables under study, and the strengths of relationships among them. In an interpretivist context surveys are appropriate as a complement to other forms of data or observations. They can serve as a way to add to one's knowledge through "triangulation" as one of several methods. Thus it is important to realize that while surveys are typically used in quantitative research, they can also help qualitative researchers as well. The panel gives further details. In particular one should review the notes below the slides in Lee's presentation as they discuss the inter-relations between these types of research and describe situations where these approaches and orientations fit together. For those interested in the qualitative approach, Myers provides an excellent site offering considerable detail on this type of research.
Overall the survey approach can be seen to have the following strengths and weaknesses:
- Surveys are easy to administer.
- Surveys are simple to score and code.
- Surveys determine the values and relations of variables and constructs.
- Responses can be generalized to other members of the population studied and often to other similar populations.
- Surveys can be reused easily, and provide an objective way of comparing responses over different groups, times, and places.
- Surveys can be used to predict behavior.
- Specific theoretical propositions can be tested in an objective fashion.
- Surveys can help confirm and quantify the findings of qualitative research.
- Surveys are just a snapshot of behavior at one place and time.
- One must be careful about assuming they are valid in different contexts. In particular, different cultures may produce different results. Kettinger, Lee, and Lee (1995) provide a good example of this by showing the effect of cultural differences in the measurement of IS service quality.
- They do not provide as rich or "thick" description of a situation as a case study.
- They do not provide as strong evidence for causality between surveyed constructs as a well designed experiment.
How are Surveys Developed and Used?
After having decided to employ survey research it is advisable to search the literature for measures of your constructs of interest. This process can be expedited by reviewing our list of constructs or using the search engine. If you are unable to find an appropriate instrument with an associated methodology, it may be necessary to develop one. To do so requires a theoretical foundation in the area of study and an understanding of the survey process. But, even when an existing instrument is being used, an appropriate theory and a correct process are necessary.
A carefully constructed theory is a precursor to the actual use of an instrument. Theories propose constructs and their expected relations. They guide the investigation of these relations as they attempt to help one understand behavior and identify regularities in it. However, this is also a two-way street: instrument development can often refine theories as well. As Straub (1989) indicates, "Attention to instrument issues ... brings greater clarity to the formulation and interpretation of research questions. In the process of validating an instrument, the researcher is engaged, in a very real sense, in a reality check. He or she finds out in relatively short order how well conceptualization of problems and solutions matches with actual experience of practitioners." (p. 148).
A number of exemplary survey-based studies describe instrument development within a theoretical context. In the following three studies researchers have taken particular care to consider theories and to identify the steps in instrument creation and validation:
- Straub (1989) uses a detailed example in the measurement of computer abuse to discuss instrument validation and its importance in confirming theories.
- Moore and Benbasat (1991) describe the development of multiple constructs to measure IT adoption. In particular they provide detail on item creation and the derivation of items based on the theory of innovation diffusion.
- Chin, Gopal and Salisbury (1997) demonstrate the development of a five item faithfulness of appropriation construct based on three separate experiments on adaptive structuration. Specifically they show the differences between a principal components analysis and structural equation modeling. They also look are the effects of positive and negative wording of items.
With a theoretical foundation in place, the activities of the survey process can be considered. Grover (1997) provides a detailed check list to be followed in the development and use of an instrument. Significant among these steps are:
- Determination of the unit of analysis (e.g., the individual, group, or organization)
- Creation and use of multi-item scales
- Pre-testing and use of pilot data
- Random sampling from a defined sample frame
- Determination of an appropriate response rate and evaluation of nonresponse bias
- Assessment of whether significant correlations imply real causal relations
- Determination of statistical power of the final analysis.
Dillman (1978) offers a number of very practical tips to facilitate the surveying process and increase response rates. Salant and Dillman (1994) provide a very good update to Dillman's initial classic.
The following references provide more detail on the theories and processes involved in surveying. These are divided into general references on surveying, sources specific to IS, and those dealing with a variety of methodology issues. A broad understanding of survey methodology as well as knowledge about specific IS issues are useful in using the survey approach.
Are there Issues in Automating Surveys?
Since surveying involves a number of detailed and sometimes tedious steps, it is reasonable to consider its automation. This is possible, however, care must be taken to ensure that computerized surveying leads to the same results as traditional, noncomputerized approaches using paper and pencil.
Bratton and Newsted (1995) summarize much of the work in this area and show that differences in results can be due as much to different interfaces as to differences in techniques. Interestingly, fewer differences seem to be found in newer work (Liefeld, 1992; Potosky and Bobko, 1997), as compared to studies before 1990 (e.g. Liefeld, 1988; Kiesler and Sproull, 1986). Recently, however, Webster and Compeau (1996) have found that, while there may not be differences in means between computerized and noncomputerized surveying, there may be differences in correlations with other variables such as those involving training performance. While there appears to be a growing acceptance of automated survey methodologies, there is a need for caution in the use of computerized techniques.
There are a number of commercial companies that provide computerized surveying for a fee. While we have not tested these products, they appear to have useful capabilities -- perhaps, especially for those wishing to do commercial surveys in IS.
For all references, please see the IS-Specific References on Survey Research.