Research >>> Surveys: Premises, definitions and choices

Think of the word ‘research’. What does it mean to you? Many people jump straight to the idea of finding something out, perhaps a bit like personal learning. Going a step further others might think of people doing market research, perhaps by means of a clipboard and questionnaire in the street or on the telephone. Those questionnaires are what many will understand to be part of a kind of survey. Surveys are a very popular and basic form of research, but for various reasons people tend to think if a survey as a very structured affair, which it actually needn’t be at all. I’m just blogging here about the different forms a survey can take and my own choices in order to open your research eyes a bit and maybe encourage you to think out of the box with your own research in and around teaching in order to give you some options.

One premise on which the concept of surveys is based is that, ‘if we wish to know what people think about certain things, we must ask them’ (Bennet, 1983). This postulates the use of surveys as a research method and tool. However, the word survey can also describe a research strategy, i.e. an ‘overall approach to doing social research’ (Robson, 2002). Robson outlines the central features of typical surveys, the idea of which most researchers are familiar with, as: the use of a fixed, quantitative design; the collection of a small amount of data in standardised form from a relatively large number of individuals; or, the selection of representative samples of individuals from known populations. Robson cites Bryman’s (1989) definition of surveys, which goes some way to clarifying Robson’s own position: “Survey research entails the collection of data on a number of units and usually at a single juncture in time, with a view to collecting systematically a body of quantifiable data in respect of a number of variables which are then examined to discern patterns of association” (Bryman 1989:104).

But the research I did for my PhD was not quantitative in nature, and it did follow a survey like approach. How? The way in which my research was inspired by a survey approach is meant in only a loose way to indicate that a number of research participants (13 in total) were found from which to collect data by interview and then work-focussed diaries over the course of a year. Before I started, it was helpful to me to consider the literature on surveys to get an overall command of the approach.

Use of Surveys and Associated Methods

Surveys can be used to collect information and observations at a single juncture in time, as stated above, but also over time; to conduct analysis of and describe current practices and events by polling surveys or by analytical surveys; or to collect factual and attitudinal data (Bennet, 1983). While much secular use of surveys is instrumental, undertaken in the interests of marketing or sales, a large amount is in the context of academic research with the aim of seeking to find out something about what is going on today (Robson, 2002), as was the case in my own research. Most surveys are carried out for descriptive purposes as part of a non-experimental fixed design, with the aim of obtaining information about the distribution of a range of ‘people characteristics’, and of relationships between them (Robson, 2002). When accompanied by the sophisticated analysis of the detailed pattern of correlations, surveys can be used to explore causal relationships (Robson, 2002).

Robson notes, however, that surveys are not well suited to carrying out general exploratory work, because he says, it is likely to be an inefficient and ineffective procedure, taking a great deal of time to analyse. He does add though that a wide range of open-ended questions may be used to explore an area, which may be a good way to find out what questions need to be asked in future stages of the research. While Robson’s comments were acknowledged, my research, *was* undertaken in a qualitative and exploratory fashion, with an element of descriptive content.

Survey Research Methods

An array of survey types and methods used in surveys are mentioned in the literature. How researchers conduct surveys can include any or all of: street or door-to-door interviews, face to face interviews, self-administered postal (mail), email or web questionnaires, telephone surveys, observational surveys, or the use of diaries in surveys (Robson, 2002). Surveys are more often undertaken by means of a questionnaire. Diaries as used in survey type research can even be thought of as a self administered questionnaire repeated at a certain frequency over a set period of time, and the diary method is cited in the survey context as a form of questionnaire/self observation log (Bennet, 1983) – I’m going to talk more about the use of diaries in another blog post on as they were a central feature of my research.

Zikmund (1988) offers a word of advice in reference to the use of self-administered questionnaires, however, and suggests that if time is a factor in the interest in research results or if attitudes are rapidly changing (e.g. toward a political event), then mail surveys may not be the best communication medium. This advice is somewhat outdated now though if the concept of mail is replaced with web or email, and can be seen as an effective method of cheap and speedy communication with willing participants. As Bennet (1983) suggests, survey techniques can be developed in a bespoke way that fits with the objectives of the research in question (Bennet, 1983).

Who is being surveyed depends again on the objectives of the research at hand and the sample population and size required, for example if undertaking quantitative research. While individuals are usually approached in surveys, an individual may be representative of an organisation, team or particular group (Robson, 2002). Alternatively, the whole population of an organisation, group or team could be surveyed (Robson, 2002). In the case of my own PhD research, the ‘population’ ‘surveyed’ were actually thought of as research participants and were self-selecting based on their interest and willingness to take part in the study, as well as fitting the criteria described in my thesis.

Weighing up the strengths, weaknesses, and threats afforded by survey methods and techniques was useful to me as this provided a framework to explain the reasons I made certain choices in my research, and allowed me to outline some of the pertinent limitations of the data collection methods used.

Strengths, Weaknesses and Threat Analysis

The literature provides many descriptions of the strengths (advantages), weaknesses (disadvantages), and threats of the survey approach and associated methods.

Strengths

The idea of the survey that inspired my qualitative research is where it compared favourably with other methods in terms of factors such as: cost, speed (Bennet, 1983) (Robson, 2002), access to information (Bennet, 1983) (Robson, 2002), simplicity of implementation, audience data accessibility, data transparency or accountability, and ease of access to attitudes, values, beliefs and motives (Robson, 2002), easy access to remote and distant geographical areas and busy individuals or other types of normally inaccessible workers (Zikmund, 1988).

Self-administered surveys also facilitate completion at the convenience of the individual (Zikmund, 1988). These factors were taken into consideration in the context of my research and influenced the decision to use diaries as a data collection tool, which were then sent and retrieved on a regular basis via email. This was especially useful where my research participants were based overseas.

With specific reference to interview surveys, face-to-face interviews between the researcher and interviewee were judged to provide an opportunity to: build rapport, clarify questions, encourage participation and involvement, utilise open-ended questions, in addition to maintaining control of the quality of the recorded response (Robson, 2002). These factors were considered when deciding to engage in interviews with the research participants of my study prior to their embarking on the main diary study. As a means by which to initiate the building of rapport between researcher and research participant, the pre-diary interviews undertaken were a success. This was then maintained to greater or lesser degrees through email contact in interaction with participants throughout the study.

Having established which type(s) of survey methods will be used it is still necessary to consider the benefits of different types of questions before proceeding to the design and construction of the survey. Open-ended responses provide a good warm up (Zikmund, 1988), can be used when all of the possible answer categories are not known, allow the respondent to respond adequately, can be used when there are too many potential answers to list on the questionnaire, are preferable for complex issues that are difficult to condense into smaller categories, and enable a greater degree of respondent creativity (Blank, 1984). This type of creativity can in turn facilitate a spontaneous response, a conversational style, and potentially lead to effective communication (Zikmund, 1988). Answers to closed-ended questions on the other hand are standard, relatively complete, and a minimum of irrelevant responses are received, they can be compared from person to person, and are easier to code and analyse (Blank, 1984). From the perspective of the respondent, they often understand the meaning of closed-ended questions and therefore they are easier to answer, in addition to the fact that providing sensitive information such as income, education and age may also be easier (Blank, 1984). Fixed-alternative questions require less interviewer skill, are quicker, and easier to answer, while responses facilitate answer comparability, coding, tabulating and interpretation (Zikmund, 1988).

These factors were considered in terms of the interview style I deployed in pre-diary interviews.

Weaknesses

In spite of all their advantages and strengths, surveys do have numerous disadvantages and weakness. Firstly, securing involvement is difficult and response rates are often low (Bennet, 1983; Robson, 2002). Other problems include lack of knowledge regarding the motivation of the participant to take part in the survey, which could be down to politeness, social desirability or even boredom, all of which could place doubt on the validity of responses in terms of honesty and as to whether they provide an accurate reflection of beliefs or behaviour (Robson, 2002). In addition to motivation, survey response data are influenced by individual memory, experience and personality (Robson, 2002).

Robson (2002) highlights some key weaknesses of face-to-face interviews. He relates that they reduce participant anonymity and may decrease openness. Their cost is also deemed to be high, especially in reference to completion time required, interviewer time, and travel time and cost. The sample distribution cannot usually be wide, and the motivation, personality, skills and experience of the interviewer can all affect the data adversely, in addition to the interviewer unwittingly or otherwise creating bias by influencing responses. The dynamics of the relationship between interviewer and interviewee can also be affected due to differences in characteristics such as class or ethnic background (Robson, 2002).

The bottom line for surveys though is wording, and, as Zikmund acknowledges, ‘A research survey is only as good as the questions it asks’ (Zikmund, 1988). And, as Robson points out, while the reliability and validity of the survey depends largely on the technical proficiency of the researcher, bad questions negate the value of the whole exercise (Robson, 2002). Question ambiguities and misinterpretations may go undetected with no opportunities to ask the interviewer questions or for the interviewer to probe for additional information or clarification (Robson, 2002; Zikmund, 1988).

Open-ended questions increase the researcher workload in terms of coding, editing, and analysing the data – the uniqueness of each respondent’s answer means there is some difficulty in categorising and summarising the answers, while interviewer bias may influence the responses (Zikmund, 1988). This style of questioning also leads to the collection of much irrelevant information, and low standardisation of data that makes comparison and coding challenging and subjective (Blank, 1984). In order to answer an open-ended question, better writing and intellectual skills and a reasonable educational level is needed (Blank, 1984).

These factors were all relevant to the research I did and therefore imply limitations on the interviews and diary study carried out. These issues were therefore revisited in the discussion chapter of my thesis.

Threats

Some threats arising from the use of surveys can be prepared for and sometimes avoided by advance actions, or at least made sense and use of retrospectively. Because ‘a research survey is only as good as the questions it asks’ (Zikmund, 1988) questionnaire design becomes crucial and Zikmund suggests that some problems may be minimised or avoided altogether if a skilled researcher composes the questions. He adds that if questions or instructions are difficult to comprehend, respondents must use their own interpretations, which may be wrong (Zikmund, 1988). Other threats associated with the use of surveys may be beyond the control of the researcher. These could include the respondent refusing to answer personal or other types of question, or giving false answers (Zikmund, 1988; Bennet, 1983). While securing involvement has to be accepted as a likely hazard of the strategy per se, other problems hampering this and the researcher timetable could include a range of other factors such as: bad weather, school holidays, industrial disputes, flu or other epidemics, and computer breakdowns (Robson, 2002).

The other threat associated with most surveys is the external validity problem concerning the lack of relation between attitude and behaviour, which is reflected in what people write they do in contrast with what they actually do (Robson, 2002). Blank (1984) lists other threats, which include the respondent: answering the way he thinks he should answer, whether it is an honest reply or not (caused by social desirability or sensitive questions); being afraid that his responses will reveal a lack of education or that he will appear stupid; saying that his time is too valuable to waste on the study, that the study is not applicable to him; saying he cannot answer because the questions are too general and vague or because he has never thought about the topic (Blank, 1984). Again, these factors were all relevant to my own research and therefore imply limitations on the study carried out, and these issues were therefore revisited in the discussion chapter of my thesis.

When you do any of your own research you should record your choices, your reasons for them, and any limitations and implications this had on the outcomes of your research. Your choices will no doubt reflect your main research question. It depends on what you want to find out as to how you go about trying to find your answer. A survey approach and method may or may not be the right one for you.

Have fun! 😎

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s