Note: This essay does not reflect our in store offerings. While we hope to update it soon, it is now a part of graduate and undergraduate curriculum at over 30 universities worldwide, leading us to decide to post it as is for the time being.

Validity 101 – What it is, why it is essential, where others fail, and how we succeed

What it is

Validity is the most important concept in research, determining whether or not the findings have any value in the real world. Without validity, research is futile, resulting in useless information that serves no purpose. The heart of validity lies in the methodology; research methods lacking validity produce invalid results, regardless of how diligently the methodology is followed.

There are a number of different types of validity, but for our purposes we will focus on two, internal and external. Internal validity refers to the relationship among variables, essentially whether or not a cause and effect relationship can be established. While there are numerous threats to internal validity, two are particularly relevant to behavioral research/usability testing, selection and experimenter bias. Selection refers to the sample used, becoming a threat when participants are not chosen at random. Experimenter bias comes into play when the experimenter must evaluate the subject but is unable to do so objectively and either overestimates or underestimates the participant in some way.

External validity refers to whether or not the findings are applicable in the real world, and it can be compromised by almost anything – improper sampling, research setting, and pretest knowledge to name a few. One notable threat to external validity is the Hawthorne effect, a term used to describe the changes in a subject’s behavior when he or she is aware that he or she is being observed. In essence, the Hawthorne effect tells us that people do not behave in the same manner when they know they are being watched as compared to when they do not know they are being watched. This concept is crucial in observational research, as it presents a huge threat to validity.

Why it is essential

A research methodology must maintain both internal and external validity for both it and its results to have any value. While conditions surrounding a research study often require that internal validity be sacrificed for external validity (and vice-versa), every effort must be made to maintain as much validity as possible. Every compromise deflates the value of the research, eventually resulting in useless data.

Where others fail

Most usability testing/customer experience evaluation methodologies suffer from serious validity problems, both internal and external.

The problems start before the research does; participants are not selected at random, but culled from records of people who have signed up in search of incentive, or rewards for participation. To start, these participants had to go through the process of signing up online, a time-consuming process that requires a certain degree of technological know-how in addition to a patient temperament. The simple fact that they have the time to do this, nonetheless go to the lab for the testing, suggests that they are far from the average sample of Internet users. Some at-home non-observational testing where participants are asked a series of questions while browsing a site requires participants to change their proxy settings, a process that the average internet user would be at least uncomfortable doing, and likely unable. This leads to a very skewed sample of tech-savvy users, not the ideal participant for testing of this type.

The next validity problem plaguing all other usability/customer experience firms is the lab setting, an unnatural and foreign environment for participants. First, their motivation for visiting the site is very different from that of the normal site visitor because they are there for the incentive, not a need to shop. This motivational difference has profound effects on the subjects’ behavior, altering the speed at which they move through a site as well as how they approach the entire exercise. Additionally, the labs are equipped with different hardware than what they are accustomed to, and, as anyone who has ever borrowed someone else’s computer will tell you, this change is quite disorienting, leading to altered behavior.

Finally, the Hawthorne effect runs rampant in these evaluations. The subjects know they are being observed and approach the entire task in a different manner than they normally would. Attention is more focused, speed increases, more attempts are made before giving up, and careless mistakes such as typos decrease. When subjects interact with experimenters during the evaluation, they often attempt to please the experimenter by conforming to expectations and being overly complimentary. The same problem can impair the experimenter as well, preventing him or her from conducting an objective assessment.

These validity issues plague all usability testing/customer experience firms. All, that is, except Kahn Research Group.

How we succeed

Kahn Research Group addresses all of these validity issues with a methodology that cannot be beat. To start, we do not rely on just one source of data, but three: surveys, site logs, and observational data. This proprietary methodology (patent pending), the Triangulation Method, provides the most accurate view of the user experience available today.

The Triangulation Method

Surveys

We start by designing two questionnaires, one for successful site visitors (purchase or registration or however you define success) and one for unsuccessful visitors. We then integrate these questionnaires into your system, programming them to “pop-up” when visitors either complete a purchase or leave the site. The questionnaires appear at random, generating a random, and hence valid, sample. To increase participation rates and honesty, an incentive is offered for completing the survey.

Site logs

Additionally, we mine the site logs for data on our respondents, again ensuring a random sample. We examine this log extraction for numerous data points, looking at, for example, time spent on pages, pathways through the site, and numerous other variables. By tying this data to the attitudinal data, we can understand how behavior differs by psychographic, demographic, and technological profile, as well as any other factor chosen.

Observational Data

The observational research, however, is where we really shine above our competitors. We utilize one of two methodologies depending upon the goals of the study and resources available to the client.

Our first method eliminates the Hawthorne effect, and, while it does take place outside of the subject’s home or office, it is conducted in an environment that is far less foreign than a lab. Rather than bring people into a lab, we bring the lab to the people, setting up web-enabled kiosks in locations appropriate for the client. For example, a bank’s informational website would be tested from within a bank branch, a travel site in an airport, or a sports-related site at a sporting event. With cameras focused to capture body, facial, and vocal language, the subject’s behavior is documented unbeknownst to him or her. In addition to providing the most valid observational data available, this method has a very positive side effect: it promotes the site directly to your target audience. When the subject is done interacting with the website he or she is intercepted and surveyed on the experience.

Our second method eliminates the lab setting concerns while minimizing the Hawthorne effect by going out and observing people using the site in their natural setting, be it home or office. Participants are chosen from registration information (ensuring a sample of actual users), by selecting demographically appropriate people that have never used the site before, or both. Due to the somewhat invasive nature of this approach, a large incentive is offered in exchange for participation. The subject is recorded for one hour while the researcher leaves the location. By leaving the premises, the researcher significantly reduces the impact of the Hawthorne effect while increasing the comfort level of the subject, thereby increasing the validity of the results as well. Upon returning to the subject’s home or office, the researcher conducts a long, open-ended interview in order to ascertain the participant’s perception of the site.

Both methods have distinct advantages and disadvantages, and one is usually more appropriate for a site than the other. Our first method is particularly effective if you have a targeted audience that can be found at a particular location. However, service-oriented sites such as online banking and online stamps might be better served if observed in the place where they are most frequently used. The cost of in-home or in-office testing must be considered as well; with a $50 or more incentive being offered, the number of subjects should be kept to a minimum, resulting in data that is more qualitative than quantitative. Our kiosk method, in contrast, is relatively inexpensive, even free when exit interviews are not conducted, and yields both qualitative and quantitative data. But either way, our methodology yields the most valid usability/customer experience findings available.