7. Potential limitations

Research 101

Consider all of the following:

The sample – is the sample size and type appropriate? A small sample size means low statistical power.
Time – are frequency, length and duration of the intervention sufficient?
Are the data analysis and the statistical tests appropriate?

Internal Validity examples

History: making sure the participants do not participate in other events that may influence the variables to be measured. (Needs to be listed as part of the Methods section)
Prevention of selection bias: Is the study randomized?
Attrition: How many participants have withdrawn from the study?
Expectancy: Have the raters (people who do the measurements) been trained? Are there multiple raters? Are the participants taking multiple versions of the same test?

External Validity examples

To show the complexity of the subject: Steps to make external validity high often result in a lowering of internal validity.
Example: If the participants taking multiple versions of the same test to increase internal validity, the practice effect might influence the results (external validity).
Are the participants a good representation of the population? If the sample is composed of a homogeneous population, like all low achievers or all high achievers, the results won’t be generalizable to the “average” person.
Are the results of the study tainted by the Hawthorne effect? Study participants may be behaving differently because they know they are in an experimental study.