7. Potential limitations
Consider all of the following:
Internal Validity examples
External Validity examples
- Are the instruments and tests utilized current, calibrated, reliable and valid?
- The sample – is the sample size and type appropriate? A small sample size means low statistical power.
- Time – are frequency, length and duration of the intervention sufficient?
- Are the data analysis and the statistical tests appropriate?
Internal Validity examples
- History: making sure the participants do not participate in other events that may influence the variables to be measured. (Needs to be listed as part of the Methods section)
- Prevention of selection bias: Is the study randomized?
- Attrition: How many participants have withdrawn from the study?
- Expectancy: Have the raters (people who do the measurements) been trained? Are there multiple raters? Are the participants taking multiple versions of the same test?
External Validity examples
- To show the complexity of the subject: Steps to make external validity high often result in a lowering of internal validity.
- Example: If the participants taking multiple versions of the same test to increase internal validity, the practice effect might influence the results (external validity).
- Are the participants a good representation of the population? If the sample is composed of a homogeneous population, like all low achievers or all high achievers, the results won’t be generalizable to the “average” person.
- Are the results of the study tainted by the Hawthorne effect? Study participants may be behaving differently because they know they are in an experimental study.