Certifying bodies vary in how they approach referencing. Some require every item on an examination be referenced, some identify a reference for items in most instances, and yet others rely on the consensus judgement of a group of subject-matter experts (SMEs). Although some items may be difficult to reference, our recommendation to clients is for all examination items to have a reference.
Why should we reference every item?
The main reason we recommend referencing every examination item is to support the validity of the examination content. Certifying bodies often receive emails and phone calls from candidates challenging examination content and thereby the outcomes of their testing experience. Although it may be tempting to respond that a group of SMEs agreed on the correct answer, pointing to an authoritative reference in the candidate’s profession that corroborates the SME judgements carries much more weight.
Another reason we recommend referencing every examination item is to ensure the defensibility of examination results. In very rare circumstances, this means defending an examination in court. In such cases, certification examinations become more defensible when SME judgements are combined with authoritative references.
What are some recommendations for item referencing?
- Have an approved reference list. Candidates should be able to use a list of approved references to prepare for their certification examination; however, reviewing those references should not be required in order to pass the examination. Certifying bodies should strive for a concise list of approved references, yet with a sufficient number to cover the entirety of the examination’s scope of content.
- Have a policy for referencing items. The certifying body should establish general guidance on acceptable item referencing practices. Such a policy may address which types of references are acceptable, how often references should be updated, how many references are needed per item, and so on. Textbooks, journal articles, and position papers are the types of references most often used in certification. Certifying bodies should consult the examination development committee or other SMEs to establish the referencing policy and how frequently it should be updated.
- Incorporate referencing into item writing. An item should be referenced when it is initially written, and that reference should be confirmed throughout the item development process. When items without references are selected during examination assembly, the examination development committee must spend time referencing items during their review of the examination, rather than focusing on examination content. This is an inefficient use of the committee’s time and makes the review process longer than necessary.
- More is sometimes better. It is easy for SMEs to reference facts and definitions straight out of a textbook. However, we often find SMEs struggle with content that requires application of knowledge, such as clinical judgement or evaluation of a novel issue or situation. Nevertheless, identifying a reference that can support practitioner judgement is helpful, even when the reference does not describe the exact situation. In cases where an item is more complex or requires the respondent to evaluate competing sets of information, it may be necessary to have references for each principle required to arrive at the correct answer.
We want to ensure each item that appears on an examination form is fair, accurate, meaningful, and current. Enforcing a policy on referencing items is an excellent method to ensure the content of each item is factually accurate and up-to-date. Item referencing also makes examination results more defensible and the examination development process more efficient. So given the benefits, why wouldn’t you reference items?!