Test development
A beginner’s guide.
Why is test content so important?
The purpose of any certification testing program is to validate skills and knowledge. To assure employers that an individual is competent in the tasks required to carry out a certain job role, to the requisite standard.
For a certification program to be effective – and successful:
- Organizations must be confident that the standards and quality of the certification reliably attest to the competence of the certificate holder.
- Employees must feel confident that the certification maintains the quality standards needed to be recognized and valued by employers.
Valid, reliable and fair test content is central to providing the quality and assurance required from certification programs. And this can only be achieved through a rigorous test development process. One that involves numerous steps, plus the engagement of multiple subject matter experts (SMEs) and psychometric and test development specialists.
This guide outlines the key steps and experts who need to be included in the development of certification test content. As well as top tips, advice and best practice examples. All based on our decades of experience developing high quality certification test content that is trusted by both organizations and the individuals that work for them.
On this page you will learn:
High quality test content must be…
- Valid – does a test assess the knowledge and skills it is supposed to measure?
- Reliable – does a test consistently assess the required knowledge and skills?
- Fair – does a test assess the required knowledge and skills without unfairly disadvantaging any individual or group of test takers?
“The consequences of poorly designed test content are serious. By certifying individuals who are not competent in the tasks required to carry out a role, you risk the reputation and future of your certification programs and your organization.
Certification programs need to be able to rely on the validity of test results. That is why the test content we develop is always based on proven assessment science and psychometric principles. To deliver a valid, reliable and fair outcome, every time.”
Isabelle Gonthier
Senior Vice President of Content Science, PSI
“The feedback from everyone on the Practice Analysis Task Force of Acute Care Nurse Practitioners was about the pride they had in such a good, solid, fair – fair, a word that they used a lot – process. These are high-level, intense people who will tell you if they’re not impressed by a process.”
JoAnne Konick-McMahan - PCCN
Certification Practice Specialist, AACN
How do SMEs define test specifications through job analysis?
The test content development lifecycle
Job analysis and test specifications:
The test development process starts by defining what needs to be assessed and how – test specifications including content outline. This is achieved through a job analysis and in-depth consultation with subject matter experts (SMEs).
For any test, including certification testing, this stage of the process identifies:
- The tasks an individual must perform in their job or specialty area.
- What they need to know to perform these tasks.
SMEs assemble
The first step in a job analysis for certification programs is the involvement of SMEs, to create a list of the knowledge and tasks required to perform a job to the necessary standard. The best practice approach to developing this list is a survey, distributed to as many SMEs working in your profession as possible. The test specifications, defining what is on the test and the structure of the test, is then created from the results of the survey. The test specifications are also included in the candidate handbook to provide the topics and subtopics that are covered in the test.
Involve the right SMEs
Ensuring the right perspectives are counted is critical to the validity and reliability of your test content. When identifying your SMEs, the first step is to determine your target population – the pool of individuals your SMEs should come from.
Individual SMEs should be able to provide insights into the target job role or areas of knowledge required, so they should always:
- Hold the target certification or a related credential with similar requirements.
- Have work experience in the profession within the past 6 to 12 months.
In addition, a group of SMEs should be reflective of the target population’s demographic characteristics. For instance, this includes:
- Years of experience
- Level of education
- Geographic location
- Workplace Setting
- Training
- Area of Speciality
Recruiting SMEs who have previous experience with test development can be useful. But to be representative, your group should also include newly credentialed people to the field.
“There’s this myth in the certification space in general, ‘Oh, it’s just a select group of individuals who create these tests; people who have been out of practice for a while and think they know what’s best’. Our partnership with PSI seeks to dispel those myths.”
JoAnne Konick-McMahan - PCCN
Certification Practice Specialist, AACN
Tips for SME recruitment
SMEs are usually involved as volunteers and the PSI team has extensive experience providing support and guidance to programs around SME engagement and management throughout the process.
Some of the successful recruitment tactics used include:
- An incentive, such as a nominal honorarium in return for their service.
- Referrals from existing SMEs or connections within your industry.
- Offer of Continuing Professional Education opportunities or credits.
- Publicising on relevant social media channels, in industry publications and via industry bodies and events.
Treat the recruitment of your SMEs as you would a paid position. Request cover letters, resumes, and/or curriculum vitae, and then interview those who appear promising. It’s a good idea to target more potential SMEs than you need, as some may not be able to complete the process.
Be clear about expectations and what’s involved. Potential SMEs may be initially eager to get involved, but some may not realize the amount of time required and resign midway through – or won’t provide the level of involvement you need.
How can I use industry best practices to power my certifications?
Case Study
PSI uses job analysis and test development to power NBSTSA certifications and ensure the right test taker skills are being assessed.
The work of rigorous job analyzes that informs test content and design is ongoing with a full job analysis every five years. And annual test development programs guarantee that only the most relevant, up-to-date content is reaching test takers. Ongoing evaluations include meeting with the NBSTSA Examination Review Committee to develop and pre-test content.
“PSI uses industry best practices to power NBSTSA certifications by measuring relevant test taker knowledge, skills, and abilities are being assessed. The rigorous development life cycle informs test content and design and includes a full job analysis and standard setting every five years. Annual test development work guarantees that only the most relevant, up-to-date content is reaching test takers. These ongoing evaluations include meeting with the NBSTSA Examination Review Committee to develop and trial new test content. “
Ben Price
CEO, NBSTSA
What steps can I take to assure validity, reliability, and fairness in the test development life cycle?
Test development
When you have assembled and trained your Subject Matter Experts (SMEs), it’s time to put them to work and create your certification test.
In this next stage of the process, new test items are created that map back to the test specifications. Test items are then reviewed and assembled into equivalent test forms. With thorough checks at every step for validity, reliability and fairness. Always led and facilitated by experienced psychometricians and testing developers.
The time and effort involved with managing the test development process should not be underestimated. Frequent communication with your SMEs is vital, to maintain engagement and momentum – and avoid resignations. Regular contact with your SMEs as a group also means they will learn from each other.
Item writing
Item writing refers to the process of constructing the individual items or questions that are required for a test. Multiple SMEs are involved as item writers.
Training in best practice and the required standards is essential for SMEs writing test items. Some of your SMEs will be participating in item writing for the first time. Others may not have been involved in item writing for several years and will need a refresher.
Training takes place in a workshop, with a full guide to item writing circulated afterwards. This covers the key elements of item writing:
- Composing a stem – the section of an item that poses the problem.
- Selecting the appropriate item type – for example multiple choice or drag and drop.
- Developing rationales – for multiple choice or multiple select items.
Security is vital during this stage of test development. Your SMEs should always use a secure item authoring and banking platform to write and submit new items.
The PSI team has experience facilitating item writing workshops and campaigns, in-person and remotely. Let us know if you need help with this or have questions about the technology needed to maintain test security during the test development process.
Item review
Content validation is essential before your test items are finalized. Items should always be reviewed by a group of SMEs. Group review can take place either in-person or remotely, where SMEs go through every draft item and provide feedback.
The process can vary at this point, but items are usually updated in response to feedback before approval and use as pre-test items.
“Many certification programs were forced to shift from in-person to remote SME meetings during the pandemic. Any initial concerns about SME participation and test content security were alleviated when clients learned that PSI had been successfully facilitating remote meetings for years. Most clients have resumed in-person meetings, and some have adopted hybrid and fully remote approaches. The ability to effectively facilitate SME meetings, in-person and remotely, creates flexibility for PSI to partner with clients and meet their unique needs.”
Pamela Stemmer
Senior Director, Psychometric Services, PSI
Pre-test analysis
During pre-test analysis, brand new items are included in a test as unscored items. This provides an opportunity to analyze item data before the item is added to your scored item pool.
Test assembly
Put simply, test assembly involves putting the items your SMEs have written, reviewed and approved for use on test forms. But it’s not as simple as that. Any test form must:
- Accurately reflect your test specifications (be valid)
- Be equivalent to any alternate test forms (be reliable)
- Not unfairly disadvantage any test takers (be fair)
Certification programs tend to use two options for test form assembly, depending on the profession they represent, number of test takers and specific program needs:
- Fixed forms – where all test takers see an identical test form, or one of a small number of alternate and equivalent test forms.
- Linear On the Fly Testing (LOFT) – where every test taker sees a unique and equivalent test form with different items. These items are selected from a pre-approved pool of items that meet established content and psychometric parameters.
Protect test integrity with LOFT
If test takers with advance knowledge of item content pass your test, the integrity of your test may be at risk – as well as your reputation and the value of your certification programs.
With LOFT, an algorithm is used to create equivalent test forms from a pool of items – for example, creating 100 item forms from a pool of 500 items. This significantly reduces the likelihood of pre-knowledge. It not only provides assurance that every test form is equivalent, but also that each test taker is fairly assessed on their knowledge.
Test review
Test review by a committee of SMEs is an important validation step for your item pools and test forms.
With fixed forms, your SME committee will go through each test form to check they are all valid and fair.
With LOFT, your committee reviews only the item pool, not each individual form. This can save the SME time, because they don’t need to review the same item multiple times if it appears on multiple forms. Any items with content issues are removed from the LOFT pool at the test review meeting, reducing the need for an additional review after the meeting.
A bit more about... working with SMEs
SME meetings can either be conducted remotely or in-person, depending on client preference and needs. SME meetings switched to fully remote during the pandemic, and many certification programs are sticking with this way of working.
How can recruiting the right SMEs maximize impact during the test development process?
Case Study
American Association of Critical-Care Nurses (AACN)
PSI’s ability to maximize the impact of certified nurses who serve as SMEs during the test development process is a foundation of our long-standing partnership with AACN. We take an approach to the peer-developed process that understands every test taker is a future acute or critical care nurse who may one day be involved in the development process themselves.
During every five-year test development cycle, SME meetings are facilitated for job analysis, standard setting, item review, and test review. During the pandemic the PSI team was committed to adapting to AACN’s needs, calibrating every phase of the test development process with their safety and security in mind.
“Our folks are always impressed by the psychometrician and test developers’ abilities to work with our experienced and often outspoken group while maintaining rigorous standards. And I know the SMEs go home and recommend it to others. The incredible work that goes into a certification test development process makes the SMEs, who are certificants themselves, even more proud.”
JoAnne Konick-McMahan - PCCN
Certification Practice Specialist, AACN
How can SMEs and the Angoff method help me to determine a minimally competent test taker?
Standard Setting
A cut score is the carefully chosen score on the scale of a test that determines whether a test taker has passed or failed. Put another way, it is the minimum score a test taker must achieve to pass the test.
SMEs are typically involved in the standard setting process for a certification test. The information that SMEs provide is based on the definition of a minimally competent test taker and what they need to know. During a facilitated committee meeting, SMEs use one of a variety of different standard setting methods to provide critical information for determining the cut score. The modified-Angoff Method is the most common standard setting method in credentialing. With this method, SMEs assign every test item a rating. These ratings inform the cut score selection.
A bit more about... the Angoff Method
The Angoff Method of standard setting brings together a group of SMEs to rate the difficulty of each item, based on the minimally competent test taker.
- The group is trained on the process and trained on creating the definition of a minimally competent test taker, so all SMEs have a shared understanding of the test taker.
- SMEs give every test item an Angoff rating between 1 and 100. This rating is an estimate of how many minimally competent test takers would get the item correct.
- Ratings are collated and tabulated, average ratings are calculated for each item, and the average of these values are used to inform the selection of the cut score.
Why is implementation and maintenance essential to enhance my testing programs?
Implementation and maintenance
After all the hard work you and your SMEs have put in, it’s time to go live with your test and deliver it to test takers. This is not a time to sit back and relax, however (although the moment might be worthy of a little celebration).
Ongoing analysis
Continued statistical analysis of your items and test forms is important after your content goes live. This gives you the information needed to make recommendations about items that must be updated, edited, put on hold, or removed from your item bank.
Examples of analysis after your test goes live include:
- Item difficulty (referred to as the p-value)
- Are your items too difficult or too easy?
- Item discrimination
- Do items discriminate? Do they differentiate higher and lower scoring test takers?
- Distractor analysis
- How do distractors perform, (i.e., the incorrect responses to a multiple-choice question).
- Feedback review
- Of test taker comments, queries and concerns.
- Item bank inventory
- To show where further item development and writing is needed.
Ongoing analysis provides an opportunity to refresh your item pool, develop new content where needed, and enhance your testing programs.
Keep pace with change
Changes to your test may be needed if new technologies or techniques are introduced to your profession. For example, IT certification is a rapidly changing environment. Emerging technologies such as machine learning and Artificial Intelligence are being introduced to many IT certification programs to ensure they remain relevant to test takers and employers.
It is best practice to conduct a job analysis and update test specifications every 5-7 years.
“To maintain test security, the continued use of a secure item authoring and banking platform is essential throughout the entire test development process.”
What can innovative credentials do to help my organization respond to a rapidly evolving sector and expand into new markets?
Case Study
Information Systems Audit and Control Association (ISACA)
In the rapidly evolving IT governance sector, the team at wanted to expand into new markets and diversify the organization’s learning and credentialing portfolio. Working closely with PSI, ISACA created five hybrid ‘stackable’ credentials including cybersecurity and data science. On passing all five tests, an Information Technology Certified Associate (ITCA) Certification is awarded.
These innovative credentials responded to an industry change with performance testing for core skill sets that are relevant in the new world of work. They also provide an opportunity for ISACA to connect with the next generation before they enter the workforce, and support those wishing to switch to a career in IT.
“Our folks are always impressed by the psychometrician and test developers’ abilities to work with our experienced and often outspoken group while maintaining rigorous standards. And I know the SMEs go home and recommend it to others. The incredible work that goes into a certification test development process makes the SMEs, who are certificants themselves, even more proud.”
JoAnne Konick-McMahan - PCCN
Certification Practice Specialist, AACN
We have a wide array of tools, techniques, and technology at our disposal to ensure certification test content is valid, reliable and fair.
By using the right tools and following the comprehensive processes outlined:
- Employers will be assured that only competent individuals are granted your certification.
- Your test takers will be assured that a certification test fairly assesses the skills and knowledge required in their chosen profession.
- Certification programs will be assured that their test content is fair, measures what’s important and relevant, and is legally defensible.
The PSI team of psychometricians and testing experts are experienced in every stage of the test development process. We are here to support you with your job analysis, SME training, ongoing statistical analysis and everything in between.
