How to manage your SMEs through the test development process
All your stakeholders – from test takers and employers to regulators and credentialing bodies – must have confidence in your testing program. They need to be assured that only individuals that have met the required level of competence will be granted your certification. And that every test you deliver is valid, reliable and fair. This assurance can only be provided through a rigorous test content development process that involves multiple and representative groups of Subject Matter Experts (SMEs) from across your industry.
Your SMEs play a vital role in defining the knowledge, tasks, and abilities required to perform a job to the required standard. They are also critical in the test development process, including writing items, reviewing items, approving tests, and setting the minimum standard for success. Your SMEs are also your stakeholders. The test content development process should be a positive experience for them. One that is well managed, engaging and rewarding from the outset.
Top tips for recruiting test development SMEs
SMEs are usually involved as volunteers. They are giving their time for free and getting them involved and engaged starts with recruitment. Treat the recruitment of SMEs as you would a paid position. Be clear about your expectations and what’s required and communicate with candidates throughout the process.
Request cover letters and resumes, and then interview qualified candidates. This level of rigor is a good way to show the importance you place on SME involvement in the test content development process.
The team here at PSI has extensive experience partnering with clients and guiding the recruitment of SMEs with our clients. Some of the successful tactics used to get SMEs onboard include:
- An incentive, such as a nominal honorarium in return for their service.
- Referrals from existing SMEs or connections within your industry.
- Offer of continuing professional education opportunities or credits.
- Communication around the benefits in participating, such as expanding network and collaborating with other experts in the field as well as the opportunity to contribute to their field through engagement in the test development process.
Involve the right SMEs for valid test content
It’s also important to ensure you recruit a representative group of SMEs to develop your test content. Including a broad range of perspectives is critical to the validity, and reliability, and fairness of your tests.
Start by determining the pool of individuals your SMEs should come from. Your group of SMEs should be reflective of your target population. For example:
- Years of experience and level of education.
- Geographic location.
- Gender, race and ethnicity.
- Industry specialty.
Recruiting SMEs who have previous experience with test development can be useful. But to be representative, your group should also include first-time contributors.
SMEs in the test content development process
Providing SME training at the beginning of a workshop is a good way to build engagement early in the test content development process. Once you assemble and train a diverse and engaged group of SMEs, it’s time to put them to work creating your test. The key stages are typically:
- Item writing – Multiple SMEs serving as item writers, constructing the individual items or questions required for a test. Using a secure item authoring and banking platform makes it easier for them to write and submit new items.
- Item review and updates – Items being reviewed by a different group of SMEs who go through draft items and provide feedback. Items are then updated in response to feedback before final sign off and pre-test analysis. Again, a secure item banking platform will make item updates simpler for your SMEs. This facilitated group meeting can take place either in-person or remotely.
- Test review and approval – After test forms or item pools have been assembled based on the content outline specifications, SMEs review and validate the content of the test for approval prior to delivery.
Frequent communication with your individual SMEs is vital to maintain engagement and momentum and avoid resignations. Keep SMEs updated with progress, even during times where their direct involvement isn’t needed. Regular contact with your SMEs as a group also means they will learn from each other.
In-person or remote SME meetings?
We were already successfully facilitating training workshops and remote meetings before COVID. This meant we were able to seamlessly transition even more clients from in-person to remote meetings at the start of the pandemic.
During the transition back to in-person meetings, we have successfully implemented a hybrid approach where needed. Although we often recommend in-person meetings, we work closely with clients to ensure successful meeting facilitation – regardless of the location and modality.
What next for your test content?
You and your SMEs deserve a pat on the back when your expertly developed test content goes live. It doesn’t stop there though. Involve your SMEs in regular test reviews to ensure you keep pace with developments in your industry and changes to standards of practice or legislation. By involving a representative group of engaged SMEs in the test development process you can be assured your test content is fair, measures what’s important and relevant, and is legally defensible.
