In the test development process, everything we create – content outlines, exam items, performance validation – can be seen as a sample of a broader professional environment. We put careful thought into sampling the right content to assess knowledge and skill. But just as important is selecting the right people, our Subject Matter Experts (SMEs), to shape that content.
The quality of the exam doesn’t just depend on what’s included, but on who helps determine it. So, to build a fair, relevant exam that reflects the real-world diversity of the workforce, we need SMEs who represent the range of backgrounds, experiences, and perspectives found in the profession. They are the essential bridge between the work and the way we measure it.
We can’t involve every professional who’s ever worked in a role. But we can take a purposeful, structured approach to ensure the SMEs we do involve reflect the population we’re trying to assess. This isn’t about ticking demographic boxes. It’s about making sure test content is fair, accurate, and relevant for everyone who takes the exam. Here’s how I approach it.
1. Start by defining your population (even if it’s messy at first)
You can’t recruit a representative group of SMEs if you don’t know who you’re trying to represent. And when you’re launching a new program, that’s often the biggest challenge.
We don’t always have great data on the target population, and that’s okay. Start with a working definition. It doesn’t need to be perfect, and it will evolve over time. Use whatever information you can get your hands on – job descriptions, workforce data, even input from early stakeholders. A few helpful resources you can use include:
- The Bureau of Labor Statistics (BLS) for U.S. workforce data.
- O*NET Online for job descriptions, tasks, knowledge areas, and national outlook.
You’re aiming to define the demographics, job functions, specialisms, and work contexts that make up the population your exam will serve.
2. Build your network to find the right SMEs
Even if you have a good sense of who your population is, that doesn’t mean it’s easy to find them.
If you’re working with an established association, you may already have access to potential SME contacts through events, mailing lists, training programs, or certification renewals. But if you’re starting fresh, it’s all about networks.
It’s kind of like playing ‘Six Degrees of Kevin Bacon’. Start with your thought leaders, those people who had the idea for the program or are deeply embedded in the community. Ask who they know. Then ask those people who they know. Before long, you’ve mapped a web of potential contributors.
3. Use a purposive sampling approach
Let’s be real: statistically perfect samples aren’t possible in test development. We’re not running national surveys, we’re running workshops and review sessions with a small number of SMEs. That’s why we use something called purposive sampling. That means we’re purposefully selecting people because they fit the demographics we’ve identified as important to the population.
Start by identifying your key demographics. These will vary depending on the exam, but might include:
- Years of experience
- Area of practice or specialty
- Geographic region
- Job role or function
- Work setting (e.g., public vs. private)
For those key demographics, aim for representativeness. For everything else – like age, gender, ethnicity – strive for diversity. You’re not going to get a perfect cross-section, but intentionality goes a long way.
4. Over-recruit to account for drop-off in the test development process
People drop out. They get busy. Life happens. That’s why one of the most practical things you can do is over-recruit. If you need 8 SMEs, recruit 12. The extras give you breathing room. And if someone can’t participate this time, you can always re-engage them in a future step. That might be item review, standard setting, or ongoing content maintenance.
5. Set clear expectations to keep people engaged
Recruiting a great group is just the beginning. The next challenge is keeping them engaged. The best way to do that? Be clear, realistic, and structured.
- Define what you’re asking them to do and how long it will take.
- Give them a starting point, not a blank page (blank pages are overwhelming).
- Be transparent about the timeline and what’s expected at each step.
Engagement tends to drop when people don’t know what they’ve signed up for or feel like they’re flying blind. Thoughtful planning and communication can help you avoid that.
Read the blog: Maximising SME time and impact in test development.
6. Use your data (if you’ve got it)
If you already have a running certification program, you probably have data – and lots of it.
- Who’s taking the exam?
- Who’s renewing?
- What does your certificant population look like over time?
This is a goldmine for improving your approach to SME selection. You can use it to refine your population definition, check how representative your current SMEs are, and course-correct where needed.
And if one stage of your process isn’t fully representative, it’s not the end of the world. You can make up for it in other steps. The goal is for the overall test development process to be representative.
Top tips for getting started
If you’re a test developer starting from nothing, here’s what I’d recommend:
- Get a definition of your population.
It won’t be perfect, but start somewhere. - Lean on your network.
Use your thought leaders and early champions to build your SME base. - Over-recruit.
More names = more flexibility. Even if you can’t use everyone now, you can involve them later.
Purposeful representation makes for better tests
You don’t need a statistically perfect sample to develop fair, relevant, and high-quality tests. But you do need to be intentional.
Define who you’re trying to represent. Structure your recruitment around that. Be transparent with your SMEs. And plan for the reality that not everyone will stay with you for the entire process.
The more purposeful you are upfront, the easier it is to maintain quality, fairness, and engagement throughout the test development process.