When it comes to high-quality test development, Subject Matter Experts (SMEs) are essential. They’re the ones with the education, the lived experience, the deep content expertise, and the insight into what’s actually relevant and realistic in the real world. As an essential member of the process, test developers often ask a lot from SMEs. If the test developers don’t make the process easy or clear for the SMEs, that’s where delays happen or SME engagement starts to slip.
If I could share just one message from my time supporting SME-led projects, it would be this: the best way to maximize your SME’s time and impact is to reduce their cognitive load. Make the process as seamless, intuitive, and guided as possible. And yes, most of that comes down to preparation on our end before the SMEs ever join a call, open a platform, or even volunteer their time. Based on experience, here’s how I approach it.
Start with prep: Your work first, then theirs
The biggest mistake I see with SMEs in test development is underestimating the amount of groundwork required before they get involved.
I can understand why. Often teams think, “You’re the expert – you know what to do.” So they send over some guidance and expect SMEs to dive in and write great items. But writing test questions is a completely different skill set. It’s unfamiliar, and it can be overwhelming, even for experienced experts in their own field.
Instead, I do everything I can up front to remove friction. That means planning every step of the process, designing workshops carefully, and giving SMEs what they need before they even know they need it. When we make it easier for them, we get better input. And a lot less frustration on both sides.
Learn how to recruit the right SMEs for your testing program.
Flip the process: Review first, then write
One of my favorite strategies is reordering the process. Instead of asking SMEs to start by writing items, I have them review items first.
This gives them a reference point. They can see what a good item looks like (and maybe what a not-so-good one looks like too). It sets expectations and gives them a mental model to follow. Then, when it’s time to write new items, they’re not starting from scratch because they’ve already internalized what success looks like.
Read best practice advice for you exam and test content reviews.
Prompt, don’t just ask: Guide their thinking
Item review is a big part of any test development project. But simply saying, “Review this item and tell us what you think” is… not helpful.
Instead, I prompt the SMEs I am working with:
- “This item’s really difficult – why might people be picking this wrong answer?”
- “What would you tweak in the stem to make it clearer?”
- “Does this distractor make sense given how the content is taught in practice?”
By asking specific questions, I reduce the effort they need to spend figuring out what to say. It’s about giving them a starting point and never making them stare at a blank page. This is easiest during a workshop, but these prompting questions can be generated before an online/remote review, too.
Set clear expectations, then repeat them
One thing I’ve learned as a psychometrician: clarity is kindness. A lot of SMEs drop out not because they’re not interested, but because they didn’t realize how much time or effort was involved.
So, how do you solve this… overcommunicate.
- When recruiting: “Here’s what we’re asking, and here’s about how long it will take.”
- In workshops: “Here’s what success looks like.”
- In follow-up: “Just a reminder of our goals and timeline.”
You want to avoid what I call the “grumpy zone.” This happens when someone realizes an activity is going to take way longer than expected. This causes frustration and discourages SMEs from wanting to continue supporting the project or worse, not wanting to participate in future efforts. But if you give time back? They’ll be thrilled.
One simple trick is to slightly overestimate how long the work will take. Not so much that it scares people off, but just enough to build in a buffer. That way, when they finish early or more quickly than planned, it feels like a win.
Find out more in our blog: Tips to work effectively with SMEs in test development.
Use psychometric data to focus their input
Here’s where I come in as a psychometrician. My job is to make the data make sense to the SMEs and help them interpret it. This helps SMEs to understand where their input will have the most impact.
For example:
- “This item is tough, but it discriminates well. Unless there is an issue with the content, let’s keep it.”
- “This one’s difficult and it doesn’t discriminate. What is the best way to revise it?”
- “This stat is only based on 30 responses. Unless the content is wrong, let’s wait and see how it performs.”
The key is to help SMEs understand what the numbers mean in context so they can make smart decisions without having to decode the psychometrics on their own. It’s one of the ways testing experts and test developers can better support SME success.
Design test development tech and workflows with SMEs in mind
SMEs often work on their own time. That means:
- Having clear user guides for any item authoring and banking software they need to use, with screenshots and troubleshooting tips.
- Scheduling optional check-ins just in case they get stuck.
- Organizing items into categories so they can pick what matches their availability and energy. Have an hour of free time? Work on harder items that may need more edits or write new items. Only have 20 minutes between meetings? Work on easier items that only require a quick content check.
When test developers take the time to support SME flexibility, everything runs smoother.
AI can help – but it doesn’t replace the human brain
A secure AI that only uses approved sources to generate items is a great tool to use, as a stepping stone and something that gets SMEs to that next place more easily. You can use AI to:
- Generate draft items for human review.
- Come up with new ideas based on content themes.
- Reduce repetition when you need hundreds of items.
But AI has limits. It’s great at patterns, but not so much at nuance. You’ll still need SMEs, the real experts, to review and refine the outputs. And when AI gets something totally off the wall? SMEs need to know it’s okay to say, “We’re not fixing this—we’re deleting it.”
So yes, AI can reduce cognitive load. But it’s not a magic wand. It works best when paired with human expertise and thoughtful processes from experienced test developers.
My top 3 strategies
If I were building an SME engagement process from the ground up, here’s what I’d focus on:
- Design everything around the SME experience.
It’s okay if it’s harder for the test developers (and it should be!), as long as it’s easier for the SMEs. - Be crystal clear and upfront about expectations.
Time commitment, outputs, timelines, lay it all out early and repeat it often. - Always give SMEs a starting point.
Never ask them to begin from zero. Even a rough draft or prompt can go a long way toward getting quality input.
Learn more in our frequently asked questions about test development.
It starts with us
At the end of the day, maximizing SME time isn’t about squeezing more out of them. It’s about making their contribution as smooth, meaningful, and productive as possible.
That means prep work. That means clear communication. That means smart tools and guided support. And above all, it means recognizing that the more effort we put in ahead of time as testing experts, the more impact they’ll be able to make when it counts.
Download our guide to developing the right test content for your certification program.