As is often the case with psychometricians, the answer to the question of whether an item bank is a testing program’s biggest asset is… it depends. It certainly can be, but only if you take the time to develop it correctly, using best practice principles.
Even the best resourced testing programs don’t have the capability to write an infinite number of test items. And doing so wouldn’t necessarily be advantageous. Developing more items isn’t always the answer when it comes to item banks. So what are the key considerations for developing a healthy item bank and how big should it be? Again, it depends.
Right size and nurture your item bank
There are many industries where having a large item bank can be a hindrance rather than a help. In industries that change rapidly due to legislative changes or evolving practices, items can age quickly. Not only does this present the risk of outdated content, but there’s also an investment associated with reviewing and updating items in response to change. It’s important to ensure that you receive a return on this investment.
As test development experts, we are often asked what the optimal number of test items is for an item bank. The solution is usually different for every testing program, and there are several factors involved in finding the answer. For example, how many forms do you have? What’s your pre-testing policy? What is your test taker volume? An initial goal could be to develop twice the number of items you need for a single form, and then grow your bank from there. We find that having three to four times the items you need is often a best-case scenario.
However, it’s important to remember that the overall quantity of items in your bank isn’t as important as how these items break down across your different content areas. And quantity alone is not enough. If those items aren’t performing well, they aren’t useful.
Read about how many items are needed to create an exam.
Use your Subject Matter Experts wisely
A useful step in establishing how many items you need to develop for your bank is to determine the areas of need. If you are developing a new test, you will need items across the entire test content specification or test blueprint. For existing tests, there may be areas that have fewer items than others. It may seem obvious, but these are the areas in which you need to focus the efforts of your Subject Matter Experts (SMEs).
A solid process, following best practice principles and involving SMEs who have had item writing training, is important at this stage. It makes it far more likely that you will have high quality items to take through to pre-testing and then on to your operational item bank. Equally, if your SMEs are disciplined and focused on the content areas where items are most needed, rather than working on random or “easier to write” content areas, it will prevent an imbalance in your item bank and over exposure in areas where you have fewer items.
It’s also important to know when to stop. Too many SMEs can be a hindrance rather than a help. There’s often a cost involved in compensating your SMEs and managing the process, which may not be worth it when you already have sufficient items.
Read our tips to manage SMEs in test development.
Common errors with item banking
As well as focusing your writing efforts on content areas with fewer items, this is where it can also be important to focus your pre-testing. A mistake testing programs sometimes make is failing to do enough pre-testing to obtain enough high-quality items across all content areas. This can hinder your flexibility in creating exam forms.
Some testing programs have pre-test blocks that are too large relative to the number of scored items. This can be a missed opportunity for maximizing the content coverage and reliability of the scored items. Again, it will vary for each testing program, but we recommend that roughly 10 out of 100 questions are non-scored items for pre-testing.
Another common mistake we see is neglect and poor maintenance of item banks. Regular revisions and updates are critical to keep your item bank in good shape. This includes reviewing items that are not performing well so you can fix any issues based on the information you have. Without maintenance, the items you have in your bank that do not perform well statistically will only increase in number. Keep in mind, sometimes it’s a better use of time to simply delete poorly performing items than it is to continually try to fix them.
Benefits of a healthy item bank
For tests that have a fixed form or forms, an adequately sized item bank is particularly important. If a form is compromised, you may need to retire all the items on the form. Testing programs often have a supplementary form available in case this happens, which isn’t possible without a healthy item bank.
It’s also worth noting that an increased emphasis on Diversity, Equity & Inclusion (DEI) might also lead to items being removed from your bank. It’s not uncommon for differential item functioning analysis or sensitivity reviews to highlight items that should be removed for DEI reasons.
All these situations, whether it’s compromised test content or a DEI review, can increase the number of items you lose from your bank, so it’s important to be prepared.
Read more about how to develop valid, reliable, and fair test content.
Item bank issues or concerns?
Talk to your test developers and psychometricians about what’s happening with your testing program. What are the questions you need to answer to determine the right size for your item bank? Test volumes, the frequency content becomes outdated, and the specifics of your program will all make a difference. Explore the steps you can take to ensure your item bank is in the best possible shape. This might be more items in specific content areas or a change to your pre-testing policy. By asking the right questions and working towards solutions, your item bank could become one of your biggest assets.
