Linear On The Fly Testing (LOFT) has been made possible through advances in technology, but it isn’t new. This method has been used to deliver professional tests and exams for over 25 years. LOFT allows us to generate unique and equivalent forms for every test taker, produced by selecting a set number of questions from a large item pool. And while LOFT has been used more in licensure testing, a growing number of certification programs are now realizing the benefits.
For a certifying body, the move to LOFT might feel like a big step, but it’s worth noting that LOFT is a tried and tested solution for exam assembly. It has already saved resources and improved test security for multiple certification programs. Increased efficiencies, particularly saving time for your exam committees, might be a sufficient reason to implement LOFT. But there are many more factors that make Linear On The Fly Testing well worth consideration.
Test evolution
LOFT is an approach to test assembly that combines the benefits of Computerized Adaptive Testing (CAT) and fixed form testing. With traditional paper-based testing, all test takers would see the same fixed test form containing exactly the same items. While this ensured test equivalency in terms of content and difficulty, the consequences of any security incident were potentially serious.
Since the 1970s, innovations in computer-based testing led to the development of CAT, which adapts the items presented to each test taker based on their previous responses. One of the benefits of CAT is to reduce item exposure and the impact of compromised test content. If there are multiple versions of a test, it makes it a lot harder for test takers to benefit from shared test content. However, CAT typically relies on Item Response Theory (IRT), and this measurement approach requires larger sample sizes to calibrate item statistics, therefore precluding its use for programs with smaller candidate volume. Like CAT, LOFT can produce unique exam forms for candidates, but can maintain statistical equivalence using Classical Test Theory statistics, which can be generated with smaller samples.
With LOFT, an algorithm is used to create equivalent forms from a pool of items – for example 20 items from Domain A, and 15 items from Domain B – while ensuring that the overall test is neither too easy nor too difficult. This provides assurance that, not only does each form test equivalent content, but also that each candidate is fairly assessed on their knowledge by maintaining the same passing standard.
Streamlined process
The process required to assemble, review, and create a test is hard work and time consuming. When developing fixed form tests, a committee of Subject Matter Experts (SMEs) usually reviews each form multiple times. Typically, the SMEs review the forms individually, then at an Exam Review meeting to make changes to the forms, and then a third time to confirm no errors were introduced when making changes. During each review, the committee evaluates all items for content to ensure each item is fair, accurate, meaningful, and current. The SMEs also address enemy items, where a set of items contain entirely duplicative content or give away the answer to another. Any items that have content issues or are an enemy to another on the form are replaced.
With LOFT, however, your committee reviews only the item pool, not each individual form. This saves the SMEs time, in that they don’t have to review the same item multiple times if it appears on multiple forms. The LOFT algorithm that assembles each unique form also controls for enemy items, ensuring no enemies appear on the same form. Therefore enemy items don’t need to be replaced, they just need to be identified. Finally, any items with content issues are simply removed from the LOFT pool at the Exam Review meeting, therefore reducing the need for a third review after the meeting.
Learn more about how to identify the best SMEs for your exam development.
Increased security
If a fixed form is compromised, whether it is shared widely on the internet or just amongst smaller groups, the consequences can be dramatic. If incompetent candidates with foreknowledge of the items take and pass your exam, it can create an issue of public safety, not to mention undermine the integrity of the exam and diminish the value of your certification. The only adequate response upon discovering your exam has become compromised may be to take the form out of circulation and go through the onerous exam development process again – right from the beginning.
With LOFT, the likelihood of this predicament is significantly reduced. For example, if a test is compromised and certain questions shared on the internet, those items can be removed from the pool. This eliminates the need to create new items, produce new forms, or go through the whole review cycle again. Furthermore, because all candidates see unique forms, if an entire form is compromised, you may be able to identify the individual responsible for compromising your exam content.
When to use LOFT
A lot of our clients ask us what features of a certification test make it a good candidate for LOFT. There are two main pieces of criteria that should be met to implement LOFT with your program:
1. Having a healthy item bank
LOFT isn’t a silver bullet; you still need a healthy bank full of multiple-choice items with known statistics. Your LOFT pool should have at least 50% more items than required for a single fixed form. If you already have at least two fixed forms with minimal overlap, then your program is probably a good candidate for LOFT.
2. Having sufficient test taker volume
You need large enough numbers of test takers to give you reliable, stable item statistics. If you have at least 200 candidates per testing period, and at least 100 candidates for each pre-test set you administer, your program may be a good candidate for LOFT.
Organizations that run multiple certification programs can still benefit from LOFT even if the tests involve a different number of items, seat times, or scoring systems. If your tests involve complex item types or small candidate volume, LOFT may still be feasible. We work with organizations who use a combination of LOFT and traditional fixed form testing depending on the need across their programs.
A twist on traditional test creation
If your organization is accustomed to fixed form testing, LOFT can be a departure that is hard to wrap your head around. That’s what the PSI team of experts is here for. We can answer your questions, address your concerns, and demonstrate that LOFT is a psychometrically defensible method of test assembly. This solution ensures the same content and statistical equivalence of fixed form testing, but boasts advantages in terms of security and efficiency of the exam development process. This is particularly evident when it comes to the involvement of your SMEs. At the same time, LOFT supports you to achieve the major objectives of any certification test – to be valid, defensible, and fair for your candidates.
