Linear On The Fly Testing (LOFT) is not just a recent technological development; it has a robust history spanning over 25 years in delivering professional tests and exams. This method allows the generation of unique and equivalent exam forms for each test taker by selecting a specific number of questions from a comprehensive item pool. Although LOFT has traditionally found extensive use in licensure testing, a growing number of certification programs are increasingly recognizing and capitalizing on its benefits.
While transitioning to LOFT can be a significant shift for certifying bodies, it’s essential to highlight that LOFT is a proven and established solution for exam assembly. Numerous certification programs have already adopted LOFT, experiencing resource savings and enhanced test security. LOFT can be a highly efficient way to increase security without an increase in effort from exam committees. For programs that already publish multiple exam forms, the efficiency gains, especially in terms of time savings for exam committees, may alone justify the adoption of LOFT. However, there are additional compelling factors that make Linear On The Fly Testing a highly worthwhile consideration for any certification program.
The evolution of LOFT
LOFT, an innovative approach to test assembly, blends the advantages of Computerized Adaptive Testing (CAT) and fixed form testing. CAT emerged as an effort to tailor item presentation based on individual responses to achieve highly precise scores over the entire range of scores. CAT allows a test sponsor to determine the precise level of competence of a test taker, whether that level is very high, very low, or anywhere in between. CAT’s reliance on Item Response Theory (IRT) often demands substantial sample sizes for item calibration, limiting its application in programs with smaller candidate volumes.
Enter LOFT. In certification testing (likewise in licensure testing), the concern is not to precisely locate the level of competence; rather, the goal is simply to determine whether or not the test taker has met the minimum requirement. That is, we seek to answer the question “did the test taker pass?” With CAT, the test taker doesn’t know how many items they will need to answer, and the test sponsor can’t control the allocation of content from one test taker to the next. With LOFT, every test taker is presented the same number of items with the same content allocation.
The difficulty of exams with LOFT can be controlled using IRT parameters, but LOFT can alternatively use Classical Test Theory statistics to maintain statistical equivalence while enabling the creation of distinct exam forms for candidates. It employs a set of rules to assemble equivalent forms from a pool of items across different domains, ensuring balanced content and consistent difficulty levels. Not only does LOFT ensure equivalent content across exam forms, but it also maintains fairness in assessing candidates’ knowledge by standardizing the testing experience and upholding consistent passing standards.
Increased security with LOFT
Publishing a small number of exam forms (i.e., as few as three, two, or even one) poses a security risk for any exam program. The repercussions of a compromised fixed form, whether widely disseminated on the internet or circulated within smaller groups, can be severe. The potential for incompetent candidates with prior knowledge of the items to successfully pass the exam not only poses a risk to public safety, but also undermines the exam’s integrity, eroding the value of the associated certification. In such cases, the only viable response may involve removing the compromised form from circulation and embarking on the demanding process of exam development anew, starting from scratch.
LOFT offers a substantial reduction in the likelihood of encountering this predicament. Since LOFT is presenting different items for each test taker, there is much less reason for one test taker to tell another what was on their test. In the event of a compromised test where certain questions are shared online, LOFT may allow for the seamless removal of those items from the overall pool, reducing the need for the laborious task of creating entirely new items, producing fresh exam forms, or undergoing a complete review cycle. LOFT thus provides a robust defense against potential security breaches and streamlines the response to such incidents, ensuring the integrity of the certification process.
Reduce SME effort
When developing fixed form tests, a committee of Subject Matter Experts (SMEs) usually review each exam form multiple times. Typically, each SME reviews each form on their own, then at an Exam Review meeting to make changes to the forms, and then a third time to confirm no errors were introduced when making changes. During each review, the committee evaluates all items for content to ensure each item is fair, accurate, meaningful, and current. The SMEs also identify and address enemy items, where a set of items contain entirely duplicative content or give away the answer to another. Any items that have content issues or are an enemy to another on the exam form are replaced.
With LOFT, however, your committee reviews only the item pool, not each individual exam form. This saves the SMEs time, in that they don’t have to review the same item multiple times if it appears on multiple forms. The LOFT algorithm that assembles each unique exam form also controls for enemy items, ensuring no enemies appear on the same form. Therefore, enemy items don’t need to be replaced, they just need to be identified. Finally, any items with content issues are simply removed from the LOFT pool at the Exam Review meeting, therefore reducing the need for an additional review after the meeting.
For programs wishing to significantly expand the number of exam forms, LOFT is an efficient method to employ, providing a swift avenue to enhance security. In essence, LOFT not only facilitates scalability but also acts as a rapid means to fortify the overall security infrastructure of your program.
Learn more about how to identify the best SMEs for your exam development.
When to Opt for LOFT in Certification Testing
Clients often inquire about the suitability of LOFT for their certification programs. To determine if LOFT aligns with your needs, consider the following criteria:
- A Robust Item Bank: LOFT is not a universal solution, but thrives with a substantial item bank. Ensure your LOFT pool contains at least 50% more items than required for a single fixed form. If your program boasts two fixed forms with minimal overlap, it indicates a favorable environment for LOFT implementation.
- Adequate Test Taker Volume: Reliable and stable item statistics necessitate a sufficiently large pool of test takers. If your program sees a minimum of 200 candidates per testing period and at least 100 candidates for each pre-test set administered, it suggests that LOFT could seamlessly integrate into your certification process.
Navigating Complexity: For organizations managing multiple certification programs, the benefits of LOFT extend beyond uniform test specifications. Even with variations in item counts, seat times, or scoring systems, LOFT can prove advantageous. In cases involving complex test specifications or smaller candidate volumes, LOFT may still remain feasible, especially when supported by an experienced team of test developers and psychometricians. Collaborating with experts, such as those at PSI, ensures that LOFT can be effectively applied to meet the diverse needs of your certification programs. Many organizations successfully employ a blend of LOFT and traditional fixed form testing based on the unique requirements across their programs. The key lies in leveraging the flexibility of LOFT while maintaining a strategic approach to certification testing.
Why LOFT?
If your organization is accustomed to fixed form testing, embracing Linear On The Fly Testing (LOFT) might initially feel like a significant departure. The good news is that the PSI team of experts is here to guide you through the transition. We’re ready to address your questions, alleviate concerns, and showcase that LOFT is not a departure, but a psychometrically defensible method of test assembly.
LOFT ensures the same content and statistical equivalence as fixed form testing while introducing notable advantages in terms of security and the efficiency of the exam development process. This is particularly evident in the streamlined involvement of your Subject Matter Experts (SMEs). Moreover, if you’re looking to dramatically increase the number of exam forms in your program, LOFT offers a quick and effective solution. By leveraging LOFT, you not only enhance the integrity of your certification process, but also achieve the major objectives of any certification test—validity, defensibility, and fairness for your candidates. Partnering with PSI and adopting LOFT could be the strategic move that propels your certification program to new heights.
