In our last blog we talked about how to develop test content through rigorous processes to ensure validity and fairness in assessment – key requirements for legal defensibility. But the defensibility of an assessment does not end with the development of fair, unbiased, and accurate test content.
The fairness and integrity of an assessment may also be impacted by the methods and modes of administration. In today’s world of technology-based assessment, it is important to ensure that test takers are prepared to use and able to access the testing system in an equitable way that doesn’t favor or disadvantage any group. And with the increasing use of newer technologies that enable remote online testing along with traditional test centers in “multi-modal” programs, it is important to consider how the use of technology and testing modality may affect test takers differently. We must also consider the concerns that test takers and other stakeholders may have regarding data privacy and security. Any of these factors may spur legal challenges to the testing program.
Online vs. test center delivery of high stakes exams
Assuring the security of exams and the integrity of the scores they produce are key concerns, especially when the results are used in high-stakes programs, such as qualifications, admissions, licensing and certification. Typical security methods include ID authentication and monitoring by a proctor, video surveillance and recording, and capturing test taker response data to analyze potential malpractice. With remote online testing, test takers use their own technology and are monitored via online proctoring to re-create the experience of in-person proctored testing. While test centers offer a physical location and standard technology for those test takers who need and may not have access to either. Online testing offers a number of advantages, including social distancing and safety, flexibility in testing times, no travel costs, and better access to credentials for many people who need these options.
Considerations. With the increase in online remote testing in multi-modal testing programs, testing organizations and other stakeholders have raised questions about the comparability, security and integrity of online remote testing and the impact on the test taker experience. To this end, research studies have begun to appear in the scientific testing literature, some of it produced by PSI, examining these issues. The results of several studies offer promising evidence that multi-modal exam programs provide psychometrically sound measurement and comparable scores across online vs. test center modes, with no appreciable differences in test security issues flagged, and equally favorable test taker experience ratings across testing modes.
Another consideration in online proctored testing is data privacy. Testing organizations should be aware of regulations in the US and Europe regarding access to, storage and use of test taker data, particularly in the context of secure proctored testing. A good source of information about these laws and regulations is the Association of Test Publishers. Careful use and management of test taker data is important in maintaining a legally defensible exam program.
Test Taker Preparation
It is vital to help test takers prepare for the exam process by providing information in advance to help ensure that they have access to the necessary technology and are familiar with it, as well as the security steps that will be taken to protect the integrity of the exam. The following are recommended to address potential test taker concerns and help to protect testing organizations:
- Prepare test takers by sharing information about the technology requirements and giving practice in taking the exam using the testing system interface.
- Educate test takers and stakeholders about the online proctoring process and why it is needed.
- Communicate with test takers about the steps involved with online proctoring, including how their identify will be confirmed, how they will be observed during a test, and the test taking behavior they should avoid (what a proctor will be looking for).
- Inform test takers about the security measures you take to collect, store, transfer, and protect data to the highest possible standards.
- Gather and record explicit consent for any data or personal information you collect and store.
- Reassure test takers that a secure browser is in use and an online proctor will not have remote access to their device;
Download Now: How to Prepare Test Takers for Online Proctoring
Enhancing security in multi-modal programs
Additional steps may be taken to help assure the quality and integrity of multi-modal exam programs, and to further assure defensibility.
- Evaluate psychometric quality of exams across modes to ensure comparability of scores and pass rates, and no advantage /or disadvantage to either modality.
- Monitor test taker data using data forensics methods to detect potential malpractice and assure fairness to all test takers. Data forensics can complement other security methods to provide a powerful defense against security risks. Read more: Data Forensics for Test Integrity
- Use Automated Test Assembly to administer unique combinations of test content to test takers. This minimizes content exposure and ensures that test takers do not have an unfair advantage by having advance knowledge of the test items. For example, methods such as linear on-the-fly (LOFT) and multi-stage adaptive testing can be used to generate test forms that are content balanced with minimum overlap. They also ensure statistical equivalence and provide a unique and fair test for each test taker.
Conclusion
Technology advances have enabled the evolution of test delivery to include online remote proctored testing in multi-modal high-stakes exam programs. In this blog we’ve reviewed important considerations that can give testing organizations confidence in using state-of-the-art online testing methods that offer many benefits. With the right science, technology, and expertise, it’s possible to ensure the necessary rigor to deploy fair and defensible multi-modal exams.