As part of the Association of Test Publishers ‘New World of Testing’ Digital Series, I recently hosted a webinar on Defeating Test Deceit with a panel of test security experts from PSI as well as client and partner organizations. We shared our knowledge and experience of the security concerns currently facing the assessment industry, as well as examples of the ingenuity our community has shown to address these challenges and foster change.The webinar was wide ranging and a highly topical discussion, with subjects covered and questions answered that shared the full force of the tactics we employ to deflect and deter organized cheating in its many forms. It’s such a hot industry topic that ATP has set up a Test Fraud Coalition, chaired by our panelist Carl Bowman, SVP of Exam Services, CompTIA.
To give you a flavor of the webinar recording, here are some key considerations in defeating test deceit:
It takes a team
A lot of discussion around test security is focused on detection. This might be test fraud, such as cheating or collusion, or harvesting test content. Tools such as online proctoring and data analytics highlight that these types of malpractice might have occurred. But what then?
Effectively tackling test fraud means being able to move rapidly from detection to action. A joined-up approach is crucial and this is reflected in the range of roles represented by the panel on our webinar. The adage goes “it takes a village” and when it comes to test security it takes a whole team. That team needs to be built on a true partnership between test sponsor and test provider that starts before any malpractice is detected.
Step 1 – Have a plan
A test sponsor must have clear policies and a plan in place that outlines the measures they will take in the event of test fraud. These documents are unique to the test sponsor and based on the outcomes of conversations with their testing partner. .
The testing partner’s role is to educate the sponsor about the possibilities, the sponsor then sets their policies and the actions they will take if the rules are broken. Different levels of test fraud will warrant different actions, from invalidating a test result to starting legal proceedings.
Our client ISACA has a very clear webpage that outlines the actions they consider to be test fraud, with a reporting form for members, test takers and test administrators. Kim Cohen, Senior Director, Credentialing at ISACA joined the webinar to talk about the importance of maintaining integrity and a level playing field for the ISACA testing program.
Step 2 – Confirm your suspicions
When you have a plan in place and data analytics or proctoring indicates you might have an issue, the first step is confirmation. That’s where, for example, the PSI Security Operations team comes in. They work closely with our team of expert psychometricians to investigate further, using both data forensics and information from test centers and online proctoring.
My colleague, Nicole Tucker, Director of Statistical Reporting and Analytics at PSI, discussed this topic on the webinar. With years of experience in the testing industry, her wealth of knowledge on the subject is enormous.
Step 3 – Gather evidence
Harvesting test content and selling it on the internet is a major issue for many test sponsors. One of the most exciting tools we have in our toolbox at this stage is web crawling. This uses web spiders or bots to look for your intellectual property (IP) and copyrighted material on the internet. Anything suspicious is flagged for further investigation. We can then buy the content and match it to actual test content. Our final webinar panelist, Jeff Marsh, Manager of Test Security, Ascend Learning, shared more light in this area.
Additional evidence we gather might be video footage, audio files, and/or photographs from test centers and online proctoring session. As well as the data analytics from the confirmation stage.
Step 4 – Take action
Any action you take against test fraud is likely to be far more effective if you:
- Have clear policies and a plan in place.
- Communicate these to test takers in advance of their test.
- Provide strong evidence to support your action.
Taking this holistic approach will place you in a more robust and legally defensible position that is less likely to be challenged whether you invalidate an individual test result, prevent a test taker from accessing further credentials, or start legal proceedings.