Data forensics
A convenient and cost-effective addition to your test security toolkit.
What is data forensics and why do testing organizations need to know?
Data forensics, sometimes known as computer forensics, refers to the use of statistical methods to investigate digital data to detect anomalies. It is often used in incidents of financial fraud, data theft and other cyber security crimes.
With the move to computer-based and online testing, data forensics is a relevant topic for testing organizations. In response to these developments, test security has become even more important. And many of the test security risks which may be amplified by online testing are now addressed by innovative security measures – including data forensics.
On this page you will learn:
How is data forensics used to improve test security?
When it comes to online test services, the application of statistical detection methods helps us recognize anomalies in testing data consistent with potential misconduct. Identifying these suspicious patterns helps us detect and investigate misconduct that may not be observed by test proctors.
Watch a short video on using data forensics to improve test security.
Why do we use data forensics in test security?
Security measures have always been needed to protect the integrity of computer based testing, exams and assessments. Especially when they are used for high stakes purposes, such as education and credentialing. In these applications, test security is needed to guard against misconduct – whether testing takes place in-person or online. Data forensics provides an additional method to monitor and evaluate test taker behavior to help guard against various types of test fraud.
With flexible computer-based testing, and more recently online and remote testing, technology gives us a greater variety of ways to protect test security. One element of this enhanced security stems from the data that arises from computer-based exams, which presents an enormous opportunity for forensic analyses.
How is test data used for data forensics in online testing?
Data forensics methods evaluate patterns of responses to test questions to identify anomalies that indicate different types of test fraud. Data forensic analyses are used to identify suspected issues with test security, as well as detecting potential issues that were not observed by test proctors.
When test takers respond to test questions in abnormal ways, this may be reflected in the data as irregularities in their answer choices, score patterns, and response times. We use this data to identify suspicious test taking behavior, detect misconduct and increase test security – for the benefit of testing organizations and test takers alike.
What types of misconduct does data forensics uncover?
When designing statistical detection methods for use in data forensics we ask, “if a person was engaging in this specific type of fraudulent behavior, what traces might be left behind in their data patterns?”. Then we design innovative ways to detect those patterns. This might include:
- Proxy testing
- Collusion
- Item harvesting
- Item pre-knowledge
How are different types of fraud and misconduct revealed in the data?
Potential fraud may be revealed by scanning the data for anomalies at different levels:
Test taker data
- Extremely similar responses among two or more test takers
- Unusual response times and patterns
- Extreme changes to score on retake
Test item and form data
- Drastic shifts in an item’s average score
- Drastic changes in an item’s average response time
- Issues associated with specific test forms
Group data
- High rates of forensic issues flagged for groups of test takers e.g., at a test center or location
- Test forms or items flagged for forensic issues for groups of test takers
What does a data forensics program for online testing look like?
A multi-faceted approach to data forensics that uses different tools is the most effective. For example, a program that incorporates a variety of statistical indices and analytics across different levels (test taker, test and group level).
Depending on test taker volume and the security requirements of the testing program and organization, scans take place on a weekly, monthly, quarterly or less frequent basis. Statistical indices are used to measure groups of data points to find trends, effects or features. For example:
- Response similarity – flags similarity between all test takers in their responses.
- Item and test time – flags unusually fast and slow response times for a whole test or specific item.
- Abnormal score patterns – flags higher and lower than expected scores for test takers.
An effective data forensics program is a high-level partnership between the client and the provider. One that proactively looks for trends related to cheating, while applying expertise and insights relevant to the specific testing organization and industry.
Nicole Tucker
Director of Statistical Reporting and Analytics, PSI Services
PSI has developed our own proprietary index of test taker response similarity that has proven to be successful in detecting certain types of cheating.
Dr. Isabelle Gonthier
Chief Assessment Officer, PSI Services
How does data forensics fit into my overall test security plan?
An effective test security plan covers the whole test life cycle – starting with prevention during exam development, continuing through secure test delivery, and on to post-test analysis and detection. For example:
- Non-disclosure agreement with item writers and subject matter experts, along with secure storage and transfer of test content during test development.
- Online proctoring and the use of alternate or unique test forms during test delivery.
- Clear communication with test takers about the rules and guidelines for online testing to reduce the risk of unintended misconduct.
Watch a short video on communicating your policies to prevent test misconduct.
In the event of failure to prevent misconduct, detecting its presence and impact on scores is equally important. Data forensics is a method we use to achieve this.
Data forensics flags a potential issue. What next?
Data forensics might detect evidence of suspicious behavior, but it doesn’t stop here. There are a range of investigations that help to confirm whether misconduct has taken place.
A test region, site, testing window or test taker associated with anomalous findings may trigger further in-depth analysis and investigation.
- Investigative measures at test taker level include a review of session recordings, computer logs or registration data.
- A review of video or photos can be used to investigate a certification center, along with a site audit, drop-in inspections or secret shopper for tests at that site.
- Web crawling can be used to systematically browse the internet for references to proprietary test content. This might include fraudulently obtained items offered for sale or discussed on social media.
The more our clients work with us, the more confident they become in the results of our data forensics.
Nicole Tucker
Director of Statistical Reporting and Analytics, PSI Services
Web crawlers can be used in data forensics to enhance test security and protect a testing organization’s intellectual property by finding exam content that is exposed on the internet. This can be extremely helpful in addressing the root cause of certain types of test fraud that appear in data forensic results.
Dr. Isabelle Gonthier
Chief Assessment Officer, PSI Services
Further investigations confirm misconduct. What next?
When data forensics and further investigation confirms misconduct has taken place, we work with a testing organization to coordinate the response, following the established plan. This might include deactivating a compromised test form or removing individual items from the bank, invalidating score results, or taking legal action.
There are some important legal considerations before you decide to act against a test taker, proxy test taker, brain dump site or other stakeholders involved in misconduct. Including the clarity of your test taker agreement and policies, and whether your test content is copyrighted. Civil litigation and government enforcement are both options you can take, should you decide to proceed with legal action.
