This is Part II in our blog series on Quality Assurance in Testing. To read Part I, click here.  
Quality assurance is essential at every stage of the assessment lifecycle. From how you develop and review your test content to how you deliver and secure your tests. A good quality test allows your test takers to demonstrate their competence in your field and go on to pursue their chosen career specialty. Robust quality assurance ensures sound test outcomes and prevents detrimental impact on your test takers, so that every test you deliver continues to be valid, reliable, and fair.
End to end audits
An audit program, where an internal audit team randomly reviews a percentage of remotely proctored tests, will help to maintain quality standards in your program.
We have seen that end-to-end audits of an entire testing session are more effective than partial audits. These full session audits check whether best practices and processes are being rigorously applied. This provides:
- The ability to rewind and replay any suspicious activity.
- Greater visibility of changes within a session.
- Important information to support security team investigations.
If an issue is identified, the audit rate can be increased. And where necessary followed by a remediation plan that might include additional coaching, training, or documentation for proctors.
The numbers tell a story
We conduct audits on 6% of live online proctored tests as standard. That’s over 42,000 audits every year. We are working to increase our audit rate to 10%. Our quality scores for live online proctors are typically 98-99%. We review our Quality Rubric (the standards we measure against) on a quarterly basis to ensure we are auditing the right things.
Where data forensics sits in your quality assurance toolkit
Your testing program is unique, so your security plan needs to be bespoke. A holistic approach to test security will maintain high standards in detecting test fraud, taking appropriate action, and making improvements should it occur. For example, data forensics services are a handy tool in the box for maintaining quality across both test centers and online proctoring.
Read more about Data Forensics here.
Support test center quality
Data forensics serves to flag potential quality concerns in a particular test center or region.
For example, statistical analysis might indicate that proxy testing or test taker collusion is occurring, or that certain proctors aren’t paying sufficient attention. This would then trigger an audit, or other appropriate mechanism, to investigate further. Data forensics means these investigations can be targeted to where they are most needed.
Support online proctoring quality
As measures to increase remote test security get more advanced, so to do the steps taken by test takers to circumvent them. Data forensics is a valuable tool in response to this.
One example involves the lockdown browser used to prevent test takers accessing unauthorized applications during their test. For example to share content or involve a proxy tester. Lockdown browsers use a list of known and prohibited applications and will prevent a test from running if one of these applications is open. However, there are always new applications that aim to get around this. When a test has been flagged by online proctoring, data forensics indices will correlate the background applications and discover what else was running during the test. These applications are then added to the list of prohibited applications. It’s a great example of using data to keep pace with test malpractice.
Taking data forensics further
At PSI, our data forensics services are developing all the time – and growing in popularity with clients.
We partner with several testing organizations who use data forensics to increase the quality of their test content. For example, with a security breach such as pre-knowledge the quality of a test will falter. Data forensics will indicate when this has happened, so the test items and forms affected can be prioritized for review by SMEs.
We also proactively look for compromised test content with our web crawling service. Again, when content is shared on the internet this can be prioritized for SME review and either rewritten or removed from the item pool.
Client success
The role of a client success team – sometimes known as account management – is to maintain healthy client relationships. This involves:
- Ensuring contractual and quality obligations are met to the satisfaction of the client.
- Using data to drive strategic discussions in partnership with the client.
- Serving as internal advocate for the client within the testing provider’s business.
- Acting as the testing provider’s advocate to the client.
Recipe for success
Regular check-ins are vital for a great partnership. These take the form of:
- Partner discussion calls briefer and more frequent calls to find out what’s going on with the client in the short to medium term.
- Account review meetings often annual and more future focused, to share and explore testing data and develop long term strategies.
Testing data for quality assurance
Your testing data holds invaluable insights which can be used to maintain and improve the quality of your testing programs. A client services team will bring in as many data points as possible, including:
- Test taker satisfaction scores.
- Ticket / case response and closure rates.
- Test taker volume increases and decreases.
- Data forensics and web crawling reports.
Trends, shifts and patterns will then be used to develop strategies to improve quality or address issues. For example, tactics to capitalize on growing test taker volumes or boost low volumes.
Industry wide improvements
A testing partner with broad industry knowledge, beyond an individual client or specific sector, will support strategic recommendations with additional intelligence and insights. An experienced testing partner will spot trends, facilitate conversations, and help identify solutions or program enhancements.
When clients have concerns, for example test taker privacy when it comes to room scans for remote testing, a knowledgeable testing partner will be able to explore precedent, share knowledge and offer reassurance.
Conclusion
Working with an experienced testing partner at every stage of the process will support your testing programs to not only achieve and maintain high standards, but also take steps for continuous quality improvement. Quality assurance is an integral part of our partnership with every client. Get in touch to find out how quality assurance could benefit your testing programs.
