Pass rate concerns in the testing industry 

There has been extensive debate across the globe about fluctuations in important test and exam results that might be attributed to the pandemic. From Multistate Bar Exams and the National Assessment of Educational Progress in the US, to A-levels and GCSEs in the UK. And of course, these discussions affect certification and licensure programs as much as they do education. Unexpected variations in pass rates can have serious consequences for the organizations and industries we work with – both in the short and long term. So much so that concerns about pass rates shared by PSI clients prompted us to conduct a study to investigate further. Through a survey and analysis of test scores, we aimed to answer a key question: What impact has COVID-19 had on the credentialing world, and how is the pandemic affecting test results?

Study method and findings

We partner with several nursing specialty certification programs, some required for state licensure and others voluntary. From around mid-2020, many of these programs noticed atypical fluctuations in pass rates. They came to us for further insights and advice on how to address this potential issue. In response to these concerns, we collected mean score and pass rates from 2018 through 2022. We also attempted to account for the main factors, unrelated to the pandemic, that are likely to affect results. For example, changes in the exam content outline or cut scores. Across the programs we examined, we found that changes in pass rates varied greatly:

• Six programs experienced large changes.
• Four programs experienced moderate changes.
• Seven programs experienced little or no change.
• One program experienced positive gains.

It is worth noting that in some programs, fluctuations in pass rates were present even before COVID-19.

Factors affecting pass rates

The primary concern for programs that experienced lower pass rates is the opportunity for learning that might have been lost during 2020 and 2021, when test takers only had access to virtual learning. While the impact of this seems to vary across different programs, and test takers at different stages of their career, lost learning opportunities have clearly had some effect. The switch to virtual learning came hand in hand with what has been dubbed “the great resignation”. With more experienced members of the workforce leaving, there may have been a loss of organizational knowledge and mentorship opportunities. Again, this not only impacted people in educational programs but particularly those at an earlier stage of their career journey. Other factors identified that may have affected pass rates include changes in academic admission or certification program requirements, a change in test takers from employer programs, and certification becoming a lower priority during a time of furlough and re-assignments.

Impact of pass rates changes

There are several reasons why we, as an industry, need to better understand changes to pass rates caused by the pandemic – and take steps to address them. If future test takers hear that a test is too hard, they may be more hesitant about attempting it. In addition, if employers believe a certain license or certificate is too difficult or costly to obtain, this is likely to lead to declining employee support. Both factors have the potential to cause a sequence of events that starts with lower test taker volumes, which would be an immediate concern for any testing program. In turn, a reduction in certificate holders and licensees could limit the available workforce in many industries. Declining professionalism has the potential to impact quality of service and in some cases even public safety. In the example of healthcare, translating into poorer patient outcomes and less confidence in practicing professionals. If credentials become less sought after, the reputation – and bottom line – of our industry will inevitably be impacted. It’s a cautionary tale that we must take heed of right now.

Potential solutions to pass rate concerns

There is more to do to understand the impact of the pandemic on pass rates, as well as the future challenges this might present. It is also becoming clearer that some industries, and even distinct credentials within an industry, have been affected more than others. In response, we will continue to monitor the programs already involved in our study and wherever possible include additional programs. At the same time as monitoring impact, we are collaborating closely with testing programs to identify possible mitigation strategies. Again, these will vary between industries and programs but might include:

• Increase test taker confidence and knowledge with practice tests and test preparation materials.
• Enrich score reports to identify gaps in learning and work with faculty and programs to fill these.
• Continue to ask questions and work to develop solutions that protect the future of the credentials and industries we work with.

Our study and discussions with colleagues in the industry show that, although highly varied, the pandemic has had an impact on pass rates. However, some clients are already seeing their pass rates stabilize or slowly increase, and we are confident that the issue can be remedied. Much as we all worked together during the pandemic, we are working together now to find solutions.