Over the past few years, former colleagues would intermittently reach out to ask about certification exams. Usually, the situation was that they were tasked to develop the organization’s first certification exam and they were looking for a starting point.
My friends and I would end up having an extended lunch discussing standards, councils, test cheating, psychometric analysis…information overload to the brim. But they just wanted to launch one certification exam, and only had a small budget for the project.
I learned from the best assessment experts and managers from my time at HP Software fifteen years ago. I’ve worked different roles in certification from exam developer to program manager. The requirements to build and deliver a professional certification exam can get overwhelming because of the legal risks* associated with offering certifications. I would like to offer this perspective on what it takes to launch your first certification exam given limited resources.
If I can boil down the essential actions to take when tasked with launching your organization’s first certification exam, they would be:
Establish a relationship with a test development expert/psychometrician
Test development experts/psychometricians possess advanced studies and experience related to industry standards, program policies, evaluation methods, and statistical analysis for developing certification exams. These experts offer a range of services to suit different budgets. It could be consulting time, training your team, or handling the end-to-end development themselves.
There are also several books and online materials on exam development and certification programs. I recommend reading up on these topics before speaking with the assessment expert to be able to communicate goals, tasks and requirements in a common language. This is useful if budgets are tight and you need to utilize consulting time efficiently.
Establish your exam development standards
Whether you will be guided by an assessment consultant or go at it internally, applying standards in exam development provides a consistent process and outcome. If exam questions will be developed by experts from different regions/organizations, having standards will help build content that is cohesive and well documented. This step will also strengthen the defensibility of your exam.
Some questions to consider when compiling the exam development standards include: How do you define the candidate who is ready to be certified? How do you determine the scope and objectives of the test? How will you know how difficult a question is supposed to be? What formats of questions will be included in the test? What is the process for ensuring the questions are vetted to be fair to all candidates? What is the process to determine the passing score? How will these information be documented?
Select a delivery platform that allows candidates to take the test fairly and securely
A growing certification program will likely deal with exam hosting, authoring and item writing, badging, vouchers, certificates, and test data, and use several types of technology. But if I’m asked, “What’s the bare minimum to deliver my team’s first certification exam to a global community of professionals?”
I would want an application that:
Offers features to mitigate or reduce the risk of test exposure, such as online proctoring, or screen lockdown
Is independent and provides all candidates an equal and unbiased chance at passing the exam
Nothing upsets me more in my work than throwing away hundreds of resource hours and the associated costs, and the work of highly-billable experts because the questions they built had been compromised and were being sold online. There is a customer success risk when underqualified candidates who become certified through nefarious means, end up working with a patient, student, or a client. Consider exam security when evaluating your test delivery service.
One common legal challenge* to certification programs involves bias, which refers to a difference in how the test performs for different groups. For example, a program decides to host and deliver its certification exam through their in-house developed learning management system. Employees can access, schedule and take the exam as often as they want while external professionals may have to register and pay a fee. A challenge can be made to the certification program if a group (employees) is able to take the exam as often as they want without cost resulting in higher pass rates while external experts have to keep their access at a minimum due to exam fees. Along with program policies, consider an exam delivery application that gives your potential candidates an equal and unbiased opportunity to pass the test.
Test delivery data
Test delivery applications gather test data. Analyzing exam delivery data—from how long candidates answered every question, to which option in a question was obviously incorrect because no candidate answered it, to the performance of a question related to the candidate’s final score—helps improve your exam.
Some examples of psychometric analysis include identifying bad-performing and good-performing questions, observing and noting suspicious activity, and pass and fail rates. Imagine if a candidate lost a job opportunity because he failed to qualify for a required certification from your company. The candidate failed the exam by one point, but that one question he missed turned out to be a poorly-performing item. With well-analyzed test data, you can take action to improve the exam. The psychometrician, with whom you are establishing a partnership, will be able to help with this task.
For your first certification offering, a development effort which prepares by seeking guidance from a test development expert, establishing and documenting a set of standards and setting up a test delivery platform, may lead to a fair and reliable exam that your global community of professionals will appreciate and support.
Summary:
If you have plans to offer your organization’s first certification exam,
Seek an assessment expert/psychometrician’s counsel. If able, have them train your team members on test development.
Delegate a team member to document the exam development standards and program policies for your team.
Research a test hosting and delivery system best suited to your budget and needs, and able to grow with your certification program.
* References:
Minimizing Legal Risks of Certification Programs, Chicago Law Partners article, 2011
The Legal Defensibility of Assessments, QuestionMark white paper from CedMa Europe 2007