What process do you use to determine the passing score—the point which differentiates the minimally qualified from the less-than-qualified?
I remember speaking with an assessment team many years ago and when I asked how they determined the passing score, one of them said their VP thought 80% was a good number.
What if the fair passing score referencing the MQC was 76%, and that individuals who scored between and 76% and 79% and failed should have been certified? This is a risk in the exam development process that can be legally-challenged if a candidate who believed they were more than sufficiently proficient, failed to make the passing mark the VP set.
As there are industry guidelines to write multiple choice questions, there are methods to determine an exam’s passing score. I’d like to note that each method requires a group of subject matter expert (SME) reviewers to provide a diverse set of viewpoints and field experience during the review.
Modified Angoff Method:
Devised by William Angoff in 1971, this process determines the passing score by having a set of SMEs predict the response probability of each question in the test. The process involves three steps.
1. Each SME predicts the response probability of each item in the test
2. Any deviation in the score of an item (one SME thinks the question is too difficult and gives it a low score, while another considers it too easy and gives a high score) is discussed until the score is adjusted or the item is rejected
3. The final scores are compiled and the average value is rounded off as the passing mark.
This was the method I learned to use. I facilitated Angoff review sessions with subject matter experts in my exam development projects.
Bookmark Method:
The Bookmark method was recommended by Lewis, Mitzel and Green in 1996. This method utilizes the assistance of a psychometrician to reduce the workload on SMEs.
A psychometrician organizes the questions by level of difficulty in a booklet. SMEs are instructed to estimate at what page of the booklet they considered as the passing marker. There may be disagreements on the level of difficulty on the questions, similar to the deviations on the Angoff scores.
Ebel Method:
Devised by Robert Ebel in 1972, this method rates each question two ways; by difficulty and importance. A simplified version of the Ebel method asks SMEs to rate questions only according to importance, categorizing them as essential, important, or indicated. After defining a minimally qualified candidate profile, SMEs then assess how many questions in each category the candidate should be expected to answer correctly. The passing score is established by averaging the ratings across the three categories. (1)
What method do you use? If it's not one of the three listed, add it as a comment. I am interested to learn about it!
Resources:
https://www.ets.org/Media/Research/pdf/passing_scores.pdf