During the job task analysis and design phase for a certification exam, an assessment facilitator guides a group of subject matter experts through discussions to determine the different aspects of a role being certified. The group notes the potential complexity of the tasks and topics discussed. The level of complexity is included when formulating exam objectives from the job task analysis.
A complex multiple choice question mimics the mental effort a person goes through as they perform a task in real life.
Read the sample exchange below.
SME 1:
A business consultant doesn’t need to know all the backend details, maybe only an awareness will do.
FACILITATOR:
What would be proof that they’re aware of backend functions?
SME 2:
They can provide a high-level description of the solution components to the client.
FACILITATOR:
So they should be able to name the product components and describe how they relate to each other?
SME 1 and 2:
Yeah, what we usually end up doing is give an overview of the main backend components. But when it comes to data and reports though, it's definitely has to be in-depth knowledge and experience. We have to be able to review the custom reports the client wants and set up templates. And then be able to recognize data issues affecting the ability to customize reports.
FACILITATOR:
For a given client scenario, they should be able to analyze custom reporting requirements. Another topic could be on recognizing data issues.
From the example above, the three exam objectives identified during the exam design are:
Describe the different product components
Given a scenario, determine the effect of data issues to reporting needs
Given a scenario, recommend custom report configurations and templates
Do you notice that describing different product components requires less mental effort to answer than analyzing custom report requirements?
Changing complexity
Objective 1 is basic and requires only recalling learned information. This isn't complex, but as discussed by the experts, it reflects what the business consultant role does in the course of their work. | Example: What are the out-of-the-box reports available in CrwdTest software? |
Objective 2 introduces complexity by applying learned knowledge to a given situation. (The example can use one more variable) | Example: A client wants a report on the percentage of bugs open and closed by developers daily. What standard report can the business consultant recommend? |
Objective 3 points to a complex question. As the task of the consultant in the real world involves considering several types of information, conditions or factors, the question would likely include a scenario with several types of information for the test taker to consider. | Example 3: A community bank uses CrwdTest software to monitor their load test lifecycle project and store its results. The IT team wants to integrate the scheduled testing cycles listed in CrwdTest and the calendar of each account manager to minimize any impact to the sales cycle. The data must be updated in real-time. What should the CrwdTest consultant recommend to the IT team? |
Based on the examples shown, can you come up with your guidelines for composing a complex multiple choice question?
Complex tasks in the real world should have equally complex test questions.
A note about difficulty vs complexity
Complex questions are understandably difficult both to author and to answer. But a basic question can be difficult too (though in my opinion, artificially so). Consider the examples below:
What is the value of Pi (π)?
3.1416
(basic, uses recall of learned information)
What is the value of Pi (π) to the tenth decimal?
3.1415926535
(basic, but one I consider difficult to recall and do not wish to see in an exam I'm taking)
I’m not a fan of this type of difficult question.
Conclusion
The complexity of a test question is determined during the design stage (exam blueprint) and developed during the writing stage (item writing).
During job task analysis, include complexity in the discussion with experts. Determine the level of mental effort needed for each exam topic identified. It is tedious but needed if you want the certification exam to validate competence to perform expected tasks.
Refer to the Revised Bloom’s Taxonomy of Educational Objectives, a taxonomy for learning, teaching and assessing, for industry-accepted levels of cognitive processes. Use the recommended verbs at each level, to aid you and the experts during the formulation of exam objectives and the development of the test questions
Further discovery
If you are new to exam development, I recommend researching the following terms included in this blog post:
exam blueprint
item writing
Revised Bloom’s Taxonomy of Educational Objectives