Best Practices in Writing Test Items

This article is based on content presented by Lori L. Alexander, MTPW, ELS, MWC, at the AMWA Medical Writing & Communication Conference. It was originally published in the AMWA Journal.

Continuing medical education credits are often awarded based on answers to test questions, so it is essential that the questions be designed well.

Research shows, however, that “almost half of all test items have flaws,” said veteran continuing medical education writer and editor Lori Alexander. This problem is driving the need to develop better test items and is creating job opportunities for medical writers and editors. In this open session, she explained how to write test items using guidelines and best practices from the National Board of Medical Examiners and the American Board of Internal Medicine.

Alexander began by describing a general approach to writing test items:

• Identify your audience and what they need to know.
• Focus on meaningful content rather than trivial facts.
• Emphasize the common clinical presentations of frequently seen diseases rather than rare disorders.

She noted that multiple-choice questions are typically the preferred test format and explained that research shows a high score on a test composed of multiple-choice questions correlates with superior clinical performance.

Before she launched into a detailed discussion of the 4 learning objectives for the session, she outlined the anatomy of a test item:

• Stem: the problem to be solved, often a clinical scenario
• Lead-in question: the prompt to elicit the correct response
• Answer options: the correct answer and a variable number of distractors
• Rationale: the explanation of why an answer is correct or not
• References: the evidence from the literature that supports the rationales

Develop Clear and Focused Educational Objectives.

The educational objective is the fundamental basis of a test item, Alexander said. It should focus on a single concept and address at least 1 of the following: a recent advance in medical knowledge, a best practice, or a clinical management principle.

The educational objective should start with a measurable verb that describes the behavior desired from the test-taker; for example, “Identify the mammographic findings of suspicious masses.”

Writers should also pay attention to the cognitive level of the educational objective. According to Alexander, “More than 90% of test items are written at a low cognitive level” and require the test-taker to simply recall facts. Clinical decisions, however, require the ability to apply knowledge, which can only be tested with higher cognitive level objectives.

Write Stems and Lead-in Questions that Follow Guidelines for Best Practices.

The stem, which is usually a clinical scenario, should be based on the educational objective. Writers should select a common clinical presentation and present it clearly and succinctly.

One of the most common problems with stems is a lack of focus, which can occur because the stem either is wordy or teaches a concept instead of generating a question. Both flaws frustrate the test-taker who must sift through unnecessary information.

The lead-in question should be phrased as a direct question; fill-in-the-blank and incomplete sentence formats are not recommended. Other guidelines for composing lead-in questions include:

• Define the problem clearly.
• Connect to the clinical scenario.
• Follow the “cover-the-options rule.”
• Address a single specific task (diagnosis, testing, management).

The cover-the-options rule refers to a guideline from the National Board of Examiners that recommends a lead-in question provide enough information to be answered without viewing the options.

Create Answer Options That Are Free of Flaws and Clues.

Answer options are “the component of test items with the most flaws,” Alexander said. Accordingly, there are guidelines for writing good answer options:

• Be simple and clear.
• Order logically.
• Craft plausible distractors.
• Create a single best answer.
• Vary the position of the correct answer.
• Make options mutually exclusive.
• Make options grammatically parallel.

Writers should also create answer options that are within the same category of response. For example, if one option is a drug, then the other options should also be drugs.

Answer clues should also be avoided. Writers may unwittingly identify distractors with grammatical errors such as subject-verb disagreement or differing verb tenses. They may also clue test-takers to the correct answer by repeating a word or phrase from the lead-in question (the “clanging clue”), or by making the correct answer longer and more detailed.

Construct Rationales That Explain All Answer Options.

Rationales help test-takers understand the material better. They are an essential component of self-assessment modules but not always included with other educational activities. Clients may prescribe that rationales be a certain length— check with them before investing a lot of time researching and writing. Some simple tips for writing rationales included:

• Explain all answer options.
• Concentrate on what the lead-in question asked.
• List in the same order as the answer options.
• Include at least 3 published references to support data.