|
 |
Multiple Choice Test
(Screen 3 of 6)
|
 |
Teaching Goals
Student Learning Outcomes:
- Demonstrate recognition and recall of knowledge, skills and abilities
- Demonstrate analysis, synthesis, and evaluation
- Demonstrate critical thinking
Instructor Teaching Outcomes:
- Assess higher-order and lower-order thinking skills
- Assess content broadly and deeply
- Assess quickly and efficiently
- Identify student misconceptions
Step-by-Step Instructions
In content assessment, you should aim at finding out which students have mastered the material and which have not, and to what extent they have or have not. The test score should reflect that and only that. The "rules" of test development, then, work in fundamentally two ways: to maximize the potential for the student to show you what they know about the content, and to minimize the possibility that other factors will influence the score. Such factors range from test forms that are hard to read and navigate to fatigue to reading ability or computer literacy. In writing items, you must eliminate "guessing" and "trick" questions. In test layout, that means avoiding student confusion about how to complete the test.
Planning the Test: The critical issue at this stage is matching the test to your teaching so that what was taught gets tested and what is tested gets taught. This is fundamentally an issue of fairness as well as matching assessments to learning outcomes. A great many of the complaints about testing stem from a mismatch here ("I didn't know that was going to be on the test!", "We didn't talk about that in class!"). Constructing a test blueprint is an excellent planning tool for testing and for teaching.
A test blueprint [sometimes called a Table of Specifications (e.g. Gronlund & Linn, 1990)] is a table that records the content areas for the test on one axis, which may be listed by topic or text chapter or other divisions. The other axis of the table categorizes the ways you expect your students to know that content. These can be traditional taxonomies like Bloom's Taxonomy of Educational Objectives or your own categories. Finally, each cell in the table records the relative weight you'll give to the intersection of each content and way of knowing. These can be expressed as proportions or as numbers of items. Here's an example of a test blueprint:
|
Example Test Blueprint for an Introductory Physics Course
|
|
Facts, Terminology, Concepts
|
Problem Solving
|
Application and Integration
|
|
| Work |
10%
|
5%
|
5%
|
20%
|
| Energy |
10%
|
40%
|
10%
|
60%
|
| Potential Energy |
4%
|
16%
|
0%
|
20%
|
| Kinetic Energy |
3%
|
12%
|
5%
|
20%
|
| Mechanical Energy |
3%
|
12%
|
5%
|
20%
|
| Power |
2%
|
8%
|
10%
|
20%
|
|
22%
|
53%
|
25%
|
|
Once you have the Test Blueprint constructed, you can use it to plan your teaching as well as to write test items. It's useful in planning teaching because it records what content you feel is most important and how you expect your students to know that content. It also informs decisions relevant to your instructional planning. And, since the goal is a match between teaching, testing and outcomes, the same emphases used in teaching should be used in testing. The Test Blueprint thus serves as a powerful way to align teaching and assessment (Nitko, 2001).
Writing the Items: Perhaps the first issue here is whether you will actually write your own items or use items from test or item banks that accompany textbooks. Using publishers' test banks comes with a number of risks. First, your test blueprint and the publishers' may look different, so you'll need to ensure that the test still retains your emphases. Second, it's doubtful that published item banks consist of tried and true, high quality items (e.g. Hansen & Dexter, 1997; Sims, 1997). Neither authors nor publishers usually receive compensation for providing item banks, and therefore little if any item development work is likely conducted. Therefore, you should use published items sparingly or, at least, use them very carefully with as much scrutiny as you would your own items.
These tips and "rules" for item writing are synthesized from research-based sources and/ or consensus-of-the-field sources (e.g. Haladyna & Dowling, 1989). For an extensive treatment of this topic with many examples and checklists, see Nitko's chapter 8 (2001).
- Determine how many total items you want. Considerations here include how much material the test will cover, how deep the coverage needs to be, how complex the items are, and how long students will have to take the test. [The rule of thumb is one minute per question or possibly more if the items are complicated (Gronlund, 1988; Oosterhof, 2001)].
- Use the Test Blueprint to determine how many items you need to write in each cell.
- Avoid being overly general or overly specific with the content for each item. This is somewhat dependent on your learning objectives, but you don't want to be asking about broad, sweeping issues nor do you want to be asking about minutia.
- Make sure each item tests one and only one concept. If the item is "double-barreled", that is tests two or more concepts, you won't know which of the two the student truly understands if she gets the item correct. (See Pros and Cons for more on this issue.)
- Decide what item variations you're going to use (see below).
Rules for Writing Item Stems
- Either write the stem in the form of a question or, if a statement is in the stem with its completion in the options, put the "blank" at the end of the stem, not in the middle.
- Poor: When looking at liquid in a test tube, the ____________ is the name of the curved surface of the liquid.
- Better: The curved surface of liquid in a test tube is called a .
- (Answer: meniscus)
- Put the "main idea" of the item in the stem, not in the options.
- Streamline the stem to avoid extraneous language, but try to put as much of the text in the stem leaving the options shorter.
- Avoid negatives like "except" or "not" or highlight them in italics, boldface or underline if you use do use them.
- Highlight important words like "not", "only", "except" etc. if you use them at all.
Rules for Writing Options
- General Option Writing Tips
- Practically speaking, there's no "magic number" of options you should use (Nitko, 2001). Write the number of options that make sense. It is acceptable to mix and match the number of options. It's better to have a three-option item than a four-option item with a poor distracter (Nitko, 2001).
- Options should be relatively equal in length. A much longer or much shorter option can attract responses because they stand out visually.
- Make sure all options are grammatically congruent with the stem. For example, the article “an” at the end of the stem will give a key that starts with a vowel away.
- Put repeating words or phrases in the stem rather than in each option.
- Avoid overlapping options.
- Poor: Water will be a liquid between ____________ and ____________ degrees centigrade.
- a) 0, 100
- b) -50; 0
- c) 100; 150
(Note that a and b both include 0 and a and c both include 100 -- they overlap.)
Better: Water will be a liquid between ____________ and ____________ degrees centigrade.
a) 1; 99
- b) -50; 0
- c) 100; 150
(Note that there is now no overlap.)
-
- Avoid "all of the above." Its use muddles the interpretation of a student's response (Nitko, 2001).
- Key-writing Tips
- Make sure the key is, in fact, correct according to the text and/ or consensus in the field (Which, we hope, is also what's been taught in class!).
- Distracter Writing Tips
- They're called distracters because they are strategically designed to attract examinees who haven't completely mastered the content. This isn't "tricky" or "deceptive" or "unfair". It stems from the premise that the goal of testing is to find out who has learned the content and who has not, perhaps along a continuum between the two. Students who mastered the material should recognize the key and those who haven't should not.
- All distracters need to be plausible. We've all seen "Mickey Mouse" listed as an option on tests, but that's seldom a believable possibility, and, in terms of finding out who has mastered the material, a waste of ink.
- Use your knowledge of common student misunderstandings in writing distracters. If you know, for example, that students often miss a step in a calculation, include a distracter that would result from that miscalculation. There are also many studies that have documented common misconceptions in science concepts. Building these into multiple-choice distracters is a possibility (Sadler, 1998).
- Avoid "linking" items where the answer to one item is found in or dependent on another item. This is something to check for near final test assembly when you are proofreading. It's very easy to do inadvertently when you're writing items across several sessions.
Constructing the Test: Here are a few tips for assembling the test.
- Provide very clear, printed instructions and directions. If you're using a variety of Multiple Choice formats, you may wish to write separate directions for each section.
- Order the items by content, by item format, and then by increasing difficulty of items (Gronlund, 1988). This rule is based on information processing principles. It is easier mentally for students to answer all of the items about one content before moving to another. They also perform similar mental tasks on similar items before changing mental tasks with other formats. Finally, putting easy items before hard items helps students gain some success early on.
- Randomize the ordering of keyed responses to overcome many of the rules of thumb for guessing, like "always choose B or C". This also eliminates students looking for or using patterns like AABBCCDD, ABADABA, etc.
- Make sure students don't get "lost" in the test because then their score doesn't reflect what they know, but how well they navigated the test.
- Don't crowd items on the page.
- Avoid using double-sided pages because students may miss the back page.
- Use navigational cues throughout like, "continue on the next page" or "page 4 of 12".
Scoring the Test: You may use a full-credit model, where the student gets the item correct or incorrect. You may also use a partial credit model, where the key receives full credit and some distracters receive partial credit. More information on scoring appears in the "Analysis" section.
Tell me more about this technique:
|
