Reviewing a test item with the Evidence Centered Design model

Introduction of the ECD model

In this post, an item will be assessed to check whether it is fitting to assess a certain target skill. This item, shown in Figure 1, is taken from an old Dutch traffic theory exam for cycling and aimed at primary school children in grade 5.  For this assessment, the evidence centered design model is used. This model consists of six sub-models: the student model, task model, evidence, presentation, assembly, and reporting model, see Figure 2 (Almond, Steinberg, & Mislevy, 2002). In this post, only part of the student, task, and evidence model will be discussed. The student model provides insight into what the minimum requirements or skills are when performing a task. The task model provides an answer to which assessment tasks are needed to gain information about the student. And the evidence model answers the questions: ‘What counts as evidence for proficiency?’ and ‘How to interpret evidence when drawing a conclusion about the target skill?’. Of those models, the following aspects will be discussed: the target skill, the traffic task and task situation, the task context, the task complexity features, the responding, and the responding process.

Figure 1. Item used for the assessment of the target skill, translated into English (VNN, 2014).

Figure 2. The sub-models that together form the ECD model

Target skill

The target skill seems to be decision making in traffic or having sufficient knowledge about traffic rules and insight in a situation to safely carry out the task. The task in this case, as described in Figure 1, is to safely pass the car that is backing out of a parking spot. The students will go through a few processes while solving the item, the perceptive process, the anticipation process, and the decisive process. In the perception process, they become aware of their environment. More specifically, their speed, the speed and direction of the other vehicles, and the fact that the rules indicate that the car should give way to them. In the anticipation process, they need to know the possible outcomes and predict the behaviour of the other people involved in this situation. The following situations are a few possibilities that may cross their mind: we can continue if the car lets us go first, we should slow down to make sure the car driver will notice us in time, we should slow down to avoid a collision if the car driver does not see us in time, we should not break abruptly since we are then risking a collision with each other (other cyclists). These thought focus on traffic flow, safety, and social participation in traffic. Lastly, a decision needs to be made on how to act and the corresponding answer needs to be chosen. The decision for continuing or slowing down will be based on the speed and movement of the car. If it stops, the cyclists can continue. If the car does not stop, the cyclists will have to slow down and eventually break. Therefore, the safest option is to at least slow down until the cyclists are noticed by the car driver. So, Sanne gives the correct answer (see Figure 1).

            The description of solving this item might seem straightforward, however, it is important to be aware of the pitfalls of this question. It is essential to pay attention to what can go wrong during these processes. It is possible that not all relevant information is noticed in the perception phase, it is possible that not all possible outcomes are thought of in the anticipation phase, and it is possible that still the wrong decision is made despite having the correct perception and anticipation. It might even be possible that, for example, reading skills of the students are not well/ fully developed, therefore the student might not understand the question, which also can be a reason to give the incorrect answer to this item.

Traffic task and task situation

The traffic situation can be described with the help of a detailed overview of the traffic task, shown in Table 1. In this table is shown that the cyclists are cycling on a road and there are other road users. The cyclists encounter a car that is backing out of a parking spot into the road and they must change their position to pass the car and avoid a collision. When they passed the car, they can go back to cycle on the right side of the road again.

Table 1

Characteristics of task situations

Main task	Subtask	Light	Weather	Road category	Road section	Other road users
Cruising	Stay on course	Normal view	Normal	30 km/h road	Straight road	Cars
Cruising	Stay on course	Normal view	Normal	30 km/h road	Straight road	Bikers
Change lateral position	Overtaking car	Normal view	Normal	30 km /h road	Straight road	Cars
Change lateral position	Overtaking car	Normal view	Normal	30 km/h road	Straight road	Bikers
Change lateral position	Merging	Normal view	Normal	30 km/h road	Straight road	Cars
Change lateral position	Merging	Normal view	Normal	30 km/h road	Straight road	Bikers

Task context

The context of the task can be real, simulated, described, or context-free. Since the item is a picture, it is clearly not a real or simulated context. I think the picture can be evaluated as a described situation as well as context-free. The actors, objects, and materials are visually shown and there is only focus on the specific task of safely continuing the route without being interrupted or hit by the car that is getting out of the parking spot. An argument that can be made for context-free is the fact that only this specific task is focused on, so the test taker should mainly be aware of the rule that the car driver should give way to the cyclists. However, since a picture with the situation is provided rather than only a rule or sign, I would say that the context of the task is more described than context-free.

            To let a student experience a learning task more realistically, the task context could be real or simulated. For example, if you are cycling and you want to turn left. You must think about a lot of things, e.g. looking for other road users, signalling, deciding if you can continue or should wait. Therefore, practising such a skill might be helpful. Doing this in a simulation has the advantages of it looking like a real situation but without the actual danger.

Task complexity features

There are various aspects that add to the complexity of carrying out the target skill in a real situation. A few of such aspects are shown in Table 2. As described at the target skill, the student needs to notice other road users and their speed and direction. To decide how to handle the situation other road users and their speed and direction should be considered. The decision that is made should be feasible to carry out regarding space and time.

The complexity of this task could be varied by changing the situation in which the target skill is carried out. To increase the complexity, it could be night- instead of daytime which decreases sight and visibility. To make the task less difficult, the car coming from the opposite direction or the parked cars could be removed to increase visibility and space to carry out the action of passing the car.

Table 2

Task complexity features

	Perception	Decision making	Action execution
Sight and visibility	X
Presence of other road users	X	X
Regulation of situation		X
Speed differences	X	X	X
Time pressure		X	X
Space to carry out actions	X	X	X

Response

The students respond to this item by selecting one of the three answers that are provided. Therefore, the answer is provided using the visual response channel. However, if the answer would be constructed by the students themselves, they have the possibility to express and explain themselves better and the question won’t be marked with a pass or fail only based on the final answer. The constructed answer could be provided to the assessor verbally or an element of interaction can be added to the question by moving certain elements in the situation. However, the item is paper-based, so, instead of selecting one option, it would be more insightful to present this item as an open question.

Response processing

There are different parts of a response that points can be assigned to: actions and strategies, the actual solution, or the consequences of the solution. In this case, since only A, B, or C can be chosen, the reasoning of the test taker is not provided to the assessor. So, either the actual solution or the consequences the answer has in this situation can be scored. However, in the case of assigning points to the answer with the best outcome, the item should be formulated differently. The wording of the question should make the aim of the item clear. If the item would be presented as an open question as suggested above, the actions and strategies used by the student could be scored together with the actual solution.

            The scoring for the alternative task mentioned in the task context (turning left when cycling) differs from the item provided in Figure 1. The alternative task has a simulated context, the actions and the consequences are important, thus the student should get points based on (one of) these parts. However, a student should be made aware of the aim of the task before carrying it out. When the assessor is interested in the reasoning behind the actions, the student could share their thinking process when deciding what to do or they could be asked later why they made certain decisions.

Conclusion

A few aspects of the student, task, and evidence model were discussed. Those aspects are an example of what must be considered when constructing an item for a test. It is important to know what skill needs to be measured to develop a fitting item or test that provides an answer to the question if the student is proficient in a certain skill.

            For this specific item, I think overall it is a good item to measure the target skill. However, it could benefit from rethinking the way the student has to respond to the item since now the reasoning of the answer that is provided is unknown to the assessor. The reasoning can also give the assessor valuable information about possible mistakes that are made during the decision-making process.

References

Almond, R. G., Steinberg, L. S., & Mislevy, R. J. (2002). Enhancing the design and delivery of assessment systems: A four process architecture. Journal of Technology, Learning, and Assessment, 1(5), 1–64.