Calculating Discrimination Index

Calculating the Discrimination Index (DI) may seem complicated but actually, it's fairly straightforward. It only seems complex because it involves numbers in tables and because there are a several steps invovled. But if you keep your cool and follow them one by one, you'll have a DI in no time and be able to judge which sentences are the best ones for determining second langauge proficiency.

Preparation for Calculating the DI

Figure 1: In preparation for calculating the DI, subjects and sentences are arranged to the initial SRT scores.

The first step involves using the test scores from your 50 or so participants that you pilot tested the 40-50 sentences with.

1. Order the participants by their scores ⇒ In Figure 1, the scores of 15 participants (labelled A to O along the top) are shown on the scale at the bottom. Note that no one scored the maximum 21 points for the 7 sentences shown in this example so the scale only needs to go up to the highest score of 20. You can see that for 50 participants doing 50 sentences, this table would be much, much larger. But in a spreadsheet program, even such a large table is very manageable. The participants are arranged in the table according to their total score. In this example, A and E scored 1 and so are on the far left hand side. L, on the other hand, scored 20 and is on the right.
2. Order the sentences by points scored ⇒ The next step is to arrange the sentences from top to bottom according to the score that all the participants achieved for each sentence. In this example, sentence 2 was the hardest. If 3 represents a perfect repetition of the sentence, then theoretically with 15 participants all repeating it perfectly, each sentence could score 45 points. But the participants only achived 14 out of a possible 45 points for this sentence making it harder than all the others. Sentence 6, by contrast, was the easiest with participants scoring 32 out of the possible 45 points.
3. Fill in each participant's individual sentence scores ⇒ Finally, you fill in the centre of the table by writing what each participant scored for each sentence. In Figure 1, participant J scored 2 for sentence 6, 1 for sentences 3, 7, 4, 5 and 2 and a 0 for sentence 1.

Once these three steps have been carried out, preparation is complete and you can go on to actually calculate the DI.

Calculating the DI

Figure 2: To calculate the DI, sentence scores are rearranged and subtracted from actual scores.

As you may have noticed when looking at Figure 1, the rows containing what each participant actually scored are labelled "actual." There are two other rows under the "Actual" row. These are labelled "Rearr." and "Diff."

1. Fill in the "Rearr." row ⇒ First, rearrange the "Actual" scores so that they are in ideal increasing order. This new order is what would have happened had you been testing in ideal conditions with sentences that perfectly discriminated between proficiency levels.
2. Calculate the differences ⇒ Next, fill in the "Diff." row by completing the difference between the "Actual" and "Rearr." scores as shown in Figure 2.
3. Sum the differences ⇒ To calculate the DI for each sentence, simply add together the figures in the "Diff." row. Sentence 4 has a DI of 4 because there were four instances where the actual and rearranged scores were different. Because the rearranged values are ideal, if the actual scores are closer to the rearranged scores, they are more ideal. Thus a lower DI shows a sentence that is better at discriminating between proficiency levels. Thus, in Figure 2, sentences 6, 5 and 2 are best, and sentence 3 is good. Sentences 7, 4 and 1 are less than ideal. In this way, you can select the best 15 sentences from your 50 or so pilot test sentences. However, you should bear in mind that in real life with so many more participants and sentences, you are unlikely to get a 0 DI for any of your sentences. In addition, you'll also need to calcuate the diffulty level of each sentence before you can select your final 15 so that you get a set of 15 sentences that are truly representative of the range of levels you want to test.
4. Calculate the difficulty level ⇒ To be able to select 15 sentences that cover the range of levels you want to assess and to rank them in an increasing order of difficulty, you need to perform a simple bit of maths to calculate the difficulty level with the following formula:

1- (number of correct points for that sentence) / (total possible points for that sentence)

For example, sentence 3 has a total possible score of 45 but only 20 was scored. This means we calculate 1-20/45 to give 0.56. The closer to 1 this value is, the harder the sentence is.

Ideally, in selecting your final 15 sentences, all the sentences should have low DIs and there should be a range of difficulty levels included among them. Your 15 should also be clear recordings and Radloff advises that you do not consider sentences with a difficulty level of more than 0.95 as these may be too hard for anyone.