Decision Guide for RTT Administration Protocols

Recorded Text Testing
Background Research
Intelligibility Interviews
Choose Kind of Test
Choose Test Points
Materials
Obtain a Text
Develop Questions
Create Introduction Text
Assemble Pilot Test
Administer Pilot Test
Select Final Questions
Build Reference Tests
Develop Post-RTT Questions
Administer Text Test Set
Process Scores

How do I know what kind of Recorded Text Testing (RTT) is right for my situation? Follow instructions and links in this decision guide to determine the best administration protocols (developed by Douglas Boone, 2004).

Decision Guide for RTT Administration Protocols

For any research method, it is important to choose appropriate protocols to ensure the reliability of the test and to enhance its validity. When researching comprehension, the RTT is no exception. The following questions will help you choose the optimal protocols for a given situation.

1. Is this survey limited by time or other resources?

Time and other resources are often insufficient for the full application of the Classical RTT protocols. The Rapid Appraisal RTT (RA RTT) is an attempt to apply the recorded text method under straitened circumstances. It builds upon the use of group interviews and observation that characterizes Rapid Appraisal surveys (Bergman 1991). The RA RTT is generally suggested when the alternative is doing no comprehension testing at all.

• If the survey is limited, proceed to Step 3a in order to choose between Options 1 and 2.
• If it is possible to conduct in-depth testing, proceed to Step 2 to help you choose from among Options 7-13.

2. Is the question and answer format natural?

Many people are unaccustomed to answering specific questions about something that has just been heard or read. Individuals or groups may feel so awkward about it that they score poorly or refuse the test. The retelling format avoids this problem. It is most commonly used in Rapid Appraisal surveys, but it may also be recommended for other surveys when it is more appropriate than the question and answer format.

• If the question and answer format is natural, proceed to Step 3c to continue with Options 9-13.
• If it is not natural, proceed to Step 3b so you can choose between Options 7 and 8.

3. Is there a common language?

3a. RA RTT. If the researchers and the subjects share a common language, it is best to use this language to prompt the retelling (Option 1). Otherwise, use an interpreter to prompt the retelling through the local language (Option 2). In either case, the subjects may reply in whatever language they prefer.

3b. Quantitative scoring of Individual Retelling. If the researchers and the subjects share a common language, it is optimal to use this language to prompt the retelling (Option 7). Otherwise, use an interpreter to prompt the retelling through the vernacular, or include the prompt in recorded instructions that accompany the text (Option 8). In either case, the subject may reply in whatever language they prefer.

3c. Quantitative scoring of Individual Answers. If the researchers and the subjects share a common language, take advantage of that situation and use that language. Proceed to Step 4 to narrow the choice from among Options 9-11.

• If there is no common language, or if you cannot be assured that each subject knows it, translate the questions into the local language with the intention of eliciting responses in the vernacular. Proceed to Step 5 to help you choose between Options 12 and 13.

4. Can the subjects be expected to be able to translate?

• If there is no doubt that subjects are bilingual to the point of being able to hear a question and answer it in the common language, then the best choice is not to use the vernacular at all (Option 11).

• If there is any doubt, they should be left to respond in the language of their choice. If subjects are able to comprehend the common language, then the questions may be posed in the common language (Option 9). Otherwise, it is more prudent to translate the questions into the vernacular (Option 10).

Since you can expect the language of stimulus to bias the choice of language of response, the convenience of the subject should dictate the choice among these three options.

5. Will people respond to a recorded stimulus?

• If people will probably realize that they are supposed to answer the recorded questions, then the Classical RTT protocols can be used (Option 13).

• Some people have a hard time recognizing that the questions are separate from the story and that they are expected to answer them. You may get more reliable results if you have someone with you to ask the questions in the subjects' language. In this case, the questions are translated into the vernacular and recorded, but a local assistant repeats each recorded question for the convenience of the subject (Option 12).