Developing an SRT

Data Collection Tools
Interviews
Observation
Questionnaires
Recorded Text Testing
Sentence Repetition Testing
Word Lists
Participatory Methods
Matched-Guise

Sentence Repetition Tests
Developing an SRT
Administering an SRT
Analysing SRT Data

Introduction

Radloff (1991:37-38) <ref>Radloff, Carla F. (1991). Sentence Repetition Testing for Studies of Community Bilingualism. Dallas: Summer Institute of Linguistics.</ref> reminds us of how much depends on us taking particular care at this stage of working with SRTs. The process of creating an SRT is time-consuming and may seem over-elaborate. But we are creating a tool which will help to determine the linguistic future of entire speech communities and we should bear this in mind no matter what tools we are developing or administering.

We should pay particular attention at the development stage to finding the right personnel. This includes finding speakers of the test language who have the right level of education to contribute to test development. Taking time to rate participants for proficiency is also worth doing. And making sure our transcription of the sentences that we include in the test development is also important.

We want the most accurate results possible from our tools and so, we should be willing to be as thorough as we can be with our development of them.

Preliminaries

It's best to develop the test in an location where the test language is a common LWC and where you can get good access to contacts to help you develop the test.

Radloff recommends including the following personnel:

• The Researcher: er... that's you! You don't need to have much proficiency at all in the language of the test. But the more you have, the easier the development and administration of the test will be. You don't need to spend weeks living in the community prior to starting development but you could read any materials that other workers in that language have produced such as grammar notes or descriptions of phonology, etc.
• Educated Mother-Tongue Speakers: You'll need at least three of these. They elicit the initial sentences, select those to be used in the test and help develop the scoring system. It's helpful that these people have some education because you'll want to be testing in a standard form of the language and this is usually acquired through education in that language. Education also ensures that these helpers have enough ability in the test language to be able to construct a test in it. It may well be that there is no formal education in the LWC for the people you are working with. In this case, consider education to equal experience and select assistants who obviously display a high level of ability in the language and who aren't challenged by the construction of such a test.
• Second Language Speakers: You'll need a number of these at each proficiency level in the test language so you can calibrate the test. Radloff recommends looking to these social groupings for such a pool of speakers: a local college, a business, a neighbourhood, an organisation, etc.
• Test Administrator: We talk more about training the test administrator on our Administering an SRT page. Of course, the researcher could also administer the test but, to do so, they should be familiar to some extent with the test language. The administrator and the researcher need to have a language in common to carry out the training and communicate the results.

Recording equipment is obviously necessary along with enough headphones for a participant, the researcher and the administrator (if these are different people). In order for everyone to hear the recording at the same time, two Y-adaptors are needed. It is vital that at each stage of data collection and ordering that you backup your recordings and sentence collection and that you keep these backup files on a separate machine/server to your data.

Eliciting Sample Sentences

Aim: get a wide range of natural well-formed sample sentences (40-50) that range from simple to very difficult

1. Collect sample texts ⇒ In order to achieve this, you'll have to elicit or find a sample text that has a wide range of language in it. This could be spoken or written, but if it's written, it needs to be natural language. You could ask your second language speaker to respond to a topic you come up with or give a description of something. Or, you could actually ask them to form sentences on a topic which will include specific grammatical constructions of increasing complexity.
2. Extract a range of sentences ⇒ Once you have your text/s, you'll need to go through it and extract a range of sentences (not questions!) from simple through medium to difficult. There should be 60-70 of these. Difficulty could be a matter of grammatical complexity or it could also be other features such as the level of formality. In many languages, these are related. One tip that Radloff gives is to select some sentences that begin with discourse markers as these tend to be more challenging. Make sure you have a wide variety of content/topics. If you don't, participants may be able to provide answers which are based on previous sentences they've heard.
3. Get the sentences written down ⇒ Having chosen your sentences, ask the second language speaker to write the sentences down in the local script. You should transcribed the sentences you've chosen phonetically using IPA so that you (and your research team) can follow these sample sentences in later stages of test development and administration.
4. Record the sentences ⇒ Ask the speaker to speak each sentence at natural speed and record them. Asking them to pause for a couple of seconds before and after they speak each sentence will help separate questions when it comes to actually constructing the test later. If you're good at using software like Audacity though, to insert blank spaces and compile audio tests like SRTs, this will be less of an issue.
5. Get a second opinion ⇒ You should then find two other educated second language speakers (preferably one male and one female) to read through all the sentences collected and judge them for suitability. Sentences which are too political, opinionated or otherwise controversial should be eliminated as well as any that simply 'sound funny.' Try to eliminate any that are too similar to each other too.
6. Refine the selection ⇒ Take one of these latter helpers at a time and play the remaining sentence recordings to them one at a time asking them to repeat them after they've heard them once. Use your phonetic transcriptions to check their repetitions carefully. Remove any sentences which neither helper can repeat because they are too long. Sentences they can almost repeat are good ones for helping you judge the upper limits of proficiency. Finally, have the whole research team look over the 40-50 sentences that should remain. If you need more or there are too many sentences focussed on one topic or are repetitious, you'll need to go back to step 1 in this section and elicit some more sentences. Have a cup of tea first though. You deserve it for getting this far!
7. Make a preliminary test ⇒ Order all the 40-50 sentences from shortest to longest. Consider the three shortest as practice sentences. Get a mother-tongue speaker of the test language (and dialect!) to record these. A man's voice is usually more widely acceptable than a woman's voice. Either while you record or later, put about a three second gap between each sentence. Each sentence should appear only once in the test but you might want to record a few samples of each sentence until you have a natural, usable sample of each.
8. Transcribe the test ⇒ Refine or re-do your transcription of the preliminary test sentences and check this with a speaker of the test language for accuracy. Ask them also to give a word-for-word translation of each of the sentences (we'll be referring to this later). Also do a free translation of each sentence. Optionally, you could add the local script version of the sentence too.

Evaluating L2 Speakers

Aim: assess the proficiency levels of the 50 or so speakers who will give you your range of proficiency levels

1. Select a proficiency scale ⇒
2. Select 50 speakers ⇒ see pages 52-3
3. Assess each participant ⇒

Pilot Testing

Aim: start calibrating the test by testing the 50 or so speakers from the previous stage

1. Decide who does the testing ⇒
2. Get consent ⇒ include here explaining what the test is for and the procedure from p52-3
3. Test high-level speakers first ⇒
4. Test all the speakers ⇒
5. Score the test responses ⇒

Refining the Test

Aim: select the final 15 sentences which will form the SRT and match SRT scores to proficiency levels

1. STEP $rArr;
2. STEP $rArr;

References

<references/>