Administering an SRT - Revision history

Admin at 01:24, 13 July 2011

2011-07-13T01:24:27Z

Admin at 01:19, 13 July 2011

2011-07-13T01:19:11Z

Admin at 23:18, 21 June 2011

2011-06-21T23:18:27Z

Admin: /* Testing */

2011-05-17T11:44:15Z

Testing

Admin: /* Scoring */

2011-05-17T11:42:33Z

Scoring

Admin: /* Testing */

2011-05-17T11:17:28Z

Testing

Admin: /* Pre-Testing */

2011-05-17T11:03:09Z

Pre-Testing

Admin at 10:40, 17 May 2011

2011-05-17T10:40:56Z

Admin at 06:33, 12 April 2011

2011-04-12T06:33:23Z

Admin at 06:31, 12 April 2011

2011-04-12T06:31:45Z

← Older revision		Revision as of 01:24, 13 July 2011
Line 88:		Line 88:
	Good training will include plenty of discussion of situations like this example.		Good training will include plenty of discussion of situations like this example.

−	~~[[Category:Methodology]]~~
	[[Category:Sentence_Repetition_Testing]]		[[Category:Sentence_Repetition_Testing]]

← Older revision		Revision as of 01:19, 13 July 2011
Line 89:		Line 89:

	[[Category:Methodology]]		[[Category:Methodology]]
		+	[[Category:Sentence_Repetition_Testing]]

← Older revision		Revision as of 23:18, 21 June 2011
Line 87:		Line 87:

	Good training will include plenty of discussion of situations like this example.		Good training will include plenty of discussion of situations like this example.
		+
		+	[[Category:Methodology]]

@@ Line 47: / Line 47: @@
 You will need a device to play back your test recording and two sets of headphones connected by a Y-connector so that both the participant and the administrator can hear the recording at the same time. You might want to consider having someone to operate the recording and a third person to score. If you do this, you will need a third set of headphones and another Y-connector so that all three can hear the recording simultaneously.
-In addition to a playback device, you might also consider using a second device to record the first 5-10 participants in each location. This serves as a reference which can be very helpful if there are slight dialectal differences which give consistent "errors" as people produce varieties of the test language that are more natural to them.
+In addition to a playback device, you might also consider using a second device to record the first 5-10 participants in each location. This serves as a reference which can be very helpful if there are slight dialectal differences which give consistent "errors" as people produce varieties of the test language that are more natural to them. It is important that you do not mark consistent dialectal differences, even lexical ones, as errors. If you do record the first group of participants' responses, don't forget to obtain [[informed consent]].
 Make sure you choose a location where there are as few distractions as possible for the participant. Onlookers should be told to remain as quiet as possible and not communicate with the participant. It is important to realise that if a participant waiting their turn overhears the repetitions of a current participant, it does not affect their future participation. They cannot hear the original recording and the amount of language and the time between them hearing it and having their own turn is too great for them to memorise what they hear. In any case, they have no knowledge of the accuracy of the responses they hear and this means that it is vital that the test administrator does not give verbal feedback on the participant's performance. Instead, general encouragement and praise for participating in the test and "knowing the language so well" should be given.

← Older revision		Revision as of 11:42, 17 May 2011
Line 52:		Line 52:

	==Scoring==		==Scoring==
−	~~(notes~~ from ~~Radloff section 2~~.6)	+	You will need to prepare a very large number of scoresheets so that you have one new one for each participant at each location you are going to. If you are on a long survey and carrying such a large amount of paper around is prohibitive, consider using a netbook or some other device to store your data. If you do so, carry an external drive of some sort to back up data and make sure that this is kept separately from the device in case you lose one or the other and in a waterproof case.
		+
		+	Scoresheets should have the test sentences written in local script or [[IPA]] depending on what the administrator's preference is. There should be enough space around the sentence to mark errors. Each scoresheet will need space to add each participant's name, age, sex, mother tongue, participant ID number, etc.
		+
		+	The administrator should sit so that the participant cannot see what is written on the scoresheet, and bear in mind that if future participants are watching, they should not be able to see the scoresheets either, particularly if the sentences are in the local script. If the administrator only marks when errors are made, the participant may realise this and react to their errors being recorded. Therefore, it is important to make or be seen to make the same number of marks for good sentences as for erroneous ones.
		+
		+	Each of the words that the participant does not repeat perfectly should be marked. Three errors or more means that they score nothing for that sentence. Errors consist of the following:
		+	* omission - leaving out words that are in the recording
		+	* substitution - replacing a word in the recording with another word that is not in that place in the sentence in the recording
		+	* change of word order - swapping the position of words from one part of the sentence to another
		+	* pronunciation - not those which may result in a different accent but only changes which result in loss of intelligibility and so prevent communication
		+	* repetition - and this includes restarting a sentence or going back to repeat part of a sentence once started. If a restart is done to correct an error, one point is deducted for the original error but not the restart.
		+	* addition - putting anything into the sentence that is not in the original
		+	Scoring is the same for all of these i.e. 1 point deducted for each no matter what type of error it is.
		+
		+	Note that in part 5 of [[Developing an SRT#Pilot Testing\|our guide to developing an SRT]] we provided a key for scoring that includes suggested symbols that administrators can use to mark types of error. However, in a field testing situation, this level of detail is not only unnecessary, it also slows down the testing procedure too much. Simply marking the place where an error occurred is enough for an actual testing situation.
		+
		+	Once an error is made in a word, one point is deducted. Therefore, should a participant make two or more errors on the same word e.g. mispronouncing it unintelligibly and repeating it also, administrators should simply mark this as one error. Further errors have to occur on other words.
		+
		+	If, during testing, there is debate about what consistutes an error or how many errors may have occurred, this means that training of administrators has not been done thoroughly enough. Prior to a survey, the entire team should be familiar enough with the sentences and the potential errors that have been flagged from the pilot test recordings to know how to score the various errors that are likely to occur. Consider this test sentence
		+
		+	''The teacher repeated the sentence to the students.''
		+
		+	and the response
		+
		+	''The teacher is saying sentence to the students.''
		+
		+	How many points would you deduct?
		+
		+	There are three errors and three points should be deducted thus:
		+	* '''omission''': ''the'' is missing
		+	* '''substitution''': ''saying'' replaces ''repeated''
		+	* '''addition''': ''is'' is inserted
		+
		+	Good training will include plenty of discussion of situations like this example.

@@ Line 43: / Line 43: @@
 ==Testing==
- (notes from Radloff section 2.5 and 3.6)
+Having [[Developing an SRT|developed your SRT]] and trained your administrators, you are now ready to actually carry out the testing. Although development takes time (as any worthwhile survey tool should), the actual testing of an individual using an SRT is very quick. However, you still need to ensure you have the right environment for your test and that the administrators know how to create this environment.
 ==Scoring==
   (notes from Radloff section 2.6)

@@ Line 6: / Line 6: @@
 Before you do anything, you need to make sure you have everything in place for training the test administrators. To train them, you will need to carry out the following steps
 # '''Prepare the training materials''' &rArr; You will have already prepared the final test in both elaborated transcription and recorded form having worked through [[Developing an SRT]]. In addition, you will have the responses for at least 20 of the participants who helped you develop the SRT. You can use 10 of these responses to train test administrator s and then the other 10 for the third step below of ensuring administrator reliability. What you'll create are in effect 10 test recordings. Each one will have the 15 final test sentences and the responses to these of one of the 10 test development participants. You'll also need to prepare score sheets for your administrator training and these can be edited down to the final 15 sentences from the scoresheets you used in test development. We recommend that you encourage your administrators during training to note down the specific type of error they hear occuring. This is unnecessary on the field where a simple mark can be used, but in training, you need to know if they understand the differences between the types of errors so that they can score accurately.
-# '''Train the administrators''' &rArr;
+# '''Train the administrators''' &rArr; Your administrators will need to be familiar with several things before they can administer a test: they'll need to be familiar with each line of the transcription including [[IPA]], they'll need to know the scoring system, they will have to be able to manage the practicalities of carrying out a test (see below for more) and they should be able to repeat all the test sentences and follow the transcript at the same time as the recording. At this point, they're able to actually listen to the 10 practice recordings that you've completed and score them one by one, comparing their answers to the master scoresheet. They should repeat this as many times as they need to to become confident scorers.
-# '''Ensure administrator reliability''' &rArr;
+# '''Ensure administrator reliability''' &rArr; Using the other 10 of the 20 recordings made during test development, you can make another set of recordings to help with administrator reliability. By using these, the administrators can test themselves to ensure that they are maintaining the standards required for scoring tests. Unlike step 2 above, the administrators should not pause the recording at any time or refer to the master scoresheet until they have scored the entire test. At this point, they can compare their score with that of the trainer to see how close they are. If an administrator is scoring either 5 above or below the trainer's total, they will need further training until they are more accurate in their assessments. You should also use these recordings to periodically re-calibrate administrator's skills if they are to administer this SRT over longer periods of time.
 ===How do you select participants?===

@@ Line 4: / Line 4: @@
 {{SRT_Steps}}
 ==Pre-Testing==
+Before you do anything, you need to make sure you have everything in place for training the test administrators. To train them, you will need to carry out the following steps
-(notes from Radloff section 3.5)
+# '''Prepare the training materials''' &rArr; You will have already prepared the final test in both elaborated transcription and recorded form having worked through [[Developing an SRT]]. In addition, you will have the responses for at least 20 of the participants who helped you develop the SRT. You can use 10 of these responses to train test administrator s and then the other 10 for the third step below of ensuring administrator reliability. What you'll create are in effect 10 test recordings. Each one will have the 15 final test sentences and the responses to these of one of the 10 test development participants. You'll also need to prepare score sheets for your administrator training and these can be edited down to the final 15 sentences from the scoresheets you used in test development. We recommend that you encourage your administrators during training to note down the specific type of error they hear occuring. This is unnecessary on the field where a simple mark can be used, but in training, you need to know if they understand the differences between the types of errors so that they can score accurately.
 ===How do you select participants?===

← Older revision		Revision as of 06:33, 12 April 2011
Line 1:		Line 1:
		+	{{data_collection_tools
		+	}}
		+
		+	{{SRT_Steps}}
	==Pre-Testing==		==Pre-Testing==

@@ Line 1: / Line 1: @@
-# Pre-Testing (notes from Radloff section 3.5)
+==Pre-Testing==
-# Testing (notes from Radloff section 2.5 and 3.6)
-# Scoring (notes from Radloff section 2.6)
+(notes from Radloff section 3.5)