Developing an SRT - Revision history

Admin at 01:24, 13 July 2011

2011-07-13T01:24:44Z

Admin at 01:19, 13 July 2011

2011-07-13T01:19:29Z

Admin at 23:36, 21 June 2011

2011-06-21T23:36:24Z

Admin: /* Refining the Test */

2011-05-17T12:58:28Z

Refining the Test

Admin: /* Pilot Testing */

2011-05-17T11:46:07Z

Pilot Testing

Admin: /* Refining the Test */

2011-05-16T13:47:20Z

Refining the Test

Admin: /* Refining the Test */

2011-05-16T13:46:12Z

Refining the Test

Admin: /* Refining the Test */

2011-05-16T12:10:04Z

Refining the Test

Admin: /* Refining the Test */

2011-05-16T11:00:32Z

Refining the Test

Admin: /* Eliciting Sample Sentences */

2011-05-16T09:21:39Z

Eliciting Sample Sentences

@@ Line 90: / Line 90: @@
 ==References==
 <references/>
 [[Category:Sentence_Repetition_Testing]]

← Older revision		Revision as of 01:19, 13 July 2011
Line 91:		Line 91:
	<references/>		<references/>
	[[Category:Methodology]]		[[Category:Methodology]]
		+	[[Category:Sentence_Repetition_Testing]]

← Older revision		Revision as of 23:36, 21 June 2011
Line 90:		Line 90:
	==References==		==References==
	<references/>		<references/>
		+	[[Category:Methodology]]

@@ Line 80: / Line 80: @@
 # '''Extract the final form score''' &rArr; see the [[Calculating Discrimination Index]] page.
 # '''Record the final test''' &rArr; use some software to extract your 15 sentences from the 50 or so pilot test sentences you started with to create a new master recording for the final test. At the start of this master recording you'll need to include three practice sentences which should have low DIs and low difficulty levels. Make the first practice sentence short and the next longer and the last longest. Make sure your last practice sentence is as long as your first test sentence.
-# '''Calibrate the test''' &rArr; You now have two sets of scores. From the [[#Evaluating L2 Speakers]] section earlier, you have proficiency level scores for your participants. You also have just calculated their scores for the 15 sentences you have selected. What you need to do for this step is to correlate these two sets of scores so that when you have a SRT score, you know what level of proficiency it relates to. The frist stage of this is to plot the participants' proficiency levels against their SRT pilot test scores for the 15 sentences in a scatter graph. Use y-axis for the proficiency level and the y-axis for the SRT scores. As you plot this, you'll be able to see what proficiency levels the scores indicate just by looking at the graph. But to improve the accuracy of the test, we can apply some statistical techniques to refine this even further. These three statistical techniques form the next three steps here...
+# '''Calibrate the test''' &rArr; You now have two sets of scores. From the [[Developing an SRT#Evaluating L2 Speakers|Evaluating L2 Speakers]] section above, you have proficiency level scores for your participants. You also have just calculated their scores for the 15 sentences you have selected. What you need to do for this step is to correlate these two sets of scores so that when you have a SRT score, you know what level of proficiency it relates to. The frist stage of this is to plot the participants' proficiency levels against their SRT pilot test scores for the 15 sentences in a scatter graph. Use y-axis for the proficiency level and the y-axis for the SRT scores. As you plot this, you'll be able to see what proficiency levels the scores indicate just by looking at the graph. But to improve the accuracy of the test, we can apply some statistical techniques to refine this even further. These three statistical techniques form the next three steps here...
 # '''Calculate the line of estimation''' &rArr; See the [[Statistical Refinement of an SRT]] page.
 # '''Calculate the standard error''' &rArr; See the [[Statistical Refinement of an SRT]] page.

@@ Line 42: / Line 42: @@
   '''Aim:''' start calibrating the test by testing the 50 or so speakers from the previous stage
 # '''Decide who does the testing''' &rArr; It is easier, particularly for relationships, if whoever has carried out the proficiency evaluation described above, also carries out the pilot testing of the SRT. For consistency, one person should score all the tests at this stage, and this person should be familiar enough with all the 40-50 preliminary test sentences to follow the transcripts at the same speed as the recording.
-# '''Get informed consent''' &rArr; Each person taking the preliminary test will need to understand the entire test process beforehand. This is the "informed" part of "informed consent." They have to know what they are consenting to. At this stage, they are consenting for their score data to be used only to help us construct a final test. It is recommended that you record the responses of a few people at each proficiency level and for about 10 random other participants. These recordings will be used to train the test administrators. The results and recordings of the preliminary test will not be used in our final reporting and not shared outside the research team. As you will have to explain this for each of the possibly 50 people you're working with, it's good to have this standardised to ensure consistency and that you don't leave anything out.
+# '''Get [[informed consent]]''' &rArr; Each person taking the preliminary test will need to understand the entire test process beforehand. This is the "informed" part of "informed consent." They have to know what they are consenting to. At this stage, they are consenting for their score data to be used only to help us construct a final test. It is recommended that you record the responses of a few people at each proficiency level and for about 10 random other participants. These recordings will be used to train the test administrators. The results and recordings of the preliminary test will not be used in our final reporting and not shared outside the research team. As you will have to explain this for each of the possibly 50 people you're working with, it's good to have this standardised to ensure consistency and that you don't leave anything out.
 # '''Test high-level speakers first''' &rArr; It's often hardest to design an SRT that adequately discriminates among the higher levels of proficiency. So, if possible, it's best to start doing preliminary testing with the people you identified as having the highest proficiency in the language. As you carry out the test, allow them space after each test sentence to repeat by pausing the recording. Try not to comment on the subject's performance, but do give encouragement that they are participating well. As their scores are calculated, plot them as described in step 5 below. If you can't find a difference between these high-level speakers, your test sentences don't have enough difficult examples in them. Get a night's sleep first, and then go back to elicit sample sentences which are more difficult running through all the steps above once more. If you still don't have a test which distinguishes them, it might be that you've chosen the wrong proficiency scale or assigned these speakers to the wrong levels. By this point, you'll have realised how important it was to be thorough at all the earlier stages of the test. Go away for a long weekend, and then start again.
 # '''Test all the speakers''' &rArr; If step 3 above shows you are good to go, test all the 50 or so participants you have remembering to identify about 10 random participants to record and to record a few from each proficiency level. As before, allow each participant space after each test sentence to repeat by pausing the recording. Try not to comment on the subject's performance, but do give encouragement that they are participating well.

@@ Line 86: / Line 86: @@
 # '''Control test the final test''' &rArr; to do this, simply administer the test with your 15 sentences to mother-tongue speakers of the test language. The group of participants chosen for this should come from a wide range of backgrounds and education levels so that you can prevent any variables such as these influencing the results. Obviously, as these are native speakers, you should be looking at test results approaching 100% with participants consistently achieving scores that place them in the highest proficiency level. Obviously, if you do not, you might need to look at the development of your test so far.
-The next phase of SRTs is to learn how to [[Administer an SRT|administer them]].
+The next phase of SRTs is to learn how to [[Administering an SRT|administer them]].
 ==References==
 <references/>

@@ Line 76: / Line 76: @@
 # '''Apply the discrimination index''' &rArr; We need to reduce the 50 or so sentences that we've been using for our preliminary test to the 15 we will use on the final test. To do this, we need some way of deciding which sentences are better than others for discriminating between levels. We use what Radloff (1991:55) calls the ''discrimination index''. Basically, the lower the value for the discrimination index, the better the sentence is for discriminating between levels. For the methodology needed to calculate discrimination indices for the sentences you are using in the preliminary test, see the [[Calculating Discrimination Index]] page.
-# '''Calculate difficulty levels''' &rArr;
+# '''Calculate difficulty levels''' &rArr; see the [[Calculating Discrimination Index]] page.
-# '''Select your final 15''' &rArr;;
+# '''Select your final 15''' &rArr;; see the [[Calculating Discrimination Index]] page.
-# '''Calculate the final score''' &rArr;
+# '''Extract the final form score''' &rArr; see the [[Calculating Discrimination Index]] page.
-# '''Record the final test''' &rArr;
+# '''Record the final test''' &rArr; use some software to extract your 15 sentences from the 50 or so pilot test sentences you started with to create a new master recording for the final test. At the start of this master recording you'll need to include three practice sentences which should have low DIs and low difficulty levels. Make the first practice sentence short and the next longer and the last longest. Make sure your last practice sentence is as long as your first test sentence.
 # '''Calibrate the test''' &rArr;
 # '''Calculate the line of regression''' &rArr;
 # '''Calculate the standard  error''' &rArr;

@@ Line 23: / Line 23: @@
   '''Aim:''' get a wide range of natural well-formed sample sentences (40-50) that range from simple to very difficult
-# '''Get informed consent''' &rArr: The people you work with at this stage will be providing spoken sentences that will ultimately end up being included in the final SRT. These sentences could very well then be played by to hundreds of people, often in a large geographic area that may even include more than one country. It is vital that, right at the start, the people who provide these sentences agree to their speech being used in this way. Make sure you get consent in a way that is culturally appropriate to the people you're working with. See section 5.1.9 in Decker & Grummitt (forthcoming) <ref>Decker, Ken and John Grummitt. (forthcoming). ''Understanding Language Choices: A guide to sociolinguistic assessment''. Dallas: Summer Institute of Linguistics.</ref> for detailed discussion of this issue.
+# '''Get informed consent''' &rArr; The people you work with at this stage will be providing spoken sentences that will ultimately end up being included in the final SRT. These sentences could very well then be played by to hundreds of people, often in a large geographic area that may even include more than one country. It is vital that, right at the start, the people who provide these sentences agree to their speech being used in this way. Make sure you get consent in a way that is culturally appropriate to the people you're working with. See section 5.1.9 in Decker & Grummitt (forthcoming) <ref>Decker, Ken and John Grummitt. (forthcoming). ''Understanding Language Choices: A guide to sociolinguistic assessment''. Dallas: Summer Institute of Linguistics.</ref> for detailed discussion of this issue.
 # '''Collect sample texts''' &rArr; In order to achieve this, you'll have to elicit or find a sample text that has a wide range of language in it. This could be spoken or written, but if it's written, it needs to be natural language. You could ask your second language speaker to respond to a topic you come up with or give a description of something. Or, you could actually ask them to form sentences on a topic which will include specific grammatical constructions of increasing complexity.
 # '''Extract a range of sentences''' &rArr; Once you have your text/s, you'll need to go through it and extract a range of sentences (not questions!) from simple through medium to difficult. There should be 60-70 of these. Difficulty could be a matter of grammatical complexity or it could also be other features such as the level of formality. In many languages, these are related. One tip that Radloff gives is to select some sentences that begin with [[Discourse Markers|discourse markers]] as these tend to be more challenging. Make sure you have a wide variety of content/topics. If you don't, participants may be able to provide answers which are based on previous sentences they've heard.