Chapter 1 Introduction 1
1.1 Background of the study 1
1.2 Purpose of the study and research questions 2
1.3 Significance of the study 3
1.3.1 Theoretical significance 3
1.3.2 Methodological significance 4
1.3.3 Practical significance 4
1.4 A note on terms 6
1.4.1 Mock essays 6
1.4.2 Rater effects/variability/biases/errors 6
1.4.3 Rating accuracy 6
1.4.4 Scoring/score validity 6
1.4.5 Scoring/rating expertise 7
1.4.6 Cognitive strategy 7
1.4.7 Meta-cognitive strategy 8
1.4.8 Rating session 8
1.4.9 Feedback iteration 8
1.4.1 0 Counter-balanced design 8
1.4.1 1 The correlation coefficient 8
1.4.1 2 The mixed feedback 8
1.5 The structure of chapters 9
Chapter 2 Literature review 10
2.1 Introduction 10
2.2 Rater variability as a serious threat to score validity in performance assessments 10
2.2.1 Models of performance assessment of writing 11
2.2.2 Factors that affect rater performance 14
2.2.3 Rater variability from the psychometric perspective 17
2.2.4 Rater variability from the hermeneutic perspective 22
2.2.5 Rater cognition studies 23
2.3 Rater training as a routine measure to control rater variability 38
2.3.1 Overview of procedures of rater training 39
2.3.2 Studies on rater training effectiveness 41
2.3.3 Critical issues in rater training 43
2.4 Rater feedback as an important part of training 45
2.4.1 Studies on rater feedback effectiveness 46
2.4.2 Some common problems 51
2.5 Methodological considerations 52
2.5.1 Content of the feedback 52
2.5.2 Delivery of the feedback 53
2.6 Summary 53
Chapter 3 Method 54
3.1 Introduction 54
3.2 Research Design 55
3.2.1 The mixed methods design of this study 55
3.2.2 The quasi-experimental nature of this study 56
3.2.3 Context of the study 56
3.2.4 Materials 62
3.2.5 Participants 66
3.2.6 Instruments 70
3.2.7 Pilot study 72
3.2.8 Procedures 78
3.3 Data analysis 80
3.3.1 MFRM analysis of ratings 80
3.3.2 Analysis of degree of certainty 84
3.3.3 Verbal protocol analysis 84
3.3.4 The questionnaire of raters'perceptions of mixed feedback 86
3.3.5 The questionnaire of raters'evaluation of feedback effectiveness 86
3.3.6 Interview with those who are negative with the feedback effectiveness 86
3.4 Summary 86
Chapter 4 Results 88
4.1 Findings for Research Question 1:The effect of the mixed feedback on rater variability 89
4.1.1 Findings for RQ1（1）:The effect of the mixed feedback on rater severity 89
4.1.2 Findings for RQ1（2）:The effect of the mixed feedback on rater internal consistency 97
4.1.3 Findings for RQ1（3）:The effect of the mixed feedback on rater bias 101
4.1.4 Findings for RQ1（4）:The effect of the mixed feedback on rater central tendency 104
4.1.5 Findings for RQ1（5）:The effect of the mixed feedback on rater severity DRIFT 106
4.2 Findings for Research Question 2:The extent to which EG raters can incorporate the mixed feedback into rating behavior when it is provided repeatedly over 3 iterations 109
4.2.1 First round results 110
4.2.2 Second round results 111
4.3 Findings for Research Question 3:The effect of the mixed feedback on raters'degree of certainty for their ratings 112
4.3.1 First round results 112
4.3.2 Second round results 114
4.4 Findings for Research Question 4:The effect of the mixed feedback on raters'deci-sion-making 115
4.4.1 Findings for RQ4（1）:The effect of the mixed feedback on raters'decision-making in terms of text features 116
4.4.2 Findings for RQ4（2）:The effect of the mixed feedback on raters'decision-making in terms of information processing behaviors 129
4.5 Findings for Research Question 5:Raters'perceptions of the mixed feedback 139
4.5.1 First round results 139
4.5.2 Second round results 145
4.6 Summary 151
Chapter 5 Discussion 153
5.1 Discussion of Research Question 1:What effect does the mixed feedback have on rater variability? 153
5.1.1 Methodological considerations 153
5.1.2 Interpretation of the results 153
5.2 Discussion of Research Question 2:To what extent can EG raters incorporate the mixed feedback into rating behavior when it is provided repeatedly over 3 iterations? 158
5.2.1 Methodological considerations 158
5.2.2 Interpretation of the results 158
5.3 Discussion of Research Question 3:What effect does the mixed feedback have on raters'degree of certainty for their ratings? 160
5.3.1 Methodological considerations 160
5.3.2 Interpretation of the results 160
5.4 Discussion of Research Question 4:What effect does the mixed feedback have on raters'decision-making? 162
5.4.1 Methodological considerations 162
5.4.2 Interpretation of the results 163
5.5 Discussion of Research Question 5:What are raters'perceptions of such feedback? 170
5.5.1 Methodological considerations 170
5.5.2 Interpretation of the results 170
5.6 General discussion:Establish an internal link between findings 173
5.6.1 Linking findings of RQ1,RQ3 and RQ4 173
5.6.2 Linking findings of RQ1,RQ3,RQ4 and RQ5 176
5.7 Summary 181
Chapter 6 Conclusions 182
6.1 Summary of findings 182
6.2 Implications of the study 183
6.2.1 Theoretical implications 183
6.2.2 Methodological implications 184
6.2.3 Practical implications 184
6.3 Limitations of the study and direction for future research 185
References 189
Appendices 203