中国高校英语专业写作测试效度检验研究PDF电子书下载

INTRODUCTION 1

PART Ⅰ LITERATURE REVIEW 5

CHAPTER 1 LITER ATURE REVIEW I—LANGUAGE TESTING AND VALIDITY EVIDENCE 7

1.0 Introduclion 7

1.1 A Short History of Language Testing 7

1.2 Validity and Validation 9

1.3 Aspects of Validity Evidence 12

1.3.1 Theory-based Validity 13

1.3.2 Context Validity 15

1.3.3 Scoring Validity 17

1.3.4 Consequential Validity 18

1.3.5 Criterion-related Validity 21

1.4 Summary 22

CHAPTER 2 LLTERATURE REVIEW Ⅱ—PARAMETERS FOR A VALIDATION FRAMEWORK FOR AWT 24

2.0 Introduction 24

2.1 Test-taker Characteristics 24

2.2 Theory-based and Context Validity Evidence 39

2.2.1 The Definition of Academic Writing 39

2.2.2 The Cognitive Process Models in Writing 41

2.2.3 A Socio-cognitive Model of Writing:Process and Context 59

2.2.4 Text Types and Text Construction in Academic Writing 64

2.2.5 A Summary of Theoretical Models of Academic Writing 74

2.2.6 The Specific Parameters of Theory-based Validity and Context Validity 78

2.3 The Parameters of Sconng Validity Evidence 89

2.4 The Parameters of Consequential Validity Evidence 91

2.5 The Parameters of Criterion-related Validity Evidence 93

2.6 A Summary Prototype Framework for Validating Academic Writing Tests 95

2.7 The Situation of Academic Writing Tests at Tertiary Level in China 97

2.7.1 General Requirements for Academic Writing 97

2.7.2 Specifications for Academic Writing in TEM4 and TEM8 98

2.7.3 Additional Syllabus Requirements for the University Graduation Dissertation 98

2.8 General Research Question and Research Sub-questions 99

2.9 Implications of the Research 100

2.10 Summary 100

PART Ⅱ PILOT STUDIES 103

CHAPTER 3 THE DEVELOPMENTAND PILOTING OF INSTRUMENTS AND PROCEDURES FOR APPLYING THE VALIDATION FRAMEWORK TO TEM8E AND GRADUATION DISSERTATIONS 105

3.0 Introduction 105

3.1 Pilot Study One:The Development and Piloting of the Instrumentsand Procedures on Test-taker Characteristics 106

3.1.1 Sampling Participants 106

3.1.2 Establishment of Variables in Test-taker Characteristics 108

3.1.3 Development and Piloting of Instruments 108

3.2 Pilot Study Two:The Development and Piloting of the Instruments and Procedures for the Theory-based Validation of TEM8E and the Dissertation 122

3.2.1 Participants 122

3.2.2 Establishment of Variables in Theory-based Validity Evidence 122

3.2.3 Development and Piloting of Instruments 123

3.3 Pilot Study Three:The Development and Piloting of the Instruments and Procedures for the Context Validation of TEM8E and the Dissertation 153

3.3.1 Participants 154

3.3.2 Establishment of Variables in Context Validity Evidence 154

3.3.3 Development and Piloting of Instruments 155

3.4 Pilot Study Four:The Development and Piloting of the lnstruments and Procedures for the Scoring Validation of TEM8E and the Dissertation 159

3.4.1 Participants 160

3.4.2 Establishment of Variables in Scoring Validity Evidence 160

3.4.3 Development and Piloting of Instruments 160

3.5 Pilot Study Five:The Development and Piloting of the lnstruments and Procedures for the Consequential Validation of TEM8E and the Dissertation 162

3.5.1 Participants 162

3.5.2 Establishment of Variables in Consequential Validity Evidence 162

3.5.3 Development and Piloting of Instruments 163

3.6 Pilot Study Six:The Development and Piloting of the Instruments and Procedures for the Criterion-related Validation of TEM8E and the Dissertation 166

3.7 Summary 168

PART Ⅲ MAIN STUDY 173

CHAPTER 4 DISCUSSION OF THE RESULTS CONCERNED WITH THE EVIFENCE OF TEST-TAKER CHARACTERISTICS 177

4.0 Introduction 177

4.1 Sampling Participants 177

4.2 Discussion of the Results from Questionnaire A1(QA1) 182

4.2.1 Discussion of Learner Background Variables 183

4.2.2 Discussion of Motivation 201

4.2.3 Discussion of Orientation to the Language Study Context 204

4.2.4 Discussion of Learning Strategies 210

4.2.5 Discussion of Learning Approaches 213

4.3 Discussion of the Results Obtained from Questionnaire A2(QA2) 216

4.3.1 Discussion of Academic Writing Leaming Strategies 216

4.3.2 Discussion of Use/Process Strategies in Academic Writing 220

4.4 Summary 224

CHAPTER 5 DISCUSSION OF THE RESULTS CONCERNED WITH THEORY-BASED VALIDITY EVIDENCE 228

5.0 Introduction 228

5.1 Participants 229

5.2 Discussion of the Results from Questionnaire B1(QB1) 229

5.2.1 Statistical Analysis of the Results from Factor Analysis of QB1 230

5.2.2 Quantitative Analysis of the Results from Regression and ANOVA on QB1 234

5.2.3 Brief Qualitative Analysis of Academic Writing Process Feedback from QB1 236

5.2.4 Discussion of the Results from SEM on QB1 and QA2 239

5.3 Discussion of the Results from the Think Aloud Protocol Experiment 248

5.4 Discussion of the Results from Questionnaire B2(QB2) 256

5.4.1 Discussion of the Results from Factor Analysis of QB2 257

5.4.2 Discussion of the Comparison of the Responses to QB1 with those to QB2 260

5.5 Discussion of the Results from Focus Groups(FG) 264

5.5.1 Participants and Administration of FG 264

5.5.2 Discussion of the Feedback from FG 265

5.6 Discussion of the Feedback from Diary B 272

5.6.1 Participants and Administration of Diary B 272

5.6.2 Discussion of the Feedback from the Diary 273

5.7 Summary 276

CHAPTER 6 DISCUSSION OF THE RESULTS CONCERNED WITH CONTEXT VALIDITY EVIDENCE 284

6.0 Introduction 284

6.1 Participants 284

6.2 Discussion of the Results from Questionnaire C1a (QC1a) 285

6.3 Discussion of the Results from Questionnaire C1b(QC1b) 288

6.4 Discussion of the Results from Questionnaire C2a(QC2a) 289

6.5 Discussion of the Results from Questionnaire C2b (QC2b) 291

6.6 Discussion of the Results from Interview C1 293

6.7 Discussion of the Results from Interview C2 293

6.8 Summary 294

CHAPTER 7 DISCUSSION OF THE RESULTS CONCERNED WITH SCORING VALIDITY-,CONSEQUENTIAL VALIDITY-,AND CRITERION-RELATED VALIDITY EVIDENCE 298

7.0 Introduction 298

7.1 Scoring Validity Evidence 299

7.1.1 Participants 299

7.1.2 Discussion of the Results from Questionnaire D1(QD1) 299

7.1.3 Discussion of the Results from Questionnaire D2(QD2) 300

7.1.4 Summary of 7.1 302

7.2 Consequential Validity Evidence 303

7.2.1 Participants 304

7.2.2 Discussion of the Results from Questionnaire E1a(QE1a) 304

7.2.3 Discussion of the Results from Questionnaire E1b(QE1b) 307

7.2.4 Discussion of the Results from Questionnaire E2a(QE2a) 309

7.2.5 Discussion of the Results from Questionnaire E2b(QE2b) 312

7.2.6 Summary of 7.2 314

7.3 Criterion-related Validity Evidence 316

7.3.1 Participants 316

7.3.2 Discussion of the Results from the Correlations of the two Assessments 317

7.4 Summary of Chapter 7 319

CHAPTER 8 CONCLUSIONS AND IMPLICATIONS 323

8.0 Introduction 323

8.1 Research Questions Revisited 323

8.1.1 Test-taker Characteristics 324

8.1.2 Theory-based Validity 326

8.1.3 Context Validity 328

8.1.4 Scoring Validity 330

8.1.5 Consequential Validity 330

8.1.6 Criterion-related Validity 331

8.2 Limitations of the Study 332

8.3 Implications 333

8.3.1 Implications for Test Developers in General 334

8.3.2 Implications for Exit Assessment at Universities 334

8.3.3 Implications for University Students 335

8.4 Future Research Directions 335

8.5 Concluding Remarks 336

REFERENCES 337

APPENDICES 358

List of Tables 2

CHAPTER 2

Table 2.1 Definitions of learning strategies 33

CHAPTER 3

Table 3.1 Some demographical characteristics of the seniors at YNU in the pilot study 107

Table 3.2 The main demographical characteristics of 11 participants in TA experiment 127

Table 3.3 Transcription conventions (adapted from Bracewell and Breuleux 1994:59) 128

Table 3.4 Think-aloud essay marking scheme 142

Table 3.5 Cronbach alpha coefficients for marking essays in piloting TA 143

Table 3.6 Frequency of score distribution of theessay in piloting TA 144

Table 3.7 Pearson correlations between TEM8E and the dissertation at YNU in 2003 166

Table 3.8 Spearman rho between the ranks of TEM8E and the dissertation at YNU in 2003 167

Table 3.9.1 Instruments developed and piloted for test-taker characteristics 168

Table 3.9.2 Instruments developed and piloted for theory-based validity 169

Table 3.9.3 Instnunents developed and piloted for context validity 170

Table 3.9.4 Instruments and/or procedures developed and/or piloted forscoring,consequential and criterion-related validity 171

CHAPTER 4

Table 4.1 The demographic overview of student-participants in the main study(QA1.1.02,03,04,05,11) 178

Table 4.2 Data collection list in the Main Study on test-taker characteristics 225

CHAPTER 5

Tables 5.1-31 appear inAppendix 5. 1

Table 5.32 Data collection list in the Main Study on theory-based validity evidence 277

CHAPTER 6

Tables 6.1-4 appear in Appendices 6.1-6. 4

Table 6.5 Data collection list in the Main Study on context validity evidence 294

CHAPTER 7

Tables 7.1-11 appear in Appendices 7.1-7. 11

Table 7.12 Data collection list in the Main Study on scoring-,consequential-,and criterion-related validity evidence 320

List of Figures 2

CHAPTER 2

Figure 2.1 The external characteristics of test-takers (O'Sullivan 2000;O'Sullivan and Weer 2002) 27

Figure 2.2 Structure of the writing model(Flower and Hayes 1981:370) 42

Figure 2.3 Structure of the knowledge-telling model(Bereiter and Scardamalia 1987:8) 50

Figure 2.4 Structure of the knowledge-transforming model(Bereiter and Scardamalia 1987:12) 52

Figure 2.5 Model of writing as communicative languageuse(Grabe and Kaplan 1996:226) 61

Figure 2.6 Elements of text structure(Grabe and Kaplan 1996:63) 70

Figure 2.7 A model of text construction(Grabe and Kaplan 1996:81) 72

Figure 2.8 A summary model of academic writing 78

Figure 2.9 Theory-based validity and context validity(adapted from Weir 2005:47) 79

Figure 2.10 Components of language knowledge(adapted from Bachman 1990:87;Bachman and Palmer 1996:68) 81

Figure 2.11 Sub-framework of scoring validity(Weir 2005:47) 90

Figure 2.12 Sub-framework of consequential validity(adapted from Weir 2005:47) 92

Figure 2.13 Sub-framework of criterion-related validity(Weir 2005:47) 94

Figure 2.14 A socio-cognitive framework for validating writing tests(adapted from Weir 2005:47) 96

CHAPTER 3

Figure 3.1 Q-Q Plot of essay scores from 11 participants in the TApilot 144

CHAPTER 4

Figure 4.1 Distribution of participants:Gender by University(QA1.1.01) 180

Figure 4.2 Distribution of participants:Ageby University(QA1.1.03) 180

Figure 4.3 Distribution of participants:Region by University(QA1.1.04) 181

Figure 4.4 Distribution of participants:Type of secondary school by University(QA1.1.05) 181

Figure 4.5 Distribution of participants:Beginning time of studying English by University(QA1.1.11) 182

Figure 4.6 Linear and curvilinear regression on QA1.3.08 'If we had no tests,I think I would actually learn more'in the variable of test-related anxiety in orientation 206

Figure 4.7 Linear and curvilinear regression on QA1.3.09 'During importanttests,I often feel so nervous that I often forget what Ireally know'in the variable of test-related anxiety in orientation 206

Figure4.8 Linear and curvilinear regression on QA1.3.10'Even when I am well prepared for a test,I feel very worried aboutit'in the variable of test-related anxiety in orientation 207

CHAPTER 5

Figure 5.1 Scree plot for component number on QB1 in Factor analysis 231

Figure 5.2 The model for causal relationships between academic writing strategies,sub-processes and score gains 243

Figure 5.3 Scree plot for component number on QB2 in Factor analysis 258