1 SPEECH AND SPEAKER RECOGNITION EVALUATION&Sadaoki Furui 1
1．Introduction 1
2．Principles of Speech Recognition 2
3．Categories of Speech Recognition Tasks 3
4．Evaluation of Speech Recognition Systems 6
5．Principles of Speaker Recognition 12
6．Categories of Speaker Recognition Tasks 13
7．Normalization and Adaptation Techniques 15
8．Evaluation of Speaker Recognition Systems 17
9．Factors Affecting the Performance and Evaluation Paradigm Design for Speech and Speaker Recognition Systems 22
10．System-Level Evaluation of Speech and Speaker Rognition 23
11．Conclusion 24
References 24
2 EVALUATION OF SPEECH SYNTHESIS&Nick Campbell 29
1．Introduction 29
2．Components of Computer Speech 30
3．Evaluation Methodologies 36
4．Organised Evaluations and Assessment 41
5．Speaking to (and on Behalf of) People 45
6．Conclusion 45
References 48
3 MODELLING AND EVALUATING VERBAL AND NON-VERBAL COMMUNICATION IN TALKING ANIMATED INTERFACE AGENTS&Bj？rn Granstr？m and David House 65
1．Introduction 65
2．KTH Parametric Multimodal Speech Synthesis 67
3．Data Collection and Data-Driven Visual Synthesis 69
4．Evaluating Intelligibility and Information Presentation 71
5．Evaluating Visual Cues for Prominence 77
6．Evaluating Prosody and Interaction 80
7．Evaluating Visual Cues to Sentence Mode 84
8．Evaluation of Agent Expressiveness and Attitude 85
9．Agent and System Evaluation Studies 87
10．Future Challenges in Modelling and Evaluation 91
References 92
4 EVALUATING PART-OF-SPEECH TAGGING AND PARSING&Patrick Paroubek 99
1．POS Tagging 99
2．Parsing 102
3．Evaluation and Natural Language Processing 105
4．POS Tagging Evaluation Methodology 110
5．Methodology and Evaluation Measures for Parsing 114
6．Conclusion 117
References 118
5 GENERAL PRINCIPLES OF USER-ORIENTED EVALUATION&Margaret King 125
1．A Historical Note 126
2．What is User-Oriented Evaluation? 128
3．A First Principle: Quality is Decided by Users 129
4．A Second Principle: Users do not Have the Same Needs 130
5．A Third Principle: Quality can be Characterized 135
6．A Fourth Principle: Quality can be Measured 148
7．Combining the Particular and the General: The Ideal 153
8．Conclusion 154
References 156
6 AN OVERVIEW OF EVALUATION METHODS IN TREC AD HOC INFO-RMATION RETRIEVAL AND TREC QUESTION ANSWERING&Simone Teufel 163
1．Introduction 163
2．Evaluation Criteria 169
3．Evaluation Metrics 172
4．Real-World Performance 182
5．Conclusion 183
References 185
7 SPOKEN DIALOGUE SYSTEMS EVALUATION&Niels Ole Bernsen,Laila Dybkj？r and Wolfgang Minker 187
1．Introduction 187
2．Evaluation Methods and Criteria 188
3．Evaluation of the NICE Hans Christian Andersen Prototype 193
4．Evaluation of the SENECA Prototype 207
5．Conclusion 216
References 218
8 LINGUISTIC RESOURCES,DEVELOPMENT,AND EVALUATION OF TEXT AND SPEECH SYSTEMS&Christopher Cieri 221
1．Introduction 222
2．The Linguistic Resource Landscape 222
3．Background on Linguistic Data and Annotation 234
4．Data Planning for Technology Development and Evaluation 237
5．Finding Resources 240
6．Building Resources 242
7．Conclusion 259
References 260
9 TOWARDS INTERNATIONAL STANDARDS FOR LANGUAGE RESOURCES&Nancy Ide and Laurent Romary 263
1．Introduction 263
2．Background 265
3．The Linguistic Annotation Framework 268
4．Putting it All Together 278
5．Conclusion 282
References 283
Index 285