（外国语言学及应用语言学专业论文）英语口语测试问话者培训问题研究.pdf

v 中 文 摘 要 英语口语测试考官（评分员及问话者）的培训是评分过程中保证考官内部一致 性、考官之间一致性以及跨时间一致性的重要因素。然而，较之相对全面的评分员 培训方面的研究，问话者培训一直以来未能引起语言测试研究者的足够重视。近些 年来，一些研究者开始关注问话者差异性对于考生表现的影响，因为会话的语义共 建使得考生表现在一定程度上有赖于考官表现，但是问话者培训的具体工作则常被 忽略。在已发表的关于问话者差异性的较少研究当中，多数倾向于分析问话者变量 对考生表现的影响，而很少有研究使用问话者培训前后对比的实验设计来考察培训 效果。 本研究目的在于首先选出现实情况中确实会影响考生表现的问话者变量，并针 对这些因素试图提出一份适当的问话者培训材料。其次，采用培训前后实验对比研 究的实验设计方案，对该培训材料的有用性进行实证研究。 本研究实验设计部分主要通过定性研究的手段和描述性统计分析的方法进行研 究。首先，通过问卷和后续访问的研究方法调查了问话者影响考生表现的积极与消 极因素。根据问卷调查结果集中反映的问题，提出了一份问话者培训材料。然后， 采用 4 名问话者和 4 名评分员对 16 名考生分别进行了两次面试。在两次面试当中， 每位考生与相同问话者交流并由同一组培训过的评分员进行打分。两次面试过程都 被录音，并由评分组成员分别评分。但是，在第二次测试中问话者经过了专门培训， 这样问话者本人、评分员小组和考生都可以就两次面试中问话者的表现进行比较并 做出评价，从而了解问话者培训是否有效。 研究结果有以下几个方面：问卷的分析结果反映出了对于考生表现最具影响力 的问话者因素，这有助于提出适当的问话者培训材料。同时，结果显示，尽管当前 对于参与交谈的考官培训情况不甚乐观，但是通过培训一定能够使得问话者表现更 佳。此外，通过对比和分析两次考试中所收集到的 16 个面试过程的资料和数据，可 以看到培训后的问话者在问话难度和对考生的支持程度方面更加标准化。结果表明， 有两名问话者在培训前的面试中未表现出对考生的足够支持，但培训后这一情况在 一定程度上有所改善，这两名问话者表现出了问话难度有所下降，对考生支持程度 有所提高的特征。 口试旨在评估考生语言能力，可以采取某些方法来保证所有考生在面试时拥有 公平的机会来充分发挥，并且在问话者的会话难度和对考生的支持程度方面得到同 等对待。本研究关注问话者培训的改进问题，并涉及在对评分员的培训过程中提高 其对问话者影响的理解。这不仅有助于引出更加真实的考生口语样本，更有助于提 高口试的评分的信度和效度。 关键词：关键词：口试；问话者；评分员；培训 iii abstract training of interviewers, either on raters or interlocutors, is considered an essential component of the rating process to ensure the consistency between them and within themselves, as well as over time. however, interlocutor training has long escaped the attention of language researchers, compared to the much more comprehensive research into rater training. in recent years, some researchers have focused their work on the effect of interlocutor variation on candidate performance, because the co-construction of discourse makes candidate performance partly depend upon interlocutor performance, but the phenomenon of interlocutor training itself has usually been neglected in research. among the few published studies of interlocutor variation, most have tended to analyze the interlocutor variables affecting test takers performance, but few have used a pre/post design of interlocutor training effects. this study aims first to identify some interlocutor variables that in fact will influence candidate performance in our situation, and to develop an interlocutor training material by providing some useful suggestions according to those variables. the second part of study is to empirically investigate the effect of the training material by applying a pre/post design to test its usefulness. the present study was carried out with a combination of qualitative data and descriptive statistics. the first part of study is based on a questionnaire and follow-up interviews to investigate the opinions on factors introduced by interlocutor that could positively or negatively influence candidate performance, and developed an interlocutor training material with reference to the results of the questionnaire and interviews. the second part is on interviews with 16 candidates, each of whom was interviewed twice by the same interlocutor and the same rater group. four interlocutors and four raters were involved. the interviews were audio-taped and multiple-rated. the interlocutors were trained using the obtained training material before the second test, which allowed themselves, rater group and candidates to make comparison of the interlocutor performance in the two tests. the major findings are as follows: the results of questionnaire identified the most influential interlocutor factors on candidate performance, which shed light on the development of interlocutor training material. it was also found that despite the result of the current training session for examiners who took the role of interlocutor was not satisfactory, still it was confidently believed that interlocutors would perform better in oral tests through training. the test data were studied, which reflects that, through training with the obtained training material, interlocutors were more standardized in terms of difficulty and support presented to candidates. a total of 16 interviews from the two tests were analyzed and compared. results indicate that before training, two interlocutors in this study did not take on the responsibilities of being a supportive interlocutor; after training, this situation seemed to be improved to some extent. these two interlocutors became less difficult and more supportive. while the intent of the oral test has been to access candidates language ability, there must be some concern to ensure that all candidates have the same chance to perform at their best, and they were treated equally with similar difficulty and support presented by interlocutor. the present study is of relevance to the improvement of interlocutor training and the training of raters in terms of increasing their understanding of interlocutor effects, which will not only lead to more authentic candidate performance to assess, but also will contribute to the reliability of ratings and the validity of oral proficiency assessment itself. key words: speaking test; interlocutor; rater; training list of figures and tables figure 2.1 a socio-cognitive framework for validating speaking tests.7 figure 2.2 mode one of scoring direct speaking tests.14 figure 2.3 mode two of scoring direct speaking tests.15 figure 2.4 mode three of scoring direct speaking tests.15 table 3.1 framework of the questionnaire .28 table 3.2 descriptive statistics of opinions on interlocutor training (n=19).29 table 3.3 descriptive statistics of opinions on interlocutor speech (n=19).31 table 3.4 descriptive statistics of opinions on interlocutor behaviour (n=19).33 table 3.5 descriptive statistics of opinions on interlocutors conversational features (n=19).35 table 4.1 interlocutors for this study: characteristics.41 table 4.2 raters for this study: shared characteristics.42 figure 4.1 the adopted form of interview.43 table 4.3 overall framework of research design.44 table 4.4 comparison between the scores of interactiveness.48 table 4.5 frequency of interviews according to interlocutors perceived level of difficulty (pre) (n=32) .48 table 4.6 frequency of interviews according to interlocutors perceived level of difficulty (post) (n=32).49 table 4.7 frequency of interviews according to interlocutors perceived level of supportiveness (pre) (n=32) .50 table 4.8 frequency of interviews according to interlocutors perceived level of supportiveness (post) (n=32).50 chapter 1 introduction 1 chapter 1 introduction 1.1 brief description speaking tests, involving other language skills such as listening and reading in the testing process, are commonly expected to have the characteristics of relatively high validity, authenticity and interactiveness. in this view, testing the ability to speak becomes a most important aspect of language testing (heaton, 1988: 88). here we are concerned with the task type of oral interviews. due to the distinctive features of nontest or conversational interaction, they are claimed to allow second language learners to demonstrate their capacity to “interact in an authentic communicative event utilizing different components of communicative competence extemporaneously” (ross, 1996: 3-4). consequently, oral interviews are widely used in many large-scale high-stakes english examinations home and aboard, as well as in small-scale ones. with such assumed strengths as relatively high validity, speaking tests face the threat of reliability through the rating process. despite of the advantages of oral interviews over other types of exams, still they are found to have many problems in scoring. firstly, recording of speech samples are not usually made, whereas speech is a fast-fading medium, so the assessments of speech sample have to be made either during or shortly after the test (weir, 2005: 192). unable to be freed from real-time constraints, speech assessment is inherently labor intensive and expensive. secondly, test takers answers to a large extent are open-ended so that testers assessment may not be comparable from one to another. that is, marking in oral tests is in a very subjective fashion. thirdly, since the individual responses are likely to include content or features not directly relevant to particular scale definitions, scoring may be influenced by many factors, including the ones coming from examiners, other than the oral proficiency presented. rater behaviour may differ significantly, and interlocutor variation should also be taken into account. all these factors combine to make scores assigned to oral responses potentially problematic. training of examiners, either on raters or interlocutors, is considered an essential component of the rating process to ensure the consistency between them and within a tentative study on interlocutor training in esl speaking tests 2 themselves, as well as over time. jacob et al. (1981: 43) argue that training, along with the use of a scoring rubric, helps to neutralize the differences in judges caused by their different backgrounds, and to ensure more consistent interpretation and application of the criteria and standards for determining the communicative effectiveness. 1.2 motivation factors of the study owing to the characteristics of oral activities and the subjectivity of scoring, speaking faces many challenges and difficulties. in recent years, many testing researchers have focused their work on the scoring variation. one of the important sources that influence scores rewarded to candidate and its interpretation is interlocutor effect. it is an increasingly recognized fact that discourse is co-constructed, so the performance of the test taker is partly dependent upon the performance of the interlocutor. interlocutors are expected to help test takers to demonstrate their capacity to interact and perform at their best to show their language abilities. research results show that interlocutor behaviour and speech vary in many ways, which brings about the question of how to ensure that interlocutor support during the interaction is consistently offered. an effective solution to this problem can be that test developers should provide interlocutors with rigorous and adequate training. the present study deals with the issue of the effects of interlocutor training in the context of an inland province. the practical considerations are important considering the fact that, the current situation of interlocutor training is far from satisfactory, still there is the question remains to be solved that how training actually works to bring interlocutors into agreement with each other. research on interlocutor training is crucial in assessing oral proficiency, as the reasons are summarized in the following aspects: it is presumed to be able 1) to clarify the question that how much should the interlocutor contribute to the interaction; 2) to minimize differences as a result of the different support and difficulty presented to candidates in conversation; and 3) to elicit ratable speech samples that can reflect the authentic language abilities of candidates. 1.3 the need for study due to the rapid development of oral language proficiency tests in china, it is of chapter 1 introduction 3 great necessity to investigate into all aspects of oral tests, one of which is the training of all staff who administer tests. the most discussed area of test administration is probably the training of raters for speaking tests and writing tests. rater training has a long history. it has been acknowledged since the beginning of what used to be called “subjective scored” tests that it should be a matter of indifference to a test taker who scores the performance (fulture and davinson, 2007: 131). however, the training of interlocutors is a much more recent phenomenon. the reason is that it has merely been assumed that interlocutors could engage test takers in conversation using the input associated with the tasks, and score the resulting performance of the test taker. besides, many examination boards do not have appropriate procedures in place for training and monitoring interlocutors. hence, the training of interlocutors has just become a concern of some test researchers. interest is now growing in interlocutor effects. recent research in this area of language testing has highlighted the importance of interlocutor variables, especially the effect of interlocutor variation on their backgrounds, behaviours and language styles and accommodations to the interaction and so on (mcnamara and lumley, 1993, 1997; morton et al., 1995, 1997; brown and lumley, 1997; lazaraton, 1996a; ross, 1992, 1996 and brown, 2003). their findings suggested that these could account for variability of interlocutor in test performance. although the effects of interlocutor variation have been discussed in the literature, the phenomenon of interlocutor training has usually been neglected in research. among the few published studies of interlocutor variation, most of them have tended to analyze the interlocutor variables affecting test takers performance, but few have used a pre/post design of interlocutor training effects. in addition, although interlocutor training has been much discussed in the literature on speaking assessment, very little information exists about how training actually works to bring interlocutors into agreement with each other. that is, interlocutor training effects remain little explored and researchers paid little attention to it, especially the pre/post study of the influence of interlocutor training, despite recent calls in the literature for studying the process of interlocutor training (weir, 2005; fulture and davidson, 2007). based on the knowledge of language testing validation in theory and in practice, the a tentative study on interlocutor training in esl speaking tests 4 present thesis intends to focus the research on interlocutor training effects. 1.4 aim of the research this thesis attempts to propose a more specific interlocutor training material with an aim to provide some suggestions to further improve the current interviewer training session, which can contribute to both validity and reliability of speaking tests. to be more specific, the aim of this study is threefold: to identify some factors of interlocutor variables that in fact will influence candidate performance in our situation; to develop an interlocutor training material by providing some useful suggestions when training interlocutors; and to investigate the effect of the training material to test its usefulness. it is hoped that the findings of this study will contribute to the understandings beginning to emerge from other research into interviewer training effects, and inform the future development of interviewer training. specifically, the research questions for the study are the following: 1) what factors that an interlocutor may influence the performance of a test taker should be included in interlocutor training material in the tests for esl learners in china? 2) through training, can interlocutors be more standardized in terms of the challenge and support presented to different candidates? 1.5 significance of the study the present study has both the theoretical and practical significance. the theoretical significance lies in that, as mentioned in 1.3, there has been little research on the effect of interlocutor training, and this study will contribute to knowledge in this area. the practical significance is that the current situation of interlocutor training is far from satisfactory. it has not aroused enough concern to monitor interlocutor performance and to standardize the challenge presented to candidates. a specific interlocutor training material is needed to address this problem. therefore, the training material obtained in this study might have implications for further development. 1.6 outline of the paper this paper consists of 5 chapters: chapter 1 presents a general picture of the present study by providing the motivation, chapter 1 introduction 5 need, aim, significance and organization of this study. chapter 2 provides a detailed review of the related literature, including the reliability and validity, scoring method, rater effects and interlocutor effects in speaking tests. chapter 3 describes the development of the interlocutor training material. chapter 4 depicts the empirical study of the obtained training material, including the data collection and the analysis of the results in the study. chapter 5 presents the researchers conclusion of this study regarding the major findings, limitations and implications of the present study, and also suggestions for further research. a tentative study on interlocutor training in esl speaking tests 6 chapter 2 literature review this chapter prov