（英语语言文学专业论文）A+Case+Study+on+the+Reliability+and+Validity+of+the+ComputerAssisted+College+English+Oral+Test.pdf

i 摘 要 大学英语口语测试是语言测试中必不可少的一部分。 然而, 局限于交际能力的 复杂性以及测试的信度难以把握，多数大学和学院将英语口语测试排斥在非英语 专业课程的语言测试之外。随着大学英语课程教学要求的不断发展，大学英语口 语测试已被视为当前大学英语教学要求之一，在教育领域引起越来越多的学者的 关注，因而许多研究人员致力于口语测试研究。 本论文在前人研究的基础上进一步研究查证口语测试的信度、效度和可行性。 本文作者抽样检验了黑龙江科技学院（hist）在 2008 年四月和六月分别进行的面 试英语口语测试(opi)和计算机辅助英语口语测试(ceot)的信度和效度, 研究的主 体为黑龙江科技学院 7867 名 2006 级及 2007 级非英语专业的本科生，统计样本包 括来自两个班级的 52 名学生并随机从全院抽取 96 名学生和 25 名评分员进行了问 卷调查， 借助社会学统计软件 spss16.0 完成了测试分数的相关性及数据统计分析。 在数据分析及问卷调查的基础上，本研究试图回答以下三个问题：1.面试型口 试和计算机辅助口试所得出的成绩相关性是否一致？2. 与面试型口试相比，计算 机辅助口试的信度和效度如何？3. 计算机辅助口试是否可行？ 研究结果表明：1. 面试型口试和计算机辅助口试所得出的成绩相关性较高， 结果令人满意。2. 黑龙江科技学院进行的计算机辅助口试的信度和效度在某种程 度上超过了面试型口试。计算机辅助口试能减轻学生的心理压力从而使学生表现 更佳，学生还认为它的评分体系更公平、更客观、更可靠。3. 只要拥有相应的计 算机系统，计算机辅助口试是可行而有效的，计算机辅助口试采取的综合评分法 在保证信度的同时增加了可行性。 在研究和分析的基础上，本文也对如何解决在机助口语考试实施过程中存在 的问题和对该领域的进一步研究提出了一些建议。 关键词：计算机辅助口试 面试型口试 信度 效度 ii abstract college english oral test is an indispensable part of language testing. however, due to the limitation of the complex nature of communicative competence and the difficulty in ensuring test reliability, it has generally been excluded from tests of english as non- major courses in most colleges and universities. with the continuous development of college english curriculum requirements, college english oral test is regarded as one of the demands of current college english teaching. it has attracted more and more attention of scholars in the field of education. consequently, many researchers devote their effort to oral test study. this thesis discusses the verification of the test validity, reliability and practicability on the basis of former research. the author of this thesis selectively examines the reliability and validity of an face- to- face oral proficiency interview (opi henceforth) and a computer- assisted english oral test (ceot henceforth) which were organized respectively in april and june,2008 in heilongjiang institute of science and technology (hist henceforth). the research population is 7867 non- english major students of grade 2006 and 2007 in hist, the statistical sample covers 52 students in two classes who were chosen as subjects and 96 students at random and 25 raters complete the questionnaires. the correlation, statistical analysis of the data of the test scores and questionnaire feedback were performed with the statistical package for the social science (spss16.0). based on the data analysis and questionnaires, this thesis attempts to find answers to the following research questions: 1. what is the correlation between the ceot and the opi in relation to the test scores obtained from both tests? 2. how reliable and valid is the ceot compared with the opi? 3. how practical is the ceot? through the data analysis, the research results indicate: 1. the correlation coefficient between the ceot and the opi is very high and satisfactory. 2. the ceot iii in hist has higher reliability and validity than the opi, it could relieve candidates psychological pressure and make them perform better. the students also thought its marking fairer, more objective and reliable. 3. the ceot is practical and effective on condition that there are computer systems. the holistic marking method adopted in the coet helped to improve the practicality of the coet while ensuring the reliability. based on the study and analysis, the thesis also has made some suggestions on how to deal with the existing problems encountered during the implementation of the ceot. finally speculations for further study in the field and for the improvement of quality of the oral test in this study are provided. key words: ceot; opi; reliability; validity 1 chapter one introduction 1.1 the topic chosen and significance of the study in 2004, the ministry of education issued college english curriculum requirements which points out that the objective of college english is to develop students applied ability, especially listening and speaking, and enable them to communicate in english, both spoken and written, in their future work, and at the same time they will be able to enhance their ability to study independently and improve their cultural quality so as to meet the needs of china s social development and international exchange (2007:18). meanwhile the college english curriculum requirements recommends a computer- and classroom- based multimedia college english teaching model to strengthen the teaching of college english listening and speaking(2007:30). college english oral test is an indispensable part of language testing. however, due to the limitation of the complex nature of communicative competence and the difficulty in ensuring test reliability, it has generally been excluded from tests of english as non- major courses in most colleges and universities, thus many teachers lack the information as to how to give students an oral test and students who score highly speak poor english. in fact, oral ability should adjust to the development of the society. on the other hand, testing oral ability contributes to help learners master the language ability, speaker s high oral ability is able to satisfy routine social demands and limited work requirements, also able to handle with confidence. besides, with the continuous development of college english curriculum requirements, college english oral test is regarded as one of the demands of current college english teaching, it has attracted more and more attention of scholars in the field of education. therefore, many researchers devote their effort to oral test study. many scholars have explored oral test and oral teaching on different aspects. weir (1990) analyzes the characteristics of communicative language testing, discusses the approaches to language test design, basic considerations in test design and introduces us 2 different test methods in communicative language testing. clerk (1980) raises speaking proficiency measures in his works. fisher (1979) shows the importance of the reliability in oral examinations. bachman (1996) makes many researches on the construct validation in the communicative oral test. arthur hughes (1989:113) points out that oral tasks should be properly representative of the population of tasks that the students are able to perform, the tasks should elicit behavior which truly represents the students ability, and he strengthens that it is essential that the samples of behavior can and will be scored validly and reliably. robert wood (1993:233) expresses his opinion in assessment and testing: a survey of research. he states in the course of discussing the oral test, four kinds of relationship should be considered: reliability vs. validity; direct vs. indirect methods of assessment; authenticity vs. inauthenticity; general vs. specific ability or subskills. in language test construction and evaluation, j.charles alderson,caroline clapham and dianne wall (2000:105) think that training oral examiners is quite essential, because it is relevant to the reliability. in china, some scholars also pay much attention to the research of oral test. liu runqing and han baocheng (2000) discuss the principles of designing and administering the oral test and systematically analyze the oral test mode, purpose, method and assessment in their book language test and its methods. wen qiufang (1999)makes many researches on the practicality of the large- scale oral test, and she raises the cross cultural communicative competence, which consists of three parts: the sensitivity，tolerance and flexibility to the cultural differences. pang jixian (2005:19- 23) focuses on the reliability and validity of the oral test. generally speaking, college english test is often costly and has enormous practical restricts. with the development of the society, as well as the development of the computer and network, computer is widely used in almost every aspect. in recent years, it has been used to help test in many fields so as to make test quicker, more exact and more effective, therefore, it greatly helps people with their test. in order to discuss whether the college english oral test based on computer is practical and acceptable, the 3 researcher makes a comparison between the face- to- face oral proficiency interview (opi henceforth) and the computer- assisted english oral test (ceot henceforth) in heilongjiang institute of science and technology (hist henceforth), makes a survey of college english oral test, including the test itself, test mode, test takers, examiners and its effects on students and educational system to determine the reliability, validity and backwash effect. the research reveals that college english oral test is essentially a reliable and valid test, it has produced beneficial backwash effect on college english learning and teaching. it s necessary to create an independent college english oral test system. 1.2 aims of the study and the research questions in hist, non- english major students have been given an oral english test (opi) each term since 2004, furthermore, hist arranged english oral course for non- english major students in 2007 , and in june,2007, it adopted the coet for its assessment. the coet in hist conforms to the current needs of the society and language teaching in that it not only serves to assess the students oral performance but also guides and encourages oral english teaching. what s more，it is also beneficial for the students to improve their english learning, their oral english learning in particular. the ceot has been organized four times in hist so far. how about the reliability and validity of the present english oral test - especially computer- assisted oral english test? the thesis will mainly investigate about this question and present an oral english test analysis in hist. generally speaking，it is of profound significance to examine the reliability and validation of the coet in hist and find out how much it has influenced oral english learning and teaching, hence to better lead students learning of spoken english. other aims of the study were to investigate whether the ceot can improve the english oral test and make it more effective. meanwhile, the research discusses whether it may ease the test takers psychological pressure from the testers when they do not 4 need to face the real testers and they can perform better and can bring their potential to full play. this study embodied the following research questions: 1. what is the correlation between the computer- assisted english oral test (ceot) and the oral proficiency interview (opi) in relation to the test scores obtained from both tests? 2. how reliable and valid is the computer- assisted english oral test compared with the oral proficiency interview? 3. how practical is the computer- assisted english oral test? 1.3 methods of the study this paper is to make further research on the basis of former research and discuss the verification of the test validity, reliability and practicability. the researcher analyzes the two types of oral tests in hist conducted respectively in april and june, 2008, and the research population is 7867 non- english major students of grade 2007 and 2008 in hist, the statistical sample covers 52 students in two classes who were chosen as subjects and 96 students at random and 25 raters complete the questionnaires. the correlation, statistical analysis of the data of the test scores and questionnaire feedback were performed with the statistical package for the social science (spss16.0). the study is both theory and empiricism oriented. the former provides the sound research direction and the latter helps to make that theory tie in with an existing situation. literature review will be the first thing to set hands on so that the thesis research reflects the most up- to- date linguistic theory which underpins the study. in order to measure our students oral ability effectively and scientifically we have collected different data of opi and ceot. based on bachman s reliability and validity theories, the researcher got the relevant coefficients to evaluate the oral tests and used quantitative approach and descriptive method to investigate the reliability and the validity of the two types of oral test. some data obtained and experience in this analysis may be helpful to the design of 5 our later oral test. 1.4 arrangements of the thesis the thesis consists of five chapters. chapter one introduction briefly introduces the topic chosen, significance and research questions of the study as well as the study methods. chapter two literature review and basic theories provide a literature review concerning the study in this field. chapter three case study of hist describes the conducting procedures of the opi and the ceot, examines the reliability and validity of the two oral tests. chapter four existing problems and suggestions points out the existing problems and offers suggestions about college english oral test. chapter five conclusion presents the major findings, limitations of the study and speculations about the further research. 6 chapter two literature review and basic theories language test plays an important role in college education in china, and it is also one methodology to test the language levels of university students. 2.1 language test 2.1.1what is language test arthur hughes (1981:161- 168) stresses test is used to refer to any structured attempt to measure language ability. every test has a theory behind it: some abstract belief of what language is, what language proficiency consists of, what language learning involves and what language users do with language. every theory contains constructs or psychological concepts, which are its principal components and relationship between these components. language test is designed to test the learners language proficiency, different psychologists give test different definition. according to anastasi (1982:26- 30):“ a psychological or educational test is a procedure designed to elicit certain behavior from which one can make inferences about certain characteristics of an individual.” lado(1961) indicates “ foreign language proficiency consists of knowledge and skills. knowledge contains grammar，vocabulary and pronunciation while skills contain listening, speaking, reading and writing.” 2.1.2 development periods of language test during the twentieth century, the foreign language test experienced different development phases. bernard spolsky (1979:321- 340) thinks that language test broke itself into three chronological periods, he calls them “ pre- scientific trend，psychometric- structuralism trend and psycho- sociolinguistic trend” . the first period is the pre- scientific or traditional period (before 1960). since language teaching itself was not a distinct discipline, language test followed whatever general principles of testing were available in the humanities or social sciences. examinations that measurement was based on were the subjective judgments of one 7 examiner alone or perhaps a group. teachers constructed their own classroom tests, which were an outgrowth of grammar- translation or reading- oriented methods that they were using, essay writing and translation and grammatical analysis were representatives. the main approach of language test was the essay- translation approach. heaton (2000:15) thinks the main characteristics of this approach are: it does not require special skill or expertise in test, the subjective judgment of the teacher is considered to be of paramount importance. the second period is the psychometric- structuralism or modern period (since 1961), from the early 1950s through the late 1960s. in this period, test focused on specific language elements such as phonological, grammatical, and lexical contrasts between two languages. the breaking point of the development of the language test was the language test delivered by robert lado in 1961, it emphasized to optimize objectivity and reliability but with little attention to validity and language knowledge generally was broken up into small parts and measured by standardized, objective discrete- point tests normally consisting of multiple- choice items. these tests always test what was easiest to measure, language teachers adopted the practice of testing single, minute elements of language in a test, the skills of listening, speaking, reading and writing are separated from one another as much as possible because it is considered essential to test one thing at a time. this approach was called structuralist approach, it draws on the work of structural linguistics. the third period (in the 1970s) is labeled the psycholinguistic- sociolinguistic or post- modern period. in this period, the main concern is to secure not only objectivity and reliability, but also validity. “ psycholinguistic” was equal to the relationship between competence and performance and “ sociolinguistic” meant the importance of taking variability of language use into account. clark (1980:432) pointed out that testers were realizing that “ the whole of the communicative event was considerably greater than the sum of its linguistic elements” . oller (1979: cited in wood1993:233) describes the psycholinguistic- sociolinguistic era as the age of the integrative test. the main 8 approach of language test was the integrative approach. according to heaton (2000:19), this approach involved the testing of language in context and is thus concerned primarily with meaning and the total communicative effect of discourse. integrative tests were often designed to assess the learner s ability to use two or more skills simultaneously. therefore, integrativ