TranslatorCertificationExaminationsandValidity翻译资格考试效度.doc

Translator Certification Examinations and ValidityAlan K. Melby (version 1c still a rough draft)There are many translator certification programs around the world. In 2005 FIT () published a report (Stejskal 2005) by Jiri Stejskal (currently one of the vice-presidents of FIT but then chair of the FIT committee on the status of the translation and interpretation profession). From that report we learn that the vast majority (96%) of the 63 FIT member organizations responding to a survey include an examination as part of the credentialing system.Clearly, then, most of professional associations that are members of FIT are interested in establishing that their examinations are sound and defensible. By sound I mean that the examination is a good indicator of professional competence, and by defensible I mean that if someone challenges their failure to receive a credential, the examination can be defended on the grounds that it was developed based on widely accepted principles and requirements of assessment. The next section explores the ISO standards that contain these widely accepted principles and requirements.ISO 17000 and 17024This paper, which is about credentialing examinations for translators, flows from the ISO 17000 series of standards. Not as well-known as the ISO 9000 series, the 17000 series is nonetheless very influential and is managed by CASCO (2012), the ISO committee on conformity assessment.The starting point standard of the series, ISO 17000:2004, defines conformity assessment as follows:(2.1) demonstration that specified requirements relating to a product, process, system, person, or body are fulfilledThe first note on this definition makes it clear that certification is used for persons while accreditation is used for assessment bodies. Thus, even though in some countries, such as Australia and South Africa, translators are accredited, this paper will use only the term translator certification. In accordance with ISO 17000, accreditation will be reserved for a third-party review of a certification body to determine whether it is properly conducting the certification process.Today, the most common standard used for accrediting certification bodies is another member of the ISO 17100 series: ISO 17204 (Conformity Assessment - General requirements for bodies operating certification of persons). For example, the Irish National Accreditation Board (www.inab.ie/aboutus/) uses ISO 17204, and the United Kingdom Accreditation Service () points out in document P16 on accreditation of approved and notified bodies that as of August 2009 its use of the EN 45000 series of standards was being re-evaluated, since they were being largely replaced by the ISO 17000 series.Indeed, as of January 1, 2010, EU Regulation 765/08 on Accreditation and Market Surveillance is in effect, which includes the use of ISO 17000 standards harmonized with EN 45000 standards.In the United States, ISO 17024 has been adopted by ANSI (), the US member body of ISO, for use in accrediting bodies that certify persons.Thus, ISO 17024 is recognized in the United States and Europe as the relevant international standard providing minimal requirements for a personnel certification body. It is assumed that a study of other areas of the world would reveal that ISO 17024 is recognized world-wide. Given the use of ISO 17204, a basic question arises: If a certification body determines whether a person is be certified, and an accreditation body determines whether a certification body is recognized then who watches over the accreditation bodies to make sure they are using ISO 17024 properly? This top-level watchdog function is performed by national governments or by the International Accreditation Forum (www.iaf.nu), which is an organization whose members are established accreditation bodies in various areas. The International Accreditation Forum is the ultimate world-wide authority that decides whether to recognize an accreditation body.Main Requirements of ISO 17024The requirements of ISO 17024 fall into two categories: the operation of the certification body as an organization and the development and maintenance of what is called the certification scheme. A certification scheme is essentially the certification examination and the procedures used to create new versions of the exam and to grade the exam.The requirements for the operation of the certification body are focused on (a) making sure the certification process is fair and equitable for all candidates, (b) documenting all procedures for granting, renewing, and suspending or withdrawing certification, and keeping proper records, and (c) involvement of all stakeholders in an advisory council.The requirements for the certification scheme itself are focused on validity and reliability. These are terms from assessment theory. Here is a vastly oversimplified description of them.An exam is valid when you are testing what you want to test.And exam in reliable when the candidate gets very much the same score regardless of who grades the exam.Thus, an exam could be reliable but not valid. For example, suppose you want to test how well someone reads Russian. Your invalid exam consists of measuring how tall the candidate is when barefoot. The score is associated with Russian as follows: the bigger the score (in centimeters) the better they speak Russian. The result will be highly reliable. No matter who the grader is the height will come out the same, so long as they measure carefully. However the result is completely invalid. There is no connection between height and Russian language reading proficiency. An exam could also be valid but not reliable. For example, suppose you want to test how well someone can read an MRI scan. Your valid exam consists of real MRI scans of patients who are healthy and scans of patients who have various diseases, such as cancer, along with questions about the scans. But further suppose that the graders are not given the answers to look for and instead compare the answers of the candidates with their own opinion. If the some graders are not very good at reading MRI scans and others are very good, the scores could vary considerably from grader to grader, even for the same set of answers. The grading would not be reliable.For much more detail on reliability and validity in various kinds of language testing, see Bachmann (1990).Translator CompetenceThe rest of this paper will treat the question of validity for a translator certification exam.In order to demonstrate that an exam in valid, you first have to clearly describe what you want to test.In the case of translator certification, you want to find out whether a translator is competent to perform the typical work of a professional translator. Regardless of the detailed breakdown of competence, establishing validity requires a job analysis, sometimes called a practice analysis. I will write more about job analysis later, but for now, lets further examine the term competence.As you look at the literature on assessment, you will find two noun forms for the adjective competent:- competence and- competencyFor some these are synonyms, but Teodorescu (2006) makes an interesting distinction between them in the context of assessment, HPT (human performance technology) in particular. She suggests that competency is associated with job analysis in which knowledge, skills, and other attributes of a person are categorized and ranked for use in hiring, training, and assessment (such as certification) and that competence goes beyond competency to an examination of how to use competency to achieve the goals of an organization.Nevertheless, we will use the noun form competence since that is what is used in ISO 17024. No distinction between competency and competence is made in 17024.The ISO 17024 definition of competence is:(3.6) demonstrated ability to apply knowledge and/or skills and, where relevant, demonstrated personal attributes, as defined in the certification schemeThis is a compromise between American and European usage. In Wang (2005), an article on developing certification exams from the journal Educational Measurement), we find that competence is broken down into knowledge, skills, and abilities (KSAs). This is a typically American approach.An official European approach is found in the:RECOMMENDATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 23 April 2008 on the establishment of the European Qualifications Framework (EQF) for lifelong learningwhere competence is explained as follows: The concept competence is thus used in an integrative manner; as an expression of the ability of individuals to combine in a self-directed way, tacitly or explicitly and in a particular context the different elements of knowledge and skills they possess. This is actually quite close to the 17024 definition, even though the official EQF definition of competence in the same document makes it look more different:competence means the proven ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations and in professional and personal development. In the context of the European Qualifications Framework, competence is described in terms of responsibility and autonomy.Lundberg (1972) provides some historical insight into the origin of descriptions of competence. He cites Summer (1956) as providing the following early description of executive competencies:(Lundberg p. 13) - knowledge- attitudes- ability (consisting of skill, art, judgment, and wisdom)Over the years, this breakdown has evolved to emphasize knowledge and skills and somehow include ability.Competence and the Nature of Translation QualitySandberg (2000) provides some philosophical background for the study of competence. Sandberg distinguishes between a rationalistic approach to competence and an interpretive approach to competence (based on phenomenology as developed by Husserl, Searle, and others in philosophy).The Sandberg article is particularly relevant to translator certification, since performance of a translator is certainly influenced by his or her attitudes and beliefs about the nature of translation and translation quality.It is proposed that for the purposes of translator certification, we assume agreement with Functionalism in translation studies (see various publications by Christiane Nord, http:/www.christiane-nord.de/?de_publications,45), with its emphasis on the translation brief that is negotiated between the requester and the provider. This negotiated brief is called the translation project specifications in ISO/TS 11669. A definition of translation quality based on Functionalism, ISO/TS 11669, and cross-industry studies of quality management would be:A quality translation demonstrates sufficient accuracy and fluency for the audience and purpose and complies with all other specifications that have been negotiated between the requester and the provider, provided that the specifications are appropriate to end-user needs.The practical consequence of Functionalism and ISO/TS 11669 is that the translation performance component of a translator certification exam should include both a source text and a set of specifications, that is, instructions to the translator. Specifications must include the audience and purpose of the translation. A list of all 21 translation parameters from 11669 can be found on a webpage (/specs).Using Job Analysis to Establish ValidityRegardless of whether you use the American or European definition of competence, or some other minor variation, the establishment of the validity of a translator certification exam involves a job analysis that begins with consulting with a small group of expert practitioners on what makes for a competent translator and then creating from the analysis of the comments from the practitioners a survey that is presented to a larger group of practitioners for ranking in importance of the elements of knowledge, skills, and other attributes of competence. Then representatives of all stakeholder categories must be consulted.Once a test blueprint has been developed from all the results of the job analysis, the actual exam must be constructed so that it is fair and reliable.Bottom Line for ISO: Accreditation of Translator Certification BodiesMultiple exams from multiple FIT member bodies could be mutually recognized if they all meet the requirements of 17024. This would also provide for more portability of translator certification credentials across national boundaries.A partnership must be developed between translation schools that grant diplomas and translator certification bodies to coordinate education and certification.Someday there may be an international organization recognized by the IAF that specializes in accrediting translator certification programs world-wide.ReferencesBachmann, Lyle (1990). Fundamental Considerations in Language Testing. Oxford University Press, Oxfor