外文资料--Evaluation of an Authentic Examination System (AES).pdf -- 5 元

EvaluationofanAuthenticExaminationSystemAESforProgrammingCoursesTorbjörnJonsson,PouriaLoghmaniandSiminNadjmTehraniDepartmentofComputerandInformationScienceLinköpingUniversity,Sweden{torjo,poulo,simin}ida.liu.seAbstractThispaperdescribesourexperiencewithanauthenticexaminationsystemforprogrammingcourses.Webrieflydescribethearchitectureofthesystem,andpresentresultsofevaluatingthesysteminrealexaminationsituations.Someofthefactorsstudiedindetailaretheonlineinteractionsbetweenthestudentsandexaminers,theresponsetimesandtheireffectsonthepressureexperiencedbystudent,theacceptanceofthemethodamongthestudents,andwhethertheexaminationformisgenderneutral.IntroductionAsexperiencedteachersinprogrammingcourseswehavenoticedthedrawbacksinthetraditionalexaminationformusedinprogrammingcourses.Thestudentslearntoprogramvialaboratoryexercises,butthefinalevaluationoftheirabilitiesandthegradingoftheexaminationareinaformthatusespaperandpeninsteadofcomputers.Consideringthatthestudentwillneverusethismodeforproducingaprogramthroughtheprofessionallife,weconsiderthistobenotasuitablemethod.AttheDepartmentofComputerScienceatLinköpingUniversity12fundamentalprogrammingcoursesforapproximately1000studentsindifferenteducationalprogramsaretaughtannually.Thispaperdealswithanewpedagogicalviewintheseprogrammingcourses,whichcanbeappliedtoanyprogramminglanguage,typeofstudentandeducationalprogram.Theideaisbasedonextensivestudiesarounddifferentexaminationforms,whereindividualgrading,efficientandusefulfeedbackandtheauthenticityoftheexaminationformareusedasbasiccriteriaforthechoiceofexaminationmethod.Webelievethatthechoiceofmethodtogetherwiththeaddedefficiencyintheassessmentprocessimprovesthequalityofourstudyprogrammes.Inparticular,webelievethatitwillchangetheexaminationprocessfromasummativetoanormativeassessmentoccasion1.Foranumberofyearswehaveexperimentedwithtestingthestudentsviacomputeraidedexaminationsinsomepilotcourses–anauthenticexaminationformforthistypeofcourse.However,thisexaminationformhasnotbecomemorewidespreadduetoinsufficientsupportforthecomputerenvironmentnecessaryforthiskindofexamination.DuringthepastyearanewauthenticexaminationsystemAEShasbeendeveloped,whereallthestudentsandtheexaminingteachersareconnectedtothesamesystem.Theprocess,includingcommunicationandgrading,issupportedbythisenvironment.Inthispaperwedescribetheexaminationsystemandourinitialevaluationsofthissysteminanumberofrelativelylargeexaminationsessions.ThecoursesinquestioncoveredprogramminginAdaandweretakenbyfirstandsecondyearstudents.DuringthepastyearwehaveevaluatedtheAES.Theinstrumentsusedfortheevaluationconsistedofquestionnairesfilledby231studentsoveraperiodof3monthsand4examinations.Thepaperisorganisedasfollows.Insection1wedescribewhythetypeofexaminationweproposeisthemostappropriateforprogrammingcoursesandcomparetosomerelatedsystems.Section2includesabrieftechnicaldescriptionoftheexaminationsystems,includingitsarchitecturaldesign.Insection3wedescribehowthecomputersystem,thatmanagestheexaminationprocessonline,hastobeaugmentedbyrulessetupineachparticularcourse.Section4coversourevaluationmethodsandisfollowedbyevaluationresultsinsection5.Section6concludesthepaper.1ExaminationformsEveryexaminationmethodhasspecificcharacteristicsthatmakeitmoreorlessappropriatetoaparticularcoursesetting.HåkanOswaldssonstudiedtherangeofpossibleexaminationformsforatypicalprogrammingcoursepriortothedevelopmentofthecurrentexaminationsysteminourdepartment5.Whileseveralmodesofexaminationcanbeconsideredaseffectivemeansforenhancedlearninge.g.homeassignments,oralexaminationsfollowingadesignassignment,etc,therearenotmanyexaminationtypesthatcombinetheneedforasummativeassessment,withadequatefeedbacktoinducelearning.Combinedwiththelargenumberofstudentsthatwearecurrentlyteaching,designofanidealexaminationsettingisatrulychallengingtask.TheworkbyDawsonHoweisanearlyattempttobringcomputersupportintotheprocessofprogrammingassignmentevaluationandadministration2.Theneedforautomatedexaminationsystemshasbecomemorepertinentduringthelate90swiththeadventofdistanceandlifelonglearning.Forexample,attheOpenUniversityinUKtherehavebeenattemptstoexchangestudentassignments,andtheirsubsequentcorrectionandassessmentbyexaminersviaMSWorddocuments8.However,theavailablereportse.g.theworkbyPriceandPetreconcentrateontheeaseofadministrationforcourseassignmentandgrading,ratherthanthepedagogicalfeedbackinanonlineauthenticexamination.Inrecentyearsseveralauthorsreportonautomaticassessmentsystems,mostlyconcentratingonpresentationofthetechnicalaspectsofthesystemandtheresultsofthestudentsintermsofgrading4,5,7,8.Whilewesharetheaspirationoftheseresearchteamsandconductsimilarstudies,ourfocushasbeenontheformalevaluationofhowthestudentsperceivedtheexaminationenvironment.Inadditionwehavestudiedhowtheywereaffectedbyfactorsspecifictoauthenticexaminations,howthesystemperformanceandtheexaminersonlinebehaviouraffectstheperceivedloadonthestudent,andothersuchaspects.2TechnicaldescriptionoftheAESAEShasbeendevelopedusingtheJ2EEplatform.Thisrepresentsasinglestandardforimplementinganddeployingcomplexenterpriseapplications.Havingbeendesignedthroughanopenprocess,J2EEmeetsawiderangeofenterpriseapplicationrequirements,includingdistributionspecificmechanismssuchasmessagingsystem,scalabilityandmodularity.TheclientsarebasedontheModelViewControllerMVCapplicationarchitecture,whichseparatesthreedistinctformsoffunctionalitywithintheapplicationTheModelrepresentsthestructureofthedataintheapplication,aswellasapplicationspecificoperationondata.TheViewaccessesdatafromthemodelandspecifieshowthatdatashouldbepresented.ViewsintheAESconsistofstandaloneapplicationsthatprovideviewfunctionality.TheControllertranslatesuseractionsonthemodelandselectstheappropriateviewbasedonuserpreferences.TheAESisdesignedasasetoflooselycoupledmodules,whicharetightlycoupledinternally.Groupingfunctionalityintomodulesprovidesintegrationbetweenclassesthatcooperate,yetdecouplesclassesthatrefertoeachotheroccasionally.Modulardesignsupportsthedesigngoalthatsoftwarewillbereusable.Eachmodulehasaninterfacethatdefinesthemodulesfunctionalrequirementsandprovidesaplacewherelatercomponentsmaybeintegrated.TheAESincludesmodulesforStudentaccountsTeacheraccountsExamsExaminationProcessingMessagingStatisticsTheAESdesignisdividedintomultipletierstheClienttier,theMiddletierconsistingofoneormoresubtiers,andtheBackendtierseefigure2.1.Partitioningthedesignintotiersallowsustochoosetheappropriatetechnologyforagivensituation.Multipletechnologiescanevenbeusedtoprovidethesameserviceindifferentsituations.Forexample,HTMLpages,JSPpages,andstandaloneapplicationscanallbeusedintheclienttier.Eachofthethreetiersplaysaspecificroleinthedesign.TheClienttierisresponsibleforpresentingdatatotheuser,interactingwiththeuser,andcommunicatingwiththeothertiersofthesystem.InthiscasetheClienttieristheonlypartofthesystemvisibletotheuser.TheAESClienttierconsistsmainlyofastandaloneapplicationthatcommunicateswiththeothertiersthroughwelldefinedinterfaces.AmessageorientedapproachbasedonJMSJavaMessagingSystemhasbeenchosentotakecareofthecommunicationbetweentheClienttierandtheMiddletier.TheMiddletierisresponsibleforanyprocessinginvolvingEnterpriseJavaBeans.EnterpriseJavaBeansaresoftwarecomponentsthatextendserverstoperformapplicationspecificfunctionality.TheinterfacebetweenthesecomponentsandtheircontainersisdefinedintheEnterpriseJavaBeansspecification.ThecontainersprovideservicestotheEnterpriseJavaBeansinstancestheycontain,suchascontrollingtransactions,managingsecurity,threadorotherresourcepooling,andhandlingpersistence,amongotherhighlevelsystemtasks.TheBackendtieristhesysteminformationinfrastructure.Thistierincludesoneormorerelationaldatabasemanagementsystemsandpotentiallyotherinformationassetsthatcouldbeuseful,e.g.thecentraluniversitycourseresultsadministrationsystemLADOK.TheEIStieralsoenforcessecurityandoffersscalability.TheBackendtierprovidesalayerofsoftwarethatmapsexistingdataandMiddleTierJNDIJavaNamingandDirectoryInterfaceBackendTierClientTierEJBContainerEnterpriseBeansMessageDrivenBeansJMSJavaMessagingServiceClientStandaloneSwingapplicationRDBMSFigure2.1TheAESdesign.applicationresourcesintothedesignofAESinanimplementationneutralway.Thesystemisseparatedintofivedifferentfunctionallayers,eachwithitsownresponsibilitiesanditsownAPI.Theselayersarephysicallysplitacrossthethreedifferenttiers.Thepersistencelayer,forexample,providesthemechanismsnecessarytopermanentlysaveobjectstate.ItprovidesbasicCRUDcreate,read,update,deleteservicesandalsodealswiththeobjecttorelationalmappingissues.Thisleadstoamoreflexibleandmaintainablesystem,e.g.layerscanbechangedwithnoeffectonotherlayers,aslongastheAPIremainsconstant.3ExaminationsetupTheexaminationsystemisonlyonepartoftheexaminationprocess.Thesecondpartisthesetuptheruleswehaveforthestudents.Wehavetriedafewsetupsoveranumberofyearsusingaprototypeforthesystemfor56years.3.1ThefirstsetupThefirstversionallowedthestudentstowritetheprogramsusingacomputerinsteadofwritingonpaper.Wefoundthismethodtobeanimprovementbecausewedidnothavetoreadillegibletextsandthesubmittedsolutionscouldbetestedafterwards.Gradeswerebasedonthenumberofcorrectlysolvedexercises.Aproblemwiththissetupwasthatallthegradingstillhadtobedoneaftertheexamwasfinished.Mostofthestudentswaitedtosendinthesolutionsuntilthelastminuteoftheexam.3.2ThesecondsetupOurintentionwastohaveanexaminationwherethestudentsshouldhavearesponsefromtheexaminerswithinafewminutesandwheregradesweregiventothestudentswhentheylefttheexam.Wealsointendedtoprovidethestudentwiththepossibilityofgettingaresponseforeachexercisewithinafewminutes,sotheycouldcorrectanearlycorrectsolution.Thesecondsetupwhichweusetodayisbasedonbothnumberofcorrectlysolvedexercisesandtheamountoftimetakentosolvethem.Anumberofdeadlinesaregiven.Ifthestudentwantsahighgradehe/shehastosolveanumberofexerciseswithinaprespecifiedtimelimit.Thecurrentexaminationprocessfollowsafewsteps1.Thestudentsendsanexaminationrequestforanexercisetotheexaminers.2.Theexaminerscanreturnoneofthefollowingresults.Passedthesolutioniscorrect.Incompletethesolutionhaserrors,andmustbecorrected.Itspossibletomakeanewattemptlater.Failthesolutionisincorrectandthestudentisnotallowedtocontinuetoworkonthisexercise.3.Everyexaminationattemptandtheresultwillcontributetothefinalexamgrade,andthestudentisinformedofhis/hercurrentgrade.Ifthestudentsubmitsanewexaminationrequestonanadditionalexercisehe/shecanreachahighergrade.ThisexaminationprocessisbuiltintoourcurrentAES,buttherulestimelimitsetc.canbechangedforseparatecourses.Thismakesthesystemflexible.TimelimitsandgradingInthecoursesthissystemwastestedtherewerethreeexercisesineachexamandtherequirementsfordifferentgradeswereForthegrade5excellentthestudentmustcompleteo3exercisescorrectin3hoursoro2exercisescorrectin2hoursForthegrade4verygoodthestudentmustcompleteo2exercisescorrectin3hoursoro1exercisecorrectin1.5hoursForthegrade3passedthestudentmustcompleteo1exercisecorrectin4hoursTheabovesetuptogetherwiththeAESsupportgivesustheopportunitytogradethestudentsduringtheexam.Studentswhohavesolvedanexerciseareinformedofthegradetheyhavereached.Iftheyaresatisfiedwiththatgradetheycanleavetheexammanystudentsleaveafteronetotwohourswhentheyhavegrade4or5.StudentquestionsInanordinarycomputeraidedexam,anumberofquestionsaresubmittedbythestudents,wheretheanswercaneitherbeclassifiedaspersonalorasinterestingforallstudents.Theexaminercandecideifhe/shewillsendtheanswertothewholegroupofstudentsorjusttoaspecificstudent.Thenumberofquestionsseemstoberelativelyconstantduringtheexamapproximately25questionsper5minutes.Mostquestionsaresentinduringthebeginningoftheexam,whichcanbeexplainedbythefactthatthestudentsaskaboutspecificthingspertainingtotheexercisesandthattherearemorestudentsinthebeginningoftheexam.Submission/approvalattemptsInanordinarycomputeraidedexamwehavealargenumberofexaminationrequestsfromthestudents.Aswecanseeinfigure3.1wehavearelativelyhighfrequencyintheperiodfrom30minutesto3hours.Afterthat,mostofthestudentsleavetheycantgetahighergradethan3afterthattime.Aroundthedeadlineswecanseethattheexaminationattemptsappearmoreoften,butnotsignificantlymoreoften.Still,theincreaseofexaminationrequestsleadstomoreworkfortheexaminers.Thiscanresultinanincreaseintheresponsetimewaitingtimeforthestudent.4EvaluationmethodsThedevelopmentofthecurrentsystemstartedinsummer2001andcontinuedthroughwinter2001/2002.Whenwebegantestingthissystemwewantedasatestexampleacoursewithalargenumberofstudents.Oneofourintroductorycoursesinprogramminghasaround270studentseachyear,sothatwasourfirstchoice.Approximately180ofthesestudentsareIndustrialManagementEngineeringstudentsandtherestareTechnicalBiologystudents.Ourstatisticsarebasedontheirfirstexaminationinthiscourse,whichtookplaceinMarch2002.Wealsousedaretakeexaminthiscoursetodoanewstudywithanewsetofquestions.ThisevaluationwasdoneinMay2002.Inthesetwostudies,studentsfilledinquestionnairesdirectlyaftertheexam.Thefinalquestionnairehadtwoparts.Thefirstpartwasmainlyquestionswhereanswersareinfreetextformat.Thesecondpartincludedquestionswithscaledanswersgradeontofive,disagreeagree,worsebetter.Thefirstpartwasusedinthreeevaluations.Themoreextensivequestionnairewithtwopartswasusedonlyforthelastevaluationi.e.forthetwolastexams.Theappendixshowsthefinalquestionnaire.Bothtypesofquestionnaireswereanonymousandthequestionnaireswerefilledinafterthegradingwasdonefortheexams.Thestudentshadalreadyreceivedtheirgradeswhentheyfilledinthequestionnaires.Webelievethatthisprovidesameasureofobjectivityonthestudentside.WealsousedthelogfilesfromtheAESfortheexamstogetstatisticaltrendsaboutgrades,gender,responsetimesforquestionsrespectivelyapprovalattemptsamongothersseesection5.5EvaluationresultsUnfortunatelyalmostallstudentshadnopreviousexperiencewithpaperbasedprogrammingexaminations,sotherepliescouldnotbeusedforcomparisonswiththatexaminationform.However,weusedtheresponsetostudyotherquestionsindetailspeciallythepartrelatedtothetime/stressfactor.First,howoftenthestudentssentinarequestquestionsorapprovalattempts,andhowlongthetimeforaresponsewasSecondly,howwellwastheexaminationsystemacceptedbythestudentsAthirdquestionwasacomparisonbygradesbetweenthegenders.Theresponserateofthequestionnaireswasquitegood.WehadfourexamsduringtheevaluationperiodwiththefollowingresponseratesExam176answersof112students67.8Exam287answersof105students82.8Exam350answersof66students75.7Exam418answersof22students81.8Thefirstthreequestionnairesweredoneatthefirstexaminationoccasionforthestudentsandthefourthonewasdoneinaretakeexaminationwhereallthestudentswerestudentswithnogradefromanearlierexam.5.1EventsduringanexaminationThenumberofevents,questionsandexaminationrequests,spreadoveranexaminationsessionof4hourscanbeaninterestingmetrictolookat.Themajornegativefactorthatwasindicatedinthequestionnaireswasthefeelingoftimepressureorstress.17ofthefreetextanswershadsomeconnectiontothisfactor.Fromatechnicalpointofviewwewerealsointerestedinfindingthatthecapacityofthesystemwasadequate.Thereforewehavesummarisedthenumberofinteractionstakingplaceineveryexam.Infigure5.1wecanseethatthenumberofquestionsishigherinthebeginningofanexamination,butwehavequestioneventsoverthewholeexaminationtime.Thenumberofexaminationrequestsisrelativetotime.Therewereafewrequestsinthefirsthalfanhourandthatthefirsttwohoursarebusyfortheexaminers.Therequestrateisquitehighwhenwereachthetimelimitsforthegradesespeciallythe4hourlimit.Fromatechnicalpointofviewthesystemperformanceundertheaboveloadshasbeenadequate.Tostudythestudentexperienceofstressduetowaitingtimewehavecalculatedtheaveragewaitingfortheanswertoaquestionandanapprovalofanexaminationrequestrespectively.Wehavealsolookedattheextremevalues.Itturnsoutthatforaquestiontheshortestansweringtimewas30secondsandthelongest6minutes.Thecorrespondingfiguresforapprovalattemptswere1minuteand10minutesrespectively.Thefirsttypeofinteractiontook2minutesand42seconds,andthesecondtype3minutesand31secondsonaverageforoneparticularexam.Thestudentresponses,fromthequestionnaires,onthisamountoftimeisthatitisacceptabletowaitaminuteorEventsduringexamsin10minuteintervals0510152025TimefromstartQuestionsExaminationsFigure5.1Studenteventsquestionsandexaminationrequestsduringanexam.