摘要

大學(xué)英語課程學(xué)業(yè)測試的有用性往往被人們忽視。基于Bachman的測試有用性

的理論，本文試圖評估與新視野大學(xué)英語系列教材配套的在線系統(tǒng)試題庫測試卷的

信度和效度，由此推斷出學(xué)習(xí)者是否達(dá)到了大學(xué)英語課程教學(xué)要求以及課程標(biāo)準(zhǔn)對

于語言能力的要求，即聽力理解能力，閱讀理解能力，翻譯能力，書面表達(dá)能力，

以及英語綜合應(yīng)用能力。

本文的研究問題為：1.新視野大學(xué)英語測試卷的信度如何？2.新視野大學(xué)英語

測試卷的效度如何？為了回答以上兩個(gè)研究問題，武漢某高校大學(xué)一年級117名學(xué)

習(xí)者接受了測試，測試卷的單項(xiàng)成績和試卷總分均被納入研究范圍。作者在研究中

收集了兩方面的數(shù)據(jù)：定量數(shù)據(jù)和定性數(shù)據(jù)。定量數(shù)據(jù)源于SPSS軟件對學(xué)生本次

期末考試成績的單項(xiàng)和總分?jǐn)?shù)據(jù)分析得出的結(jié)果，包括正態(tài)分布數(shù)據(jù)描述，信度系

數(shù)Cronbacha,以及相關(guān)分析和因子分析得到的數(shù)據(jù)；定性數(shù)據(jù)來自于對其中6位

學(xué)習(xí)者的訪談。

本研究結(jié)果表明：新視野大學(xué)英語題庫試題具有一定的信度和結(jié)構(gòu)效度，教師

從學(xué)生的成績反映可以推斷出學(xué)生的英語語言能力水平。本次試卷未能準(zhǔn)確測量學(xué)

生的閱讀能力，原因是試題本身過于容易。不僅如此，研究還發(fā)現(xiàn)用合適詞以及正

確形式填空一題難度過大，接近半數(shù)學(xué)習(xí)者得分低，個(gè)別試題存在設(shè)計(jì)缺陷。通過

本次研究，作者的建議是：在正式測試之前最好進(jìn)行預(yù)測以提高試卷質(zhì)量；出版商

應(yīng)格外注意新視野題庫第二級的閱讀題題庫的質(zhì)量問題。

關(guān)鍵詞：新視野大學(xué)英語題庫試題測試的有用性信度效度

Contents

Abstract................................................................................................................................i

摘要..................................................................?

Contents..............................................................................................................................iii

Introduction........................................................................................................................1

1LiteratureReview......................................................................................................7

1.1Languagetestingdevelopment....................................................................................7

1.2Languageability.......................................................................................................13

1.3Reliabilityandvalidity..............................................................................................14

1.4Testdevelopment.......................................................................................................17

2Methodology.............................................................................................................35

2.1Researchquestions.....................................................................................................35

2.2Subjects.....................................................................................................................36

2.3Instruments...............................................................................................................39

2.4Datacollection..........................................................................................................40

2.5Dataanalysis.............................................................................................................41

3ResultsandDiscussion............................................................................................43

3.1Descriptionofnormaldistribution............................................................................43

3.2ReliabilityoftheNHCEtest.....................................................................................46

3.3ValidityoftheNHCEtest.........................................................................................47

3.4Itemanalysisofmultiplechoicesandblankfillingtestingitems.............................55

Conclusion........................................................................................................................59

References........................................................................................................................62

Acknowledgments............................................................................................................67

AppendixINHCEtestpaper........................................................................................68

iii

AppendixIIAnswersforNHCE2finalexam.............................................................81

AppendixIIIIndividualandglobalscores..................................................................83

AppendixIVTotalscorefrequency.............................................................................87

AppendixVFacilityvalue............................................................................................89

AppendixVIInterviewquestions.................................................................................90

AppendixVIISummaryofinterviewees9answers.....................................................91

4

ListofTables

Table1:CompositionOftheNHCETestPaper......................................................................25

Table2:DescriptiveStatistics....................................................................................................44

Table3:M-Estimators................................................................................................................45

Table4:ReliabilityCoefficient...................................................................................................46

Table5:Inter-ItemCorrelationMatrix....................................................................................49

Table6:KmoAndBarletfSTest..............................................................................................51

Table7:Communalities..............................................................................................................51

Table8:TotalVarianceExplained............................................................................................52

Table9:ComponentMatrix11......................................................................................................52

Table10:RotatedComponentMatrix3.....................................................................................53

Table11:RotatedComponentMatrix”.....................................................................................54

Table12:Multiple-ChoiceAnalysis..........................................................................................56

Table13:BlankFillingStatistics..............................................................................................57

5

Introduction

SincetheeducationofEnglishwasfirstintroducedwiththeestablishmentof”the

firstschoolofforeignlanguagesJingShiTongWenGuan(BeijingNormalLanguage

School)in1862”(Cheng2008),theforeignlanguageteachinginChinahaschanged

greatlyoverthepast151years.Withtherapiddevelopmentofeconomyandsocietyof

China,learnersgraduallyrealizethevitalroleofEnglishintheirstudy,intheirfuturejobs,

andinthedevelopmentofthewholenation.AsChinaiscaiTyingoutareforminavery

difficultperiodofhistory,itiscertainlytruethattheteachingofEnglishasaforeign

languagechangesalotinourcountrytoo,fromthetraditionalteacher-centeredclasstothe

currentlearner-centeredone.ThistrendofEnglishteachingdoeshappeninevitably.

Actuallyithasaverydeeprootcause,whichpointstothedevelopmentofthetheory

concerninghowlanguageisacquired.Inaword,thedevelopmentoflinguistictheories

leadstoagreatleapinthewaypeopleteachEnglishtoEFLlearners,whichthen

ultimatelyinfluencethelanguagetestingtheories.Inordertogetaclearunderstandingof

thedevelopmentoflanguagetestingknowledge,thehistoryoflanguageteachingtheories

mustbefullyunderstoodbyusfirst,whichindicatesthatthecomprehensionoflanguage

theoriesisneededaswell.

Languageteachingandlanguagetestingcannotbeseparatedfromeachothersince

theyco-existinarelationofpartnership.Inordertoevaluatethelearners9achievementof

Englishlearningattheendofonesemester,atestisnormallygiventolanguagelearners

whichmaintainsausefulwayofjudginglearners,masteryoflanguagetaughtbyteachers

inclassroomsorawayofdiagnosinglanguagelearners9weaknessesintheirstudy,orof

makingdifferentkindsofdecisionsonlanguageteachingaccordingly.Theseareallbased

onthepurposeofthetest.Itisthepurposeoflanguagetestingthatcategorizesvarious

typesoflanguagetesting.Normallytherearethefollowingkinds:proficiencytest,

achievementtest,diagnosistestandplacementtest.

“Thefundamentaluseoftestinginaneducationalprogramistoprovideinformation

formakingdecisions,thatis,evaluation^^(Bachman1990:54).Intheclassroomsettings,

1

themosttraditionalandfrequentlyusedtestistheachievementtestheldmainlybythe

educationaladministrativeofficeofeachcollegeanduniversitytoevaluatewhether

learnersachievethecoursestandardsandrequirementsoftheschoolorthenation.Inmost

cases,thetestpapersaredevelopedbyprofessionalpersonsindifferentforeignlanguage

schoolsoftheuniversityoreducationandresearchinstitutions.ForEnglishteachingand

testing,themainpurposeofEnglishteachersismainlyjudgingthelearners9language

abilityrequiredbythecoursesyllabusandthecoursestandardsaftertheirlearningcertain

Englishtextbooks.IndailyEnglishlanguageteaching,textbookspublishedbyseveral

pressesallaroundthecountryareusedbyEnglishteachers.Nowadays,thetextbooksare

nolongersimpleprintingpapers,instead,asthemulti-mediatechnologyembedsinnearly

allaspectsofourlife,theEnglisheducationhasbeengreatlyimproved.Today'stextbook

mayhavetraditionalpaperbook,modernCDversion,andthelatestdevelopmentof

computertechnologyintroducestheonlineversion,whichcombinesaudioandvideo

together.TheNewHorizonCollegeEnglish(NHCEforshort)seriesisonetypical

exampleofthismulti-technology-basedtextbook,whichincludesanitembankuniquely.

AsoneofthemostpopularandwidespreadEnglishteachingmaterials,NHCEseriesis

usedbyahugenumberofChineselanguagelearnersinuniversitiesorcollegesallaround

thecountry.Asitisintegratedwithanaffiliatedonlinesystem,thisresearchselectsa

NHCEtestpaperdevelopedbyitembankofNHCEonlinesystemasitsresearchtarget

withanexplicitdescriptionofthetestdevelopmentproceduresofNHCEtestsincethe

waythatatestisdevelopedandusedcanproduceagreatinfluenceonthetest,which

includestestdesign,testoperationalizationandtestadministration.

Moreover,itisofgreatsignificancetoknowthatthelanguageabilityoflearnerscan

onlybeinferredsincelearners9languageabilityisinvisiblewhichcannotbetested

directlyinourtest.Basically,“ourtestscantestlanguagelearnersbehaviors,whichare

consideredasthemanifestationoflearnerslanguageabilityunderlyingit”(Li2001:30).

Totestlanguagelearners9ability,thetraditionalfourskillsoflanguageareusuallythe

mainfocusoftest,whicharelistening,speaking,readingandwriting.ThestudyofNHCE

testinthispapermeanstoevaluatetheskillsoflanguagelearnerswhichexcludespeaking

2

sincenospeakingtesttaskisgiveninthisstudyforthelackofresourceandqualified

raters.

Inaddition,onefactmustbepaidclosestattentiontoisthatthereisno"perfecttest”

intheworld.AsBachmanindicatedinhisbookthatuWewonderifthereisasinglebest

testfbranylanguagetestingsituation”,andthereisamisconceptionaboutlanguage

testingthat"languagetestershavesomealmostmagicalproceduresandformulaefbr

creatingthe'best'tesf^Bachman&Palmer1996:3).Nevertheless,itdoesn'tmeanthat

thereisnovaluefordoinglanguagetestingresearchfbritcanbringthefeedbackaboutthe

languagelearners9performance,whichcanhelpthelanguageteachersadjusttheir

teachingprogramsinthefuture.Eventhoughthereexistsnoperfectlanguagetest,allthe

testersaretryingtheirbesttominimizethosenegativefactorsthatmightinfluencethe

qualityoftesting,suchastheunsuitabletestingenvironment,theinconsistencyamong

raters,ortheimpropertestingmethod.Inordertomakesurethatatestisofgoodquality,

somepubliclyacknowledgednormsarewidelyusedtoevaluatethetestusefulnessofone

singletest,whichincludesreliability,constructvalidity,authenticity,interactiveness,

impactandpracticality(BachmanandPalmer1996:18).Thefirsttwoofthesesix

qualitiesbyBachmanareconsideredbymostresearchersasthefundamentalqualitiesof

tests.Bystudyingthereliability,thebasicqualityoftestperformancescorescanbe

revealedwhichisthemostsignificantandessentialqualityofagiventest,whilethe

constructvalidityemphasizesthetheoriesthattesterswanttotestinadvance.Asaresult,

itistestuser'sresponsibilitytopreciselydefinethetheoreticalframeworkoftheconstruct

validityofonecertaintest.Then,thelanguagetestdevelopmentproceduresmustbe

carefullyarrangedandstrictlyexecuted.Thispaperwillmainlydiscusstwoofthesesix

qualities,whicharereliabilityandconstructvalidity.Besides,attentionhasalsobeen

giventotwoaspectsofvalidity,whicharefacevalidityandcontentvalidity.

Inaword,thestudyconductedaimstoevaluatethereliabilityandvalidityqualities

ofafinalexamtestpaperdevelopedbyitembankofNHCEonlinesystemsecondedition

whosepurposeistocheckwhetherlearnersmeettherequirementsoflanguageability

requiredbycoursestandardsandCollegeEnglishCurriculumRequirements2007

3

(Requirements2007forshort),whichareabilitiesoflistening,reading,translation,writing

andusingEnglishinawell-roundedway(Requirements2007:18).

ThereasonsforstudyingthereliabilityandvalidityqualitiesofNHCEtestpaperare

thefollowings:

1.Thereliabilityandvalidityarethemostimportantqualitiessincethepurposeof

givingatestcannotbeachievedwhenthetestisnotreliableandvalidenough.Inanother

word,decisionsmadebyresultsfromuselesstestsmayleadtosomenegative

consequences.

2.NHCEisawidelyusedtextbookinChina,withanonlineassistancesystemfor

bothteachersandlearnerswhoarethepotentialusers.Byevaluatingthereliabilityand

validityqualitiesofitembankofNHCEonlinesystem,thebasicqualityofNHCEtest

papercanberevealedtoseewhetheritmeetsthestandardsofBachman'stestusefulness

norms,thecoreofanytest.

3.AclearandthoroughviewofthisNHCEitembankcanbegottenasthetestercan

decidewhentoapplyitintheirtestingorganizationwork,ormakesomechangesintheir

tests,orevengivesomeusefulfeedbackstothepublisherofthissystem.

Thestudiesofreliabilityandvalidityqualitieshavebeenconductedbymany

domesticandforeignscholars.

Fulcher(1997)reportedareliabilityandvaliditystudyofaplacementtestusedat

UniversityofSurrey.Chapelle,JamiesonandHegelheimer(2003)examinedthewaysin

whichthepurposeofalow-stakesweb-basedESL(Englishasasecondlanguage)test

guideditsdesignandthevalidityprocess.Stricker(2004)replicatedpreviousresearchon

theconstructvalidityofthepaper-basedversionoftheTOEFLandextendedittothe

computer-basedTOEFL.Domestically,YangandJin(2001)interpretedthescoreof

CollegeEnglishTestBand4(CET-4forshort)andCET-6startingfromtheassuranceof

validityandreliabilityoftest.XueandLiu(2006)usedMessick9sframeworktovalidate

therevisedCET-4.Overall,Weigle(2000)reviewedtheMichiganEnglishlanguage

assessmentbattery(MELABforshort)indiscussionsoftestusefulnessandtest

development.FoxandFraser(2009)reviewedtheVersantSpanish?Testinfivequalities

oftestusefulnessandacomprehensivedescriptionofthetestdevelopment.InChina,Hu

4

(1996)madeasummaryofaseminarreportconductedbyBachmanabouthistheoriesof

languagetestingdesignanddevelopment,includingnearlyallaspectsoflanguagetesting

suchastestusefulness,testtasks,targetlanguageuse(TLUforshort),andtesttaker

characteristics.Xue(2008)discussedaboutthetheoreticalmodelsoflanguagetestingand

thesixnormsoftestusefulnessasassessmentstandardsfbrcommunicativelanguage

testing.

Nevertheless,itcanbefoundthatfewtouchuponthetestusefulnessofanitem-bank

developedtest,especiallythereliabilityandvalidityqualities,andstudiesconcerningthe

testusefulnessinChinaalmostfocusonhigh-stakestests,suchasCET-4.Inthissense,

thispaperfocusesonthereliabilityandvalidityqualitiesofalow-stakesachievementtest,

theNHCEtestdevelopedbyitembankofNHCEonlinesystem.Thepossiblereasonswhy

therearefewpreviousresearchesoftestdevelopedbyNHCEonlinesystemare:firstly,

notallcollegesoruniversitiescouldaffordtheNHCEsystem,whichmaycostahuge

amountofmoney.Secondly,NHCEsystemrequirestechnicalsupportsofserversand

otherrelateddailymanagement,whichasksformoreexperiencedandprofessionalhuman

resourcestooperateit.Thirdly,achievementtestisusuallyignoredbymostresearchersor

languageteachers,whomaynot"realizetheimportanceofachievementtestinmaking

decisionsfbrlanguageteachingandleaming^^(Yuan2002),and“thequalityofcollege

Englishachievementtestisalwaysfbrgotten^^(ZhouandZhang2002).

Tobespecific,thesignificanceofthisstudyare:

1.Fewpeopletoucheduponthestudyofachievementtest,whichisoftenignoredby

scholars.Inthissense,theauthorselectsanachievementtestasitsstudyobjecttoevaluate

itsreliabilityandvalidityqualities.

2.TheNHCEseriesiswidelyusedinmanyChinesecollegesanduniversities,the

onlinesystemofwhichisintegrateddeeplywithNHCEtextbooksasaveryusefultoolfor

learners9after-classstudyandtheevaluationoflearners9masteryoflanguageability

requiredbycoursestandardsandRequirements(2007).

Thestructureofthispaperisdescribedasfollows:

5

Inthefirstpartliteraturereview,keytermsandschoolsoflanguageandlanguage

testingtheorieswillbediscussedfirst,whichincludesthedevelopmentoflanguagetesting

theory,languageability,reliabilityandvalidity,andtestdevelopment.

Inthesecondpart,themethodologyofthispaperisclearlyshown,whichare

researchquestions,subjects,instruments,datacollectionanddataanalysis.Next,the

resultsofthisresearchandalsoadiscussionabouttheresearchresultswillbegiven

comprehensively,whichincludereliabilityCronbachAlpha,facevalidity,contentvalidity

andconstructvalidity.Meanwhile,itemanalysisofreadingcomprehensionandblank

fillingtesttaskswillbeanalyzedrespectivelysincemultiple-choicetestingitemsremain

themajorityoftheNHCEtest.Inaddition,blankfillingtesttaskisinvolvedinNHCEtest

forthefirsttime,whichaimstotestthelearners9knowledgeatthecohesionand

grammaticallevelandtoimpulseslearnerstomemorizevocabularyinascientificway.

BesidesquantitativeanalysisofNHCEtestpaper,feedbackscollectedfromtest

takers9interviewsinaqualitativeanalysiswillbediscussedasasupplementaryof

quantitativestatistics.Inthelastpartofthispaper,theconclusionwillbegivenbymaking

asummaryofthisstudyaswellassomesuggestions.

6

1LiteratureReview

Inthispart,fourmajorissueswillbediscussedcomprehensively,whicharethe

languagetestingdevelopment,languageability,reliabilityandvalidity,andtest

development.Thediscussionoflanguagetestingdevelopmentcannotignoresome

linguisticstheoriesandlanguageteachingideologiesforatestismainlyusedtomeasure

thelanguageabilityoftesttakersatacertainlevelwhichisrequiredbyteachersand

educationalofficersformakingdecisionssuchasdisplacementanddiagnosis,orfor

makinganevaluationabouthowmuchlanguagelearnersmeettherequirementsof

curriculumandcoursestandards.Inaddition,themethodsoflanguageteachingaremainly

guidedbylinguisticstheories.Generallyspeaking,todiscussaboutthereliabilityand

validityqualitiesofatest,theissuesofhowlanguageisviewedbylinguistsand

researchersshouldbetoucheduponsinceviewpointsortheoriesoflanguage,language

teachingandlearningwillprovideastrongfoundationfbrthedesignofalanguagetest.In

thissense,inthefirstsubpart,relatedviewpointsandtheoriesofhowlanguageistreated

andacquiredwillbementioned,whichisthenfollowedbyadiscussiononthelanguage

ability,theobjectoflanguagetestingmeasurement.Inthethirdsubpart,reliabilityand

validitywhichremainthekeyissuesofthispaperwillbediscussed,includingthebasic

theoreticalmeaningandpreviousstudiesconcerningit.Inthelastsubpartofliterature

review,adetaileddiscussionontheconceptoftestdevelopmentandacomprehensive

descriptionofNHCEtestdevelopingprocedureswillbegivenaseachstepindeveloping

atestwillaffecttheultimatequalityofatestinasignificantsense.

1.1Languagetestingdevelopment

Theresearchfieldoflanguagetestingdevelopswiththedevelopmentoflinguistic

andlanguageteachingtheories,whichbecomesahighlightofbothtestorganizersand

teachers.Astheoriesofhowlanguageisviewedandlearnedchangealot,from

structuralismtocommunicativelanguageabilitymodel,theoriesconcerninglanguage

teachingandlearninghavealsodevelopedfromthegrammar-translationapproachto

7

communicativeapproach,whichtheninfluencethedevelopmentoflanguagetesting,from

theessay-translationapproachtocommunicativeapproachaswell.

1.1.1Theoriesconcerninghowlanguageisviewed

Thetheoriesconcerninghowlanguageisviewedremainaverybigquestionwhich

maynotbeexplainedclearlyfortheyhavebothabstractandconcreteconceptions.

Meanwhile,indifferentperiodsoftime,people'sunderstandingaboutlanguagehasgone

throughagreatchangeduringthehistory,whichcanbetracedbackto1940s.

Before1940s,thetheoryoflanguagewasnotfoundedandpeopletendedtotreat

languageasakindofknowledge.

Then,duringthe1940s,thetheoryoflanguagemadeitsfirstgreatleap.Duringthis

period,itwasrealizedthat"theknowledgeoflanguagerulesdidnotequaltheknowledge

oflanguageitself9(Li2001:4).Consequently,somescholarssuggestedthat“teachthe

language,notaboutthelanguage^^(Moulton1963,ascitedinLi2001).Asaresult,the

schoolofStructuralismcameintopeople'svision.Saussureviewedlanguageasasystem

ofsigns,whichisconsistingof"signifier"and"signified”.Meanwhile,thefocusof

researchoflanguageistheunderlyingsystemoflanguagewhichiscalled“Langue”by

Saussureratherthantheuseoflanguage,named“Parole”.

Nevertheless,inthelate1950s,NoamChomskyfiercelyattackedtheStructuralism,

whoindicatedthatStructuralismcouldnotexplainonesimplequestion:Howcould

childrencreatenewsentencesthattheyhadnotbeentaughtbefore?Chomskyclaimedthat

wasthepovertyofthestimulus,whichmeantthatthereexistedaLanguageAcquisition

Device(LADforshort)inpeople'sbrainphysically,oramentalmechanism.Itwasthis

LADthathelpedchildrenmakenewsentenceswhiletheinputtheyreceivedwasnot

adequateenoughformakingsuchcreation.ThenChomskyintroducedapairof

conceptions,whichwerecompetenceandperformance.Whilecompetencemeansthe

speakers9underlyingknowledgeoflanguage,theperformancemeanstheactualrealization

ofthisknowledgeinlanguageuseundersomespecificcircumstances.

AfterChomsky,manyscholarsexpressedtheirattackstowardshistheory,who

claimedthatChomsky'slinguisticcompetencewastooabstractwhichignoredthecontext

oflanguageuse.ItwasHymeswhofirstestablishedtheconceptofcommunicative

8

competencein1972,whichincludedthelinguisticcompetencethatwasstronglyrequired

incommunicationandalsoabilitiesbeyondlinguisticcompetence.Later,Canaleand

Swain(1980)definedcommunicativecompetenceintermsoffourcomponents:

grammaticalcompetence,sociolinguisticcompetence,discoursecompetenceandstrategic

competence.

AfterCanaleandSwain,Bachmanfurtherdevelopedthetheoreticalframeworkintoa

moresystematicalandadvancedone.AshereportedinhisbookFundamental

ConsiderationsinLanguageTesting,"recentformulationsofcommunicativecompetence

provideamuchmoreinclusivedescriptionoftheknowledgerequiredtouselanguage,the

knowledgeofhowlanguageisusedtoachieveparticularcommunicativegoals"(Bachman

1990:83).Inhistheory,Bachman'scommunicativelanguageabilityconsistsoftwomain

components,whichareorganizationalandpragmaticcompetence.“Theorganizational

competencecompromisesabilitiesincontrollingtheformalstructurefbrproducingor

recognizinggrammaticallycorrectsentences,comprehendingtheirpropositionalcontent

andorderingthemtoformtexts”(Bachman1990:87),suchabilitiesaregrammatical

competenceandtextualcompetence.Forpragmaticcompetence,€4itmainlyfocusesonthe

organizationanduseoflanguageinrealcommunication,whichenablesustocreate

discoursebyrelatingutterancesorsentencesandtextstotheirmeanings,totheintentions

oflanguageusers,andtorelevantofthelanguageusesetting”(BachmanandPalmer1996:

69).Overall,Bachman'stheoreticalframeworkofcommunicativecompetencetakesboth

internalandexternalknowledgeoflanguageintothoroughconsideration,whichgivesusa

systematicalviewoflanguageandlanguageuse.

Fromthethreemainstagesoflanguagetheoriesdevelopmentabove,itisclearthat

theextensionofthelanguageknowledgebroadensvastlyanditbecomesmore

comprehensiveandreasonabletoinvolvesocial,culturalandevencognitivefactorsinto

theconceptoflinguisticcompetence.

1.1.2Theoriesoflanguageteachingandlearning

Thedevelopmentoflinguisticstheoriesconcerningthelanguagedefinitely

determinesthechangeoflanguageteachingandlearningtheories,fbrtheultimategoalof

languageteachingistofacilitatelanguageleamers,languageproficiencyinorderthatthey