IntraclasscorrelationcoefficientsReliabilityandmore…AlexaRomberg710DiscussionSection2/11/09REVISED2/18/092*Sources*Shrout&Fleiss(1979)PsychBulletin(oncoursewebsite)McGraw&Wong(1996)PsychMethods(wille-mail,N.B.hassomecorrectionsthatwerepublishedlaterthatyear)(directlytranslatestoSPSS)3ICC’svs.Pearson’srPearson’srmeasuresinterclasscorrelationsVariablesarefromdifferent“classes”,theymeasuredifferentconstructs:e.g.,IQandGPA;heightandweight;celebritypopularityandnumberofmarriagesICC’s(therearemanyofthem)measurecorrelationswithina“class”Variablesaredifferentmeasurementsofthesamething/constructTheycouldberepeatedmeasuresonthesameparticipant,suchasforreliabilityOrtheycouldbemeasuresonmultiplepeoplewithinthesamegroup,e.g.,twins,studentsinaclassroom4ICC’sGeneralgoalistodeterminetheproportionofvariationinameasurethatisduetobeingamemberofaparticulargroupForexample,considerastudyof5families,eachofwhichhas3children,inwhichyoumeasuretheIQandageofeachchildIfyouareinterestedinhowfamilymembershipandIQarerelated,youwoulduseanICCIfyouareinterestedinhowageandIQarerelated,youwouldusePearson’sr5ManydifferentkindsofICC’sSomehavenamesyoumayrecognize:Cronbach’salphaHeritabilitycoefficientsKish’srateofhomogeneityOthersdon’tappeartohavenamesotherthansomewhatconfusingindices:ICC(C,1),ICC(1)6ExamplesofdatalayoutforICCWhattypeofanalysisdothesetablescalltomind?7CalculatinganICCStep1:PartitionthevariancebyenteringdataintoanappropriateANOVAmodelStep2:EntertherelevantMeanSquaresintotheformulafortheICCofinterestThiscanbedone(opaquely)inSPSSunder:Analyze->Scale->ReliabilityAnalysis->Model(Alpha)->Statistics8CalculatinganICC(reliabilityframework)ThisboilsdowntounderstandingthesourcesofsystematicvarianceintheratingsNeedtoconsider:1)howwereyourratersdetermined(i.e.,howareyouinterestedingeneralizingfromthem?)2)isvariabilityintheratingsduetospecificratersimportanttoyouornot?3)areyouinterestedinthereliabilityofindividualratersorinthereliabilityofthemeanofagroupofraters?9Consideration1:TheratersAreratersdrawn“randomly”fromalargerpool?E.g.,Doyouhave5RA’sallworkingonalargestudybutonly2rateeachparticipant?Or,doallratersrateallparticipants?(e.g.,youhave2RA’sworkingforyouandtheyratealloftheparticipants)Thedifferencehereisaquestionofgeneralization-doyouwanttogeneralizefromtheratingsonwhichyouarecalculatinganICC?10Consideration1:TheratersIfyoudowanttogeneralize(i.e.,theratersaredrawnfromapool)didthesamesubsetofratersrateallparticipants,ordiddifferentsubsetsofratersrateeachparticipant?Notsamesubset:Youhave5RA’sworkingonaprojectandeachtargetisratedbyarandomsubsetof2ofthem.Samesubset:Youhave5RA’sworkingonaproject,butthedataonwhichyouarecalculatingtheICCisfromthesame2raters.11Consideration2:VariabilityduetoindividualratersThisisanalogoustoaskingwhetheryouthinkthatvariabilitybetweencolumnsinyourtableisrelevanttotheratingsWhenyouhaveratersgivingsubjectiveratingsonascale(say,1-10),differentraterswillusedifferentpartsofthatscale.ImaginethatRater1rateseveryonehigh(i.e.,uses#’s5-10onthescale)andRater2rateseveryonelow(i.e.,uses#’51-5onthescale).Withthisconsideration,youaresayingwhetheryoucareaboutthedifferenceinabsolutevaluebetweentheseratersorwhetheryouonlycarethattheratersrankedthetargetsinthesameway(i.e.,thesametargetgota10fromRater1anda5fromRater2).12Consideration2:VariabilityduetoindividualratersYouroptionsforwhetherindividualratervariabilityisimportanttoyou:NOTIMPORTANT,thenyouarebasicallytestingwhethereachraterrankedeachtargetinasimilarway(referredtoasconsistency)ISIMPORTANT,thenyouaretestingwhethereachratergaveeachtargetexactlythesamescore(referredtoasabsoluteagreement)Completelyirrelevant:whenyouhavedifferentsubsetsofratersratingeachparticipant,youareunabletopartitionvarianceduetoindividualratersandthischoicedoesnotapply13ICCModelDecisionTree(Consideration1:Raters)Wereratersdrawnfromlargerpool?Didthesamesubsetofratersrateeachtarget?Two-waymixedNoYesOne-wayRandomTwo-wayrandomNoYesFortwo-waymodelsyoumustchooseTYPEaswell(Consideration2:Individualratervariability):

consistency(interestedinwhethertargetsrankedthesame)

absoluteagreement(interestedinwhethertargetsgotexactsamescores)14Consideration3:Whattypeofscore?Areyouinterestedinthereliabilityofanindividualraterorthereliabilityofthemeanofallraters?Typically,ifyouhavemorethanonerating,youwouldaveragethemforyouranalyses,sincetheaveragewillalwaysbeamorereliablemeasure.SPSSgivesyouboth,allthetime.SingleMeasureIntraclassCorrelation=reliabilityofanindividualraterAverageMeasureIntraclassCorrelation=reliabilityofthemeanoftheraters15ChooseyourICC:Scenario1Aresearcherhas10RA’sdoingratingsoutattheprison.MaryandSuerateprisoner1,SueandChrisrateprisoner2,KathyandPeterateprisoner3,etc.Theresearcherwillusethemeanoftheratingsinhisanalysisandwantstoknowthereliabilityofthatmean.WhichANOVAmodel?WhichTYPE?WhichMEASURE?One-wayRandom(notype-onlyconsistency)AVERAGEMeasure16ChooseyourICC:Scenario2Aresearcherhas10RA’sdoingratingsoutattheprison.Totestthereliabilityofthispoolof10RA’s,theresearcherhas3raters,Mary,SueandPete,ratethesame5prisoners.Theresearcherwantstoknowhowreliablethemeansofthe3ratersare.WhichANOVAmodel?WhichTYPE?WhichMEASURE?Two-WayRandomConsistencyorAbsoluteAgreementAverageMeasure17ChooseyourICC:Scenario3Ourresearcherhashadhisfundingcut,andnowcanonlyafford3RA’s,whowillbedoingalltheworkforhisstudy.HesendstheRA’stotheprison,wheretheyeachratethesame5prisoners.Theresearcherwantstoknowhowreliableanyindividualraterisandisconcernedthattheymaybeusingdifferentanchorpointsintheirratings.WhichANOVAmodel?WhichTYPE?WhichMEASURE?Two-WayMIXEDAbsoluteAgreementSINGLEMeasure18Bonusfunfact!TheICCsappropriateinScenarios2and3(thetwo-wayrandomandmixed)actuallygiveyouTHESAMECORRELATION.Thedifferencecome