GamesPlane:AugmentedRealityGamesman

MillerHollinger

ElectricalEngineeringandComputerSciencesUniversityofCalifornia,Berkeley

TechnicalReportNo.UCB/EECS-2025-136

/Pubs/TechRpts/2025/EECS-2025-136.html

June4,2025

Copyright?2025,bytheauthor(s).

Allrightsreserved.

Permissiontomakedigitalorhardcopiesofallorpartofthisworkfor

personalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesare

notmadeordistributedforprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationonthefirstpage.Tocopyotherwise,torepublish,topostonserversortoredistributetolists,requirespriorspecificpermission.

Acknowledgement

ToDanGarcia,forbeinganexcellentresearchadvisor.

ToEricPaulosforkindlyreviewingthispaper.

ToJoshZhangforhisassistanceincameracalibrationwhichgreatlyimprovedthesystem’s

accuracy.

ToAlvaroEstrellaforextendingtheGamesmanUniAPI.

ToSriyaKantipudiforaddingnewgamestoGamesPlane.

ToAbrahamHsuforconsultingwithmeaboutvectormath.

ToGamesCraftersasawhole.

ToBarakStout,my9th-gradecomputerscienceteacherwhosepatienceandknowledgeledmeintothewonderfulworldofcomputation.

Aboveall,tomyparents,JohnandKarenHollinger,forsupportingme

throughoutmyacademiccareerandbeinganendlesssourceofkindnessandadvice.

1

GamesPlane:AugmentedRealityGamesman

byMillerHollinger

ResearchProject

SubmittedtotheDepartmentofElectricalEngineeringandComputerSciences,UniversityofCalifornia,Berkeley,inpartialsatisfactionoftherequirementsforthedegreeofMasterofScience,PlanII.

ApprovalfortheReportandComprehensiveExamination:

Committee:

TeachingProfessorDanGarciaResearchAdvisor

Date

*******

ProfessorEricPaulosSecondReader

Date

2

Abstract

Inthisreport,IintroduceGamesPlane,asystemthatusesaugmentedrealitytooverlaythepre-calculatedcoloredvalueofmovesforaplayerplayingaphysicaltwo-person,

complete-informationboardgame.Thevaluescomefromapplicationprogramminginterface(API)callstoourGamesmansystemthathasstronglysolvedthegame.Itprovides20framespersecondupdatesforboardsofallshapesandsizeswithnear-100%accuracy.Finally,itishighlyconfigurable,withtoolsanddocumentationprovidedforeasyextensibility.

3

Acknowledgements

ToDanGarcia,forbeinganexcellentresearchadvisor.Hisguidance,encouragement,and

positivitymadeGamesPlanepossible,andmyloveforcomputerscienceresearchisthankstohim.

ToEricPaulosforkindlyreviewingthispaperandhisfeedbackontheproject’sdevelopment.

ToJoshZhangforworkingalongsidemetodevelopandtesttheGamesPlaneproject,andespeciallyforhisassistanceincameracalibrationwhichgreatlyimprovedthesystem’s

accuracy.

ToAlvaroEstrellaforextendingtheGamesmanUniAPItointegratewithGamesPlane.

ToSriyaKantipudiforaddingnewgamestoGamesPlaneandworkingwithmetoimproveitsaccuracy.

ToAbrahamHsuforconsultingwithmeaboutvectormath.

ToGamesCraftersasawhole;fortheMonday-Wednesday-Fridaylunchmeetingsandthelate-nighthangouts.Mycollegeexperiencewouldnothavebeenthesamewithoutthem.

ToBarakStout,my9th-gradecomputerscienceteacherwhosepatienceandknowledgeledmeintothewonderfulworldofcomputation.

Aboveall,tomyparents,JohnandKarenHollinger,forsupportingmethroughoutmyacademic

careerandbeinganendlesssourceofkindnessandadvice.

4

Contents

Abstract 2

Acknowledgements 3

Contents 4

Chapter1:Introduction 6

Chapter2:Background 8

2.1:GamesCrafers,Gamesman,GamesmanUni,anGamesPlane td8

2.2:BringingGamesmanoPhysicalBoarGames td8

Chapter3:RelatedWork 10

3.1:Makingrealgamesvirual:Trackingoargamepieces1 1tbd[]0

3.2:AugmeneRealiyChessAnalyzerARChessAnalyzer2 1tdt()[]0

3.:AnInelligenChessPieceDeecionTool 13tttt[3]0

3.4:Chessoaranchesspiecerecogniionwihhesupporofneuralneworks4 11bddttttt[]

3.:ChessPieceDeecion 115tt[5]

3.:ComparisonoGamesPlane 116t

3.7:SysemComparisonChar 1tt3

Chapter4:DevelopmentandChallenges 14

4.1:ConcepualizaionofGamesPlane 14tt

4.2:FirsIeraion 14ttt

4.:TransiionoArUco-OnlyApproach 13tt5

4.4:GamesPlaneArchiecure 1tt6

4.:PrinaleGamesPlanes 175tb

4.:SofwareDevelopmen 16tt8

4.7:HomePage 20

4.:Crafsman 218t

4.:Documenaion 29tt3

4.1:LaunchGame 240

4.11:SarerGuie 2ttd6

Chapter5:Results 27

5.1:BoarSaeDeecion 27dtttt

5.2:OpimalAccuracy 27t

5.:HeighTolerance 23t8

5.4:PieceRoaionTolerance 2tt8

5.:PoorlyAlignePiece 25d9

5.:CameraRoaions 26tt9

5.7:OherShorcomings 1tt3

5.:Spee 18d3

Chapter6:FurtherWork 32

Chapter7:Conclusion 33

5

Bibliography 34

Appendix:User’sGuide 35

Appenix1:SaringGamesPlaneLocally dtt35

Appenix2:PlayingaGamewihGamesPlane dt35

Appenix:AingaNewGame d3dd35

Sep1:CreaeaPhysicalBoaranPieces ttdd36

Sep2:Creaehe.sonFileinCrafsman tttjt36

Sep:WrieaUWAPIConverer 7t3tt3

Sep4:RegiserhegameinAppLaunchGame.py ttt/38

Sep:PlayYourNewGame! t538

Appenix4:Trouleshooing dbt38

6

Chapter1:Introduction

GamesCraftersisaresearchgroupdedicatedtosolving,analyzing,andplayingtwo-playerturn-basedcomplete-informationgamessuchasTic-Tac-Toe,NineMen’sMorris,orOthello.GamesCrafters’systemGamesman[8]accomplishesthisgoal,usingacodedescriptionofagametocreateasolution.Overtime,itsscopeexpandedanditaddedpuzzlesandmore

complexgames.Apublicwebsite,GamesmanUni[9],wascreatedtotransitionGamesmanawayfromadownload-and-recompileX11-based-GUImodelandtowardsthemodernageofwebapps.Figure1.1showsaviewofthesite.

Figure1.1.GamesmanUnisportsnearly100playablegamesandpuzzles,allstronglysolved.

Despitethisgrowth,Gamesmanhadyettocrossacriticalboundary…intothereal

world.Althoughmostofthegamesitsolvedwerenearlyalwaysplayedface-to-face,findingthevaluesofmovesthroughGamesmanrequiredopeningawebsite(orrunningalocalscript)toinputineachmoveoneatatime,thencross-referencetheresultswiththereal-worldboard.

WithGamesPlane,weentertherealworldusingArUcotracking-poweredboardsandpieces.GamesPlaneisasystemforcreating,readingpiecedatafrom,anddisplaying

informationontospeciallydesignedgameboards.UsingGamesPlane,youcandefinethe

informationforagameboard,useacameratolocatewherethepiecesare,andthenseethebestmovesinaugmentedreality.Indoingso,GamesPlaneservesasareal-lifeinterfacetoGamesman,bringingtheperfectplayhintsoncerelegatedtocomputerscreenstophysicalgameboards.

7

Figure1.2.TheoverlaydisplayedbyGamesPlaneonBlack’sturn.Yellowmovesare

“draw”moves,andredmovesare“losing”moves.Greenmovesare“winning”moves,thoughno

suchmovesexistinthisparticularposition.

AuserofGamesPlanecanprintoutoneofmanygameboardandpiecesets,thenrunGamesPlaneontheirdevice.GamesPlanefindsthepiecesandsendsthatinformationofftoGamesmanUni,whichreturnsanimageshowingvalidmovesintheposition,withgreen

“winning”,yellow“tie”and“draw”moves,andred“losing”moves.Thisimageisdisplayedin

augmentedreality(i.e.,projectedoverthecamerapicture)usinginformationabouttheboard,creatingthefinalresult:areal-lifeimageoverlaidwithanARgraphicshowingmovesbasedonthepieces’positions(Figure1.2).

GamesPlaneworksusingArUcomarkers,whicharesquareblack-and-whitetags

featuringspecificpatternseasilyrecognizablebycameras.DetectionofArUcomarkersis

implementedbyOpenCV,makingthemaneasytoolforcomputervisionprojectssuchas

GamesPlane.Auserplacesanchormarkersontoagameboardandattachesthemtopieces.Thesemarkers’relativepositionsinanimageareusedtodeterminewherethepiecesareinspace,andthenonthegameboard.

GamesPlanehastheuniquequalityofbeingextensiblebydesign:withitsintegrated

documentation,starterguide,andCraftsmantool,futureuserswillhaveaneasytimelearningtousethesystemandsetupboards.

Thesystemrunsatahighframerate—atleast20framespersecond,evenonlow-endsystems.Therearecurrently5gamesthatfunctiononGamesPlane.Templatesareprovidedtoprint8.5x11”boardsforavarietyofgames,buthypotheticallyanysizeorshapecanfunctionaslongascameraqualityishighenough.ItcanfailwhenArUcomarkersareobscuredorthe

cameraisattoolowofanangle.butitisalsofairlytolerant:itfunctionsevenwhenpiecesare

imperfectlyaligned,unevenlyrotated,orofdifferentsizes.

8

Chapter2:Background

2.1:GamesCrafters,Gamesman,GamesmanUni,andGamesPlane

GamesCrafters’centralgoalhasalwaysbeentostronglysolvedeterministictwo-player

turn-basedgameswithcompleteinformationlikeTic-Tac-Toe,orConnectFour.Weusea

systemcalledGamesman[8]tosolvethesegames;itexhaustivelysearchesallpossibleboardstatesforagame,eventuallycreatingadatabasestoringeverypositionandifitisawin,lose,tie,ordrawinperfectplay(andhowmanymovesitisfromtheendofthegame).UsersmostoftenaccessthesedatabasesusingGamesmanUni[9],apubliclyavailablewebsitethatlets

usersplayboardgamesonlinethroughgraphicaluserinterfaces(GUIs).Duringtheirgames,playersareshownvaluemoves:themovestheycanmakecoloredred,yellow,orgreenforiftheyarelosing,drawing/tieing,orwinningrespectively.ThisflowofinformationisshowninFigure2.1.

Figure2.1.InformationaboutthegameisgeneratedbyGamesmanandthenservedbyUWAPIto

GamesmanUni.TheuserinterfacecreatedbyGamesmanUniissenttoGamesPlane.

2.2:BringingGamesmantoPhysicalBoardGames

Foralongtimenow,ithasbeenagoalofthelabtobringthepowerofGamesmantothephysicaldomain.Thereasonforthisisfairlystraightforward:boardgamesaregenerallyplayedface-to-face,notonline.BeforeGamesPlane,touseGamesmanwithareal-lifegamewould

requiretheusertoenterintoGamesmanUnieverymovetheymake,whichisacumbersomeapproachthatslowsdownthepaceofgameplay.

Attemptsatcrossingthebarrierbetweendigitalandphysicalhavebeenmadeinafewinstances.Forexample,inFall2024,aprojecttodisplayvaluemovesforConnectFourwasattemptedusingcolordetection.Althoughthismethodwassuccessful,itwasnotgeneralizabletoothergames,asitworkedbydetectingtheaveragecolor(redoryellow)ineachConnect

Fourboardspace.Additionally,thissystemonlyworkedonvideorecordings,notonlivevideo.

9

Theobjectiveofcreatingasystemthatwouldallowmanygamestobeplayedinrealliferemained.

Asidefromthegoalofcrossingintothephysicaldomainandgeneralitytomanygames,athirdgoalwasaccessibility.AnotherkeyideaofGamesCraftersisthatgamesshouldbe

enjoyedbyeveryone.GamesPlane,then,aimstobeaseasytouseaspossiblefornewusers.Thissuggeststheinclusionofone-clickinterfaces,easysetup,andexhaustivedocumentation.Together,usabilityfeaturessuchasthesecontributetotheongoinguseanddevelopmentofthe

software,bothbycasualandtechnicalusers.

10

Chapter3:RelatedWork

Theconceptofasystemthatconvertsvideoofboardgamesintodigitalboardstatesisinitselfnotnovel.Multiplepapershavebeenauthoredonthetopic,generallyfocusingon

implementationsforpopularboardgameslikeChessorGo.GamesPlane’simplementation

provesitselfuniquethroughitsabilitytobegeneralizedtomanyboardgames,rapidresponsetime,andvaluemovedisplays.

3.1:Makingrealgamesvirtual:Trackingboardgamepieces[1]

ThisstudentprojectfromUSCSmakesuseofRANSACandHiddenMarkovModelstofindthegridofaGoboard,andthendeterminethecolorsofpiecesonthejunctionsbetween

spaces.ThesystemisfunctionalonstandardunmarkedGostones,evenatoff-angles,andis

abletodetecttheentire19x19Goboard.Ithasgooddetectionaccuracyatabout91%.Its

accuracyisboostedusinganA*algorithmthatusespreviouslyknownboardstatestopredict

thelikelihoodofdetectedboardstates:ifastateisdetectedbutismanymovesawayfromtheinitialstate,itisdeemedlesslikelyandasimplerexplanationisused.Themaindrawbackof

thissystemisitstimetooperate,taking40secondstorun20iterationsofanA*algorithmperpicture.Italsodoesnothaveanyreal-timeboardoverlay,insteadonlyfocusingonrecordingthegameovertime.

3.2:AugmentedRealityChessAnalyzer(ARChessAnalyzer)[2]

ARChessAnalyzermakesuseofaConvolutionalNeuralNetwork(CNN)approach

alongsideanARoverlaytoshowrecommendedmovesoverlaidonimagesofchessboards.

TheirCNNapproachenablesthemtouseexisting,unmodifiedchessboardsandpieces.Theytouta93.45%accuracyinstaterecognition,whichisexcellentconsideringthatupto32piecesmustberecognizedandpositionedcorrectlyforaboardstatetobecorrect.Whencomparedwithsomeoftheotherrelatedworks,thisaccuracyisespeciallyimpressive.However,the

complexcalculationsinvolvedinrunningaCNNresultsina3-4.5secondwaitingperiod

betweentakingapictureofthegameandseeinganARoverlay.Consideringthataugmentedrealityreliesonrealisticallysuperimposingthedigitalworldovertherealworld,thisdelay

becomesanotabledrawbackasitbreakstheappearanceofthedigitalobjectsappearing“inreallife.”

3.3:AnIntelligentChessPieceDetectionTool[3]

ThispaperusesCNNstolocateandcategorizechesspieces,withthegoalofcreatingaboardstate.Thepaperfocusesprimarilyonthemethoditusestocategorizepieces:aYOLO

objectdetectionalgorithm.YOLOmeans“YouOnlyLookOnce,”andreferstoanalgorithmthatrunstheimagethroughitsnetworkasingletime.BeforeYOLO,approachessuchasR-CNN

wereusedwhichwouldoftenneedtopropagateasingleimagethroughanetworkthousandsoftimes.WithYOLO,fasterorreal-timeobjectdetectionbecomespossible.Foundchesspiecesaredisplayedonaseparatechessboardtothesideoftheimageofthegameboard.Ithas

acceptableaccuracyper-pieceat84.29percentcorrectinthebestcase,however,fullboardstatedetectioncanbeerroneousconsideringupto32piecescanbeonachessboardatonce.ItrequirestheuseofacustomCNN,whichtakesbetween2and12hourstotrainand3to21secondstorun.Consideringthatitalsorequiresamassivedataset(about140,000picturesinthecaseofchess)totrain,itbecomesveryinefficienttoconvertthisCNN-basedapproachtoothergames.

11

3.4:Chessboardandchesspiecerecognitionwiththesupportofneuralnetworks[4]

ThispaperfromtheInstituteofComputingScienceatPoznanUniversityofTechnologyusesanovellatticedetectortofindachessboardinanimage,andthenasupportvector

machineandaconvolutionalneuralnetworktolocatethepieces.Theymakeuseofachess

enginetodeterminewhatboardpositionsaremostlikelytoincreaseaccuracy(e.g.havingthreewhitebishopsisveryuncommon).Withthisapproach,theyachieveanastounding99.57%

accuracyinchessboarddetectionand95%accuracyinpiecedetection.Drawbacksoftheirapproachincludethefactthatitdoesnottransferwelltoothergames,requirealattice-shapedboard,anditsexecutiontime,whichsometimesreaches4.5secondsforasingleframe.

3.5:ChessPieceDetection[5]

ThisapproachusesaYOLOCNNtodetectthespacesofachessboardaswellasthepiecesonit.Generally,itsapproachissplitintothreesteps:a“boarddetection”stepwheretheedgesoftheboardarelocated,a“griddetection”stepwherethespacesoftheboardare

delineated,anda“chesspiecedetection”stepwhereindividualpiecesineachspaceare

recognized.Itleveragesthefactthatchessboardshavealternatinglightanddarkcolorsto

detectavarietyofboardsusingOpenCV.AswiththeotherCNNapproaches,themajor

drawbackinthiscaseisthatitreliesonaspecificallytrainedCNNdesignedforchesspieces,

andsoconversiontoothergamesisacostlyaffair.Additionally,itsuseofcontourtracingmeanstheapproachcanonlypracticallyfunctiononchessboardsorothersquareboards/

3.6:ComparisontoGamesPlane

Consideringpreviousworkinthefieldofgameboardrecognitionandmovedisplay,GamesPlanedifferentiatesitselfinafewcriticalways.

Generality:GamesPlanecanfunctionwithpracticallyanygame,notjustChessand/orGo.EvengamesthatarenotcompatiblewithGamesmancanstillhavetheirboardstates

extracted.Critically,thisappliesevenforgameswithirregularboards:anyboardshape(e.g.hexagonal,triangular,orcompletelyirregular)canfunctionwithGamesPlane.Pieceswithflattops(likecheckers,disks,ortiles)workbestwithGamesPlane,butothershapescanworkaswellaslongasArUcosareflatwhenattached.

Accuracy:ThankstoitsArUco-basedapproach,GamesPlanehasexcellentaccuracyatavarietyofangles,whileothersystemstendtofocusonfunctioningatonespecificangle.Withthecamerapositionedfromthetopdown,itsrecognitionaccuracyisessentiallyperfect.

Accuracyisextremelyimportantinthecaseofstrategygames,asevenasinglemisplacedpiececancompletelyalterwhatthebestmoveis.

Speed:GamesPlanerunsatahighframerate,around25framespersecond.

Additionally,thereisnovisibledelaybetweenreallifeandthevideodisplay,evenonthe

low-endlaptopusedfortestingtheproject.Ahigher-endcomputercouldhypotheticallyachieveevenhigherframespersecond.ItonlyhastoqueryGamesmanUnioncetodisplayARvaluemoves,however,sothisisstillgenerallyfasterthantheCNN-basedapproaches.Additionally,GamesPlanecancacheitsoverlays,leadingtoinstantresponsetimes.

AROverlay:ByusingGamesmanUni’soverlays,GamesPlanegainsaccesstothe

interfaceofeverygameimplementedinGamesmanUniwithoutneedingtowriteadditional

interfacecode.Thatis,wedon’thavetore-drawtheinterface’sarrowsforslidingmovesanddotsforplacementmoves.Weinsteadusethegraphicdrawnbytheexistingsystemand

overlaythat,leveragingabstractionanda“don’trepeatyourself”softwarephilosophy.

Additionally,becauseGamesmansolvesgamesfully,ouroverlayshowsperfectvaluemoves,notalgorithmicorAIsuggestions.

12

AdownsideofoursystemisthattheusermustprepareaGamesPlaneboardwithArUcomarkers.However,withapre-madePDFreadytoprint,eventhisprocesstakesnomorethanafewminutespergame.ExistinggameboardscanalsobeconvertedtoGamesPlane

compatibilitybyaddingArUcoanchormarkers.GamesPlanethereforemakesaworthwhiletradeoffthatresultsinavarietyofbenefitsvaluabletoboardgameplayers.

13

3.7:SystemComparisonChart

System

ApplicableGames

State

RecognitionAccuracy

OperationSpeed

ValueMovesDisplay

Underlying

Technology

Makingreal

gamesvirtual:

Trackingboardgamepieces

Go

90.57%

40seconds

perframe

None

RANSAC,MarkovModels,A*

Augmented

RealityChess

Analyzer

(ARChessAnalyzer)

Chess

93.45%

4.5secondsperframe

Displayschessenginemoves

ConvolutionalNeuralNetwork

AnIntelligent

ChessPiece

DetectionTool

Chess

84.29%perpiece

3-4secondsperframe

Separateboarddisplay

ConvolutionalNeuralNetwork

Chessboardandchesspiece

recognitionwiththesupportofneuralnetworks

Chess

95%

4.57secondsperframe

Nodisplay;onlylocatespieces.

SupportVector

Machine,

ConvolutionalNeuralNetwork

Chesspiecedetection

Chess

81%

Real-Time

Nodisplay;onlylocatespieces.

ConvolutionalNeuralNetwork

GamesPlane

Any

Gamesman

Game

97-100%

Real-Time

Displaysvaluemovesin

augmentedreality

ArUcowithOpenCV,Gamesman

14

Chapter4:DevelopmentandChallenges

4.1:ConceptualizationofGamesPlane

Whencreatingtheinitialconceptforthisproject,IknewIwouldneedaboardthatwouldprovidesomekindofvisualanchorthatacameracoulddetect.I’dseenAprilTags[11]fromrobotics

applicationsbefore,andsodecidedtouseArUcotagsasthey’reintegratedintoOpenCV[12].PythonisalanguagealreadywidelyusedforvariousapplicationsinGamesCrafters,sousingPythonwithOpenCVwasanaturalapproachasitwouldensurefutureGamesCrafters

memberswouldbeabletoreadthecode.

4.2:FirstIteration

GamesPlaneitselfbeganasasingleobject,called“TheGamesPlane.”Thiswasawooden

board,aboutafoottoaside,featuringa5-by-5gridwith2-inchwidesquares(seeFigure4.1).ItiscompatiblewithgamessuchasTic-Tac-Toe,4x4Othello,Chung-Toi,orabout10others

supportedbyGamesman.Theboardhadspecialslotsinwhich5ArUcomarkerscouldbeplaced.IassumedthattheArUcotrackingwouldbeveryerroneous,andthathavingmoremarkerswouldessentiallyallowformultiplesamples.

Figure4.1.TheoriginalGamesPlane:Awoodenboard,intendedtobetheonlyboard

compatiblewiththesystem.

Forthepieces,I3D-printedcustompieces.AsseeninFigure4.2,thesewerestyledintheshapeofarook,butshorter.MythoughtprocesswasthathavingtallpieceswouldobscuretheArUcomarkerspastedontotheboardwhenviewedfromanangle,soshorterpieceswerepreferable.Atthetime,IwasunsureifIwouldbeplacingArUcomarkersontothepiecesorifIwouldbeusingsomekindofimagedetection,andsoIkeptthetopsflattoallowspaceforanArUcomarkertobeattached.Iadditionallyaddedridgesontotheedgeinthehopethatitwouldmakethepiece’sshapemoredefined,makingiteasiertodetectwithanyimagedetection

methodImightuse.

15

Figure4.2TheoriginalGamesPlanepiece,meantforusewithHaarCascades[6]asexplained

below.Itwasmeanttoevoketheshapeofachessrook.

ThefirstpipelineIenvisionedfortheGamesPlanewasasfollows:theArUcomarkers

wouldhavehardcodedreal-lifepositions.Iwouldthenuseanobjectdetectionmethodcalled

HaarCascades[6]tolocatethepiecesintheframe,andusetheirpositionsintheframe

comparedtothatoftheArUcostoestimatetheirreal-worldposition.ThereasonforthisdecisionwasthatIwantedthegamepiecestolooklikenormalgamepieces—Ithoughtthatadding

ArUcomarkersontopofthemwouldmakethemlooktoodifferentfromtraditionalboardgamepieces.

4.3:TransitiontoArUco-OnlyApproach

Unfortunately,thisinitialapproachworkedverypoorly.ThemainissuewaswiththeHaar

Cascades.Inordertodetectanewobject,it’snecessarytotrainanewcascadeusingimagescontainingtheobject(positiveexamples)andimagesnotcontainingtheobject(negative

examples).Ithereforetookabout250picturesofthepiecearoundmyapartment,andused

imagesfromtheinternetasnegativeexamples.Thisfailedtoadequatelytrainthecascade:itendedupwithafalse-positiverateofabout70%onimagesnotcontainingthepiece.TheissuewasdowntothevarietyofimagesIfedin.Bytakingmanysimilarpictures,Iinadvertently

trainedthecascadetodetectafewmuchsimplerfeatures:specifically,ashadowthatappearedontheedgeofmytableandanothershadowthatappearedalongsidethewall(seeFigure4.3).Withhowmanysituationscreateshadowssimilartothese,itwouldregularlyfindthepiecein

locationsitwasnot.Atthispoint,thetwowaysforwardwereeithertotakemuchmorevariedpicturesofthepieceinmanycontextsortosimplychangetoanall-ArUcoapproach.

Consideringthescopeoftheproject,itseemedmoresensibletoswitchtorelyingonArUcosentirely.

16

Figure4.3.TheHaar-basedapproachhadverylowaccuracy,andwouldusuallydetecta

randomcrackinthetable.ThegreensquareiswheretheHaarclassifierthinksthepieceis—

it’sactuallyinmyhand.

IbeganbyattachingArUcomarkerstothetopofthe3Dprintedpieces.However,therewasanissuehereaswell—ArUcomarkersneedwhitespacearoundtheiredges.When

detectingamarker,contrastbetweenwhiteandblackisused,andsoifthemarkerdoesnot

haveawhiteborderitfailstodetectit.Thetopsofthepiecesweretoosmall,soplacinga

markerwithaborderontopofthemmadethemarkersinvisibletothecamerafromevenashortdistance.Torectifythis,IstoppedusingthepiecesandstartedusingtheArUcomarkersontheirownwithnopiecesupportingthem—thisway,atleastwhiletesting,theywouldbelargerandmorevisible.

Atthispoint,withasolelyArUco-basedapproach,IwentaboutimplementingOpenCV’sArUcodetection.Thechallengeitpresentedwasoneofcoordinatespaces.MyfinalgoalwastoextractwhatIcall“boardcoordinates”:whereagamepieceisonaboard(e.g.f4inchess).Togetthesecoordinates,I’dneedtofirstobtainboard-centeredworldcoordinates,whichare(x,y,z)coordinatesforwhereagamepieceisinspacealignedtothegameboard’sframeof

reference.Thesewouldthencomefromcamera-centeredworldcoordinates,whichcanbe

obtainedbyestimatingtherelativepositionandposeofanArUcomarkertoacamerabyusinganimagethecamerahastaken.Convertingbetweenthesefourcoordinatespaces,and

displayinginformationfromeachinasensibleway,madeupmuchoftheworkoftheproject.Smallmathematicalerrorsweredifficulttodetectbuthadamassiveeffectontheaccuracyofpiecelocation.

Afterresolvingtheseconversions,Ifinallywasabletolocateapiece,butinaccurately.

Thedetectionwouldregularlyplaceapieceseveralspacesawayfromitstruelocation.In

strategygameswhereasinglespacecanmakeamassivedifferenceinthebestmove,thiswasexpectedlyunacceptable.Thebreakthroughinraisingaccuracycamewhendiscussingthe

camera’scalibrationfile.Acalibrationfilecontainstheinformationaboutacamera’sintrinsic

properties,specificallyitsoutputimagesizeandfocallength.Analyzingthefilerevealedthatthecamerahadbeencalibratedverypoorly,possiblyduetoanissueintheGitHubrepositoryIusedto