Makingbigdataworkforcommodities

FromautomationtoAI,andcodingwithPythonto

democratiseddatasets,commoditytradingisundergoingadigitaltransformation

TheExxonValdezranagroundonMarch24,1989.Asitdidso,250,000barrelsofcrudeoilgushedfromthedamagedshipandimpactedtheAlaskashoreline.

Theaccidentwasanenvironmental,legalandreputationaldisaster–butitalsosetoffachainofeventsthatwouldupendtheglobalshippingindustryforever.

Introduction

Analysisiskeytocreatinginsight–andthere’smoretoanalysethanever

Theyearafterthevesselcrashed,politiciansintheUnitedStatespassedlawsaimingtopreventsimilarsignificantoilspills.Withintheselawswastheobligationthatsometankersshouldusevesselmonitoringandtrackingtechnology.Ifsuchshipscouldbetrackedelectronically,theargumentwent,therewouldbegreatertransparencyabouttheirmovements,alowerchanceofcollisions,andportscouldgainabetterunderstandofaship’sactions.Similarsystemsweresimultaneouslybeingdevelopedandtestedbyotherauthoritiesaroundtheworld.

By2000,everythinghadchanged.TheInternationalMaritimeOrganisation(IMO)adoptedanewrequirementforallvesselsoveracertainsizetouseautomaticidentificationsystemtransponders,knownasAIS.Thetechnology,whoseadoptionbecameeffectivein2004,workslikeGPStrackingforships,andincludesavessel’sidentity,location,speed,directionoftravelandplanneddestination.

Now,AISdatapowerswebsitesthattrackshipsallaroundtheworld,andtheinformationisfedintothecommercialsystemsofmultipleindustries–includingthoseoftradersandcommoditiestraders.Theradicalimprovementsintechnologyhaveenabledthetrackingofshipsinreal-timetobecomeroutine.Forcommoditiestraders,theavailabilityofthisinformationopensupthepotentialtomakesmarter,data-informeddecisionsaboutthemovementofgoods–fromcottontooilandmetal–aroundtheworld.Theavailabilityofthistypeofinformation,amongavarietyofsimilarsources,isincreasinglybecomingthecornerstoneoffundamentalstrading,whichseekstobuildapictureoftheworldusingdata.

AISisonlyonedatasource–marinerscan,ofcourse,turnofftheirAISbroadcaststoevadetracking,buttherearenowfewerwaystohidethaneverbefore.Theintroductionoflowcostsatellitesandhigh-resolutioncamerasallowstheEarth(andseas)tobephotographedatalltimes.“Youcanbeobservedbysatelliteimagery,”saysLeighHenson,theglobalheadofcommoditiestradingdivision,atRefinitiv.“Andevenifthere’scloudcover,theycanlookatyourlocationusingradar”.

Machinelearningisevenmakingitpossibletocombinethesedatasourcesandautomaticallywatchthemovementofvesselsovertime.Thesedatasciencetechniqueshavecaughtshipsillegallyfishing,smugglinggoodsandbreakinginternationallaws.Mostrecently,shipshavebeenfoundtamperingwithAISsignalsandbroadcastingfalseones;combiningdataandmachinelearninghasallowedthemtobecaughtintheact.

Theexplosionofdataisnotuniquetoshipping.Acrossallcommoditiestherehasbeenanincreaseintheavailabilityofdataaboutwhatishappeningintherealworld.Cheapsensors,ubiquitousinternetconnectionsandreal-timemonitoringmakeitpossibletotrackandquantifyvastswathesofthecommoditiesindustry.Oilstoragetankscanbemonitored.Methaneemissionscanbetracked.Theflowsofoilpipelinescanbewatched.“Increasinglysophisticatedtypesofobservationsarejustsproutingupallthetime,”Hensonsays.

Asthispictureoftheworldhasgrowninscope,sotoohasfundamentalstradingbecomeevermoresophisticated.Itisnowbasedonthousandsofdailynewsreports,billionsofdatapointsfromvastnumbersofsensors,almost-instantaneousupdatesfromfinancialmarketsandtheproprietarydataheldbycommoditiesfirms.Increasingly,thatsamedataisbeingprovidedbymultiplesources;themeasurementsofoneoilsilocanbetakenbymultiplecompanieswhoprovidetheirdatatothemarket.

Butsimplyamassingdataisnotenoughtogivetradersanadvantageoverthecompetition.Traderswhohavebeenequippedwiththerightanalyticaltools,includingtheabilitytoautomateresearchandvisualisegoodsbeingmoved,areabletounderstandtheworldinamuchmoregranularway.Beingabletoeffectivelyutilisethisknowledgewillbecomeincreasinglycriticalduringthecomingyears.Astechnologymakestradingmoreautomated,everysecondmatters.Companiesneedtobeabletoquicklyanalyseandvisualisethedatatheyhave,whilestaffneedtheskillstooperateanalyticaltoolsandgaininsights.Andthewholeprocessstartswithdatamanagement.

Thise-book,whichispartofafour-partseries,outlinesthekeydatapracticesandtechniquesthattradersandanalystsshouldbeputtinginplacetogainanedge.Itexploreshowthecommoditiesindustryhasbeenrevolutionisedbydataandthestepsneededtoingest,standardise,analyseandvisualisedata.Withouttheseprocessesinplace–eitherinternallyorbeingprovidedbythird-parties–tradingcompaniesarelikelytofallbehindthecompetitioninanincreasinglydigitisedworld.

Meetingthechallengeofanever-growingdatadeluge

Withannualinputsnowbeingmeasuredinzettabytes,there’smoreinformationavailablethanhumantraderscanrealisticallymanage

Theageofdatahasfundamentallychangedhowcommoditiestradingworks.Tento15yearsago,saysAlessandroSanos,globaldirector,salesstrategyandexecutionforcommoditiesatRefinitiv,traderswouldbasetheirdecisionsaroundexclusiveaccesstoinformation.“Thiswaseitherbecausetheyheldthephysicalassets–mines,plantations,infrastructure–orbecausetheyhadaccesstoinformationthroughanetworkofcontacts,orbootsontheground,”Sanossays.Whenitwastimefortradestobemade,theypickedupthephone.

Now,itisdataandtechnologythatdrivedecisionsandtransactions.Inallareasoflife,colossalamountsofdataarebeingproducedeverysingleday.AnalysisfirmIDChaspredictedthat,by2025,theworldwillbecreating163zettabytesofdataeveryyear,agiganticleapupfromforecastsof33zettabytesofdatabeingcreatedin2018.Toputitintophysicalterms,ifallthedatastoredbyRefinitivalonewasburnedontoDVDs,andthediscsstackedontopofeachother,theywouldtowerthreetimeshigherthanLondon’s310-metretallShardskyscraper.

“Dataisarguablythenewcurrency,”saysHilaryTill,theSolichscholarattheUniversityofColoradoDenver’sJ.P.MorganCenterforCommodities.“Tradersareattemptingtoacquireevermorenoveldatasets,especiallyinnichemarkets”.Thismassiveavailabilityofdatahasessentiallyledtoitsdemocratisation–itiseasierthaneverbeforeforcompaniestogatherdataaboutthemovementofcommodities.“Knowledgeisthemostvaluablecommodity,”Sanossays.“Today,thereareliterallythousandsofsourcesofdataavailable.Thosecompaniesabletoassimilatethisinformationtsunamianddetectthesignalfromthenoisewillemergeasthefutureleaders.”

Broadly,therearefourdifferenttypesofexternaldataproviderswhichcommoditiestradingspecialiststurnto.Eachprovidesadifferentkindofinformation,andinsomecases,maybetheonlysourceof

thatdata.Thefirstarethegovernmentagenciesthatcontroltheflowofinformationfromtheirlocalities.Secondly,therearecommercialorganisationsandbigdataprovidersthatholdcertaindatasets,suchastheweatherornewsdata,instructuredways.Thirdly,thereareinternationalmonitoringorganisationssuchastheInternationalMaritimeOrganisation,thebodythatintroducedAISdatatoshipping.Finally,therearetheexchangesthatdealwithtradingdata.Someofthesedatasourcesareavailableforfreeonline,butothersrequiresubscriptionsandcommercialdeals.

“Inonesense,dataismoredifficulttocomeby,sincedatavendorsfullyappreciateitsvalue,”Tillexplains.“However,inanothersense,dataiseasiertocomebyinthatsomedatasetssimplydidnotexistinthepast–atleastcommercially–thatdosonow.”Collectingallthisdataandmakingitavailabletopeoplehasclearadvantages.Byprovidingdetailedmarketinformationalongsidefundamentaldata,Refinitivisabletobringgreatertransparencytothemarket.Ithasneverbeeneasierforcompaniestounderstandwhereandhowcommoditiesarebeingmovedandsold.

MichaelAdjemian,anassociateprofessorofagriculturalandappliedeconomicsattheUniversityofGeorgia,saysdatacollectionwithinagricultureischangingboththeagriculturalindustryandtheworldoftrading.“Satellitedataonweatherandproductionpatterns,aswellasdatageneratedbyGPS-equippedtractors,dronesandsoil-sensingapplications,areregularlyintegratedintoreal-timedashboardstoimprovetheinformationavailabletodecisionmakersinthefinancialeconomies,”hesays.Adjemianaddsthatthiswillbecrucialgivengrowingworldwidedemandsforfoodandthechallengeoftheclimatecrisis.

Butcollectingmassesofdatacausesitsownchallenges.Thevolumeofdatahasbecomeincomprehensibleforhumans,andbeingabletomakethemostofitrequiresbeingabletoeffectivelyanalyseit.It’snowcrucialthatdatacanworkwithotherdata.“Thecompetitiveadvantageisreallyevolvingfrombeingabletoaccessthoseadditionalsourcesofdata,tohowwellcompaniescanintegrateit,commingleitwiththedatathattheyalreadyproducethemselves,andthenapplytechnologytogenerateinsights,”Refinitiv’sSanossays.

Thechallenge:findinginsightsindatafromawidevarietyofsources

Preparingforsuccess:gettingyourdatareadyforbusiness

Withthepropertools,datafrommultiplesourcescanbestandardisedandmerged,pavingtheawayforautomationandartificialintelligence

Commoditiestradinghasbeentransformedinthelasttwodecades–butthereisevengreaterchangecoming.Artificialintelligence,sophisticatedanalyticstoolsandvastlyimprovedvisualisationmethodspointtoafuturethatincludesincreasedautomation–butthisdoesn’tnecessarilymeanreplacinghumantradersandanalysts.Automationcanincludefarmoreeffectivedataprocessingandasophisticatedapplicationofavailableinformation.Essentially,machinescanminedataforintelligence;humanscanthenactuponit.

Buttheuseofdatacan’tbesuperchargedifitsfundamentalcomponentsaremissing.Companiesandtradersneedtoproperlyingestdataintotheirsystems,makesureitisstandardised,usedatasciencetoanalyseit,andunderstandthetoolsneededtovisualisethisanalysisandmakeitunderstandableforpeopleontheground.

Thetaskofpreparingdataforthefutureisn’taneasyone,Sanossays.Itis,however,achallengethatRefintivhasasolutionfor.Refinitiv’sDataManagementSolution(RDMS)isaplatformthatallowscompaniestocombineRefinitiv’smultiplesourcesofcommoditydataintotheirownprocesses.Datacanbemergedandnormalised,anditactsasawayforclientstostandardisedatafromRefinitivalongsidethird-partydataandtheirowninformation.

Justhavingthedataisn’tenough:italsoneedstobeoptimised

13

Disparatedataunited

Combiningmultiple,continuouslyupdatedsourcesintovastdatabasescanyieldexceptionalinsights–iftheycanbenavigatedandparsed

Thedataexplosionhascreatednewchallengesforcompanies.“Youalsoneedsomewheretoputitall,”saysRefinitiv’sHenson.Companiesneedtomakethemostofthedatatheyarecollecting–oraccessingfromthirdpartiessuchasRefinitiv–andneedtocreatesuitableenvironmentswhereitcanbecollectedandaggregated.However,Hensonexplains,increasinglycompaniesarelookingtoaccessandmanipulatedataremotely,addingthattradersandanalystswanttoconnecttodatafeeds–andtheningestitintoevenbiggerdatabases.

It’sherewherethecloud’squickstartuptimesandseamlessabilitytoexpandasrequiredcomeintoplay.BytakingadvantageofcloudhostingandmakingdataavailableinApplicationProgrammingInterfaces(APIs),itispossiblefortraderstoeasilyaccesshugedatastreamsandusetheminthewaystheydesire.Forinstance,Refinitiv’sRDMSallowsanalyststocreatewaystoseehowmuchoilisbeingmovedfromonelocationtoanother.HensonalsoexplainsthatRefinitivprovidesdataenvironmentsinitsterminalproduct,Eikon,whichholdsmorethan2,000pricingdatasources,with1,300providerssendingreports.Itcanalsobeusedinvirtualofficestoaidremoteuse.ThecompanyestimatesthatEikonDatacanbeaccessedandsharedeasily,andsubsequentlyactedupon.Thisdatacanthenbeaccessedandsharedeasily,andsubsequentlyactedupon.

Combiningdatastreamsallowstraderstobuildcustomdatabases

Buildinginsightsonstandardiseddata

Normaliseddatasetsarethefoundationsofinformeddecision-making

Makingonesetofdataworkwithothersetsofdataisn’taneasytask.Asanexample,datamustfirstbenormalised,andfieldsinonedatabaseneedtomatchthoseinanotherdatabaseiftheyaregoingtobesuccessfullycombined.Ifthatdoesn’thappen,theresultmaybeaninaccuratepictureoftheworld–anditmaysubsequentlybeharderfortraderstomakeaccuratedecisionsasaresult.

Sanossaysthechieftechnologyofficershespeakstoarefrustratedabouttheamountoftimetheirdatascientistsandstaffarespendingcleaningupdatasetsinordertooptimisetheirusefulness.“They’retellingmetheiranalystsspendupto90percentoftheirtimejustdoingthisaggregationandnormalisationofdata,”heexplains.Spendinglongermakingdatacompatiblemeanslesstimeisspentanalysingthedataandmakingdecisionsbasedontheintelligenceitprovides.Formanycompanies,theprocessofamassingdatasourcesandstandardisingtheinformationisbestlefttothirdparties.However,Refinitivhasdedicatedteams–locatedinBangalore,Manila,IndiaandPoland–thatanalyseincomingdatasetsandensuretheyarecompatible.Theytakedisparatesourcesofdataandmakeitpossibleforthemtobeeasilycomparedtootherdata.

Fortraders,takinginalreadystandardiseddatasets(throughAPIsordesktopsoftware)thatarereadytobemixedwithproprietaryorthird-partydatacanvastlycutdownontheamountoftimeneededtomaketradingdecisions.“Previously,ourcustomersmayhavebeenhappywithjustonesourceofoilflowsandcargotracking,”explainsSimonWilson,headofoiltradingatRefinitiv.“Now,theywanttotakemultiplesources.”Headds:“Weintegratetwoorthreesourcessothattheycangetablendedviewofthecargotracking.”Thisapproachcanhelptradersmakemoreinformeddecisions.

Oncedatahasbeeningestedandnormalised,itcanthenbepassedthroughanalyticstools,visualisationprocessesand–increasingly–artificialintelligenceandmachinelearningsystems.

Accuratetrackingofcargoneedsdatathathasbeenstandardised

13

Codingiskeyforanalytics

Beingabletointerrogatedataisavitalskillfortraders,andPythonisthetoolofchoice,offeringversatilityandeaseofuseforidentifyinginsights

GonearethedaysofsolelyusingMicrosoftExcelforanalysis–thedatasetsandsourcesavailabletotradersarenowtoolargeforthespreadsheetsoftware,andsonewertoolsareneededtoeffectivelyanalysebigdata.ThecodinglanguagePython,whichisusedacrosswebdevelopmentandanumberofplatforms,isincreasinglybecomingthesystemofchoiceforanalystssiftingtheirwaythroughdataintheworldoffinancialservices.It’soutstrippingbothJavaandC++asthego-tocodinglanguage.

Python’sgrowthisdowntotheeaseofwritingandthehugeamountofpre-writtencode,availablethroughdatasciencelibraries,thatisaccessibleonline.ThelanguageisalsoflexibleenoughtoenablecodetoberuninthecloudorputintoAPIs,andit’soneofthecornerstonesofemergingmachinelearningandartificialintelligencetools.Itisincreasinglybeingusedtomodelfinancialmarketsandhandlethevastamountofdatathat’sgathered.

TheriseofPythonischangingtheskillsetsoftradingworkforces.Thosewhocancodeareindemandandarebeinghiredbyfinancialservicecompanies–numerouslargebanksareutilisingPython-basedinfrastructurefortheirwork.Infact,somepredictthatmanycommoditiestradinghousesandfinancialserviceswillmostlybehiringonlythosewithPythonskillsinthecomingyears.

UsingPythonhelpsanalyststomanagelargervolumesofdata

Interact.Analysis.Action.

Smarterwaysofpresentingdatarevealsfarmorethanrawnumbersonaspreadsheetcould,meaninginsightsandforecastscanbebetterutilised

Thevalueofanalysingcommoditiesdatacomesfromtheinsightsitcanunlock.Increasinglypowerfulvisualisationtools,alongwithcustominstructioncodewritteninPython,arebeingusedtobringdatatolife.Bymakingtheresultsofanalysisvisual,it’spossiblefortraderstobetterunderstandwhatishappening.

Take,forexample,thetrackingofships.ThroughtheuseofAISdata,eachvesselatseacanbeaccuratelyplacedonmapsoftheworld.It’spossibletoclickonashipandsearchforitsdetails,seeitsposition,movementsanddestination.Divingdeeper,itisevenpossibletounderstandthecargothatitmaybecarryingandinferwhatitsmovementsmightmeanfortradingmarkets.

Butthisisjustthetipoftheiceberg.ToolssuchasPowerBI,TableauandothersareallowingstandardiseddatatohavePythonscriptsrunagainstitandproducenewresults.Thetoolsmakeitpossibletoseebiggerpatternswithinthedataandputthemintoachartforlegibility.Forinstance,aSankeydiagramcouldshowthemovementofindividualgradesofcrudeoilmovingfromWestAfricatoNorthernEurope.Datapresentedinthisfashionisofteneasiertocomprehendthannumbersinatable,andtheprocessallowscurrentandforecastviewsofthemarkettobecreated.Withbetterforecasts,it’sanothertoolthatmakesiteasierfortraderstomakeinformeddecisions.

Visualisationcanenabledeepinsightsfromdensedatasets

Tradingmovestowardsamoreautomated,technologicalfuture

Forbothestablishedcompaniesandrecentupstarts,data-empoweredautomationandvisualisationareanessentialcompetitiveedge

Evengreaterlevelsofdisruptionarecoming.Thelastdecadehasseenthefundamentalscommoditymarketbecomefloodedwiththesenewtypesofdata.Butthedataactsonlyasastartingpoint–whatcanbebuiltontopofitwillprovidenewopportunities.

Entermachinelearningandautomation:since2000,machinelearningtechniques,whichusesophisticatedalgorithmstoexaminedataandidentifypatternswithinit,haveimprovedexponentially,andtheywillonlycontinuetogrowmorepowerfulandadvanced.Muchliketheinfluxofdatafuellingthecommoditiesindustry,thissub-fieldofartificialintelligencehasbeendemocratised.Itispossibleforindividualsorbusinessestofindallthetoolstheyneedtorunmachinelearningapplicationsonlineand,often,forfree.

Throughoutthe2020s,theuseofmachinelearningwillincreaseacrossallindustries.Forcommodities,thiscouldmeanmoreautomatedprocessingofdataandpotentiallyevenchangingthewaythattradeshappen.“Ithinkwe’llseemorealgorithms,”Hensonsays.“We’llseemorecomputerstradinginsteadofpeople.It’sbeengrowingintheequitiesmarketsandit’sdefinitelybeengrowinginthecommoditiesworld,too.”AIcanstepintoperformtasksthataretoocomplexortootimeconsumingforhumans.

Thelevelofdisruptionthatwillactuallyhappenisdifficulttopredict.However,theultimategoalmaybeusingartificialintelligencetoaccuratelyforecastthetradingpriceofindividualcommodities.Itislikelyhumanswon’tbetakenentirelyoutoftheloop,butinsteadreceivegrea