1、1stanford cs223b computer vision, winter 2006lecture 7 optical flowprofessor sebastian thruncas: dan maynes-aminzade, mitul saha, greg corradoslides by: gary bradski, intel research and stanford sail23optical flow: outline examples formal definition, 1d case from 1d to 2d: aperture problem course mo

2、tion and pyramids flow segmentation4* picture from khurram hassan-shafique cap5415 computer vision 2003optical flow5optical flow, example6another example in robotics1) rich local sensor data determines drivability2) use optical flow for near-far association3) associate known driveability with visual

3、ly similar regions7optical flow, exampleharris cornersdavid stavens, andrew lookingbill, david lieb (cs223b 2004)8optical flow, exampledavid stavens, andrew lookingbill, david lieb (cs223b 2004)optical flow9optical flow, exampledavid stavens, andrew lookingbill, david lieb (cs223b 2004)“reward”10opt

4、ical flow, exampledavid stavens, andrew lookingbill, david lieb (cs223b 2004)particle filter11optical flow, exampledavid stavens, andrew lookingbill, david lieb (cs223b 2004)result12conference paperdavid stavens, andrew lookingbill, david lieb (cs223b 2004)13optical flow: outline examples formal def

5、inition, 1d case from 1d to 2d: aperture problem course motion and pyramids flow segmentation14optical flowimage sequence(single camera)tracked sequence3d structure+ 3d trajectoryimage tracking3d computation15) 1( tiwhat is optical flow?),(ipti1p2p3p4p1v2v3v4viv) 1,(),(tvpitpiiii16* from marc pollef

6、eys comp 256 2003optical flow break down?17optical flow assumptions:brightness constancy* slide from michael black, cs143 200318optical flow assumptions: * slide from michael black, cs143 200319optical flow assumptions: * slide from michael black, cs143 200320optical flow: 1d case brightness constan

7、cy assumption:),(),()(dttdttxittxitf0)(txttitxxiixvitxtiiv 0)(txfbecause no change in brightness with time21vtracking in the 1d case:x),(txi) 1,(txip22vxititracking in the 1d case:x),(txi) 1,(txiptxxiipxttiixtiiv23tracking in the 1d case:x),(txi) 1,(txipxitixtpreviousiivv24algorithm for 1d tracking:

8、xtiivvppxi0v) 1,() 1,( tvxitxi),() 1,( tpitpiitvxixi25optical flow: outline examples formal definition, 1d case from 1d to 2d: aperture problem course motion and pyramids flow segmentation26from 1d to 2d tracking0)(txttitxxi1d:0)(txtttityyitxxi2d:0)(txtttivyiuxishoot! one equation, two velocity (u,v

9、) unknowns27from 1d to 2d tracking* slide from michael black, cs143 2003we get at most “normal flow” with one point we can only detect movement perpendicular to the brightness gradient. solution is to take a patch of pixelsaround the pixel of interest. 28how does this show up visually?known as the “

10、aperture problem”29aperture problem exposedmotion along just an edge is ambiguous30aperture problem: example31aperture problem in real life32from 1d to 2d trackingvx),(txi) 1,(txi1v2v3v4vxy) 1,(tyxi),(tyxixtiivbgv1pyyxyxxiiiiiig aroundwindow 22ptytxiiiib aroundwindow 33more detail:solving the apertu

11、re problemhow to get more equations for a pixel? basic idea: impose additional constraints most common is to assume that the flow field is smooth locally one method: pretend the pixels neighbors have the same (u,v) if we use a 5x5 window, that gives us 25 equations per pixel!* from khurram hassan-sh

12、afique cap5415 computer vision 200334lukas-kanade flowprob: we have more equations than unknowns the summations are over all pixels in the k x k window this technique was first proposed by lukas & kanade (1981) described in trucco & verri readingsolution: solve least squares problem minimum

13、least squares solution given by solution (in d) of:* from khurram hassan-shafique cap5415 computer vision 200335conditions for solvability optimal (u, v) satisfies lucas-kanade equationwhen is this solvable? ata should be invertible ata should not be too small due to noise eigenvalues l1 and l2 of a

14、ta should not be too small ata should be well-conditioned l1/ l2 should not be too large (l1 = larger eigenvalue)* from khurram hassan-shafique cap5415 computer vision 200336eigenvectors of ata gradients along edge all point the same direction gradients away from edge have small magnitude is an eige

15、nvector with eigenvalue whats the other eigenvector of ata? let n be perpendicular to n is the second eigenvector with eigenvalue 0the eigenvectors of ata relate to edge direction and magnitude suppose (x,y) is on an edge. what is ata?* from khurram hassan-shafique cap5415 computer vision 200337edge

16、 large gradients, all the same large l1, small l2* from khurram hassan-shafique cap5415 computer vision 200338low texture region gradients have small magnitude small l1, small l2* from khurram hassan-shafique cap5415 computer vision 200339high textured region gradients are different, large magnitude

17、s large l1, large l2* from khurram hassan-shafique cap5415 computer vision 200340observation this is a two image problem but can measure sensitivity by just looking at one of the images! this tells us which pixels are easy to track, which are hard very useful later on when we do feature tracking. on

18、ce suggestion: track harris corners!* from khurram hassan-shafique cap5415 computer vision 200341optical flow, exampleharris cornersdavid stavens, andrew lookingbill, david lieb (cs223b 2004)42optical flow, exampledavid stavens, andrew lookingbill, david lieb (cs223b 2004)optical flow43optical flow:

19、 outline examples formal definition, 1d case from 1d to 2d: aperture problem course motion and pyramids flow segmentation44revisiting the small motion assumptionis this motion small enough? probably notits much larger than one pixel (2nd order terms dominate) how might we solve this problem?* from k

20、hurram hassan-shafique cap5415 computer vision 200345reduce the resolution!* from khurram hassan-shafique cap5415 computer vision 200346image it-1image igaussian pyramid of image it-1gaussian pyramid of image iimage iimage it-1u=10 pixelsu=5 pixelsu=2.5 pixelsu=1.25 pixelscoarse-to-fine optical flow

21、 estimation47image iimage jgaussian pyramid of image it-1gaussian pyramid of image iimage iimage it-1coarse-to-fine optical flow estimationrun iterative l-krun iterative l-kwarp & upsample.48multi-resolution lucas kanade algorithm49optical flow results* from khurram hassan-shafique cap5415 compu

22、ter vision 200350optical flow results* from khurram hassan-shafique cap5415 computer vision 200351optical flow: outline examples formal definition, 1d case from 1d to 2d: aperture problem course motion and pyramids flow segmentation52problem multiple motion types in the image (e.g., moving cars on 280 or el camino real)53motion template idea54 stamp the current motion history template with the system time and overl