OpenCV

OpenCV (Open Soure Computer Vision Library), at http://www.openv.org/, is an

extensive library of omputer vision and mahine learning algorithms. It implements

both lassi and state-of-the-artalgorithms, all highlyoptimized and easy to use. It is

released under the BSD liense, making it free for all. Some of the modules of interest

are:

ˆ OpenCV's HighGUI library provides easy to use routines for reating GUIs and

apturing images from ameras. This library is used for aessing the Gira's

amera.

ˆ OpenCV'sImage Proessinglibrary providesahost ofimage proessingand

anal-ysis routines. Of partiular interest here are the feature extrators.

ˆ OpenCV's Video Analysis library provides routines for motion analysis. Among

otherthings,ithas routinestoalulateoptialow, and even animplementation

of the Kalmanlter.

ˆ OpenCV's 3D Reonstrution library provides routies to alibrateameras,

om-pare stereo images, and alulate projetions and bakprojetions. It ould be

used toompensatefor the sheye eet of the wide angle lens.

ˆ OpenCV's2DFeaturesFrameworklibraryprovidesmoreadvanedfeature

extra-tors and pattern mathers.

ˆ OpenCV's Objet Detetion and Mahine Learning libraries provides many

ad-vaned mahine learningalgorithms.

SeveralbookshavebeenwrittenaboutOpenCV[28,29,30℄. Thislibraryisthebakbone

of many interesting projets, and so I hose it for this projet aswell.

4.1. Introdution

The Girais amobile telepresene robotdeveloped by GiraTehnologiesAB, Sweden

(http://www.gira.org/). It is designed tobe remoteontrolled by aregivers, allowing

them tohek up onare reipientswithoutphysially being there. Caregivers may use

their own omputers to onnet to any reipient's Gira robot, move it around using

their omputer's mouse, and see its environment and talkto people through the robot.

TheGiraisalreadyinvolvedinseveral otherresearhprojets. Theunit I'vehad

a-esstoisoperatedbyNST(NorwegianCentreforIntegratedCareandTelemediine)and

primarilyinvolved intheEU'sVitoryaHomeprojet,aprojetforputtingrobotsinthe

homesofarereipientstoat asproxiesforhumanaregiverswhenthey'renotpresent,

automatiallyalertingthemwheneverneeded. Forinformationabouttheprojet,see,for

example, http://www.itfunk.org/dos/prosjekter/AAL-VitoryaHome.htm. Some more

information about how the Gira, in partiular, is used in this projet is available at

http://www.robotdalen.se/en/News/Press-releases/2013/Gira-key-player-in-new-EU-projet-VitoryaHome-/

It is hoped that the Giraan beused tollroles suh as

ˆ Provide soial interation opportunities for people who live isolatedor that don't

get out of their houses muh for health reasons, suh as old age, COPD, or

dis-ability. Caregivers, family, and friends an simply log on to their omputers to

talk, without having to drive there. For aregivers, this saves valuable time and

allows them to eiently are for more people, whih may live all over a wide

area. Althoughthisan'tompletelyreplaethehumantouh,andpersonalvisits

will still be important from time to time, this an supplement them and greatly

inrease the eetiveness of resoure-starved health aredepartments,asthe need

for health are ontinues togrow faster than the resoures toprovide them.

ˆ Allow physiians to hek up on patients under their are that aren't in their

Figure4.1.: Photo of the Gira (frommaterialprovided by GiraTehnologies)

Figure 4.2.:Drawing of the Gira(from materialprovided by GiraTehnologies)

hospitals, suh as in elder are enters. To supplement the regular visits to the

are enters, the physiian may use the robot to talk to people and solve simple

problems without needing to drive thereevery time.

Unfortunately, the Gira's standard software provides little automation and an be

tedious to use, beause every movement it an do needs to be expliitly ommanded.

It ishoped that adding more automation and autonomy to the Giraan make its use

simpler, allowing the users to fous more on the tasks they want to aomplish, and

less on the ne details of steering the Gira around. It might even help save lives if it

ouldautonomouslyrespondtopersonsindistressandreportthesituationtoemergeny

personnel.

4.2. Design

As an be seen in Figures 4.1 and 4.2, the Gira has a base unit, a long nek, and a

head. The base unit houses a omputer, ontrol buttons, and motors for the 4 wheels.

The headisonnetedtoatiltablepanelwithamonitorandaamera. The totalheight

of the Gira is a little over 1.6m. When a aregiver is ommuniating with another

personthrough the robot,this allows omfortableinteration. The tiltablepanel allows

the aregivertolookup ordownasneeded. Theontrolbuttonsonthehassisallowthe

are reipient to all the aregiver, aept and disonnet alls, and adjust the volume

level. These funtions are also available through a remote ontrol. When the robot is

not inuse, it stays inits doking station,faingthe wall.

4.3. Computer Speiations

The exat speiations of the Gira's main omputer were not available, but by

a-essing the operating system's Control Panel, it was possible to extrat the following

relevant information.

CPU Intel Core 2T7200, 2 GHz

GPU Intel i945Express

RAM 1 GB

StorageType Patriot Memory USB devie

StorageCapaity Primary partition3.5GB (1.5GBfree)

Operating System Mirosoft Windows EmbeddedStandard

The Giraalsohas twoUSB ports. The rearportismeanttohold awirelessnetwork

adapter, andthe front port anbeusedfor onnetinginputdevies likekeyboardsand

mie, whenneessary for administration[27℄.

Thesespeiationssuggestthatthe Giramightbepowerfulenoughtoallow

reason-ablyadvanedappliationstorun onthedevie itself. A suientlyeient navigation

appliationouldrunonitdiretly;remote-ontrolsolutionsmaynotbeneessary. This

would be an advantage, as a remote-ontrolsolution for autonomous navigation would

require more hardware and beless robust.

4.4. Camera

Aording toGira Tehnologies, the sensor hip isa Cynove USB devie with alisted

sensor size of 1/3.2" and a resolution of 1600x1200. It is tted with a 1.8mm wide

anglelens. Fordigitalimagesensors,the listedsensorsize isusuallyabout1.5timesthe

atual sensor size, so the atual diagonal of the sensor would be about 5.68mm. Thus,

the diagonal eld of view is approximately

2 arctan ₂ ^d _f = 2 arctan ₂ ^{5 . 68mm}

· 1 . 8mm ≈ 115 ^◦

^{. The}

horizontaleld of view isapproximately

2 arctan ₂ ^{4 . 54mm}

· 1 . 8mm ≈ 103 ^◦

^.

Beause of the amera's wide angle, it would seem like a good idea to apture video

at high resolution, in order to detet relatively distant landmarks with reasonable

a-uray, though this may need to be balaned with the slower transfer speed and higher

omputationalworkload of ahigher resolution. Testing suggests that the highest video

resolutiontheameraisable todeliveratapratialrateis800x600,atabout10frames

perseond.

When using the amera for navigation, it is neessary to orret for the distortion

(sheyeeet) ausedby the lens. An advantage ofthe wideangle isthatthe robotan

more easily keep landmarksand obstales inview whileturning and moving.

4.5. Motor Controller

The Gira'swheels areontrolled by anAVRmiroontrollerrunningustomsoftware.

It ommuniates with the main omputer through a RS232-type serial port interfae,

using a line-based ASCII protool[31℄. The ontroller aepts operations like movinga

spei distane, turninga speiangle, oraombinationof both (urved motion). In

buered mode, up tofour suh operations an be plaed in queue. All operations have

ramp-up and ramp-down times, so that jerky motions annot happen. The

miroon-troller also ontrols the tiltof the Gira'shead, and gives aess tothe buttonson the

hassis.

With some aveats, the miroontroller an help estimating the robot's position by

keeping trak of the distane travelled by its drive wheels. The ontroller an provide

this information to the main omputer on request. For navigational purposes, this is

usuallyknownasodometry,andanbeusedfordeadrekoning,whihisneessarywhen

nootherpositionestimateisavailable(i.e., noknownlandmarksare insight). However,

testing shows thatthis is,unfortunately, not reliableenough tobeused on itsown.

AnotherissueisthattheGira'sdefaultremoteontrolsoftwaregainexlusiveontrol

over ommuniation with the motor ontroller while it is running. Thus, the default

software would need to be shut down before other navigation software an ontrol the

motor, or some way of multiplexing the motor ontroller port needs to be developed.

One way todothis may be toreate avirtual motorontroller portthat both piees of

software an onnet to. Then the software behind the virtual ontrollerommuniates

with the real ontroller, and routes ommands and responses to whihever piee of

software needs it. Another option might be to make a new navigation system a fully

funtional substitute for the default software, so that runningthe default software will

just never be neessary.

5.1. Funtionality

Theimplementedsystemworksasdesribed,andanbeusedtosteertheGirathrough

the are enter, and reord the journey for later playbak, if a keyboard is onneted.

The imagesin Figures3.4, 5.1and 5.2are fromsuh a reorded journey.

5.2. Extensibility

Requirement: As the developed system is meant tobe a platform on aswhih a larger

system ould be built, it should be possible to implement other omponents on top of

it.

Figure 5.1 shows the results of adding an image proessing algorithm (the Canny

edge detetor [32℄, availablein OpenCV),as anexample ofhow suhalgorithmsan be

added. (Also,theabilitytodetetmovingedgesmightbeuseful forobstaleavoidane.)

Figure5.2demonstratesapartiulartypeoffeatureextrator(theFASTornerdetetor

[7℄, also available in OpenCV) that may be used as part of a navigation system. The

features shown in the gure (pink irles) ould be mathed with previously known

features, and their oordinates given to a SLAMimplementation,whih ould then use

them to determine the robot's urrent position.

5.3. Reording and playbak

Requirement: When playingbakareording,theresultingvisualsandmotorodometry

should be idential towhat was seen when the reording was rst reated.

Some sample reords are available on the attahed CD-ROM. While testing shows

that they do appear to be the same, there are still some lag spikes while reording,

meaning that the Gira does not work fast enough to do a smooth reording. Adding

multithreading to the motor reording omponent might mitigate this. But sine the

Figure 5.1.: Playbak with Canny edge detetor

Figure 5.2.: Playbak with FAST orner detetor (orners highlighted withpink)

apturedvideoframeshavetimestampsinthemotorreord,thisproblemdoesnotause

any driftin the timingof the playbak.

5.4. Motor ontrol

Requirement: Movement ommands from the user should be properly interpreted and

ause the Giratomove inthe desiredway.

Commands an urrently only be given using a onneted keyboard, but this should

sueforevaluation. Thesamplereords,availableontheCD-ROM,showsthatmoving

the Giraaroundthiswayworks. However, beauseoftheproblemswithurvedmotion

desribedinSetion3.2.1.4,turnsaresomewhatdiulttopredit,andsomemovement

jerks often happen when ending them. Possibly a future navigation solution would be

able to plan movesin advane insuh away that these jerks an be avoided.

5.5. Motor simulation

Requirement: Themotor simulatorshouldemulate the atualmotorontrolleras

faith-fully as possible.

Testing shows thatthe simulatorislose tothe real thing,with afewaveats. Unlike

the realthing,the simulatordoesnot makeerrors. Forexample,formehanialreasons,

the real motor ontroller is usually not able to hit the exat distane requested. If

you request a ertain distane, it will usually report an odometry that's o by a few

millimeters. The simulator, however, willalways report the exat requested distane in

its odometry. Also, there are ertain bugs in the real ontroller that's not repliated

faithfully in the simulator, suh as the quirky behaviour of the Clothoid Deeleration

Point ommand used for urved motion. And some minor features, suh as hanging

the head tilt angle, reporting presses of the hassis buttons, and heking the battery

status,arenotsimulatedatall. Thesearefairlyminorissues,however, andthesimulator

works ne for its intended purpose of simulating the result of navigation ommands.

5.6. Disussion

Clearly, many more things ouldhave been explored orimplemented inthis projet. In

partiular, it would have been very interesting to try an atual SLAM implementation

on the Gira. Unfortunately,beause of the Gira's limitedavailability,and the motor

ontroltookmuhmoretimethanexpeted,inpartbeausetheoriginaldoumentation

wasmissingsomevitalinformation. However,usingtheplatformdesribedinthisthesis,

and thedoumentationprovided inAppendix A,I believeimplementingand evaluation

navigation algorithmson the Giraan nowbe done more eiently.

In retrospet, it might have been a good idea to prioritize dierently. For example,

spendingless time ontuningthe motorontrollerand simulator wouldmean moretime

for trying out navigation algorithms, and for desribing what has been done. It would

alsohavebeeninterestingtosetup afewexperiments,suhastryingtodoasimple

pre-programmedpatrol using dead rekoning. Sine deadrekoning is unreliable,espeially

given theeetsoftheasters,theGirawouldprobablynotgoexatlywhereitshould,

but it would be agooddemonstrationof the funtionalityof the motorontroller.

For this thesis, I've built a platform for developing navigation solutions for the Gira,

a telepresene robot. I've designed and implemented a system to interfae with its

hardware, and also investigated many of the hallenges involved in making it able to

navigate a large building without human assistane, inluding loalization and route

planning. I examined some of the algorithms and tehnologies that ould be used to

solve those problems some that require adding more sensors to the Gira, and some

that don't.

The implemented platform shows amera imagesand motor odometry onthe sreen,

and allows theuser use thekeyboard toontrolthe Gira'smotorsand moveitaround.

It anreordandplaybakvideoandmotordata,and whenrunonaregularomputer,

it an simulate the Gira's motors. This allows oine development and evaluation of

loalizationand navigationsolutions, failitatingfuture work onthe Gira.

Based on a literature study of loalization approahes, it appears that adding extra

sensors may allowmore robust algorithmsto beused, but given the ontrolled

environ-menttheGiraismeanttooperatein,addingsensorsisbynomeansneessary. A

single-amera SLAM approahould work quitewell. In partiular,it mightbeinteresting to

try onvertingthe EKFMonoSLAM soure ode (found onhttp://www.openslam.org/)

from Matlab to C++ for use on the Gira. Sine this approah also allows the Gira

to be used unmodied, whih is heaper and more onvenient for the users, this seems

likethe preferredapproah. Ifwe were toadd asensor, however, aninfrared laserrange

nder would probably be most useful, in order to minimize the risk of rashing into

things.

[1℄ Kunnskapsdepartementet, Meld. St. 13 (2011-2012): Utdanning for velferd, 17

Feb. 2012.

[2℄ S. S. Srinivasa, D. Ferguson, C. J. Helfrih, D. Berenson, A. Collet, R. Diankov,

G.Gallagher,G.Hollinger,J.Kuner,andM.V.Weghe,HERB:ahomeexploring

roboti butler, Autonomous Robots, vol.28,pp. 520,Jan. 2010.

[3℄ K. Yamazaki, R. Ueda, S. Nozawa, M. Kojima, K. Okada, K. Matsumoto,

M. Ishikawa, I. Shimoyama, and M. Inaba, Home-assistant robot for an aging

soiety, Proeedings of the IEEE, vol.100, no. 8, pp. 24292441, 2012.

[4℄ M. L. Benmessaoud, A. Lamrani, K. Nemra, and A. Souii, Single-amera

EKF-vSLAM, Proeedings of World Aademy of Siene: Engineering & Tehnology,

vol. 42,pp. 924 929, June 2008.

[5℄ A. Davison, Real-time simultaneous loalisation and mapping with a single

am-era, inComputerVision, 2003.Proeedings. NinthIEEE International Conferene

on,pp. 1403 1410 vol.2, Ot. 2003.

[6℄ A. Ali and M. Nordin, Sift based monoular slam with multi-louds features for

indoornavigation, inTENCON 2010- 2010IEEE Region 10Conferene,pp.2326

2331,Nov. 2010.

[7℄ E.Rosten and T. Drummond,Mahine learningfor high-speed orner detetion,

inIn European Conferene on Computer Vision,pp. 430443,2006.

[8℄ L.-F. Gao, Y.-X. Gai, and S. Fu, Simultaneous loalization and mapping for

au-tonomousmobilerobots usingbinoularstereovisionsystem, inMehatronisand

Automation, 2007. ICMA 2007. International Conferene on, pp. 326 330, Aug.

2007.

[9℄ X. Kuai, K. Yang, S. Fu, R. Zheng, and G. Yang, Simultaneous loalizationand

mapping (SLAM) for indoor autonomous mobile robot navigation in wireless

sen-sor networks, in Networking, Sensing and Control (ICNSC), 2010 International

Conferene on,pp. 128 132,Apr. 2010.

[10℄ W.JeongandK.M.Lee, CV-SLAM:anew eilingvision-basedSLAMtehnique,

in Intelligent Robots and Systems, 2005. (IROS 2005). 2005 IEEE/RSJ

Interna-tional Conferene on, pp. 3195 3200, Aug.2005.

[11℄ C.-J.Wuand W.-H.Tsai,Loationestimationforindoorautonomousvehile

nav-igation by omni-diretional vision using irular landmarks on eilings, Robotis

and Autonomous Systems,vol. 57,pp. 546 555, May 2009.

[12℄ S. Fu, H. ying Liu, L.-F. Gao, and Y.-X. Gai, Slam for mobile robots using laser

range nder and monoular vision, in Mehatronisand Mahine Vision in

Pra-tie, 2007. M2VIP 2007. 14th International Conferene on,pp. 9196, De. 2007.

[13℄ Z.Zalevsky, A. Shpunt, A. Maizels, and J. Garia, Method and system for objet

reonstrution. Patent WO2007043036, Apr.2007.

[14℄ T. Yap and C. Shelton, SLAM in large indoor environments with low-ost,noisy,

and sparse sonars, in Robotis and Automation, 2009. ICRA '09. IEEE

Interna-tional Conferene on, pp. 1395 1401, May 2009.

[15℄ S.-Y.Hwang,J.-T.Park,andJ.-B.Song,Autonomousnavigationofamobilerobot

using an upward-looking amera and sonar sensors, in Advaned Robotis and its

Soial Impats (ARSO), 2010 IEEE Workshopon, pp. 40 45, Ot. 2010.

[16℄ R.E.Kalman,Anewapproahtolinearlteringandpreditionproblems, Journal

of Basi Engineering, vol. 82,pp. 3545, Mar.1960.

[17℄ A. H. Jazwinski, Stohasti Proesses and Filtering Theory. AademiPress, Apr.

1970.

[18℄ S. J. Julier and J. K. Uhlmann, Unsented ltering and nonlinear estimation,

Proeedings of the IEEE, vol. 92,pp. 401 422, Mar.2004.

[19℄ J.Campbell,R.Sukthankar,I.Nourbakhsh, andA.Pahwa,Arobustvisual

odom-etry and preipie detetion system using onsumer-grade monoular vision, in

Robotis and Automation, 2005. ICRA 2005. Proeedings of the 2005 IEEE

Inter-national Conferene on, pp. 3421 3427,Apr. 2005.

[20℄ D.Nister,O.Naroditsky,andJ.Bergen,Visualodometry,inComputerVisionand

Pattern Reognition, 2004. CVPR 2004. Proeedings of the 2004 IEEE Computer

Soiety Conferene on, vol.1, pp. I652 I659Vol.1, June 2004.

[21℄ A.J. Davisonand N. Kita,Sequentialloalisationand map-buildingfor real-time

omputervisionand robotis, Robotis and AutonomousSystems, vol.36,pp. 171

183, Sept. 2001.

[22℄ P. Pinies and J. Tardos, Salable SLAM building onditionally independent loal

maps, in Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ

Interna-tionalConferene on, pp. 3466 3471,Ot. 2007.

[23℄ H.Strasdat,J.M.M.Montiel,andA.Davison,Saledrift-awarelargesale

mono-ular slam, in Proeedings of Robotis: Siene and Systems, (Zaragoza, Spain),

June 2010.

[24℄ J. Civera, O. G. Grasa, A. J. Davison, and J. M. M. Montiel, 1-point ransa

for extended kalman ltering: Appliationto real-time struture from motion and

visualodometry, J.Field Robot., vol.27,pp. 609631,Sept. 2010.

[25℄ J.Civera,A.Davison,andJ.Montiel,Inversedepthparametrizationformonoular

slam, Robotis, IEEE Transations on,vol.24, no. 5, pp. 932945,2008.

[26℄ A.Stentz, Optimaland eientpath planningforpartially-known environments,

inRobotis and Automation, 1994. Proeedings., 1994 IEEE International

Confer-ene on,pp. 3310 3317vol.4, May 1994.

[27℄ GiraTehnologiesAB, Advaned OperationalGuide For Gira Version 3.1, June

2011.

[28℄ G.Bradskiand A.Kaehler, Learning OpenCV: Computer Vision withthe OpenCV

[28℄ G.Bradskiand A.Kaehler, Learning OpenCV: Computer Vision withthe OpenCV

In document Feature Extractor (sider 40-0)