Dynamic primitives for gestural interaction

Strachan, S. and Murray-Smith, R. and Oakley, I. and Angesleva, J. (2004) Dynamic primitives for gestural interaction. Mobile Human-Computer Interaction – MobileHCI 2004: 6th International Symposium, Glasgow, UK, September 13 - 16, 2004. 3160:pp. 325-330.

http://eprints.gla.ac.uk/2950/

Glasgow ePrints Service http://eprints.gla.ac.uk

DynamicPrimitivesforGesturalInteraction

¨StevenStrachan,1,2RoderickMurray-Smith,1,3IanOakley,2JussiAngeslev¨a2

HamiltonInstitute,NUIMaynooth,Maynooth,Ireland

steven.strachan@may.ie

2

MediaLabEurope,PalpableMachinesGroup,Dublin,Ireland

{ian,jussi}@medialabeurope.org

Dept.ComputingScience,UniversityOfGlasgow,Glasgow,Scotland

rod@dcs.gla.ac.uk

1

3

Abstract.Wedescribetheimplementationofaninteractiontechniquewhichallowsuserstostoreandretrieveinformationandcomputationalfunctionalityondifferentpartsoftheirbody.Wepresentadynamicsys-temsapproachtogesturalinteractionusingDynamicMovementPrimi-tives,whichmodelagestureasasecondorderdynamicsystemfollowedbyalearnednonlineartransformation.Wedemonstratethatitispossi-bletolearnmodels,evenfromsingleexamples,whichcansimulateandclassifythegesturesneededfortheBodySpaceproject,runningonaPocketPCwitha3-degreeoffreedomlinearaccelerometer.

1Introduction

Mobiletelephones,PersonalDigitalAssistantsandhandheldcomputersarecur-rentlyoneofthefastestgrowthareasofcomputingandthisgrowthisextendingintofullywearablesystems.Existingdeviceshavelimitedinputandoutputcapa-bilities,makingthemcumbersomeandhardtousewhenmobile.Consequently,acurrentrequirementinthisfieldisthedevelopmentofnewinteractiontechniquesspecificallydesignedformobilescenarios.Oneimportantaspectofinteractionwithamobileorwearabledeviceisthatithasthepotentialtobecontinuous,withtheuserinconstant,tightlycoupledinteractionwiththesystem.Inthesescenarios,interactionneednolongerconsistofanexchangeofdiscretemessages,butcanformarichandcontinuousdialogue.

TheBodyMnemonicsproject[1]developsanewconceptininteractiondesign.Essentially,itexplorestheideaofallowinguserstostoreandretrieveinformationandcomputationalfunctionalityondifferentpartsoftheirbodies.Inthisdesign,informationcanbestoredandsubsequentlyaccessedbymovingahandhelddevicetodifferentlocationsaroundthebody.Thisworkaddressesthreeproblemareasinmobilecomputing:thehighlevelsofattentionrequiredusingthedevices,theimpersonalnatureoftheirinterfaces,andthesociallyexclusivemodesofinteractiontheysupport.

Theworkdescribedinthispaperrepresentsfirststepstoprovidingthetech-nologytosupportthegesturalinteractionrequiredbythebodymnemonicsconcept.Itisconcernedwithdevelopingalgorithmstoinferthelocationofa

handhelddevice.Toprovideasystemthatrequiresnoadditionalequipment(suchasworntagsormarkers)tofacilitatetheidentificationofdifferentloca-tions,itreliesoninertialsensing.Inertialsensingisarelativelynewparadigmforinteractingwithmobilecomputers.Furthermore,itisagoodexampleofcon-tinuousinput;thedevicegathersinformationaboutuserbehaviourwheneveritisbeingheldorcarried.

Anumberofresearchers,suchasHinkleyetal.[2]andRekimoto[3],havedemonstratedthatinertialsensorscanprovidealternativestothephysicalandon-screenbuttonsinhandhelddevices.Theyhavedescribedsystemswherebyshakingandtiltingthedevicetriggersdifferentcommands.However,thesein-terfacesstillpossessastronggraphicalcomponentandlittleworkhasbeenconductedon‘screen-free’gesturalinterfaces.Pirhonenetal.[4]demonstratedamobilemp3playerwheregesturesweresufficienttoenableuserstocontroltheplayerwithoutlookingatthescreen.Wewishtodeveloptheideaofscreenfreeinteractiontoprovideincreasedusabilitywhen‘onthemove’.

2InitialExplorations

OurinitialinvestigationswereconductedusinganiPAQ5550equippedwitha3-axisXsensP3Clinearaccelerometerattachedtotheserialport.Weareconcentratingonshorttrajectoriesoriginatingandterminatingatspecificbodylocations.Severallocationswereconsideredasthesourceofeachgesture.Theseweretheleftorrighthip,whereadevicemaynaturallybeheldwhennotinuseandthecentreofthechest,whereadeviceisoftenheldtoenableoptimalviewingofitsscreen.Toavoidissuesofhandedness,wechosetomodelourgesturesasalloriginatedfromthecentreofthechest.

Fourbodyareaswerechosenasgestureendpoints-leftshoulder,rightshoulder,backpocketandbackofhead.Forthepurposesofthisexploration,allgestureswereperformedfromthecentreofthechestusingthelefthandwhilststandingstill.

The‘brute-force’approachofintegratingtheinertialmeasurementsintopo-sitionaltrajectoriesandreferringthesetoaspatialmapofthebodyisnotastrongoption.Acombinationofuncertaintyastothepreciseinitialpositionofthedeviceandintegrationdriftledtoasubstantialerrormargin.Figure1dis-playsthetrajectoriesinferredfromaccelerationmeasurements,formovementstothefourdifferentpartsofthebody,with10examplesforeachclassofgesture,andmakescleartheresultinginaccuracyattheend-points.

3DynamicMovementPrimitives

Thefocusofthisprojectwastochoosearecognitionalgorithmthatwasflexibleenoughtomodeltherequiredtrajectories,butalsoconstrainedenoughthatitcouldbetrainedwithminimaleffort,usingasmallnumberofexamplegesturesbyanoviceuser.

Fig.1.Exampleofthedriftencounteredwhenaccelerationtracesareintegratedintopositions.Significantintegrationdriftisobserved,leadingtoend-pointuncertainty.

TheDynamicMovementPrimitives(DMP)algorithmproposedbySchaaletal.,is“aformulationofmovementprimitiveswithautonomousnon-lineardiffer-entialequationswhosetimeevolutioncreatessmoothkinematiccontrolpolicies”[5,6].Theideawasdevelopedforimitation-basedlearninginrobotics,andisanaturalcandidateforapplicationtogesturerecognitioninmobiledevices.Itallowsustomodeleachgesturetrajectoryastheunfoldingofadynamicsystem,andisbetterabletoaccountforthenormalvariabilityofsuchgestures.Impor-tantly,theprimitivesapproachmodelsfromorigintogoalasopposedtothetraditionalpoint-to-pointgesturesusedinothersystems.This,alongwiththecompactandverywell-suitedmodelstructureenablesustotrainasystemwithveryfewexamples,withaminimalamountofusertrainingandalsoprovidesuswiththeopportunitytoaddricherfeedbackmechanismstotheinteractionduringthegesture.

DMP’sarelinearlyparameterisedenablinganaturalapplicationtosuper-visedlearningfromdemonstration.Gesturerecognitionismadepossiblebythetemporal,scaleandtranslationalinvarianceofthedifferentialequationswithrespecttothemodelparameters.

ADynamicMovementPrimitiveconsistsoftwosetsofdifferentialequa-tions,namelyacanonicalsystem,τx˙=h(x)andatransformationsystem,τy˙=g(y,f(x)).Apointattractivesystemisinstantiatedbythesecondorderdynamics

τz˙=αz(βz(g−y)−z),τy˙=z+f,(1)wheregisaknowngoalstate(theleftshoulder,forexample),αzandβzare

timeconstants,τisatemporalscalingfactor,yandy˙arethedesiredpositionandvelocityofthemovementandfisalinearfunctionapproximator.Inthecaseofanon-lineardiscretemovementorgesturethelinearfunctionisconvertedtoanon-lineardeformingfunction

󰀁N

f(x,v,g)=

i=1ψiwiv,󰀁Nψii=1

whereψi=e

󰀀󰀀2

−hi(xg−ci)

(2)

Fig.2.Fiverealisationsofthefourgesturesonthex-coordinateareshownalongwithanexamplesimulatedgesturefromtheDMPmodel.Aprincipalcomponentplotshowstheseparabilityofthemodelparameters.Similarresultscanbedemonstratedfortheyandzcoordinatesalso.

Theseequationsallowustorepresentcharacteristicnon-linearbehaviourthatdefinesthegesture,whilemaintainingthesimplicityofthecanonical2ndordersystemdrivingitfromstarttogoal.Thetransformationsystemforthesediscretegesturesis

τz˙=αz(βz(r−y)−z)+f,τy˙=z,τr˙=αg(g−r)

(3)

wherez˙,zandyrepresentthedesiredacceleration,velocityandpositionrespec-tively.

Theapproachtolearningandpredictingthedynamicmovementprimitiveistoprovideastepchangeinreferenceandpassthisthroughthenon-lineardeformingfunction.Valuesforthef’scanbecalculatedalongwithsetsofx’sandv’sfromthecanonicalsystemandthisisthenpassedthroughaLocallyWeightedProjectionRegression(LWPR)algorithm[7]thatlearnstheattractorlandscapeandallowsustomakepredictionsofthefunctionfgivenvaluesforxandv.

4Results,FutureWorkandConclusions

OurimplementationoftheSchaalDMPalgorithm,runningonapocketPCwithinertialsensing,providesthebasisforanefficient,robustandrapidlytrainablegesturerecognitionsystemforthefourbasicgestureswetested.

Figure2showsexamplesofaccelerationtime-seriescorrespondingtothex-coordinateaccelerationtraceforeachclassofgesture,alongwiththesimulated

curvefromthelearnedmodelinthesecondcolumn.Thegoodmatchbetweenthemeasuredandsimulatedcurvesprovidesencouragingevidenceofitssuitabil-ityforgesturerecognition,especiallyaseachsimulatedgesturewasgeneratedusingamodeltrainedononlyoneexampleofthefiveshown,andinonlyfiveiterationsoftheLWPRalgorithm.Theseparabilityofthemodelparametersforclassificationpurposesisvisibleintheplotofthefirsttwoprincipalcomponentsforeachofthefourclassesofgesture.

Theadditionalbenefitofthisdynamicapproachisthatitprovidesthede-signerwiththeopportunitytoincorporaterich,continuousfeedbackmechanismsintotheinteractionwiththeuser.Wecannowdelivercontinuousaudioortactilefeedbackrelatingtotheuser’smotion,proximitytogoals,orgesturetrajectories[8].Webelievethiskindoftightlycoupledcontrolloopwillsupportauser’slearningprocessesandconveyagreatersenseofbeingincontrolofthesystem.ForthiswewillbeusingtheMESHhardwareplatform[9],whichfeaturesa3-axisaccelerometer,3-axisgyroscope,2-axismagnetometerandanintegratedvibro-tactiletransducerwithalarge(54dB)dynamicrange.Therichersensorinputwillbroadenthescopeofinteractionpossibilities,andthesystemfeaturesthedynamicvibrotactileoutputrequiredtodisplaytheprobabilisticfeedbackfromourDMPmodels.

References

¨1.Angeslev¨a,J.,Oakley,I.,Hughes,S.,O’Modhrain,S.:BodyMnemonics.In:MobileHCIConference2003,Udine,Italy.(2003)

2.Hinckley,K.,Sinclaire,M.,Horvitz,E.:Sensingtechniquesformobileinteraction.In:ACMSymposiumonUserInterfaceSoftwareandTechnology,CHILetters2(2.(2000)91–100

3.Rekimoto,J.:Tiltingoperationsforsmallscreeninterfaces.In:ACMSymposiumonUserInterfaceSoftwareandTechnology.(1996)167–168

4.Pirhonen,A.,Brewster,S.,Holguin,C.:GesturalandAudioMetaphorsasaMeansofControlforMobileDevices.In:ProceedingsofACMCHI2002(Minneapolis,MN),ACMPress,Addison-Wesley(2002)291–298

5.Schaal,S.,Peters,J.,Nakanishi,J.,Ijspeert,A.:LearningMovementPrimitives.In:InternationalSymposiumonRoboticsResearch(ISRR2003).(2004)

6.Ijspeert,A.,Nakanishi,J.,Schaal,S.:Learningattractorlandscapesforlearningmotorprimitives.InS.Becker,S.T.,Obermayer,K.,eds.:AdvancesinNeuralInformationProcessingSystems15.MITPress,Cambridge,MA(2003)1523–15307.Vijaykumar,S.,Schaal,S.:LocallyWeightedProjectionRegression:AnO(n)Al-gorithmforIncrementalRealTimeLearninginHighDimensionalSpace.In:17thInter.Conf.onMachineLearning(ICML2000),Stanford,California.(2004)

8.Williamson,J.,Murray-Smith,R.:Granularsynthesisfordisplayoftime-varyingprobabilitydensities.InHunt,A.,Hermann,T.,eds.:InternationalWorkshoponInteractiveSonification(HumanInteractionwithAuditoryDisplays).(2004)

¨9.Oakley,I.,Angeslev¨a,J.,Hughes,S.,O’Modhrain,S.:TiltandFeel:Scrollingwith

VibrotactileDisplay.In:ProceedingsofEurohaptics2004,MunichGermany.(2004)

WeacknowledgetheHEA’ssupportoftheBodySpaceproject,ECTMRgrantHPRN-CT-1999-00107,EPSRCgrantGR/R98105/01.SFIgrants00/PI.1/C067andSC/2003/271.