2022年人体姿态估计技术和用例 MobiDev

TableofContentsWhatisHumanPoseEstimation?How3DHumanPoseEstimationWorks3DposeestimationperformanceandaccuracyReal-time3DhumanposeestimationHumanposeestimationusecasesAIfitnessandself-coachingRehabilitationandphysiotherapyAugmentedrealityAnimationandgamingSurveillanceandhumanactivityanalysisHowtotrainahumanposeestimationmodel?Howtoavoidtraininghumanposeestimationfromscratch?HumanPoseEstimationisacomputervision-basedtechnologythatidentifiesandclassifiesspecificpointsonthehumanbody.Thesepointsrepresentourlimbsandjointstocalculatetheangleofflexion,andestimate,well,humanpose.Whileitsoundsawkward,knowingtherightangleofajointinaspecificexerciseisthebasisofworkforphysiotherapists,fitnesstrainers,andartists.Implementingsuchcapabilitiesforamachineresultsinsurprisinglyusefulapplicationsindifferentfields.Inthisarticlewe’llexplorehumanposeestimationindepth.We’llfigureoutitsprincipleofworkandcapabilitiestounderstandsuitablebusinesscases.Also,we’llanalyzedifferentapproachestoHumanPoseEstimationasamachinelearningtechnology,andtrytodefinetheapplicationsforeach.2WhatisHumanPoseEstimation?HumanPoseEstimation(HPE)isataskincomputervisionthatfocusesonidentifyingthepositionofahumanbodyinaspecificscene.MostoftheHPEmethodsarebasedonrecordinganRGBimagewiththeopticalsensortodetectbodypartsandtheoverallpose.Thiscanbeusedinconjunctionwithothercomputervisiontechnologiesforfitnessandrehabilitation,augmentedrealityapplications,andsurveillance.Theessenceofthetechnologyliesindetectingpointsofinterestonthelimbs,joints,andevenfaceofahuman.Thesekeypointsareusedtoproducea2Dor3Drepresentationofahumanbodymodel.3Thesemodelsarebasicallyamapofbodyjointswetrackduringthemovement.Thisisdoneforacomputernotonlytofindthedifferencebetweenapersonjustsittingandsquatting,butalsotocalculatetheangleofflexioninaspecificjoint,andtellifthemovementisperformedcorrectly.Therearethreecommontypesofhumanmodels:skeleton-basedmodel,contour-based,andvolume-based.Theskeleton-basedmodelisthemostusedoneinhumanposeestimationbecauseofitsflexibility.Thisisbecauseitconsistsofasetofjointslikeankles,knees,shoulders,elbows,wrists,andlimborientationscomprisingtheskeletalstructureofahumanbody.nposeestimationAskeleton-basedmodelisusedfor2D,aswellas3Drepresentation.Butgenerally,2Dand3Dmethodsareusedinconjunction.3Dhumanposeestimationgrantsbetteraccuracytotheapplicationmeasurementssinceitconsidersthedepthcoordinateandfetchesthoseresultsintocalculation.Forthemajorityofmovements,depthisimportant,becausethehumanbodydoesn’tmoveina2Ddimension.Sonowlet’sfindouthow3Dhumanposeestimationworksfromatechnicalperspective,andfindoutthecurrentcapabilitiesofsuchsystems.4How3DHumanPoseEstimationWorksTheoverallflowofabodyposeestimationsystemstartswithcapturingtheinitialdataanduploadingitforasystemtoprocess.Aswe’redealingwithmotiondetection,weneedtoanalyzeasequenceofimagesratherthanastillphoto.Sinceweneedtoextracthowkeypointschangeduringthemovementpattern.Oncetheimageisuploaded,theHPEsystemwilldetectandtracktherequiredkeypointsforanalysis.Inanutshell,differentsoftwaremodulesareresponsiblefortracking2Dkeypoints,creatingabodyrepresentation,andconvertingitintoa3Dspace.Sogenerally,whenwespeakaboutcreatingabodyposeestimationmodel,wemeanimplementingtwodifferentmodulesfor2Dand3Dplanes.nsSoforthemajorityofhumanposeestimationtasks,theflowwillbebrokenintotwoparts:1.Detectingandextracting2Dkeypointsfromthesequenceofimages.Thisentailsusinghorizontalandverticalcoordinatesthatbuildupaskeletonstructure.2.Converting2Dkeypointsinto3Daddingthedepthdimension.Duringthisprocess,theapplicationwillmaketherequiredcalculationstoperformposeestimation.Estimatinghumanposeduringtheexerciseisjustoneexampleinthefitnessindustry.Somemodelscanalsodetectkeypointsonthehumanfaceandtrackheadposition,whichcanbeappliedforentertainmentapplicationslikeSnapchatmasks.Butwe’lldiscusstheusecasesofHPElaterinthearticle.tfortheprocessingtimetoseetheposeDependingonthechosenalgorithm,theHPEsystemwillprovidedifferentperformanceandaccuracyresults.Let’sseehowtheycorrelateintermsofourexperimentwithtwoofthemostpopularhumanposeestimationmodels,VideoPose3DandBlazePose.We’vetestedBlazePoseandVideoPose3Dmodelsonthesamehardwareusinga5-secondvideowith2160*3840dimensionsand60framespersecond.VideoPose3Dgotatotaltimeof8minutesforvideoprocessingandagoodaccuracyresult.Incontrast,BlazePoseprocessingtimereached3-4framespersecond,whichallowstheuseinreal-timeapplications.Buttheaccuracyresultsshownbelowdon’tcorrespondtotheobjectivesofanyHPEtask.Theprocessingtimedependsonthemovementcomplexity,videoandlightingquality,andthe2Dposedetectormodule.GiventhefactthatBlazePoseandVideoPose3Dhavedifferent2Ddetectors,thisstageappearstobeaperformancebottleneckinbothcases.OneofthepossiblewaystooptimizeHPEperformanceistheaccelerationof2Dkeypointdetection.Existing2Ddetectorscanbemodifiedoramplifiedwiththepostprocessingstagestoimprovegeneralaccuracy.7Real-time3DhumanposeestimationWhetherwedealwithafitnessapp,anappforrehabilitation,facemasks,orsurveillance,real-timeprocessingishighlyrequired.Ofcourse,theperformanceofthemodelwilldependonthechosenalgorithmandhardware,butthemajorityofexistingopen-sourcemodelsprovidequitealongresponsetime.Intheoppositescenario,theaccuracysuffers.Soisitpossibletoimproveexisting3Dhumanposeestimationmodelstoachieveacceptableaccuracywithreal-timeprocessing?WhilemodelslikeBlazePoseareabletoprovidereal-timeprocessing,theaccuracyofitstrackingisnotsuitableforcommercialuseorcomplextasks.Intermsofourexperiment,wetestedthe2DcomponentofaBlazePosewithamodified3D-pose-baselinemodelusingPythonlanguage.Intermsofspeed,ourmodelachievesabout46FPSontheabove-mentionedhardwarewithoutvideorenderingwherethe2Dposedetectionmodelproduceskeypointswithabout50FPS.Incomparisontothe2Dposedetectionmodel,themodified3Dbaselinemodelcanproducekeypointswithabout780FPS.Thedetailedinformationaboutthespentprocessingtimeofourapproachispresentedbelow.Whilethisapproachdoesn’tguaranteereliabilityincomplexscenarioswithdimlightingorunusualposes,standardvideoscanbeprocessedinrealtime.But,generally,theaccuracyofmodelpredictionswilldependonthetrainingandthechosenarchitecture.Understandingthetruecapabilitiesofhumanposeestimation,wecananalyzesomecommonbusinessapplicationsandgeneralusecasesforthistechnology.HumanposeestimationusecasesHPEcanbeconsideredaquitematuretechnologysincetherearegroundworksintheareasofapplicationslikefitness,rehabilitation,augmentedreality,animation,gaming,robotics,andevensurveillance.Sonowlet’stalkabouttheexistingusecases.FitnessapplicationsandAI-drivencoachesaresomeofthemostobvioususecasesforbodyposeestimation.Themodelimplementedinthephoneappcanusethehardwarecameraasasensortorecordsomeonedoinganexerciseandperformitsanalyses.Trackingthemovementofahumanbody,theexercisecanbesplitintophasesofeccentricandconcentricmovementstoanalyzedifferentanglesofflexionandoverallposture.Thisisdoneviatrackingthekeypointsandprovidinganalyticsintheformofhintsorgraphicanalysis.Thiscanbehandledinreal-timeoraftersomedelay,providinganalyticsonthemajormovementpatternsandbodymechanicsfortheuser.Thephysiotherapyindustryisanotherhumanactivitytrackingusecasewithsimilarrulesofapplication.Intheeraoftelemedicine,in-homeconsultationsbecomemuchmoreflexibleanddiverse.AItechnologieshaveenabledmorecomplexwaysthattreatmentcanbedoneonline.Theanalysisofrehabactivitiesappliessimilarconceptstofitnessapplications,exceptfortherequirementstoaccuracy.Sincewe’redealingwithrecoveringfromtheinjury,thiscategoryofapplicationswillfallintothehealthcarecategory.9Whichmeansithastomeetthestandardsofthehealthcareindustryandgeneraldataprotectionlawsinacertaincountry.AUGMENTEDREALITYAugmentedrealityapplicationslikevirtualfittingroomscanbenefitfromhumanestimationasoneofthemostadvancedmethodsofdetectingandrecognizingthepositionofahumanbodyinspace.Thiscanbeusedinecommercewhereshoppersstruggletofittheirclothesbeforebuying.Humanposeestimationcanbeappliedtotrackkeypointsonthehumanbodyandpassthisdatatotheaugmentedrealityenginethatwillfitclothesontheuser.Thiscanbeappliedtoanybodypartandtypeofclothes,orevenfacemasks.We’vedescribedourexperienceofusinghumanposeestimationforvirtualfittingsroomsinadedicatedarticle. ANIMATIONANDGAMINGGamedevelopmentisatoughindustrywithalotofcomplextasksthatrequireknowledgeofhumanbodymechanics.Bodyposeestimationiswidelyusedinanimationofgamecharacterstosimplifythisprocessbytransferringtrackedkeypointsinacertainpositiontotheanimatedmodel.Theprocessofthisworkresemblesmotiontrackingtechnologyusedinvideoproduction,butdoesn’trequirealargenumberofsensorsplacedonthemodel.Instead,wecanusemultiplecamerastodetectthemotionpatternandrecognizeitautomatically.Thedatafetchedthencanbetransformedandtransferredtotheactual3Dmodelinthegameengine.Somesurveillancecasesdon’trequirespottingacrimeinacrowdofpeople.Instead,camerascanbeusedtoautomateeverydayprocesseslikeshoppingatagrocerystore.CashierlessstoresystemslikeAmazonGO,forexample,applyhumanposeestimationtounderstandwhetherapersontooksomeitemfromashelf.HPEisusedincombinationwithothercomputervisiontechnologies,whichallowsAmazontoautomatetheprocessofcheckoutintheirstoresusinganetworkofcamerasensors,IoTdevices,andHumanposeestimationisresponsibleforthepartoftheprocesswheretheactualareaofcontactwiththeproductisnotvisibletothecamera.Sohere,theHPEmodelanalyzesthepositionofcustomers’handsandheadstounderstandiftheytooktheproductfromtheshelf,orleftitinplace.Howtotrainahumanposeestimationmodel?Humanposeestimationisamachinelearningtechnology,whichmeansyou’llneeddatatotrainit.Sincehumanposeestimationcompletesquitedifficulttasksofdetectingandrecognizingmultipleobjectsonthescreen,andneuralnetworksareusedasanengineofit.Traininganeuralnetworkrequiresenormousamountsofdata,sothemostoptimalwayistouseavailabledatasetslikethefollowingones:●HumanEva●MPIHumanPose,and●Human3.6MThemajorityofthesedatasetsaresuitableforfitnessandrehabapplicationswithhumanposeestimation.Butthisdoesn’tguaranteehighaccuracyintermsofmoreunusualmovementsorspecifictaskslikesurveillanceormulti-personposeestimation.Fortherestofthecases,datacollectionisinevitablesinceaneuralnetworkwillrequirequalitysamplestoprovideaccurateobjectdetectionandtracking.Here,experienceddatascienceandmachinelearningteamscanbehelpful,sincethe