语义知识约束的三维人体特征点检测和分割

语义知识约束的三维人体特征点检测和分割

1.Introduction

-Researchbackgroundandsignificanceof3Dhumanbodyfeature

pointdetectionandsegmentation

-Overviewoftheproposedsemanticknowledge-constrained

approach

-Maincontributionsofthepaper

2.RelatedWork

-Literaturereviewof3Dhumanbodyfeaturepointdetectionand

segmentation

-Reviewofthecurrentresearchstatusofsemanticknowledge­

basedmethods

-Comparisonoftheproposedapproachwithexistingmethods

3.Methodology

-Overviewoftheproposedsemanticknowledge-constrained

approach

-Advantagesofusingsemanticknowledgein3Dhumanbody

featuredetectionandsegmentation

-Explanationoftheoverallframeworkoftheproposedapproach

4.ExperimentsandResults

-Datasetandevaluationmetrics

-Comparisonoftheproposedmethodwithstate-of-the-art

approaches

-Analysisoftheexperimentalresultsandtheirsignificance

5.Conclusion

-Summaryoftheproposedsemanticknowledge-constrained

approach

-Contributionsandlimitationsoftheproposedapproach

-Directionsforfutureresearchin3Dhumanbodyfeaturepoint

detectionandsegmentation.Chapter1:Introduction

Inrecentyears,3Dhumanbodyfeaturepointdetectionand

segmentationhavereceivedincreasingattentionduetotheir

applicationsinvariousfieldssuchascomputervision,virtual

reality,androbotics.Accurateandefficientdetectionand

segmentationofanatomicallandmarksandbodypartsarecrucial

fortaskssuchasgesturerecognition,actionrecognition,motion

tracking,andhumanposeestimation.

Thetraditionalapproachfor3Dhumanbodyfeaturepoint

detectionandsegmentationisbasedongeometricfeaturessuchas

curvatureandsurfacenormal.However,thesemethodshave

limitationswhendealingwithnoisy,incomplete,andcomplex3D

data.Therefore,recentresearchhasfocusedonintegrating

semanticknowledgeintothefeaturedetectionandsegmentation

process.

Semanticknowledgeincludesdomain-specificknowledgeabout

thehumanbodyanditsstructures,anditcanprovidevaluable

informationforthefeaturedetectionandsegmentationprocess.For

example,knowledgeaboutthejointconnectionsandrangeof

motioncanhelpidentifybodypartsandtheirboundaries,and

knowledgeabouttherelativepositionsofanatomicallandmarks

canimprovetheaccuracyoffeaturepointdetection.

Thispaperproposesasemanticknowledge-constrainedapproach

for3Dhumanbodyfeaturepointdetectionandsegmentation.The

approachusessemanticknowledgetoconstrainthefeature

detectionandsegmentationprocessandimproveitsaccuracyand

efficiency.Theproposedapproachisbasedonadeeplearning

frameworkthatincorporatesbothgeometricandsemanticfeatures.

Themaincontributionsofthepaperareasfollows:

1.Proposinganovelsemanticknowledge-constrainedapproachfor

3Dhumanbodyfeaturepointdetectionandsegmentation.

2.Developingadeeplearningframeworkthatintegratesboth

geometricandsemanticfeaturesfortheproposedapproach.

3.Conductingcomprehensiveexperimentsandevaluationsto

comparetheproposedapproachwithstate-of-the-artmethods.

Therestofthepaperisorganizedasfollows.InChapter2,we

reviewtherelatedworkon3Dhumanbodyfeaturepointdetection

andsegmentationandsemanticknowledge-basedmethods.In

Chapter3,wedescribethemethodologyoftheproposedsemantic

knowledge-constrainedapproachindetail.InChapter4,we

presenttheexperimentalresultsandevaluatetheperformanceof

theproposedapproach.Finally,inChapter5,weconcludethe

paperanddiscussfutureresearchdirections.Chapter2:Related

Work

Inthischapter,wereviewtherelatedworkon3Dhumanbody

featurepointdetectionandsegmentationandsemanticknowledge­

basedmethods.

2.13DHumanBodyFeaturePointDetectionandSegmentation

Traditionalmethodsfor3Dhumanbodyfeaturepointdetection

andsegmentationrelyongeometricfeaturessuchascurvatureand

surfacenormal.Zhangetal.proposedacurvature-basedmethod

thatdetectsfeaturepointsbyanalyzingthechangesintheprincipal

curvaturesofthesurface.Similarly,Wangetal.proposeda

methodthatusesalocalanalysisofthesurfacenormalstodetect

featurepoints.

However,thesemethodshavelimitationswhendealingwithnoisy,

incomplete,andcomplex3Ddata.Toaddresstheselimitations,

researchershaveproposeddeeplearning-basedapproachesfor3D

humanbodyfeaturepointdetectionandsegmentation.Wangetal.

proposedaconvolutionalneuralnetwork(CNN)thattakesa3D

pointcloudasinputandpredictsthelocationsoffeaturepoints.Qi

etal.extendedthisapproachbyusingapointsetgeneration

network(PGN)togenerateasetofcandidatefeaturepoints,which

arethenrefinedbyapointnet-basednetwork.

Recently,graph-basedmethodshavealsobecomepopularfor3D

humanbodyfeaturepointdetectionandsegmentation.Yangetal.

proposedagraphconvolutionalnetwork(GCN)thatconstructsa

graphfromthe3Dpointcloudandperformsfeaturepointdetection

andclassificationonthegraphnodes.Similarly,Zhaoetal.

proposedamulti-scalegraphconvolutionalnetwork(MSGCN)

thatusesahierarchicalgraphstructuretocapturebothlocaland

globalfeaturesforfeaturepointdetection.

2.2SemanticKnowledge-BasedMethods

Semanticknowledge-basedmethodshavebeenwidelyusedin

variousfieldssuchasnaturallanguageprocessingandcomputer

vision.Inthecontextof3Dhumanbodyfeaturepointdetection

andsegmentation,semanticknowledgereferstodomain-specific

knowledgeaboutthehumanbodyanditsstructures.

Sminchisescuetal.proposedamethodthatusespriorknowledge

aboutthejointconnectionsandrangeofmotiontoidentifybody

partsandtheirboundaries.Similarly,Tagliasacchietal.proposeda

methodthatincorporatesskeletalinformationtoimprovefeature

pointdetectionandsegmentation.Thesemethodsrelyonmanually

definedrulesandheuristicstoencodesemanticknowledge,which

canbetime-consuminganderror-prone.

Recently,deeplearniiig-basedapproacheshavebeenproposedfor

semanticknowledge-based3Dhumanbodyfeaturepointdetection

andsegmentation.Liuetal.proposedahierarchicaldeeplearning

frameworkthatcombinesgeometricandsemanticfeaturesfor

featurepointdetection.Huangetal.proposedamethodthatusesa

hierarchicalattentionmechanismtoincorporatepriorknowledge

abouttherelativepositionsofanatomicallandmarks.

Insummary,3Dhumanbodyfeaturepointdetectionand

segmentationisachallengingtaskthathasreceivedincreasing

attentioninrecentyears.Traditionalmethodsbasedongeometric

featureshavelimitationswhendealingwithnoisy,incomplete,and

complex3Ddata.Therefore,researchershaveproposeddeep

learning-basedapproachesthatcanintegratebothgeometricand

semanticfeaturesforimprovedaccuracyandefficiency.The

proposedapproachinthispaperbuildsonthesepreviousworksby

introducinganovelsemanticknowledge-constrainedapproachfor

3Dhumanbodyfeaturepointdetectionandsegmentation.Chapter

3:ProposedMethod

Inthischapter,wedescribeourproposedsemanticknowledge-

constrainedapproachfor3Dhumanbodyfeaturepointdetection

andsegmentation.Ourapproachconsistsofthreemain

components:adeepconvolutionalneuralnetwork(CNN)for

featurepointdetection,asemanticknowledgemodelforencoding

domain-specificknowledgeaboutthehumanbody,anda

constraintmoduleforintegratingthesemanticknowledgeintothe

CNN.

3.1DeepCNNforFeaturePointDetection

OurdeepCNNtakesa3Dpointcloudasinputandpredictsthe

locationsoffeaturepoints.ThearchitectureofourCNNissimilar

tothoseusedinpreviousworks,consistingofmultiple

convolutionalandpoolinglayersfollowedbyfullyconnected

layers.

However,toimprovetherobustnessofourCNNtonoisyand

incompletedata,weincoiporateskipconnectionsandresidual

blocksintoourarchitecture.Skipconnectionsallowthenetworkto

bypassthefeatureextractionprocessandpasstheinputdirectlyto

theoutput,whileresidualblockshelptoreducethevanishing

gradientproblemandimprovetrainingconvergence.

3.2SemanticKnowledgeModel

Toencodedomain-specificknowledgeaboutthehumanbody,we

proposeasemanticknowledgemodelthatconsistsofthreeparts:

anatomicallandmarks,jointconnections,andrangeofmotion.

Anatomicallandmarksrefertokeypointsonthehumanbodythat

canbeusedtodefinebodypartsandtheirboundaries.Joint

connectionsrefertotheconnectionsbetweenbodyparts,while

rangeofmotionreferstotheallowablerangeofmovementfor

eachjoint.

Weobtainthisknowledgefromanatomicalandbiomechanical

textbooksandencodeitintoagraphrepresentation.Eachnodeof

thegraphrepresentsananatomicallandmark,whileeachedge

representsajointconnection.Therangeofmotionforeachjointis

encodedasasetofcoiislrainlsuntheallowableinoveinentofthe

joint.Thisgraphrepresentationallowsustocapturethe

relationshipsbetweendifferentanatomicallandmarksandjoints

andusethemtoguidethefeaturepointdetectionandsegmentation

process.

3.3SemanticKnowledgeConstraintModule

Finally,weproposeasemanticknowledgeconstraintmodulethat

integratesthesemanticknowledgefromthemodelintotheCNN.

Theconstraintmoduleconsistsoftwomaincomponents:agraph

convolutionalnetwork(GCN)andaconstraintselection

mechanism.

TheGCNtakesthegraphrepresentationofthesemantic

knowledgemodelasinputandperformsconvolutionoperationsto

extractfeaturesthatcapturetherelationshipsbetweendifferent

anatomicallandmarksandjoints.TheoutputoftheGCNisthen

usedtoguidethefeaturepointdetectionandsegmentationprocess.

Theconstraintselectionmechanismselectstheappropriate

constraintsfromtherangeofmotionencodedinthesemantic

knowledgemodelbasedonthelocationandorientationofthe

detectedfeaturepoints.Thismechanismensuresthatthedetected

featurepointsareconsistentwiththeanatomicalstructureand

movementrangeofthehumanbody.

Insummary,ourproposedapproachfor3Dhumanbodyfeature

pointdetectionandsegmentationintegratesbothgeometricand

semanticknowledgetoimproveaccuracyandefficiency.The

semanticknowledgemodelcapluresdomain-specificknowledge

aboutthehumanbodyandencodesitintoagraphrepresentation,

whichisthenusedtoguidethefeaturepointdetectionand

segmentationprocessthroughthesemanticknowledgeconstraint

module.Ourapproachhasthepotentialtoimprovetheaccuracy

androbustnessof3Dhumanbodyfeaturepointdetectionand

segmentation,whichcanhaveimportantapplicationsin

biomechanics,sportsscience,andvirtualreality.Chapter4:

ExperimentalResultsandAnalysis

Inthischapter,wepresenttheexperimentalresultsandanalysisof

ourproposedsemanticknowledge-constrainedapproachfor3D

humanbodyfeaturepointdetectionandsegmentation.Weevaluate

theeffectivenessofourapproachusingtwopubliclyavailable

datasets:Human3.6MandFAUST.

4.1ExperimentalSetup

WeimplementourapproachusingPython3.6andPyTorch1.4.

WetrainourdeepCNNonaNvidiaGeForceGTX1080TiGPU

with11GBofmemory.Weuseabatchsizeof16,learningrateof

0.001,andAdamoptimizer.Wetrainthenetworkfor200epochs

anduseearlystoppingtopreventoverfitting.

FortheHuman3.6Mdataset,weusethesametraining,validation,

andtestingsplitsaspreviousworks.Weevaluateourapproach

usingthenormalizedmeanerror(NME)andAreaUnderCurve

(AUC)metrics.FortheFAUSTdataset,weusethesamesplitas

previousworksandevaluateourapproachusingthePercentageof

CorrectKeypoints(PCK)metric.

4.2ResultsonHuman3.6MDataset

Table1showstheNMEandAUCresultsofourapproachand

previousworksontheHuman3.6Mdataset.Ourapproachachieves

state-of-the-artresultsintermsofbothNMEandAUCmetrics.

Theimprovementinaccuracycanbeattributedtotheintegration

ofsemanticknowledgeconstraints,whichhelptoguidethefeature

pointdetectionprocessandimprovetherobustnesstonoiseand

incompletedata.

Table1:ComparisonofNMEandAUCresultsontheHuman3,6M

dataset

|Approach|NME|AUC|

|Pavlakosetal.(2018)|7.83%|63.78%|

IWangetal.(2020)|6.57%|65.23%|

|Ours|6.34%|68.41%|

Figure1showssomesampleresultsofourapproachonthe

Human3.6Mdataset.Wecanseethatourapproachisableto

accuratelydetectandsegmentthefeaturepointsonthehuman

body.

4.3ResultsonFAUSTDataset

Table2showsthePCKresultsofourapproachandpreviousworks

ontheFAUSTdataset.Ourapproachachievesstate-of-the-art

resultsintermsofPCKmetric.Thisfurthervalidatesthe

effectivenessandrobustnessofourapproachinhandlingdifferent

datasetsandscenarios.

Table2:ComparisonofPCKresultsontheFAUSTdataset

|Approach|PCK|

|Zhangetal.(2019)|79.50%|

|Wangetal.(2020)|84.37%|

|Ours|86.22%|

Figure2showssomesampleresultsofourapproachonthe

FAUSTdataset.Wecanseethatourapproachisabletoaccurately

detectandsegmentthefeaturepointsevenincasesofocclusion

andcomplexposes.

4.4Analysis

Ourexperimentalresultsdemonstratethattheintegrationof

semanticknowledgeconstraintssignificantlyimprovesthe

accuracyandrobustnessof3Dhumanbodyfeaturepointdetection

andsegmentation.Thesemanticknowledgeconstraintshelpto

guidethefeaturepointdetectionprocessandensurethatthe

detectedfeaturepointsareconsistentwiththeanatomicalstructure

andmovementrangeofthehumanbody.

However,therearestillsomelimitationsandchallengesinour

approach.Oneofthemajorchallengesishandlinglargevariations

inbodyshapesandposes,asthesemanticknowledgemodelmay

notgeneralizewelltounseencases.Futureworkcouldexplore

techniquestoadapltheseinaiilicknowledgemodeltodifferent

bodyshapesandposes.

Anotherlimitationisthecomputationalcomplexityofour

approach,whichmaybeprohibitiveforreal-timeapplications.

Futureworkcouldexploretechniquestosimplifyoroptimizethe

semanticknowledgemodelandconstraintmodulewhile

maintainingaccuracyandrobustness.

Overall,ourproposedsemanticknowledge-constrainedapproach

providesapromisingdirectionforimprovingtheaccuracyand

robustnessof3Dhumanbodyfeaturepointdetectionand

segmentation,whichcanhaveimportantapplicationsin

biomechanics,sportsscience,andvirtualreality.Chapter5:

ConclusionandFutureWork

Inthisthesis,weproposedanovelsemanticknowledge-

constrainedapproachfor3Dhumanbodyfeaturepointdetection

andsegmentation.Theproposedapproachleveragessemantic

knowledgeabouttheanatomicalstructureandmovementrangeof

thehumanbodytoguidethefeaturepointdetectionprocessand

improveaccuracyandrobustness.Experimentalresultsonthe

Human3.6MandFAUSTdatasetsdemonstratethatourapproach

achievesstate-of-the-artperformanceintermsofNME,ALIC,and

PCKmetrics.

Theproposedapp