计算摄像学专题第14讲——MIT Media Lab.ppt

计算摄像学专题第十四讲 MITMediaLab 戴琼海清华大学自动化系2011 04 13 提纲 MITMediaLab 世界上独一无二的创新研究实验室MediaArts Sciences媒体实验室的研究是没有方向 undirected 没有学科的概念spansacrossdisciplinesandacademic industriallines 65sponsorcompanies Foundedin1985byNicholasNegroponteandJeromeWiesner ResearchGroupsAffectiveComputingBiomechatronicsCameraCultureChangingPlacesCivicMediaCognitiveMachinesFluidInterfacesHigh LowTechHumanDynamicsInformationEcologyLifelongKindergartenMacroConnectionsMediatedMatterMolecularMachinesMusic MindandMachineNewMediaMedicineObject BasedMedia OperaoftheFuturePersonalRobotsResponsiveEnvironmentsSoftwareAgentsSpeech MobilitySyntheticNeurobiologyTangibleMediaViralCommunications CentersandJointProgramsCentersandjointprogramscombinetheworkofMediaLabresearcherswithcollaboratorsthroughoutMIT CenterforFutureCivicMediaCenterforFutureStorytelling MediaLab研究组 一共有25个教授 带领着25个小组 每个小组6个研究员 都是MIT的研究生和博士生 MITMediaLab的企业支持 RESEARCHSTRATEGY 65corporatesponsorsincludesomeofthemostcreativecompaniesintheworld ADAPTABILTIY Hyper Adaptability Bio Mechatronics Music Mind Health People Sense SociableRobots Neuro Media HUMANADAPTABILITY Bio Mechatronics Neuro Media SociableRobots People Sense Music Mind Health Biomechatronics Neuroengineering SmartFabrics RethinkingCameras Human2 0 Multimedia UbiquitousComputing SocialMediaComputerVisionSensorNetworksSoftwareAgents MITMediaLab的发展愿景 MITMediaLab专注于创新 发明 多学科深度交叉 学术与应用交融 Wefocusoncreatingtoolstobettercaptureandsharevisualinformationviaanewclassofimagingplatforms RameshRaskar RameshRaskarhttp raskar info MITMediaLabCameracultureGroup 极具创造性的研究 研究领域 研究内容 ComputationalLightTransport ComputationalPhotography 实验室人员 RameshRaskar 6Master Phdstudents5visitingresearchers3scientistsandPost docs RameshRaskar MITMediaLab AssociateProfessorCameraCulturegroup LeaderJoinedtheMediaLabfromMitsubishiElectricResearchLaboratoriesin2008asheadoftheLab sCameraCultureresearchgroupIn2004 TR100AwardfromTechnologyReview In2009 SloanResearchFellowship In2009 MarrPrizehonorablemention In2010 DarpaYoungFacultyaward Amazingnews In2011 4siggraphpapers 90 intotal CPsessionatmost4papers 提纲 提纲 ComputationalPhotographyCodedTime Exposure CodinginSpaceCodedIlluminationCodedWavelengthInverseProblemsinComputationalLightTransportTransientImagingSpatialHeterodyningRank constraintof3DDisplaysComputationalProbesRandomizedProjectionsOthersSecondSkin ABio I OPlatformforWearableFabricSix DimensionalDisplaysBio inspiredVision bettercaptureandsharevisualinformationviaanewclassofimagingplatforms 提纲 ComputationalPhotographyCodedTime Exposure CodinginSpaceCodedIlluminationCodedWavelengthInverseProblemsinComputationalLightTransportTransientImagingSpatialHeterodyningRank constraintof3DDisplaysComputationalProbesRandomizedProjectionsOthersSecondSkin ABio I OPlatformforWearableFabricSix DimensionalDisplaysBio inspiredVision ComputationalPhotography CodedTime Exposure ShortExposure TraditionalCamera CodedExposure FourierTransform Informationislost FourierTransform Codedexposurephotography motiondeblurringusingflutteredshutter SIGGRAPH06 PreservesAllFrequencies ComputationalPhotography CodinginSpace Receiver Demodulation HighFreqCarrier100 1MHz ReferenceCarrier IncomingSignal PhotographicSignal LightField Carrier IncidentModulatedSignal ReferenceCarrier 99MHz Mask basedOpticalHeterodyningforLightFieldcapture SIGGRAPH 2007 ComputationalPhotography CodinginSpace f ModulatedLightField fx f 0 fx0 ModulationFunction SensorSlicecapturesentireLightField Mask basedOpticalHeterodyningforLightFieldcapture SIGGRAPH 2007 ComputationalPhotography CodinginSpace DemodulationtorecoverLightField f fx Rearrange1DFourierTransforminto2D 1DFourierTransformofSensorSignal Mask basedOpticalHeterodyningforLightFieldcapture SIGGRAPH 2007 ComputationalPhotography CodinginSpace Computing4DLightField 2DSensorPhoto 1800 1800 2DFourierTransform 2DFFT Rearrange2Dtilesinto4Dplanes 200 200 9 9 4DIFFT 4DLightField 9 9 81spectralcopies 200 200 9 9 Mask basedOpticalHeterodyningforLightFieldcapture SIGGRAPH 2007 ComputationalPhotography CodinginSpace 光线的空间调制 在2Dsensor上采集4D光场Mask是改变光线传播方向 对其进行调制的关键器件 Mask basedOpticalHeterodyningforLightFieldcapture SIGGRAPH 2007 ComputationalPhotography CodinginIllumination Non photorealisticCamera DepthEdgeDetectionandStylizedRenderingusingMulti FlashImaging SIGGRAPH2004 Non photorealisticCamera DepthEdgeDetectionandStylizedRenderingusingMulti FlashImaging SIGGRAPH2004 Non photorealisticCamera DepthEdgeDetectionandStylizedRenderingusingMulti FlashImaging SIGGRAPH2004 Non photorealisticCamera DepthEdgeDetectionandStylizedRenderingusingMulti FlashImaging SIGGRAPH2004 Non photorealisticCamera DepthEdgeDetectionandStylizedRenderingusingMulti FlashImaging SIGGRAPH2004 CannyIntensityEdgeDetection result Photo Result Ourmethod Photo ComputationalPhotography CodinginIllumination Non photorealisticCamera DepthEdgeDetectionandStylizedRenderingusingMulti FlashImaging SIGGRAPH2004 多闪光的光照位置 方向不同 成像结果提取编码光照下场景的本质信息 深度边缘基于场景结构信息进行非真实感渲染 ComputationalPhotography CodinginIllumination Non photorealisticCamera DepthEdgeDetectionandStylizedRenderingusingMulti FlashImaging SIGGRAPH2004 WatchVideo ComputationalPhotography CodinginWavelength ComputationalPhotography CodinginWavelength AgileSpectrumImaging ProgrammableWavelengthModulationforCamerasandProjectors EG2008 不同波段的光 光路设计 引导传播路径 在空间上分离原空间耦合的光信号 通过光路调制 实现空间分离通过空间滤波可以自由选择光谱 ComputationalPhotography CodinginWavelength AgileSpectrumImaging ProgrammableWavelengthModulationforCamerasandProjectors EG2008 提纲 ComputationalPhotographyCodedTime Exposure CodinginSpaceCodedIlluminationCodedWavelengthInverseProblemsinComputationalLightTransportTransientImagingSpatialHeterodyningRank constraintof3DDisplaysComputationalProbesRandomizedProjectionsOthersSecondSkin ABio I OPlatformforWearableFabricSix DimensionalDisplaysBio inspiredVision bettercaptureandsharevisualinformationviaanewclassofimagingplatforms InverseProblemsinComputationalLightTransport TransientImaging TransientImaging exploitsthefinitespeedoflightandanalyzes echoesoflight Combinetherecentadvancesinultra fastlightsensingandilluminationwithanoveltheoreticalframework LookingAroundCornersusingFemto Photography MarrPrize ICCV2009 InverseProblemsinComputationalLightTransport TransientImaging LookingAroundCornersusingFemto Photography MarrPrize ICCV2009 InverseProblemsinComputationalLightTransport TransientImaging LookingAroundCornersusingFemto Photography MarrPrize ICCV2009 InverseProblemsinComputationalLightTransport TransientImaging LookingAroundCornersusingFemto Photography MarrPrize ICCV2009 InverseProblemsinComputationalLightTransport TransientImaging 暂态成像 光的有限传播速度 极高速信号发生 作用 采样 采样光的传输路径突破光直线传播的成像局限性 对于非视线上的场景信号进行计算重构 看到被遮挡物体提出了新的光路计算框架 被遮挡信息体现于光与场景的多次相互作用中 多次反射 LookingAroundCornersusingFemto Photography MarrPrize ICCV2009 InverseProblemsinComputationalLightTransport SpatialHeterodyning LightSensitiveDisplays DepthCameras Multitouch SeamlesstransitionofmultitouchtogestureThinpackage LCD BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 InverseProblemsinComputationalLightTransport SpatialHeterodyning Capture3 DtoenabledepthandlightingawareinteractionCollocatedimagecaptureanddisplayWalk upinteractionMobilityandportability thindevice BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 DesignOverview Displaywithembeddedopticalsensors SharpMicroelectronicsOpticalMulti touchPrototype BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 DesignOverview Displaywithembeddedopticalsensors LCD displayingmask Opticalsensorarray 2 5cm 50cm BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 DesignVision Object CollocatedCaptureandDisplay BareSensor SpatialLightModulator BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 Theory Depthfromlight fieldcapture LCDusedforSpatialHeterodyneLightFieldCaptureandDisplayPinholesortiledbroadbandmasksSeparateSensorandMaskFourierRefocusingDepthfromfocusBlobtracking gesture BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 DesiredPrototype ActualPrototype BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 Pipeline MaxContrastOperator Pipeline BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 Manipulatinganobjectwith3Dgesture InverseProblemsinComputationalLightTransport SpatialHeterodyning 交互式系统 采集 显示 采集多触式 深度感知手势等更灵活的外界控制信号 实时计算控制显示输出空间外差调制 通过光场采集 2Dsensor上采集4D光场信号 实现深度信息的获取LCD复用 既用作显示又用作采集的调制器 实现大尺度光场信号采集 从而实现深度感知交互式设备的迅猛发展及应用需求推动了该创新研究成果 仅仅一个LCD即可实现更高层的人机交互应用 BiDiScreen AThin Depth SensingLCDfor3DInteractionusingLightFields SIGGRAPH2009 InverseProblemsinComputationalLightTransport Rank constraintof3DDisplays Content AdaptiveParallaxBarriers OptimizingDual Layer3DDisplaysusingLow RankLightFieldFactorization SIGGRAPHAsia2010 InverseProblemsinComputationalLightTransport Rank constraintof3DDisplays Conventional Content AdaptiveParallaxBarriers OptimizingDual Layer3DDisplaysusingLow RankLightFieldFactorization SIGGRAPHAsia2010 InverseProblemsinComputationalLightTransport Rank constraintof3DDisplays Content Adaptive Content AdaptiveParallaxBarriers OptimizingDual Layer3DDisplaysusingLow RankLightFieldFactorization SIGGRAPHAsia2010 InverseProblemsinComputationalLightTransport Rank constraintof3DDisplays Content AdaptiveParallaxBarriers OptimizingDual Layer3DDisplaysusingLow RankLightFieldFactorization SIGGRAPHAsia2010 ViewsonicVX2265wm120HzLCD InverseProblemsinComputationalLightTransport Rank constraintof3DDisplays Content AdaptiveParallaxBarriers OptimizingDual Layer3DDisplaysusingLow RankLightFieldFactorization SIGGRAPHAsia2010 自由视点立体电视的传统实现原理及控制显示方式理论分析前后LCD所构成的立体视频信息的本质结构 秩1在秩1约束下 如何对前后LCD层进行共同优化提高立体视频显示质量在现有立体显示器上的巧妙改进与优化控制 InverseProblemsinComputationalLightTransport Rank constraintof3DDisplays Content AdaptiveParallaxBarriers OptimizingDual Layer3DDisplaysusingLow RankLightFieldFactorization SIGGRAPHAsia2010 InverseProblemsinComputationalLightTransport ComputationalProbes Bokode ImperceptibleVisualTagsforCamera basedInteractionfromaDistance SIGGRAPH2009NETRA InteractiveDisplayforSelf evaluationofanEyeforVisualAccommodationandFocalRange SIGGRAPH2010CATRA CataractProbewithaLightfieldDisplayandaSnap onEyepieceforMobilePhones V Pamplona E Passos J Zizka M Oliveira E Lawson E Clua R Raskar SIGGRAPH2011 NETRA InteractiveDisplayforEstimatingRefractiveErrorsandFocalRange VitorPamplonaAnkitMohanManuelM OliveiraRameshRaskar 57 NETRA InteractiveDisplayforSelf evaluationofanEyeforVisualAccommodationandFocalRange SIGGRAPH2010 InverseProblemsinComputationalLightTransport ComputationalProbes CNN NETRA Cheap PortableEyeExamSystemBBC SmartvisionformobilephonesinthedevelopingworldFoxNews NewdevicemayeliminateeyedoctorappointmentsBusinessWeek PCWorld IDG PerfectSightusescellphonetodiagnoseeyeglassprescription 29researchteamsin14countriesalreadyhaveourprototypes Shack Hartmann LightfieldsLevoyetal2009ZhangandLevoy2009 ObservableLightFieldOh Raskar Barbastathis2009 AugmentedLightField Shack HartmannWavefrontSensor Laser Sensor 59 MicrolensArray PlanarWavefront Shack Platt1971Liangetal1994DavidWilliamsetal Rochester SpotDiagram Laser Sensor 60 Displacement LocalSlopeoftheWavefront SpotDiagram Shack HartmannWavefrontSensor NETRA InteractiveDisplayforSelf evaluationofanEyeforVisualAccommodationandFocalRange SIGGRAPH2010 NETRA InverseofShack Hartmann 61 SpotDiagramonLCD CellPhoneDisplay EyePiece 62 SpotDiagramonLCD InverseofShack HartmannUserinteractivelycreatestheSpotDiagram Displace25pointsbut3parameters RefractiveErrorsandShiftedRange Perfectvision Myopia Hyperopia 10cm Infinity 63 10D 0D Diopter 1 Distance 3D 1D 1m 33cm 1D 3D Distance Diopter NETRA InteractiveDisplayforSelf evaluationofanEyeforVisualAccommodationandFocalRange SIGGRAPH2010 SummaryofInteraction FarthestPoint myopia hyperopia astigmatism NearestPoint presbyopia AccommodationRange 64 Overview 65 InverseofShackHartmannWavefrontSensorHi resdisplays interactionMeasuringSphericalErrorNomovingparts lasersBlur Alignmentproblem LightfieldDisplayforSingleEyeAstigmatismNovelPatternsFocalRangeUserStudy InverseProblemsinComputationalLightTransport ComputationalProbes CATRA CataractMapswithSnap onEyepieceforMobilePhones siggraph2011 CATRA CataractProbewithaLightfieldDisplayandaSnap onEyepieceforMobilePhones V Pamplona E Passos J Zizka M Oliveira E Lawson E Clua R Raskar SIGGRAPH2011 Bokode AnkitMohanCameraCultureGroup MITMediaLab ImperceptibleVisualTagsforCameraBasedInteractionFromaDistance Bokode ImperceptibleVisualTagsforCamera basedInteractionfromaDistance SIGGRAPH2009 idealbarcode invisibletothehumaneye easilydecodablebyamachine Bokode ImperceptibleVisualTagsforCamera basedInteractionfromaDistance SIGGRAPH2009 visiblelightbarcodes angle Bokode ImperceptibleVisualTagsforCamera basedInteractionfromaDistance SIGGRAPH2009 Bokode ImperceptibleVisualTagsforCamera basedInteractionfromaDistance SIGGRAPH2009 InverseProblemsinComputationalLightTransport RandomizedProjections AisMxN M N 高速周期信号 编码频闪 压缩感知 单相机高速运动采集 WatchVideo CodedStrobingPhotography CompressiveSensingofHighSpeedPeriodicVideos PAMI2011 提纲 ComputationalPhotographyCodedTime Exposure CodinginSpaceCodedIlluminationCodedWavelengthInverseProblemsinComputationalLightTransportTransientImagingSpatialHeterodyningRank constraintof3DDisplaysComputationalProbesRandomizedProjectionsOthersSecondSkin ABio I OPlatformforWearableFabricSix DimensionalDisplaysBio inspiredVision bettercaptureandsharevisualinformationviaanewclassofimagingplatforms SecondSkin ABio I OPlatformforWearableFabric Prakash Lighting AwareMotionCaptureUsingPhotosensingMarkersandMultiplexedIlluminators Siggraph2007 HiddenMarkerTagsOutdoorsUniqueId High speedIRCamera MedicalRehabilitation AthleteAnalysis PerformanceCapture BiomechanicalAnalysis SecondSkin ABio I OPlatformforWearableFabric Prakash Lighting AwareMotionCaptureUsingPhotosensingMarkersandMultiplexedIlluminators Siggraph2007 InverseOpticalMo Cap HighSpeedCameraReflecting EmittingMarkerOnlyLocation HighSpeedProjectorPhotosensingMarkerLocation Orientation Illum SecondSkin ABio I OPlatformforWearableFabric Prakash Lighting AwareMotionCaptureUsingPhotosensingMarkersandMultiplexedIlluminators Siggraph2007 LabelingSpace IndoorGPS Eachlocationreceivesauniquetemporalcode But60Hzvideoprojectoristooslow Projector Tag x 0 x 255 Time SecondSkin ABio I OPlatformforWearableFabric Prakash Lighting AwareMotionCaptureUsingPhotosensingMarkersandMultiplexedIlluminators Siggraph2007 PatternMSB PatternMSB 1 PatternLSB ForeachtagFromprojectedpatternsequence decodexcoordinateFromprojectedpatternsequence decodeycoordinateTransmitback Id x y 0 1 1 0 0 X 12 BinaryGray codes SecondSkin ABio I OPlatformforWearableFabric Prakash Lighting AwareMotionCaptureUsingPhotosensingMarkersandMultiplexedIlluminators Siggraph2007 FastPatternProjector 60Hz 3location sec10 000Hz 500locations sec SecondSkin ABio I OPlatformforWearableFabric Prakash Lighting AwareMotionCaptureUsingPhotosensingMarkersandMultiplexedIlluminators Siggraph2007 FixedMasks BlinkingLEDs FastSwitchingusingNon colocatedEmittersforStructuredLight Tag SecondSkin ABio I OPlatformforWearableFabric Prakash Lighting AwareMotionCaptureUs