外文翻译--视觉跟踪问题向鲁棒非线性控制挑战

国际性刊物 .关于鲁棒非线性控制 Int.J. 鲁棒非线性控制 2000； 10:875-888 视觉跟踪问题 :向鲁棒非线性控制挑战 艾伦 泰勒保蒙 美国亚特兰大 GA 30082号格鲁吉亚工学院电子和计算机研究部门 视觉跟踪问题 :向鲁棒非线性控制挑战 摘要 ： 视觉跟踪是在控制活动图象中关键性问题之一。因为模型中存在不可靠性和噪音信号、使它变成鲁棒控制中一挑战性问题。在此论文中、我们将概略一些涉及的关键问题和一些可能的解答方案，我们想需要联系机器视觉和多刻度图像处理技术来实现这个工作任务。特别地，我们会概略一些 必需的方法例如：计算机视觉和图像处理包含光学流程图在内、活动的轮廓（横向振荡），和几何学上的被驱动流程。该论文如此也便会有指导意义 。版权 2000 约翰怀利和几个有限公司 关键字 ：目视跟踪 鲁棒控制 视觉运动 活动轮廓 光学流程 通用换算图像处理 1. 引言 在这论文里、我们考虑那些被控制的跟踪眼运动活动图象问题。这个问题的解决方案需综合控制理论、信号处理和计算机视觉等技术。我们想表明有可能由利用能适应的鲁棒控制与信号处理通用换算方法以及计算机视觉形状识别理论协同一起来处理此问题，跟踪是一基点控制问题即我 们需要的输出的遵循或跟踪基准信号，或相当于我们想使那跟踪误差相对于一些定义明确的标准（说能量、能力、峰值、等等 .)尽可能小的。即使镜头推进存在的一些干扰是一典型控制问题、这个手头上的问题是很难和挑战性，因为有非常不确定的自然干扰性。一个可以认为眼球运动跟踪问题是在一人机连接的环境之下。特别地，我们将推断计算机用户视觉运动，哪个动作是用户打算完成的。例如：在电脑荧屏上打开一个文件夹、展开一下拉菜单项等等。这类电脑跟踪系统将作为包含我们的视觉跟踪方法学一个具体的实例。重要强调是已经有相当多在测量视觉运动方面的研 究已经在被控制的实验室环境下完成、何况研究上已经在应用中用到此知识。从此我们将考虑接下来在大范围中跟踪问题的适应性，包括机器人、远距操纵的车辆，并且飞行员带瞄准器的飞行帽当前也正在发展中。后者是系统的结合钢盔支柱头和视觉跟踪能力以分辩隶属的正确的视线。显而易见，正当的开发跟踪观察者眼睛的人类视觉系统的那动态特性将导致大大幅度削减传输中所需要的信息。关于视觉跟踪和眼运动（与大量的信息一样）的讨论多半被发现在参考 1-4中 . 我们应该明白视觉跟踪问题不同于标准追踪问题，因为反馈信号是用图像感应器测量的。尤其 是，它必须在经由计算机视觉运算法提取而且在被用于控制回之前被一个推论运算法解释。反应速度是关键性因素。因为明显的理由、一视觉跟踪系统应该是尽可能的非侵入的。眼球运动通过由灰度或红外线照相机或由一系列感应器得到的获取图象来完成跟踪的 .该图像是经过解析来求出眼睛和头部的相对运动的。该低能级数据获取和识别的完成是经由一通用换算技术得到的。此技术提供许多优越性。第一，它使得信号处理的查找和医学上的图象获取过程明朗化、它导致实时信号捕获和了解程序快速化。识别将利用一通用换算方法和一新的计算理论形式来完成。 因此、从该 视频信息交换窗的控制点出发、我们有一跟踪问题在非常不可预测的干扰哪个我们想使其衰减的。注意到该不可靠应归于传感器噪音（典型的）、该算法的构成如上所述（不可测性求出起重要作用、各种各样的假设的可能性、等等 .)，和模型不定性。我们应该注意到论文将拥有指导气息。实际上，我们的一个关键目的是引进从活动图象到该鲁棒控制共同体的一些核心概念。我们坚信共同体便会从研究彼此方法和问题得益。尤其是、因为视觉跟踪和该眼球运动跟踪的特有的课题、该鲁棒控制在明确地处理不可靠性的技术我们将要在接下来做相应的讨论。 2. 视觉跟踪 视 觉跟踪存在许多论文，通过利用视力在货车交货、特别是在机器人共同体（看例如、涉及 1, 2, 5-9，和那参考在那里面的 .)代表性地、该问题定位在本研究中是通过利用视觉跟踪服务机械手（或一等值的难题）。该动机于使用视力在这一体制中是非常清晰的，那就是说，该综合计算机视觉和控制可以是用到改善测量，因为改善图像处理技术和硬件中、机器人技术到达指向就是要视觉信息可以变成不可分割那部分反馈信号。因为问题带有少量不定性、单纯的 PID控制器已经使用，并且越来越用于嘈杂的系统、能适应的方案以及随机建立的已经使用的线性二次 高斯控制器。 许多控制原理图已经建议使用视觉信息来控制回路。这已经绵亘传感零件到表征多级控制结构应用范围而言、傅里叶描述符，和局部图像分析法；看参考 9-11，和那里面的参考。以光学的流程为基础的有好的方法已经被当作机械手的计算一个主要成份使用驱动信号（阅读详细的，请参考 7, 8，和我们以下的论述）。的确，因为在一个图像中的一个物体由光亮式样组成 , 如同物体搬进空间一样光亮也是式样。 光学的流程然后是光亮的明显运动式样 ；参考 12。在标准的假定之下 , 被提供的一个静态的目标和一台感人的照相机 ( 或相等地 ,一个静态的物体和一台感人的照相机 ),一个可以在快速时写下象平面上对象凸出点的时差。若干方法已经讨论适合于计算此速度。此知识也将能用来跟踪图像。 让我们考虑追踪一个计算机荧屏上的目标具体问题。然后我们有唯一的一二维的跟踪目标点，我们为简单起见假定那目标周旋于原子能飞机垂直于光轴照相机，如果那照相机移动然后带有翻译速度的移动 : Xy 和旋转速度盎司（就那照相机设计）而言、可能形成二维的跟踪问题如同下述适合于一个目标。让电压互感器标志目标的投影 在象平面区域， T然后在视觉上跟踪保持稳定的电压互感器达到查找照相机翻译和转动盎司（就那照相机设计）。有类似的描述适合于跟踪的零件。现在由此机构，就可以写下运动的照相机光流产生的 Z形线时差，而其控制变量是由包括照相机的跟踪运动在内的流程所给定的。那可执行代码程序带表格可能在参考 13、 7查到，并且无须涉及我们现在所讨论的课题 .重点是产生的系统可能被写入标准状态的空白表格之后离散化 (同那定时在 .之间二相邻帧 ) 之后接纳那表格。 x(n+1)=x(n)+ Tru(n)=Td(n)+v(n) z(n)=x(n)+w(n) 这里 ,d 是外界产生的干扰 , v 是噪音 ，称做适合于模型不定性， z 是是那量度与噪声分量 w 一起。 ( 所有的矢量都包含在。 .状态向量由 x ， y 和摇晃因素形成的跟踪误差构成的 )。 有许多重要的控制问题与这样的设备有关系、当然，一个人量度延迟（我们想工作实时）和抽样时间选择的问题。但是我们感觉有许多深刻地和很多难题，必须被编址合理可选择的控制策略成功之前。换句话说 ,一般说来不定性（ v and w）是塑造成均匀频谱噪声。这模型是保 守的，没有加入使用帐户任何可能噪声环境的结构。现代鲁棒控制一个关键贡献考虑了的结构不定性（参考 14和以及里面其他内容）。在我们现在的情形中，我们倡导深刻的研究有关不定性方面的问题。这将导致信号处理的关键元素，尤其是，新的强大的计算机通用换算方法 .计算机视觉形状识别理论以汉密尔顿 -雅各比理论为基准将会起到一个关键角色。以下我们将讨论此话题，尤其是讨论更多明确地处理那眼球运动信息并在反馈电路使用它的方法。 3、眼睛运动信息 当许多研究已经完成眼睛运动测量的时候，少量研究已经在应用中采用这些知识。一些侵入的 视觉跟踪技术测量角膜和视网膜电势差而其他的人正在使用 fit精密地轨道飞行器角膜的隐形眼镜。非侵入的技术识别零件位于眼睛的图像上，比如虹膜或瞳孔或那角膜的界限条件反射的眼睛敏感的光线，并且从此零件推论出运动的眼睛，所以此零件的运动还可以应归于头部的运动，重要的是跟踪若干零件并求出此零件差速运动。例如：、电学上近红外眼睛跟踪系统使用角膜表面反射的光的相对运动而且反射视网膜跟随用户的目光。关于眼睛跟踪技术的好的调查报告参考 3, 15.也可以参考手册 16。 大部分从事利用眼球运动知识已经集中于联络残疾人 使用者，例如，参考 17, 18。其他的研究强调应用于类似于我们在考虑包含 19-23中的。这开创性研究尝试通过利用眼球运动在人机通信并且证明理论的可能性 . 他们也确定这些趋近于人机通信的实用真实问题和缺陷，其必须解决以便使其实用逼真以至可以对抗和补充其他的人机连接工具比如鼠标、钢笔和图形输入卡、驾驶杆以及键盘。 趋近于人机通信的视觉跟踪幕后的理论基础是目标被一受支配的似人的结构发现。如果是这样，在视网膜成像的小的区域称作小凹。因此使用者的眼睛位置提供有关他集中于屏蔽的区域的信息。该信息是精确到内部 小凹角宽度的（大约一度）。此精度已经被认为是适宜于上述研究中的人机通信（好的精确度可能需要眼肌的研究，但是我们这里不讨论 .)生物学研究者指出大量普遍的眼睛运动分成二类扫视与定位。（参考 3, 24, 4与该参考中一些扫视与定影的论述以及他们和生物学幻像的关系。这些研究同时包含大规模视觉跟踪的一些基本问题 .)扫视是突然运动视觉从场景中的一个感兴趣区到另一个。扫视通常伴随间隔为 200到 600毫秒的定影，该视觉仍然完成包括小于一度的小的神经过敏运动。其他的形式的眼睛运动可能在人机通信期中可能被忽略。 建立人机 通信的视觉跟踪的基本问题是确定定影周期（亦即，用户感觉他在视觉上已经定位在一目标上的时间），然后在肌肉神经过敏视觉运动定位在目标上。注意人类忽视这定影期间的神经过敏视觉运动，那就是说我们注视一个目标我们以为我们是早看现场的单个斑点。也要注意到纯理论的过滤器接近于除去该眼睛神经过敏不是令人满意地，既它也可能减缓正确位置扫视的响应时间。 4. 数据获得 有二基本对数据获得的方法 : 与那一个类似的活跃方式藉着追踪系统和图 像技术的无源现在眼睛用。 在有效的方法中，一种无害近红外线灯用来照明用户的脸。两个来自眼睛的 反射波被求得。是前者反射信号起因于角膜表面被称作反射。后者反射发生来自视网膜被称作明亮视觉成分。为最小化背景辐射效果，目前的系统一般需要暗淡的光。本文中，我们将钻研无源的方法。 5. 追踪及光学流程 一旦我们想要跟踪与对象相当的信息已经确定，我们使用一个光学流程估计各零件从两个连续图象，只包含那个零件。该图象是当时和以前的 .被全部删除唯一的在研究中的轮廓成像中得到的。这产生运动判断问题比用纯理论的强度成像的传统的流程判断适应度好了很多。该光学流程的计算机应用已经被用于解决活动幻像中出现的问题的一个重要的工具 。该光流域是一序列的成像的明亮图像的视在运动的速度矢量场 41。假定该明亮图像是相对运动的结果，足够大的空间来寄存图像上光亮度在空间分配方面的改变。因此，一个目标和一架相机的相对运动能够引起光学流程。同样地，一工人静电照相机成像的场景中目标的相对运动可以引起光学流程。在我们光学流程的计算机应用上，我们使用通用化粘性溶液的汉密尔顿 -雅各比样式方程式，此技术似乎理论上适于这种变化，欧拉拉格朗日接近于此问题（参见 41-43里面相关的内容）。利用这种通用的方法，我们已能处理那些在光学流程区域出现的若干相 异的变分公式的奇异性和规律性问题。这类方法论已经被用于描影法造型问题 44, 45，和边缘检测 46。 分割和数据分析 从一数据列提取是有干扰的视觉轨迹，然后问题变为数据断片进入扫视和定位间隙和测定预定注视点。一个非常接近的通用换算也被用于这些步骤中。该定位判断问题由于若干干扰来源而变得复杂。特别地，眼光闪烁及其他人为现象可能没有终止地发生在定位过程中。在这些人为现象有的能够持久 200毫秒，数据可能丢失。为更好的目标定位和分裂眼睛运动跟踪，可以使用点连续离散最大概似法接近于细化开始和结束间歇以 及定点的测定可能值。该方式是结点连续非连续的测定问题因为注视点能够取连续的值，然而分割成 定位 和 非定位 间隔是一个二元问题。它是建立在视觉活动定位和持续的统计学上的模型，它是建立在最理想的贝叶斯定理概念上的。它用一个预测步骤来填充在扫视过程中丢失的数据。该推算步骤以假设视觉或者在定位方式或者在扫视方式为基准提出两个推定量。该假设任何一个取舍是在监视若干读取信息后计算各可能性之后取舍的。延迟关联文件这过程限于大约毫秒，这个过程将不会为用户所注意到。 如上所述还可以容易地贝叶斯统计放到递减率蛇形线来处理抽 取的轮廓；参考 50。因此建立边缘检测算法的通用换算几何学上的横向振荡很自然地变成其贝叶斯定理的框架。注意我们不需要发送有关扫视运动的相关信息。当注视点改变时，我们唯一的提供有关跟踪程序初始的和最后的定点的信息。然而，加快人机通信过程，我们使用扫视运动去推算下一个定点将会在哪。 扫视运动可以由一隐藏的马尔可夫模型描述参考 51中相关内容。此外，马尔可夫模型可能用来综述在当前上内容可能的用户操作。在隐藏的马尔可夫模型描述的这扫视各状态关联到特别的运动形式阶段的随机模型。经结合用户意图的随机的描述是以呈 现背景和扫视的隐藏的马尔可夫模型为基准的，我们可以导出一推算下一个注视点将在的位置的判断法则。 结论 在论文中，我们考虑用一通用方法论来研究眼光跟踪问题。我们应该注意因为信号和模型中的不定性，此类问题为在鲁棒控制方面的研究工作者提供了一个极其难得的机会。我们论述个综合鲁棒控制、计算机视觉，和信号处理通用换算的方法。最近、存在多种人员参加在研究幻像和控制。大部分研究到目前已经分支。当分支已经结束时很多基本控制算法已经用于混合的结果。利用视觉信息在反馈电路中能够提供珍贵的新的研究来源，它有潜力推动控制研究的一 个新的领域。在本问中论述的视觉跟踪问题可以作为一范例适合于活动图象控制的整个范围。的确，该问题向鲁棒控制系统开启了一次有力的挑战。 感谢 我们要感谢明尼苏达大学的加里 .巴拉教授和安培 .特费科教授，他们提供了许多非常有用的关于视觉跟踪的交谈。该研究在某种程度上得到以下各单位的授权和支持：国家科学基金会电子计算机系统 -9700588，国家科学基金会局部信息处理系统，自动跟踪科学研究空军办公室 F49620 - 98 - 1 - 0168、军队 DAAG55 - 98 - 1 - 0169研究室。 INTERNATIONAL JOURNAL OF ROBUST AND NONLINEAR CONTROL Int. J. Robust Nonlinear Control 2000； 10:875-888 On the eye tracking problem: a challenge for robust control Allen Tannenbaum Department of Electrical and Computer Engineering, Georgia Institute of Technology,Attanta,GA30082,U.S.A. SUMMARY： Eye tracking is one of the key problems in controlled active vision. Because of modeling uncertainty and noise in the signals, it becomes a challenging problem for robust control. In this paper, we outline some of the key issues involved as well as some possible solutions. We will need to make contact with techniques from machine vision and multi-scale image processing in carrying out this task. In particular, we will sketch some of the necessary methods from computer vision and image processing including optical flow, active contours (snakes), and geometric driven flows. The paper will thus have a tutorial flavor as well. Copyright 2000John Wiley & Sons, Ltd. KEY WORDS: visual tracking； robust control； eye movements； active contours； optical flow； multi scale image processing 1. INTRODUCTION In this paper, we consider the controlled active vision problem of tracking eye movements. The solution of this problem requires the integration of techniques from control theory, signal processing and computer vision. We will show that it is possible to treat this problem by using adaptive and robust control in conjunction with multi scale methods from signal processing, and shape recognition theory from computer vision. Tracking is a basic control problem in which we want the output to follow or track a reference signal, or equivalently we want to make the tracking error as small as possible relative to some well-defined criterion (say energy, power, peak value, etc.). Even though tracking in the presence of a disturbance is a classical control issue, the problem at hand is very difficult and challenging because of the highly uncertain nature of the disturbance. One can consider the eye movement tracking problem in the context of a man-computer interface. In particular, we would like to infer from the movement of the eyes of a computer user, which actions the user intends to accomplish, e.g., opening a folder on a computer screen, expanding a pull-down menu item, etc. This type of computer tracking system should serve as a concrete illustration of what is involved in our eye tracking methodology. It is important to emphasize that while considerable research has been performed in the area of measuring eye movements in a controlled laboratory environment, much less research has been done on using this information in applications. Hence the techniques which we will discuss below should have a wide range of applicability in a number of tracking problems including those in robotics, remotely controlled vehicles, and pilot tracking helmets currently being developed. The latter are systems combining helmet mounted head and eye track capability to define a subjects true line of sight. Clearly, the proper exploitation of the dynamic characteristics of the human visual system by tracking the position of the viewers eyes leads to drastic reduction in the amount of information that needs to be transmitted. Discussions of visual tracking and eye movements (together with an extensive list of references) may be found in References 1-4. We should note that the problem of visual tracking differs from standard tracking problems in that the feedback signal is measured using imaging sensors. In particular, it has to be extracted via computer vision algorithms and interpreted by a reasoning algorithm before being used in the control loop. Furthermore, the response speed is a critical aspect. For obvious reasons, an eye tracking system should be as non-invasive as possible. The eye movement is tracked by studying images acquired by grey scale or infra-red cameras or by an array of sensors. The images are analyzed to extract the relative motions of the eyes and the head. The low-level data acquisition and recognition is accomplished via a multi scale technique. This technique offers a number of advantages. First, as is becoming clear from research in signal processing and medical image acquisition procedures, it leads to fast real time signal acquisition and understanding procedures. Recognition will be accomplished using a multi scale approach and a new computational theory of shape. Consequently, from the control point of view, we have a tracking problem in the presence of a highly uncertain disturbance which we want to attenuate. Note that the uncertainty is due to the sensor noise (classical), the algorithmic component described above (uncertainty in extracted features, likelihood of various hypotheses, etc.), and modeling uncertainty. We should note that this paper will have a tutorial flavor. Indeed, one of our key motivations is to introduce some key concepts from active vision to the robust control community. We strongly believe that both communities will derive much benefit by studying each others methods and problems. In particular, for visual tracking and for the specific proble m of eye movement tracking, the techniques of robust control which explicitly treat uncertainty can be relevant as we will argue below. 2. VISUAL TRACKING There have been a number of papers on the use of vision in tracking, especially in the robotics community. (See e.g., References 1, 2, 5-9, and the references therein.) Typically, the issue addressed in this work is the use of vision for servings a manipulator for tracking (or an equivalent problem). The motivation for using vision in such a framework is clear, that is, the combination of computer vision coupled with control can be employed to improve the measurements. Indeed, because of improvements in image-processing techniques and hardware, robotic technology is reaching the point where vision information may become an integral part of the feedback signal. For problems with little uncertainty, simple PID controllers have been used, and for more noisy systems, adaptive schemes as well as stochastic based LQG controllers have been utilized. A number of control schemes have been proposed for the utilization of visual information in control loops. These have ranged from the use of sensory features to characterize hierarchical control structures, Fourier descriptors, and image segmentation； see References 9-11, and the references therein. A promising approach based on optical flow has been used as a key element in the calculation of the robots driving signals (see in particular, References 7, 8, and our discussion below). Indeed, since an object in an image is made up of brightness patterns, as the object moves in space so do the brightness patterns. The optical flow is then the apparent motion of the brightness patterns； see Reference 12. Under standard assumptions, given a static target and a moving camera (or equivalently, a static object and a moving camera), one can write down equations for the velocity of a projected point of the object onto the image plane. Several methods have been discussed for the computations of this velocity. This information can then be used to track the image. Let us consider the concrete problem of tracking targets on a computer screen. Then we have only a two-dimensional tracking question. We assume for simplicity that the object moves in a plane which is perpendicular to the optical axis of the camera. If the camera then moves : Xy with translation velocity and rotational velocity oz (with respect to the camera frame), one may pose the two-dimensional tracking problem as follows for an object 13. Let Pt denote the area on the image plane which is the projection of the target. Then visually tracking this feature amounts to finding the camera translation q and rotation oz (with respect to the camera frame) which keeps Pt stationary. There are similar characterizations for the tracking of features. Now from this set-up, one can write down linear zed equations of the optical flow generated by the motion of the camera where the control variables are given by those components of the optical flow induced by the cameras tracking motion. The exact form may be found in References 13, 7, and need not concern us now. The point is that the resulting system may be written in standard state space form and after discretization (with the time between two consecutive frames) takes on the form x(n+1)=x(n)+ T (n)=Td(n)+v(n) z(n)=x(n)+w(n) where is the reference, d is the exogenous disturbance, v is a &noise term for the model uncertainty, z is the measurement together with noise component w. (All the vectors are in.The components of the state vector are made up of the x, y, and roll component of the tracking error.) There are a number of important control issues related to such a set-up. Of course, one has the problem of measurement delays (we want to work in real time) and choice of sampling time. But we feel there is a much deeper and more difficult problem which must be addressed before a reasonable choice of control strategy can be made. Namely, in general the uncertainty (v and w) is modeled as white noise. This model is conservative and does not bring into account any of the possible structure of noise environment. One of the key contributions in modern robust control has the consideration of structure in uncertainty (see Reference 14 and the references therein). In our case, we are proposing a much deeper analysis of the uncertainty connected to such problems. This brings the key element of signal processing and in particular, the new powerful methods of multi scale computations. Shape recognition theory in computer vision based on Hamilton-Jacobin theory will also play a key role in this program as will be argued below. We will in particular discuss ways of more explicitly processing the eye movement information and employing it in a feedback loop. 3. EYE MOVEMENT INFORMATION While many studies have been performed on the measurement of eye movements, little research has been done on employing this information in applications. Some of the invasive eye tracking techniques measure potential differences between the cornea and retina while others use contact lenses that fit precisely over the cornea. Non-invasive techniques identify features located on an image of the eye, such as the boundary of the iris or pupil or the corneal reflection of a light shone at the eye, and infer the movement of the eye from that of these features. Since the movement of these features could also be due to a movement of the head, it is important to track several features and extract the differential motion of these features. For example, active near-infrared eye tracking systems use the relative motion of the light reflected off the corneal surface and that re flected off the retina to follow the users eye-gaze. A good survey of eye-tracking techniques can be found in References 3, 15. See also the manual 16. Much of the work on using eye movement information has focused on communications with handicapped users, e.g., References 17, 18. Other works that emphasized applications similar to the one that we are considering include 19-23. These pioneering research efforts addressed the use of eye movement in man-computer communications and demonstrated the feasibility of the principle. They also identified the problems and limitations of this approach to user-computer communications that must be solved in order to make it a practical reality that can compete and complement other man-machine interface tools such as the mouse, pen and tablet, joy-stick and keyboard. The rationale behind eye tracking approaches to user-machine communication is that an object is seen by a human subject if it is imaged on a small area of the retina called the fovea. Hence, a users eye position provides information about the area of the screen that he is focusing on. The information is of course accurate to within the angular width of the fovea (about one-degree). This degree of accuracy has been found to be satisfactory for man-computer interaction by previous researchers. (Finer accuracy may be needed for the study of the eye muscles but this is not our concern here.) Biological studies indicate that the most common eye movements fall into two categories: saccade and fixation. (See References 3, 24, 4 and the references therein for discussions of saccade and fixation and their relation to biological vision. These works also contain extensive results on the general problem of eye tracking.) A saccade is a sudden movement of the eye from one area of interest in the scene to another. A saccade is usually followed by a fixation interval of 200 to 600 ms during which the eye still makes small jittery movements covering less than one degree. Other types of eye movements may be neglected in man-machine communication. The basic problem in eye-tracking based user-machine communication then is that of identifying fixation periods (i.e. periods during which the user feels that he is visually fixing at an object), and locating the object of fixation in the presence of the jittery eye motions due to the muscles. Note that humans do ignore these jittery eye motions during fixation, i.e. when we fix an object we are under the impression that we are looking at a single spot in the scene. Note also that a pure filtering approach to eliminate the eye jitters is not satisfactory since it will also slow the response to a true saccade. 4. DATA ACQUISITION There are two basic approaches to data acquisition: an active approach similar to the one used by current eye tracking systems and a passive imaging technique. In the active approach, a harmless near-infrared light is used to illuminate the users face. Two re flections from the eyes are then extracted. The first reflected signal is due to the corneal surface and is called the glint. The second reflection occurs off the retina and is called the bright eye component. To minimize the effect of background radiation, current systems typically require a dim lighting. In this paper, we will concentrate on the passive approach. 5. TRACKING AND OPTICAL FLOW Once the contours corresponding to the various features that we wish to track have been identified, we use an optical flow procedure to estimate the motion of each feature from two consecutive images that contain only that feature. The images are obtained from the current and previous contour images by deleting all contours except for the one of interest. The resulting motion estimation problem is much better conditioned than the traditional optical flow estimation from pure intensity images. The computation of optical flow has proved to be an important tool for problems arising in active vision. The optical flow field is the velocity vector field of apparent motion of brightness patterns in a sequence of images 41. One assumes that the motion of the brightness patterns is the result of relative motion, large enough to register a change in the spatial distribution of intensities on the images. Thus, relative motion between an object and a camera can give rise to optical flow. Similarly, relative motion among objects in a scene being imaged by a static camera can give rise to optical flow. In our computation of the optical flow we use work on generalized viscosity solutions to Hamilton-Jacobin type equations. Indeed, these techniques seem ideally suited for the variation Euler-Lagrange approaches to this problem (see also 41-43 and the references therein). Utilizing such generalized solutions, we have been able to handle the singularities and regularity problems for several distinct variation formulations of the optical flow that occur in this area. This type of methodology has already been applied to the shape-from-shading problem 44, 45, and edge detection 46. 6. SEGMENTATION AND DATA ANALYSIS Given a noisy eye trajectory which is extracted from a sequence of data, the problem then becomes that of segmenting the data into saccade and fixation intervals and estimating the mean fixation point. A multi scale approach is used in this step as well. The fixation estimation problem is complicated by several sources of disturbance. Specifically, eye blinks and other artifacts may occur during a fixation without terminating it. During these artifacts which can last up to 200 ms, data may be absent. To better localize the object of fixation and segment the eye movement track, one can use a joint continuous/discrete maximum likelihood approach to refine the estimates of the beginning and end of fixation intervals and of the location of the fixation point. The approach is a joint continuous/discrete estimation problem because the fixation point can take continuous values whereas the segmentation into fixation and non-fixation intervals is a binary problem. It is based on statistical models of fixation eye activity and duration and is optimal in a Bayesian sense. It uses a prediction step to fill in missing data during blinks. The prediction step produces two estimates based on the hypotheses that the eye is either in a fixation mode or a saccade mode. The hypotheses are either accepted or rejected by computing the likelihood of each after observing a certain number of readings in the future. The delay associated with the procedure is limited to about 150 ms, which should not be noticeable to the u