IROS2019国际学术会议论文集1971

Sequential clustering for tactile image compression to enable direct adaptive feedback Andreas Geier1,2, Gang Yan1, Tito Pradhono Tomo1, Shun Ogasa1, Sophon Somlor1, Alexander Schmitz1, and Shigeki Sugano1 AbstractThe sense of touch is often crucial for humans to perform manipulation tasks. Providing tactile feedback during teleoperation or for users of prosthetic devices would be benefi cial. However, the representation of tactile information constitutes a major technical challenge, since the numerous and possibly multimodal sensor readings are massive compared to the available tactile display technology. We introduce an algorithm that deploys two stages of K-means clustering along and across tactile image frames that render tactile sensor information at each time instant. In this manner, the massive tactile information is adaptively compressed in real-time while preserving its physical meaning, thus, remains intuitive and di- rect. We experimentally verify and examine the characteristics of our algorithm by evaluating the original and compressed tactile data. The data was gathered during the active tactile exploration of several objects of daily living by an Allegro robot hand that was covered with 15 uSkin sensor modules providing 240 3-axis force vector measurements at each time instant. Our novel algorithm is straight forward enough to be implemented into tactile feedback systems. Finally, our algorithm allows for the direct feedback of massive tactile sensor data for a broad variety of tactile sensors and tactile displays, thereby, enables the compressed yet intuitive representation of massive tactile sensor information for real-time applications. I. INTRODUCTION The sense of touch provides humans with rich and direct feedback while physically interacting with their environment. Clearly, the resembling, e.g. in teleoperation applications, or restoration, e.g. after injury, of tactile information would provide humans with the tactile feedback required for safe and sovereign interaction. Recently, several works presented the large-scale implementation of tactile sensors on robotic hands 1 2 3 4. However, the meaningful and intuitive representation of the massive raw sensor information to humans presents itself diffi cult. In particular, the display of meaningful feedback via visual cues is a limitation in itself and tactile displays suffer from a limited number of actuators and a limited quantization of force production 5. Extracting and displaying meaningful and intuitive infor- mation from the massive amounts of sensory data poses challenges on the software, which has to process this data This research was supported by the JSPS Grant-in-Aid for Young Sci- entists (B) No.17K18183 and (B) No. 19K14948 and the Grant-in-Aid for Scientifi c Research No. 19H02116 and No. 19H01130. Additional fi nancial support was provided by the Ministry of Education, Science, Sports and Culture of Japan (Monbukagakusho). 1The authors are with Faculty of Science and Engineering, Modern Mechanical Engineering, Waseda University, 169 Tokyo, Japan. contact: a geiersugano.mech.waseda.ac.jp 2The author is with Rostock Medical Center, Department of Orthopaedics, 18057 Rostock, Germany to make it usable. The information that is relevant to a task is often obscured behind redundancy and sensory noise. Compression techniques are a key tool in many applications 6, such as image processing 7 8 or generally infor- mation retrieval 9 10. Apart from the immense variety of compression techniques, K-means clustering is due to its fl exibility an often preferred choice over more compli- cated algorithms for clustering, e.g. Bayesian probabilistic approaches. This paper introduces an algorithm that deploys two stages of K-means clustering along and across tactile image frames that render the tactile information at each time instant. In this manner, the massive tactile information is adaptively compressed in real-time while preserving its physical mean- ing; i.e. the tactile information remains intuitive and direct. The algorithm is verifi ed by evaluating the original and compressed tactile data that was gathered during the active tactile exploration of several objects of daily living by a multi-fi ngered robot hand which is covered with distributed tactile sensors 11. The paper is organized as follows: In Section II we review related works of high-resolution tactile sensors and state- of-the-art tactile displays. Following this, in Section III, we present the experimental setup and procedure that were used to evaluate our algorithm on experimental data. Then, Section IV describes the foundations and advantages of the sequential clustering algorithm for tactile space compression. Next, Section V shows the results and evaluates them in the light of current tactile display limitations. Finally, Section VI discusses our novel approach, draws conclusion, and gives directions for future works. II. RELATED WORKS Many tactile sensors have been developed 1 12; some of them are implemented as distributed skin sensors and enable dense tactile measurements in multi-fi ngered robot hands. For example, 241 tactile sensors cover most of the surface of the TWENDY-ONE hand 2 the Gifu III hand has an immense number of 859 pressure sensing points 13 and in 14 a 3-fi ngered gripper with multimodal tactile sensors was presented. Tactile sensors were also integrated in the small fi ngertips and the palms of the humanoid iCub 3, and the Shadow hand was covered with Tekscan sensors 15. A vision-based tactile sensor called GelSlim can provide high resolution tactile images using a webcam and a coated elastomeric gel with the size of 5050mm and a thickness of 20mm 4. GelSlim can provide texture and topological 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Macau, China, November 4-8, 2019 978-1-7281-4003-2/19/$31.00 2019 IEEE8111 information about the object in contact with the sensor. This kind of rich tactile information is challenging to be directly represented by existing tactile displays. Recent reviews on tactile displays and related technology can be found in 5 16 17. Only few haptic displays can display a lot of information on a condensed area: For example, 285 ultrasound transducers are integrated in a 1919cm2area 18, in 19 400 pins were arrayed over a 1cm2area, 20 implemented 60 laterally moving skin contactors with a spatial resolution of 1.81.2mm2, and an electro-tactile display with 512 electrodes has been realized in 21. However, the manufacturing of high-density tactile dis- plays is still challenging. Up-to-date, many of them are still in the prototype stage and usually not compact enough or even wearable. In this context, compression of tactile information has been rarely investigated 22. It is an open question how tactile compression techniques can preserve the physical meaning of massive sensory data in order to provide useful information to the human by means of rather compact tactile displays. III. SYSTEMARCHITECTURE To test our algorithm, we use a previously presented exper- imental setup 11. In this section, we provide a quick review of our system architecture and the experimental procedure. A. Allegro robot hand with uSkin sensor modules Our lab developed a compact, soft, and distributed 3-axis skin sensor (called uSkin). Two versions of uSkin modules are available: a fl at module for phalanges with 44 taxels 23 and a curved module for fi ngertips with 46 taxels 24. We mounted uSkin modules on an Allegro robot hand, a commercial robot hand from Wonik Robotics1as depicted in Fig. 1. In total, there are eleven fl at modules mounted on the phalanges and four curved modules mounted on the fi ngertips. We utilized the MTB3 25, which is a small microcontroller that was originally developed by the Italian Institute of Technology for their humanoid robot iCub for managing taxel readings via I2C (Inter-Integrated Circuit) communication. In the current implementation, one MTB3 can only read up to 16 taxels simultaneously. Therefore, we activated only 16 taxels for all fi ngertip modules to reduce the number of required microcontrollers. Thus, although our Allegro hand can deliver up to 272 3-axis tactile readings in total, we only used 240 3-axis readings in the current confi guration. Each microcontroller was daisy-chained via CAN (Controlled Area Network) bus. We connected the mi- crocontroller to a Windows PC using a CAN/USB (Universal Serial Bus) converter that is available from ESD Electronics2. Up to eight microcontrollers can be connected simultane- ously using one CAN/USB converter. Therefore, we used two CAN/USB converters to read all 15 uSkin modules. In our previous experimental setup 11, we controlled and measured the Allegro hands joint angles using a Linux PC, 1http:/www.simlab.co.kr/ 2https:/esd.eu/en Fig. 1.Allegro robot hand with uSkin sensor modules (left) and the arrangement of the active taxels within the uSkin modules (right). for technical reasons. However, uSkin makes use of the ESD- CAN library that requires a Windows environment. Due to these compatibility issues, we implemented a TCP/IP com- munication in which the uSkin sensor data was bypassed to the Linux PC via LAN (local area network) communication. This enabled the simultaneous recording of the uSkin sensor and joint angle data at a sampling rate of roughly 30Hz. B. Experimental setup and procedure For the data collection, we mounted the Allegro robot hand on a sturdy aluminum profi le, the fi ngers pointed upward as shown in Fig. 1. The Allegro hand was connected to a Linux PC and was controlled using ROS (Robot Operating System) via a PEAK CAN/USB converter. We pre-recorded a grasping motion that dynamically moved all four fi ngers to actively explore an object. The data collection sequence started with all fi ngers in a fully open state. Then, we placed 1 out of 20 everyday objects (Yale-CMU-Berkeley (YCB) object model set) into the Allegro hand and triggered the motion sequence together with the data logging software that streamed the skin sensor readings and the 16 servo motors joint angles into a .csv (comma separated value)- fi le. One data set consisted of the readings of 240 3-axis skin sensors and 16 joint angles. Note that, however, the joint angle information was not used in this application. C. Sequential clustering for tactile image compression The goal of the algorithm is to enable direct and adaptive feedback while dynamically interacting with arbitrary ob- jects, yet is simple enough to enable rapid computation on a number of devices. While there is a relatively high number of 720 sensor readings across our Allegro robot hand available, the information displayed to the user must be compressed, either for technical reasons such as the technical limitations of tactile displays, or for reasons specifi c to the application such as in prosthetics where the tactile sensation must be reproduced on another body part. The compression, however, should preserve the physical meaning. In addressing the initially outlined limitations of current tactile displays, we sequentially deploy two K-means clus- tering algorithms that, fi rst, adaptively adjust the range of measured sensor readouts to the force range that can be generated by the tactile display and, second, estimate the location of contact points in regard of the actuator limitations 8112 of the tactile display. In this manner, the tactile sensor information in the contacting areas is adaptively compressed, however, in a way that exploits the full range of the available tactile display space. D. Evaluation study For the evaluation of the algorithm, we exemplarily ana- lyze the grasping of the football from the YCB object model set, Fig. 2. Specifi cally, we analyze the time instants at which the tip of the index fi nger engages and disengages with the football by comparing the raw sensor information of the 240 taxels to the compressed information in the tactile image space. Concretely, the x,y,z-taxel readings are treated similar to RGB-pixels that allow the entirety of sensor readings to be rendered into a RGB-like color image. These 15163- image frames serve as 16 bit input for our compression algorithm and are processed and output into 8 bit grayscale images that represent the tactile display space, thus, called tactile images. Finally, the algorithm is verifi ed by analyzing tactile images that refer to time instants in which the football is tightly grasped. We evaluate the algorithm in terms of the capacity to estimate the location of the contact points and in terms of how the sensor readings at the contact point are effi ciently represented after force quantization. IV. ALGORITHM DESCRIPTION K-means clustering is an iterative method of vector quan- tization; it belongs to the unsupervised learning algorithms and partitions an N-dimensional data set into K distinguished clusters of equal size and shape in which each data point be- longs to the cluster with the shortest Euclidean distance, i.e. the distance measured as the sum of the squared differences of the coordinates in each direction. Let us denote the data set at each time instant t as X = x1,.,xN where each of the N coordinates xnis a D- dimensional vector and is going to be assigned to exactly one of the clusters k 1,.,K. The assignment of data point xn to cluster k is denoted by an= k and permits the calculation of a cluster centroid k. Within each iteration, the algorithm alternates between updating the cluster assignments anand updating the cluster centroids kuntil the maximum number of iterations I is reached. K-means clustering is conceptually simple and its compu- tational cost scales with the dimensionality and size of the data set. Relevant key features of K-means clustering are the assumption of an a-priorily fi xed number of K clusters, an isotropic data space, and a geometric distance measure prone to be biased by outliers. Even though, some of the above mentioned features may be disadvantageous for many applications, we are going to show that they can be made use of in order to meet the hardware limitations of state-of-the-art tactile displays, namely, the need for force quantization addressing the limited range of force generation capacity and actuator quantization addressing the limited number of actuators or distinct contact points. Table I summarizes the complete algorithm for sequential clustering for tactile image compression in a fl ow chart. TABLE I ALGORITHM FLOWCHART VariablesNS: number of sensors NC: number of contacts Inputst: tactile frame force quantizationID: number of iterations KD: number of clusters actuator quantizationIA: number of iterations KA: number of clusters Outputs00 t: tactile image Initializationt relative tactile frame, eqn. (1) contact threshold, eqn. (2) fnS mask faulty sensors force quantizationKD initialization , eqn. (3) (*) for iD 1,.,ID for nS 1,.,NS if (snS andfnS6= 1) for kD 1,.,KD dnS,kD euclidean distance, eqn. (4) anS assignment, eqn. (5) else 0t(snS) 0 for kD 1,.,KD kD centroid update, eqn. (6) (*) 0t(snS) recovery, eqn. (7) actuator quantizationif (NC KA) KA initialization, eqn. (8) for iA 1,.,IA for nC 1,.,NC for kA 1,.,KA dnC,kA euclidean distance, eqn. (9) anC assignment, eqn. (10) for kA 1,.,KA kA centroid update, eqn. (11) else 00 t(snS) = 0t(cn), eqn. (12) (*) The centroids kD can be determined offl ine and kept constant during run-time to maintain a consistent force range. Details can be found in Section IV - C. Force quantization. A. Data arrangement As depicted in Fig. 1 and 3, at each time instant t, the 44 3-axis sensor readouts for each of the 15 uSkin modules are rendered into an image-like input frame of the size (nW= 16)(nH= 16)(nD= 3), where nWrefers to the width, nHrefers to the height, and nDrefers to the channel number. Since tactile displays usually do not provide the ability to discriminate between shear and normal forces, the input frame is simplifi ed into a (nW=16)(nH=16) tactile frame, summarized as t RnWnH, where each of the NS= 256 entries corresponds to the magnitude snS= ? ?sn S ? ?2 2 of its respective sensor readings snS= ?s x sysz?T nS. Please note, that the phalanx of the thumb carries only three uSkin modules for which reason the missing sensor readings are treated similar to faulty sensor readings. 8113 Fig. 2.Allegro robot hand actively grasping a football. Fig. 3.Arrangement of tactile sensor readouts into RGB-like input frames. B. Initialization Generally, the following initialization procedures should be run after implementation of the tactile sensors onto the robot hand and before contact with any object is made. First, faulty sensors were automatically identifi ed by sta- tistically evaluating the 240 sensor readings in the x,y,z-axis for the initial frame 0, respectively. Within this work, a sensor reading, and hence the complete sensor element, is considered as faulty and excluded from further calculations, if one of the readings sx,nS,sy,nS,sz,nSexceeds the arithmetic mean x,y,zamong all N = 240 sensor readings by one standard deviation x,y,z, respectively. Accordingly, a sensor snSis assigned with the attribute fnS= 1, if faulty and fnS= 0, if operational. Second, the threshold upon which exceedence a contact is established, is found by constructing the relative tactile frame tas the absolute difference between the initial frame 0and a frame tas t=1= |t=10|.(1) Then, the contact threshold is defi ned as the maximum magnitude snSaccording to = max(1(snS) for nS 1,.,NS.(2) Once the contact threshold is defi ned, the parts of the Allegro robot hand covered with the uSkin modules can detect contact events. C. Force quantization The force quantization refers to the