RequirementsonMulti-viewVideoCodingv.2_W6834.doc

INTERNATIONAL ORGANISATION FOR STANDARDISATIONORGANISATION INTERNATIONALE DE NORMALISATIONISO/IEC JTC1/SC29/WG11CODING OF MOVING PICTURES AND AUDIOISO/IEC JTC1/SC29/WG11 N6834Palma de Mallorca, Spain, October 2004Source:Requirements Title:Requirements on Multi-view Video Coding v.2Status:Approved1Introduction12Applications22.1Free Viewpoint Video (FVV) / Free Viewpoint Television (FTV)22.23DTV32.3Immersive teleconference33Requirements for Multi-view Video Coding33.1Compression related requirements33.1.1Compression efficiency33.1.2Scalability33.1.3Performance efficiency33.1.4Low delay33.1.5Robustness33.1.6Resolution, Color space and depth33.1.7Quality consistency among views33.1.8View random access, partial decoding and rendering43.1.9Temporal random access43.1.10Camera motion43.1.11Resource management43.2System support related requirements43.2.1Synchronization43.2.2View generation43.2.3Non-planar imaging and display systems43.2.4Camera parameters44References41 IntroductionThere have been many input documents brought to MPEG in the last 2 years on free viewpoint video in which multiple-view video coding techniques show improved coding efficiency over existing MPEG compression tools. In this time, it has been recognized that Multi-view Video Coding (MVC) is a key technology that serves a wide variety of applications, including FTV (free-viewpoint television), 3DTV (3D television) and surveillance. In response to a “Call for Comments on 3DAV” 3, a large number of companies have expressed their need for standards that enable FTV (free viewpoint television) and 3DTV. Multi-view video coding (MVC) is an encoding framework for multiple video streams and associated camera parameters. This document first presents some cases where MVC is applicable, and then the requirements for MVC.A list of 3DAV applications and the reasons why they require multiple-view video coding are given in the Application and Requirements document 1. Details on the technology itself are described in the “Report on 3DAV Exploration” 2. 2 Applications2.1 Free Viewpoint Video (FVV) / Free Viewpoint Television (FTV)In this application scenario, the viewpoint and view direction can be interactively changed which may be different from any of the input ones, i.e., those at which the original videos are shot. During such viewing, the viewers experience the free viewpoint navigation within the range covered by the shooting cameras. Such a scenario can appear in the below applications:1. Entertainment concert, sport, multi-user game, movie2. Education cultural archives, manual with real video, instruction of sports playing, medical surgery3. Sightseeing zoo, aquarium, botanical garden, museum.4. Surveillance traffic intersection, underground parking, bank5. Archive space archive, living national treasures, traditional entertainment6. Art/Content creation of new type of media art and digital contentThe basic components of an example FTV system are depicted in Figure 1. The output images from the MVC decoder are used for FTV view generation; this view generation procedure may interpolate images from different views. To achieve high-quality view generation results, a correction process (i.e., rectification of misalignment （未对准校正）and normalization of colors（颜色标准化）) is necessary in most cases. In the example FTV system shown in Figure 1, the correction is applied prior to encoding. Figure 1: Basic components of an example FTV systemA more detailed architecture of an example FTV decoder is depicted in Figure 2. Input streams to the FTV decoder include multi-view video elementary information, video resource management information, timing information, and camera parameters information. In this architecture, the MVC decoder provides reconstructed video data, which is then used in the view generation process. Note that camera parameters may also be used during the MVC decoding process. Video resource management information may be used for managing the picture memory in an efficient way and for generating predictive images for the MVC decoder. Finally, view generation is performed according to the video data information and associated camera parameters information.Figure2: Example architecture of an FTV decoder 2.2 3DTVThis can be thought of as an extension of the current stereoscopic movie. In stereoscopic movie, all viewers share the same viewpoint. In 3DTV, multiple cameras are used to capture the light field of the scene. When such a light field is displayed, multiple viewers can see different stereoscopic views consistent with their relative locations. The application potentials are similar to above. 2.3 Immersive teleconference In both scenarios above, there is interactivity between the viewers and the video content, but not between the views themselves. In immersive teleconference, participants at different geographical sites meet virtually and see one another in either free viewpoint or 3DTV style. The immersiveness provides a more natural way of communications.1. corporate teleconference2. remote trainingRefer to 1 for more information on relevant applications.3 Requirements for Multi-view Video Coding Note that in the sequel, we use “shall” if a certain requirement is very important, and “should” if a certain requirement is desirable.3.1 Compression related requirements3.1.1 Compression efficiency MVC shall provide high compression efficiency. Some overhead may be necessary to ease view interpolation, (i.e., trading coding efficiency for functionality). However, the overhead data should be limited, in order to increase acceptance of the new services.3.1.2 ScalabilityVarious types of scalability should be supported including SNR scalability, spatial scalability, temporal scalability, complexity scalability, and view scalability. This enables to display the same content on a multitude of terminals and network condition that exhibit a variety of capabilities. 3.1.3 Performance efficiency MVC should be efficient in terms of computation complexity and resource consumption. 3.1.4 Low delay MVC should support modes that have low encoding and decoding delay, view change delay, and end-to-end delay.3.1.5 Robustness Robustness to errors (also known as error resilience) should be supported. This enables the delivery of 3D content on error-prone networks such as wireless networks and other networks.3.1.6 Resolution, Color space and depth MVC should support a variety of resolutions (e.g. QCIF, CIF, SD, or HD), color space (e.g. YCrCb 4:4:4, 4:2:2 and 4:2:0 samplings, or RGB) with and color depth up to 16 bits per pixel component. 3.1.7 Quality consistency among views MVC should provide perceptually similar visual quality over different views to be presented at the same time frame. 3.1.8 View random access, partial decoding and rendering MVC should support random view access (e.g. view switching), and partial decoding of a certain subset of views.3.1.9 Temporal random accessMVC should support random access at a certain time.3.1.10 Camera motionMVC should support encodi