Video4Linux2_en.doc

The Video4Linux2 API: an introductionPosted October 11, 2006 by corbetYour editor has recently had the opportunity to write a Linux driver for a camera device - the camera which will be packaged with the One Laptop Per Child system, in particular. This driver works with the internal kernel API designed for such purposes: the Video4Linux2 API. In the process of writing this code, your editor made the shocking discovery that, in fact, this API is not particularly well documented - though the user-space side is, instead,quite well documented indeed. In an attempt to remedy the situation somewhat, LWN will, over the coming months, publish a series of articles describing how to write drivers for the V4L2 interface.V4L2 has a long history - the first gleam came into Bill Dirkss eye back around August of 1998. Development proceeded for years, and the V4L2 API was finally merged into the mainline in November, 2002, when2.5.46was released. To this day, however, quite a few Linux drivers do not support the newer API; the conversion process is an ongoing task. Meanwhile, the V4L2 API continues to evolve, with some major changes being made in 2.6.18. Applications which work with V4L2 remain relatively scarce.V4L2 is designed to support a wide variety of devices, only some of which are truly video in nature: Thevideo capture interfacegrabs video data from a tuner or camera device. For many, video capture will be the primary application for V4L2. Since your editors experience is strongest in this area, this series will tend to emphasize the capture API, but there is more to V4L2 than that. Thevideo output interfaceallows applications to drive peripherals which can provide video images - perhaps in the form of a television signal - outside of the computer. A variant of the capture interface can be found in thevideo overlay interface, whose job is to facilitate the direct display of video data from a capture device. Video data moves directly from the capture device to the display, without passing through the systems CPU. TheVBI interfacesprovide access to data transmitted during the video blanking interval. There are two of them, the raw and sliced interfaces, which differ in the amount of processing of the VBI data performed in hardware. Theradio interfaceprovides access to audio streams from AM and FM tuner devices.Other types of devices are possible. The V4L2 API has some stubs for codec and effect devices, both of which perform transformations on video data streams. Those areas have not yet been completely specified, however, much less implemented. There are also the teletext and radio data system interfaces currently implemented in the older V4L1 API; those have not been moved to V4L2 and there do not appear to be any immediate plans to do so.Video devices differ from many others in the vast number of ways in which they can be configured. As a result, much of a V4L2 driver implements code which enables applications to discover a given devices capabilities and to configure that device to operate in the desired manner. The V4L2 API defines several dozen callbacks for the configuration of parameters like tuner frequencies, windowing and cropping, frame rates, video compression, image parameters (brightness, contrast, .), video standards, video formats, etc. Much of this series will be devoted to looking at how this configuration process happens.Then, there is the small task of actually performing I/O at video rates in an efficient manner. The V4L2 API defines three different ways of moving video data between user space and the peripheral, some of which can be on the complex side. Separate articles will look at video I/O and the video-buf layer which has been provided to handle common tasks.Subsequent articles will appear every few weeks, and will be added to the list below: Part 2: registration andopen() Part 3: Basic ioctl() handling Part 4: Inputs and Outputs Part 5a: Colors and formats Part 5b: Format negotiation Part 6a: Basic frame I/O Part 6b: Streaming I/O Part 7: ControlsVideo4Linux2 part 2: registration and open()Posted October 18, 2006 by corbetTheLWN.net Video4Linux2 API series.This is the second article in the LWN series on writing drivers for the Video4Linux2 kernel interface; those who have not yet seenthe introductory articlemay wish to start there. This installment will look at the overall structure of a Video4Linux driver and the device registration process. Before starting, it is worth noting that there are two resources which will prove invaluable for anybody working with video drivers: TheV4L2 API Specification. This document covers the API from the user-space point of view, but, to a great extent, V4L2 drivers implement that API directly. So most of the structures are the same, and the semantics of the V4L2 calls are clearly laid out. Print a copy (consider cutting out the Free Documentation License text to save trees) and keep it somewhere within easy reach. The vivi driver found in the kernel source asdrivers/media/video/vivi.c. It is a virtual driver, in that it generates test patterns and does not actually interface to any hardware. As such, it serves as a relatively clear illustration of how V4L2 drivers should be written.To start, every V4L2 driver must include the requisite header file: #include Much of the needed information is there. When digging through the headers as a driver author, however, youll also want to have a look atinclude/media/v4l2-dev.h, which defines many of the structures youll be working with.A video driver will probably have sections which deal with the PCI or USB bus (for example); well not spend much time on that part of the driver here. There is often an internal i2c interface, whichwillbe examined later on in this article series. Then, there is the interface to the V4L2 subsystem. That interface is built aroundstruct video_device, which represents a V4L2 device. Covering everything that goes into this structure will be the topic of several articles; here well just have an overview.Thenamefield ofstruct video_deviceis a name for the type of device; it will appear in kernel log messages and in sysfs. The name usually matches the name of the driver.There are two fields to describe what type of device is being represented. The first (type) looks like a holdover from the Video4Linux1 API; it can have one of four values: VFL_TYPE_GRABBERindicates a frame grabber device - including cameras, tuners, and such. VFL_TYPE_VBIis for devices which pull information transmitted during the video blanking interval. VFL_TYPE_RADIOfor radio devices. VFL_TYPE_VTXfor videotext devices.If your device can perform more than one of the above functions, a separate V4L2 device should be registered for each of the supported functions. In V4L2, however, any of the registered devices can be called upon to function in any of the supported modes. What it comes down to is that, for V4L2, there is really only need for a single device, but compatibility with the older Video4Linux API requires that individual devices be registered for each function.The second field, calledtype2, is a bitmask describing the devices capabilities in more detail. It can contain any of the following values: VID_TYPE_CAPTURE: the device can capture video data. VID_TYPE_TUNER: it can tune to different frequencies. VID_TYPE_TELETEXT: it can grab teletext data. VID_TYPE_OVERLAY: it can overlay video data directly into the frame buffer. VID_TYPE_CHROMAKEY: a special form of overlay capability where the video data is only displayed where the underlying frame buffer contains pixels of a specific color. VID_TYPE_CLIPPING: it can clip overlay data. VID_TYPE_FRAMERAM: it uses memory located in the frame buffer device. VID_TYPE_SCALES: it can scale video data. VID_TYPE_MONOCHROME: it is a monochrome-only device. VID_TYPE_SUBCAPTURE: it can capture sub-areas of the image. VID_TYPE_MPEG_DECODER: it can decode MPEG streams. VID_TYPE_MPEG_ENCODER: it can encode MPEG streams. VID_TYPE_MJPEG_DECODER: it can decode MJPEG streams. VID_TYPE_MJPEG_ENCODER: it can encode MJPEG streams.Another field initialized by all V4L2 drivers isminor, which is the desired minor number for the device. Usually this field will be set to -1, which causes the Video4Linux subsystem to allocate a minor number at registration time.There are also three distinct sets of function pointers found withinstruct video_device. The first, consisting of a single function, is therelease()method. If a device lacks arelease()function, the kernel will complain (your editor was amused to note that it refers offending programmers to an LWN article). Therelease()function is important: for various reasons, references to avideo_devicestructure can remain long after that last video application has closed its file descriptor. Those references can remain after the device has been unregistered. For this reason, it is not safe to free the structure until therelease()method has been called. So, often, this function consists of a simplekfree()call.Thevideo_devicestructure contains within it afile_operationsstructure with the usual function pointers. Video drivers will always needopen()andrelease()operations; note thatthisrelease()is called whenever the device is closed, not when it can be freed as with the other function with the same name described above. There will often be aread()orwrite()method, depending on whether the device performs input or output; note, however, that for streaming video devices, there are other ways of transferring data. Most devices which handle streaming video data will need to implementpoll()andmmap(). AndeveryV4l2 device needs anioctl()method - but they can usevideo_ioctl2(), which is provided by the V4L2 subsystem.The third set of methods, stored in thevideo_devicestructure itself, makes up the core of the V4L2 API. There are several dozen of them, handling various device configuration operations, streaming I/O, and more.Finally, a useful field to know from the beginning isdebug. Setting it to either (or both - its a bitmask) ofV4L2_DEBUG_IOCTLandV4L2_DEBUG_IOCTL_ARGwill yield a fair amount of debugging output which can help a befuddled programmer figure out why a driver and an application are failing to understand each other.Video device registrationOnce thevideo_devicestructure has been set up, it should be registered with: int video_register_device(struct video_device *vfd, int type, int nr);Here,vfdis the device structure,typeis the same value found in itstypefield, andnris, again, the desired minor number (or -1 for dynamic allocation). The return value should be zero; a negative error code indicates that something went badly wrong. As always, one should be aware that the devices methods can be called immediately once the device is registered; do not callvideo_register_device()until everything is ready to go.A device can be unregistered with: void video_unregister_device(struct video_device *vfd);Stay tuned for the next article in this series, which will begin to look at the implementation of some of these methods.open() and release()Every V4L2 device will need anopen()method, which will have the usual prototype: int (*open)(struct inode *inode, struct file *filp);The first thing anopen()method will normally do is to locate an internal device corresponding to the giveninode; this is done by keying on the minor number stored ininode. A certain amount of initialization can be performed; this can also be a good time to power up the hardware if it has a power-down option.The V4L2 specification defines some conventions which are relevant here. One is that, by design, all V4L2 devices can have multiple open file descriptors at any given time. The purpose here is to allow one application to display (or generate) video data while another one, perhaps, tweaks control values. So, while certain V4L2 operations (actually reading and writing video data, in particular) can be made exclusive to a single file descriptor, the device as a whole should support multiple open descriptors.Another convention worth mentioning is that theopen()method should not, in general, make changes to the operating parameters currently set in the hardware. It should be possible to run a command-line program which configures a camera according to a certain set of desires (resolution, video format, etc.), then run an entirely separate application to, for example, capture a frame from the camera. This mode would not work if the cameras settings were reset in the middle, so a V4L2 driver should endeavor to keep existing settings until an application explicitly resets them.Therelease()method performs any needed cleanup. Since video devices can have multiple open file descriptors,release()will need to decrement a counter and check before doing anything radical. If the just-closed file descriptor was being used to transfer data, it may necessary to shut down the DMA engine and perform other cleanups.The next installment in this series will start into the long process of querying device capabilities and configuring operating modes. Stay tuned.Video4Linux2 part 3: Basic ioctl() handlingPosted October 30, 2006 by corbetTheLWN.net Video4Linux2 API series.Anybody who has spent any amount of time working through theVideo4Linux2 API specificationwill have certainly noted that V4L2 makes heavy use of theioctl()interface. Perhaps more than just about any other type of peripheral, video hardware has a vast number of knobs to tweak. Video streams have many parameters associated with them, and, often, there is quite a bit of processing done in the hardware. Trying to operate video hardware outside of its well-supported modes can lead to poor performance at best, and often no performance at all. So there is no alternative to exposing many of the hardwares features and quirks to the end application. Traditionally, video drivers have includedioctl()functions of approximately the same length as a Neal Stephenson novel; while the functions often come to more satisfying conclusions than the novels, they do tend to drag a lot in the middle. So the V4L2 API was changed in 2.6.18; the interminableioctl()function has been replaced with a large set of callbacks which implement the individualioctl()functions. There are, in fact, 79 of them in 2.6.19-rc3. Fortunately, most drivers need not implement all - or even most - of the possible callbacks.What has really happened is that the longioctl()function has been moved intodrivers/media/video/videodev.c. This code handles the movement of data between user and kernel space and dispatches individualioctl()calls to the driver. To use it, the driver need only usevideo_ioctl2()as itsioctl()method in thevideo_devicestructure. Actually, most drivers should be able to use it asunlocked_ioctl()instead; the locking within the Video4Linux2 layer can handle it, and drivers should have proper locking in place as well.The first callback your driver is likely to implement is: int (*vidioc_querycap)(struct file *file, void *priv, struct v4l2_capability *cap);This function handles theVIDIOC_QUERYCAPioctl(), which asks a simple who are you and what can you do? question. Implementing it is mandatory for V4L2 drivers. In this function, as with all other V4L2 callbacks, theprivargument is the contents offile-private_datafield; the usual practice is to point it at the drivers internal structure representing the device atopen()time.The driver should respond by filling in the structurecapand returning the usual zero or negative error code value. On successful return, the V4L2 layer will take care of copying the response back into user space.Thev4l2_capabilitystructure (defined in) looks like this: struct v4l2_capability _u8 driver16; /* i.e. bttv */ _u8 card32; /* i.e. Hauppauge WinTV */ _u8 bus_info32; /* PCI: + pci_name(pci_dev) */ _u32 version; /* should use KERNEL_VERSION() */ _u32 capabilities; /* Device capabilities */ _u32 reserved4; ;Thedriverfield should be filled in with the name of the device driver, while thecardfield should have a description of the hardware behind this particular device. Not all drivers bother with thebus_infofield; those that do usually use something like: sprintf(cap-bus_info, PCI:%s, pci_name(&my_dev);Theversionfield holds a version number for the driver. Thecapabilitiesfield is a bitmask describing various things that the driver can do: V4L2_CAP_VIDEO_CAPTURE: The device can capture video data. V4L2_CAP_VIDEO_OUTPUT: The device can perform video output. V4L2_CAP_VIDEO_OVERLAY: It can do video overlay onto the frame buffer. V4L2_CAP_VBI_CAPTURE: It can capture raw video blanking interval data. V4L2_CAP_VBI_OUTPUT: It can do raw VBI output. V4L2_CAP_SLICED_VBI_CAPTURE: It can do sliced VBI capture. V4L2_CAP_SLICED_VBI_OUTPUT: It can do sliced VBI output. V4L2_CAP_RDS_CAPTURE: It can capture Radio Data System (RDS) data. V4L2_CAP_TUNER: It has a computer-controllable tuner. V4L2_CAP_AUDIO: It can capture audio data. V4L2_CAP_RADIO: It is a radio device. V4L2_CAP_READWRITE: It supports theread()and/orwrite()system calls; very few devices will support both. It makes little sense to write to a camera, normally. V4L2_CAP_ASYNCIO: It sup