AudioandVideoCaptureSupport.doc

Audio and Video Capture SupportApplies to: Devices based on Windows Mobile Version 5.0 and laterAlthough the DirectShow implementation on Windows CE is primarily designed to support media playback, it is also possible to create audio and video capture applications.The following topics provide additional information for capturing audio and video. Video Capture Windows Media Applications Pin Property Set 1. Video CaptureApplies to: Devices based on Windows Mobile Version 5.0 and laterThe term video capture, as used in this documentation, describes any application where video is received from a hardware device. Video capture devices are typically cameras but can also be other video sources. The captured video can be saved to disk or previewed live. This section describes some of the concepts you must understand to use DirectShow for video capture: Capture Performance Capture Graph Builder Building Graphs with the Capture Graph Builder Video Capture Filters Capture Device Selection Video Previews Camera Rotation Interfaces for Capture Applications Capturing Video to a File Controlling a Capture Graph Video Capture Tasks Advanced Capture Topics ArchitectureDirectShow uses an object called the Capture Graph Builder to help create filter graphs for video capture. For more information, see Capture Graph Builder. The structure of the filter graph it creates depends on the number of output pins provided by the camera driver. A two-pin camera driver has one pin for video capture and another pin for still image capture. For two-pin drivers, the Capture Graph Builder inserts the Smart Tee filter into the capture graph. This filter splits the incoming video stream into two separate output streams, one for the actual data capture and the other for a live preview of the data, typically for a cameras LCD preview display.The following diagram shows the architecture of a typical capture graph for a two-pin camera driver.A three-pin camera driver has an output pin for video capture, an output pin for still images, and a preview pin to directly. Because the preview pin provides direct driver-level support for a devices preview pipeline, the Capture Graph Builder does not include the Smart Tee filter in the capture graph.The following diagram shows the architecture of a typical capture graph for a three-pin camera driver. The major parts of both the two-pin and three-pin architectures are described below: Camera Driver The camera driver provides the low level interface between the OS and the actual camera hardware. Smart Tee The Smart Tee filter takes a single stream of incoming image data and splits it into two output streams. One stream is used for the actual video capture and the other is used to drive the cameras viewfinder or preview window. Buffer filter The buffer filter is only added to the filter graph if the encoder DMO it is connecting to exposes the major media a filter that This filter is inserted into the filter graph to allow camera devices to support captures that do not take place in real time. This allows camera applications and devices to be developed on lower cost hardware. Color Space Converter Filter (CC) This filter transforms captured video data from one RGB or YUV color space to another. The Capture Graph Builder automatically inserts this filter into the capture and still image pipelines if the video capture filters output pin does not have the same media type as the input pin on the encoder or image sink. The practical implication of this is that you can improve the performance of your capture graph by making sure that these output and input pins have matching media types, thereby eliminating the need for a color conversion step. The video renderer for the preview display must be able to fall back to GDI-based rendering. Because GDI uses RGB color formats, the Capture Graph Builder always inserts the color space converter filter into the preview pipeline if the video capture filters output pin uses a YUV color format. The color space converter filter is omitted from the graph if the preview output from the video capture filter is in an RGB format and bit depth that GDI supports. File Writer The file writer is a multiplexer that combines audio and video data and writes it out to an Advanced Systems Format (ASF) file. DMO Wrapper Filter This filter provides an interface between DirectShow and a DirectX Media Object (DMO). The Windows Media Video encoder is provided as a DMO. Video Renderer The video renderer supplies image data to a cameras viewfinder or preview window. The Video Renderer relies on GDI when it initiates its connections with the filter graph. To support this, the filter upstream from the video renderer must support the RGB color space. Depending on the video capture filters output color format, the Capture Graph Builder may need to insert the color space converter filters before the video renderer. At allocation time, the Video Renderer enumerates all media types supported by the upstream filter. During this enumeration, the Video Renderer attempts to create a DirectDraw surface for each media it encounters. If it can successfully create the surface then it will stop enumerating and use that DirectDraw surface as its rendering target. There are two reasons that the video renderer has to support a GDI mode. First, the display hardware may not have native support for certain stretching and scaling operations. In these cases, the video render cannot use DirectDraw calls for those graphics operations and must instead rely on software-based routines through GDI. Second, in cases where there are complex clipping regions, the video renderer will fall back from DirectDraw-based rendering to GDI-based rendering. When initiating the connection, the Video Renderer needs to use GDI, and as such, the upstream filter needs to support RGB. As RGB is not always available from the video capture driver, the Color Converter needs to be inserted. Before going into Run mode, at allocation time, the Video Renderer will enumerate all media types supported by the upstream filter, and attempt to create DDraw surfaces with those types. If there is a match, then the DDraw surface will be used. Image Sink The image sink is the interface to the Imaging API, which is used to write still images. You can see from the descriptions above that the Capture Graph Builder can configure the capture graph with a number of different filters to accommodate the hardware and encoding parameters of your device. You can improve the performance of your capture graph by carefully planning and coordinating the hardware capabilities and requirements on your device. The best capture graph performance comes occurs when the video renderer can render its output using DirectDraw, and when the color space converter filter is either not required in the graph or is simply acting as a pass-through filter.2. Capture Graph BuilderApplies to: Devices based on Windows Mobile Version 5.0 and laterA filter graph that performs video or audio capture is called a capture graph. Capture graphs are often more complicated than file playback graphs. To make it easier for applications to build capture graphs, DirectShow provides a helper object called the Capture Graph Builder. The Capture Graph Builder exposes the ICaptureGraphBuilder2 interface, which contains methods for building and controlling a capture graph. Start by calling CoCreateInstance to create new instances of the Capture Graph Builder and the Filter Graph Manager. Then initialize the Capture Graph Builder by calling ICaptureGraphBuilder2:SetFiltergraph with a pointer to the Filter Graph Managers IGraphBuilder Interface. The following code shows a helper function to perform these steps.HRESULT InitCaptureGraphBuilder( IGraphBuilder *ppGraph, / Receives the pointer. ICaptureGraphBuilder2 *ppBuild / Receives the pointer.) if (!ppGraph | !ppBuild) return E_POINTER; IGraphBuilder *pGraph = NULL; ICaptureGraphBuilder2 *pBuild = NULL; / Create the Capture Graph Builder. HRESULT hr = CoCreateInstance(CLSID_CaptureGraphBuilder, NULL, CLSCTX_INPROC_SERVER, IID_IcaptureGraphBuilder2, (void*)&pBuild); if (SUCCEEDED(hr) / Create the Filter Graph Manager. hr = CoCreateInstance(CLSID_FilterGraph, 0, CLSCTX_INPROC_SERVER, IID_IGraphBuilder, (void*)&pGraph); if (SUCCEEDED(hr) / Initialize the Capture Graph Builder. pBuild-SetFiltergraph(pGraph); / Return both interface pointers to the caller. *ppBuild = pBuild; *ppGraph = pGraph; / The caller must release both interfaces. return S_OK; else pBuild-Release(); return hr; / FailedThroughout this section on video capture, it is assumed that you are using the Capture Graph Builder to create the capture graph. However, it is possible to build capture graphs entirely by using IGraphBuilder methods. This is considered an advanced topic, however, and the Capture Graph Builder methods are preferred. For more information, see Advanced Capture Topics.3. Building Graphs with the Capture Graph BuilderApplies to: Devices based on Windows Mobile Version 5.0 and laterDespite its name, the Capture Graph Builder is useful for building many kinds of custom filter graphs, not only capture graphs. This topic provides a brief overview of how to use this object. The Capture Graph Builder exposes the ICaptureGraphBuilder2 interface. Start by calling CoCreateInstance to create the Capture Graph Builder and the Filter Graph Manager. Then initialize the Capture Graph Builder by calling ICaptureGraphBuilder2:SetFiltergraph with a pointer to the Filter Graph Manager, as shown in the following code:IGraphBuilder *pGraph = NULL;ICaptureGraphBuilder2 *pBuilder = NULL;/ Create the Filter Graph Manager.HRESULT hr = CoCreateInstance(CLSID_FilterGraph, NULL, CLSCTX_INPROC_SERVER, IID_IGraphBuilder, (void *)&pGraph);if (SUCCEEDED(hr) / Create the Capture Graph Builder. hr = CoCreateInstance(CLSID_CaptureGraphBuilder2, NULL, CLSCTX_INPROC_SERVER, IID_ICaptureGraphBuilder2, (void *)&pBuilder); if (SUCCEEDED(hr) pBuilder-SetFiltergraph(pGraph); ;Connecting FiltersThe ICaptureGraphBuilder2:RenderStream method connects two or three filters together in a chain. Generally, the method works best when each filter has no more than one input pin or output pin of the same type. This discussion begins by ignoring the first two parameters of ICaptureGraphBuilder2:RenderStream and focusing on the last three parameters. The third parameter is an IUnknown pointer, which can specify either a filter (as an IBaseFilter Interface pointer) or an output pin (as an IPin Interface pointer). The fourth and fifth parameters specify IBaseFilter pointers. The ICaptureGraphBuilder2:RenderStream method connects all three filters in a chain. For example, suppose that A, B, and C are filters. Assume for now that each filter has exactly one input pin and one output pin. The following call connects A to B, and then B to C:RenderStream(NULL, NULL, A, B, C) All connections are intelligent, meaning that additional filters are added to the graph as needed. For more information, see Graph Building with Intelligent Connect. To connect just two filters, set the middle value to NULL. For example, this call connects A to C:RenderStream(NULL, NULL, A, NULL, C) You can create longer chains by calling the method twice:RenderStream(NULL, NULL, A, B, C)RenderStream(NULL, NULL, C, D, E)If the last parameter is NULL, the method automatically locates a default renderer. It uses the DirectShow Video Renderer Filter for video and the Waveform Audio Renderer for audio. Thus:RenderStream(NULL, NULL, A, NULL, NULL) is equivalent toRenderStream(NULL, NULL, A, NULL, R) where R is the appropriate renderer. If you specify a filter in the third parameter, rather than a pin, you may need to indicate which output pin should be used for the connection. That is the purpose of the methods first two parameters. The first parameter applies only to capture filters. It specifies a GUID that indicates a pin category. For a complete list of categories, see Pin Property Set. Two of the categories are valid for all capture filters: PIN_CATEGORY_CAPTURE PIN_CATEGORY_PREVIEW If a capture filter does not supply separate pins for capture and preview, the ICaptureGraphBuilder2:RenderStream method inserts a Smart Tee Filter, which splits the stream into a capture stream and a preview stream. From the applications standpoint, you can simply treat all capture filters as having separate pins and ignore the underlying topology of the graph. For file capture, connect the capture pin to a mux filter. For live preview, connect the preview pin to a renderer. If you switch the two categories, the graph might drop an excessive number of frames during the file capture; but if the graph is connected properly, it drops preview frames as needed in order to maintain throughput on the capture stream. The following example shows how to connect both streams:/ Capture to file:pBuilder-RenderStream(&PIN_CATEGORY_CAPTURE, NULL, pCapFilter, NULL,pMux);/ Preview:pBuilder-RenderStream(&PIN_CATEGORY_PREVIEW, NULL, pCapFilter, NULL, NULL);Some capture filters also support closed captions, indicated by PIN_CATEGORY_VBI. To capture the closed captions to a file, render this category to the mux filter. To view the closed captions in your preview window, connect to the renderer:/ Capture to file:pBuilder-RenderStream(&PIN_CATEGORY_VBI, NULL, pCapFilter, NULL, pMux);/ Preview on screen:pBuilder-RenderStream(&PIN_CATEGORY_VBI, NULL, pCapFilter, NULL, NULL);The second parameter to ICaptureGraphBuilder2:RenderFile identifies the media type, and is typically one of the following: MEDIATYPE_Audio MEDIATYPE_Video MEDIATYPE_Interleaved (DV) You can use this parameter whenever the filters output pins support the enumeration of preferred media types. For file sources, the Capture Graph Builder automatically adds a parser filter if needed, and then queries the media types on the parser. Also, if the last filter in the chain has several input pins, the method attempts to enumerate their media types. However, not all filters support this functionality.Finding Interfaces on Filters and Pins After you build a graph, you will typically need to locate various interfaces exposed by filters and pins in the graph. For example, a capture filter might expose the IAMDroppedFrames Interface, while the filters output pins might expose the IAMStreamConfig Interface. The simplest way to find an interface is to use the ICaptureGraphBuilder2:FindInterface method. This method walks the graph (filters and pins) until it locates the desired interface. You can specify the starting point for the search, and you can limit the search to filters upstream or downstream from the starting point.The following example searches for the IAMStreamConfig Interface on a video preview pin:IAMStreamConfig *pConfig = NULL;HRESULT hr = pBuild-FindInterface( &PIN_CATEGORY_PREVIEW, &MEDIATYPE_Video, pVCap, IID_IAMStreamConfig, (void*)&pConfig);if (SUCCESSFUL(hr) /* . */ pConfig-Release();Finding PinsLess commonly, you may need to locate an individual pin on a filter, although in most cases the ICaptureGraphBuilder2:RenderStream and ICaptureGraphBuilder2:FindInterface methods will save you the trouble. If you do need to find a particular pin on a filter, the ICaptureGraphBuilder2:FindPin helper method is useful. Specify the category, the media type (video or audio), the direction, and whether the pin must be unconnected. For example, the following code searches for an unconnected video preview pin on a capture filter:IPin *pPin = NULL;hr = pBuild-FindPin( pCap, / Pointer to the filter to search. PINDIR_OUTPUT, / Search for an output pin. &PIN_CATEGORY_PREVIEW, / Search for a preview pin. &MEDIATYPE_Video, / Search for a video pin. TRUE, / The pin must be unconnected. 0, / Return the first matching pin (index 0). &pPin); / This variable receives the IPin pointer.if (SUCCESSFUL(hr) /* . */ pPin-Release();4. Video Capture FiltersApplies to: Devices based on Windows Mobile Version 5.0 and laterCapture filters in DirectShow have some features that distinguish them from other kinds of filters. Although the Capture Graph Builder hides many of the details, it is a good idea to read this section to get a general understanding of DirectShow capture graphs.Pin CategoriesA capture filter often has two or more output pins that deliver the same kind of data for example, a preview pin and a capture pin. Therefore, media types are not a good way to distinguish the pins. Instead, the pins are distinguished by their functionality, which is identified using a GUID, called the pin category. For a discussion of how to query pins for their category, see Working with Pin Categories. For most applications, however, you will not have to query pins directly. Instead, various ICaptureGraphBuilder2 methods take parameters that specify the pin category on which to operate. The Capture Graph Builder automatically locates the correct pin.Preview Pins and Capture PinsSome video capture devices have separate output pins for preview and capture. The preview pin is used to render video to the screen,