流媒体相关内容总结

1、流媒体相关内容总结1 rtsp协议rtsp（real time stream protocol，实时流协议rfc 2326）是研究如何有效地在ip网络上传输流媒体数据的应用层协议。rtsp的消息有两大类,一是请求消息(request),一是回应消息(response),两种消息的格式不同.请求消息:方法urirtsp版本cr lf消息头cr lfcr lf 消息体cr lf回应消息: rtsp版本 状态码 解释cr lf 消息头cr lfcr lf 消息体cr lf其中rtsp版本一般都是rtsp/1.0,状态码是一个数值,200表示成功,解释是与状态码对应的文本解释。1.1 流媒体服务器wi

2、ndows media server：支持mms协议helix server：推送协议为私有协议，播放协议使用rtspdarvin streaming server：支持rtsp推送和播放。如果是点播播放，本地文件需要索引化。1.2 sdp格式sdp的格式v=o= s=i=u=e=p=c= b=:t= r= z= .k=k=:a=a=:m= v = （协议版本） o = （所有者/创建者和会话标识符） s = （会话名称） i = *（会话信息） u = * （uri 描述） e = * （email 地址） p = * （电话号码） c = * （连接信息） b = * （带宽信息） z =

3、 * （时间区域调整） k = * （加密密钥） a = * （0 个或多个会话属性行）时间描述：t = （会话活动时间） r = * （0或多次重复次数） 媒体描述：m = （媒体名称和传输地址） i = * （媒体标题） c = * （连接信息 如果包含在会话层则该字段可选） b = * （带宽信息） k = * （加密密钥） a = * （0 个或多个媒体属性行）1.3 例子以aac推送流为例。具体代码：devicehandler/aachandler & streamsender1.3.1 推送过程2009-12-29 9:58:02:234announce rtsp:/202.102

4、.126.10/0123456789abcdef.sdp rtsp/1.0cseq:3452816846content-type:application/sdpcontent-length:303v=0o=streamsender 0 0 in ip4 0.0.0.0c=in ip4 202.102.126.10a=tool:sdr v2.4a6m=audio 0 rtp/avp 96a=control:trackid=0a=rtpmap:96 mpeg4-generic/44100/2a=fmtp:96 streamtype=5; profile-level-id=15; mode=aac-

5、hbr; config=1210; sizelength=13; indexlength=3; indexdeltalength=3;a=mpeg4-esid:1012009-12-29 9:58:02:265rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 34528168462009-12-29 9:58:02:265setup rtsp:/202.102.126.10/0123456789abcdef.sdp/trackid=0 rtsp/1

6、.0cseq:3452816847transport:rtp/avp/tcp;unicast;mode=record;interleaved=0-12009-12-29 9:58:02:265rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 3452816847cache-control: no-cachesession: 134866268081914date: tue, 29 dec 2009 01:58:17 gmtexpires: tue,

7、 29 dec 2009 01:58:17 gmttransport: rtp/avp/tcp;unicast;mode=record;interleaved=0-12009-12-29 9:58:02:265record rtsp:/202.102.126.10/0123456789abcdef.sdp rtsp/1.0cseq: 3452816848session: 1348662680819142009-12-29 9:58:02:281rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darw

8、in; state/beta; )cseq: 3452816848session: 134866268081914rtp-info: url=rtsp:/202.102.126.10/0123456789abcdef.sdp/trackid=01.3.2 播放过程2009-12-29 9:58:47:765options rtsp:/202.102.126.10:1554/0123456789abcdef rtsp/1.0cseq: 6user-agent: vlc media player (live555 streaming media v2009.04.20)2009-12-29 9:5

9、8:47:765rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 6public: describe, setup, teardown, play, pause, options, announce, record2009-12-29 9:58:47:765describe rtsp:/202.102.126.10:1554/0123456789abcdef rtsp/1.0cseq: 7accept: application/sdpuser-ag

10、ent: vlc media player (live555 streaming media v2009.04.20)2009-12-29 9:58:47:765rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 7cache-control: no-cachecontent-length: 341date: tue, 29 dec 2009 01:59:02 gmtexpires: tue, 29 dec 2009 01:59:02 gmtcont

11、ent-type: application/sdpx-accept-retransmit: our-retransmitx-accept-dynamic-rate: 1content-base: rtsp:/202.102.126.10:1554/0123456789abcdef/v=0o=streamsender 0 0 in ip4 0.0.0.0s=0123456789abcdef.sdpc=in ip4 0.0.0.0t=0 0a=tool:sdr v2.4a6a=control:*m=audio 0 rtp/avp 96a=control:trackid=0a=rtpmap:96 m

12、peg4-generic/44100/2a=fmtp:96 streamtype=5; profile-level-id=15; mode=aac-hbr; config=1210; sizelength=13; indexlength=3; indexdeltalength=3;a=mpeg4-esid:1012009-12-29 9:58:47:781setup rtsp:/202.102.126.10:1554/0123456789abcdef/trackid=0 rtsp/1.0cseq: 8transport: rtp/avp/tcp;unicast;interleaved=0-1u

13、ser-agent: vlc media player (live555 streaming media v2009.04.20)2009-12-29 9:58:47:781rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 8cache-control: no-cachesession: 14027363211279date: tue, 29 dec 2009 01:58:17 gmtexpires: tue, 29 dec 2009 01:58:

14、17 gmttransport: rtp/avp/tcp;unicast;interleaved=0-12009-12-29 9:58:47:781play rtsp:/202.102.126.10:1554/0123456789abcdef/ rtsp/1.0cseq: 9session: 14027363211279range: npt=0.000-user-agent: vlc media player (live555 streaming media v2009.04.20)2009-12-29 9:58:47:796rtsp/1.0 200 okserver: dss/5.5.5 (

15、build/489.16; platform/win32; release/darwin; state/beta; )cseq: 9session: 14027363211279range: npt=now-rtp-info: url=rtsp:/202.102.126.10:1554/0123456789abcdef/trackid=02 rtp协议rtp(real-time transport protocol，实时传输协议rfc 1889、rfc 3550、rfc 3551)，它是由ietf的多媒体传输工作小组1996年在rfc 1889中公布的。rtp 本身并没有提供按时发送机制或其它

16、服务质量（qos）保证，它依赖于低层服务去实现这一过程。 rtp 并不保证传送或防止无序传送，也不确定底层网络的可靠性。 rtp 实行有序传送， rtp 中的序列号允许接收方重组发送方的包序列。参考streamsender项目。/* tcp通道中传输rtp数据。size:4 */struct _rtptcpheadu_int8 symbol;/$,0x24u_int8 channelid;/interlevelu_int16 len; /rtp包长度，不包括_rtptcphead的长度。;/* rtp包头。size:12 */struct _rtphead/* rtp */u_int8 rtp

17、_cc:4;/0u_int8 rtp_px:2; /0u_int8 rtp_ver:2; /版本号：2u_int8 rtp_pt:7; /负载类型u_int8 rtp_mark:1;/是否是帧的最后分片u_int16 rtp_sequence; /序列号，网络序u_int32 rtp_timestamp;/时间戳，网络序，时间戳增量和采样率关系很大u_int32 rtp_ssrc;/同步源标识，网络序;图形表示为：3 rtp负载首先要说明的是，音频帧的固定长度是1024（编码前）。3.1 h264参考rfc 3984，代码参考devicehandler/streamsender。h264原始流

18、以00 00 00 01(字节)或00 00 01(3字节)开头，紧跟的是1字节的网络抽象层单元nalu。rtp传输h264流的时候，将“0001”头去掉。sdp有两个参数profile-level-id和sprop-parameter-sets，这两个参数由0x670x68结构的数据生成，生成方法参考streamh264。3.1.1 结构定义下面定义了单帧、分片的结构，组包的结构复杂些。/* * h264头, fu-a */* 网络抽象层单元nalu*/struct _naluu_int8 type:5;u_int8 nri:2;u_int8 f:1;单帧模式就是原h264流/* 单一nal

19、 单元模式*/typedef _nalu _h264singleframehead;/* 分片模式*/分片模式有两种类型：28 | fu-a | fragmentation unit29 | fu-b | fragmentation unit一般使用的是fu-a类型。typedef _nalu _fuindicator;struct _fuheaderu_int8 type:5;u_int8 r:1;/0u_int8 e:1;/结束位u_int8 s:1;/开始位;struct _h264fuhead_fuindicator fuindicator;_fuheader fuheader;/*

20、组包模式*/组包模式有四种，两种单时间组合包，24 | stap-a | single-time aggregation packet25 | stap-b | single-time aggregation packet和两种多时间组合包，26 | mtap16 | multi-time aggregation packet 27 | mtap24 | multi-time aggregation packet一般使用的是stap-a，这种模式的格式为：rtpheader(12字节) stap-a nalu(1字节) framesize(2)frameframesizeframe-，与mpe

21、g4组包是不同的，可对比着看。3.1.2 推送过程2009-12-30 15:09:45:843announce rtsp:/202.102.126.10/0123456789abcdef.sdp rtsp/1.0cseq:1content-type:application/sdpcontent-length:328v=0o=streamsender 0 0 in ip4 0.0.0.0c=in ip4 202.102.126.10a=tool:sdr v2.4a6m=video 554 rtp/avp 96a=control:trackid=0a=rtpmap:96 h264/90000a=

22、fmtp:96 packetization-mode=1;profile-level-id=42e014;sprop-parameter-sets=z0lgfnofgle=,am4wpa=a=mpeg4-esid:201a=width:integer;352a=height:integer;288a=framerate:52009-12-30 15:09:45:843rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 12009-12-30 15:0

23、9:45:875setup rtsp:/202.102.126.10/0123456789abcdef.sdp/trackid=0 rtsp/1.0cseq:2transport:rtp/avp/tcp;unicast;mode=record;interleaved=0-12009-12-30 15:09:45:890rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 2cache-control: no-cachesession: 26813480

24、845923date: wed, 30 dec 2009 07:10:00 gmtexpires: wed, 30 dec 2009 07:10:00 gmttransport: rtp/avp/tcp;unicast;mode=record;interleaved=0-12009-12-30 15:09:45:906record rtsp:/202.102.126.10/0123456789abcdef.sdp rtsp/1.0cseq: 3session: 268134808459232009-12-30 15:09:45:906rtsp/1.0 200 okserver: dss/5.5

25、.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 3session: 26813480845923rtp-info: url=rtsp:/202.102.126.10/0123456789abcdef.sdp/trackid=03.1.3 播放过程2009-12-30 15:10:30:859options rtsp:/202.102.126.10:1554/0123456789abcdef rtsp/1.0cseq: 1user-agent: vlc media player (live555 strea

26、ming media v2009.04.20)2009-12-30 15:10:30:859rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 1public: describe, setup, teardown, play, pause, options, announce, record2009-12-30 15:10:30:859describe rtsp:/202.102.126.10:1554/0123456789abcdef rtsp/1

27、.0cseq: 2accept: application/sdpuser-agent: vlc media player (live555 streaming media v2009.04.20)2009-12-30 15:10:30:875rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 2cache-control: no-cachecontent-length: 364date: wed, 30 dec 2009 07:10:00 gmtex

28、pires: wed, 30 dec 2009 07:10:00 gmtcontent-type: application/sdpx-accept-retransmit: our-retransmitx-accept-dynamic-rate: 1content-base: rtsp:/202.102.126.10:1554/0123456789abcdef/v=0o=streamsender 0 0 in ip4 0.0.0.0s=0123456789abcdef.sdpc=in ip4 0.0.0.0t=0 0a=tool:sdr v2.4a6a=control:*m=video 0 rt

29、p/avp 96a=control:trackid=0a=rtpmap:96 h264/90000a=fmtp:96 packetization-mode=1;profile-level-id=42e014;sprop-parameter-sets=z0lgfnofgle=,am4wpa=a=mpeg4-esid:201a=width:integer;352a=height:integer;288a=framerate:52009-12-30 15:10:30:906setup rtsp:/202.102.126.10:1554/0123456789abcdef/trackid=0 rtsp/

30、1.0cseq: 3transport: rtp/avp;unicast;client_port=2770-2771user-agent: vlc media player (live555 streaming media v2009.04.20)2009-12-30 15:10:30:921rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 3cache-control: no-cachesession: 60374355309628date: w

31、ed, 30 dec 2009 07:10:45 gmtexpires: wed, 30 dec 2009 07:10:45 gmttransport: rtp/avp;unicast;source=202.102.126.10;client_port=2770-2771;server_port=6970-69712009-12-30 15:10:30:921play rtsp:/202.102.126.10:1554/0123456789abcdef/ rtsp/1.0cseq: 4session: 60374355309628range: npt=0.000-user-agent: vlc

32、 media player (live555 streaming media v2009.04.20)2009-12-30 15:10:30:921rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 4session: 60374355309628range: npt=now-rtp-info: url=rtsp:/202.102.126.10:1554/0123456789abcdef/trackid=03.2 mpeg4mpeg4传输参考rfc

33、3016、rfc 3640两篇文档，其中，rfc 3016支持“mp4v-es”视频传输方式，还支持“mp4a-latm”aac音频传输方式，但是，据说它没有使用mpeg4的同步技术和流管理技术；rfc 3640支持“mpeg4-generic”方式传输音视频，还支持传输其它的es流，如系统流等。mpeg4视频一般采用mp4v-es方式，aac一般采用mpeg4-generic方式。图3-2 mpeg4的rtp负载结构图3-2是mpeg4的rtp负载结构，其中，au header length是字节长度，表示au header的总位数，并且是网络字节序。au header包含一些可选的字段，长

34、度不固定，包含字段具体在sdp头中指定。如果au header的总位数不是8的倍数，那么au header最后要添加填充位，位值为0，这时au header length长度不包括填充位的长度。另外，rfc 3640上面还指出“the auxiliary section”段，因为这个段一般不用，所以没有考虑。rfc 3640 “3.2.1.1 the au-header”一节中详细的介绍了au header的字段。3.2.1 aac参考rfc 3640。rfc 3640介绍了负载aac的两种模式，low bit-rate aac和high bit-rate aac，我使用的是high bit-

35、rate aac。high bit-rate aac模式支持aac帧大小可变，帧最大长度是8191字节。high bit-rate aac模式需要在sdp中指定mode=aac-hbr, au header只包含了两个字段au-size(13位)和au-index(3位)，这样au header的长度正好为2字节，au-index填充为0, au-size是网络字节序。传输aac帧不需要包含adts头，rtp单帧、分片、组包传输除了rtp头字段中的rtp_mark没有特别需要注意的地方。aac网络流使用adts格式保存，adts指audio data transport stream，它依次包

36、含固定头(28位)和可变头(28位)两部分，它们的具体结构如下：adts_fixed_header()syncword; 12位id; 1位layer; 2位protection_absent; 1位profile; 2位sampling_frequency_index; 4位private_bit; 1位channel_configuration; 3位original/copy; 1位home; 1位adts_variable_header()copyright_identification_bit; 1位copyright_identification_start; 1位frame_le

37、ngth; 13位adts_buffer_fullness; 11位number_of_raw_data_blocks_in_frame; 2位详细说明下adts头的重要数据部分：syncword 同步字the bit string 1111 1111 1111，说明一个adts帧的开始。id mpeg 标示符, 设置为1.layer indicates which layer is used. set to 00protection_absent 表示是否误码校验profile 表示使用哪个级别的aac，如01 low complexity(lc)- aaclc0 main profile

38、1 low complexity profile (lc) 2 scalable sampling rate profile (ssr)3 (reserved)sampling_frequency_index 表示使用的采样率下标sampling_frequency_index sampling frequeny hz0x0 960000x1 882000x2 640000x3 480000x4 441000x5 320000x6 240000x7 220500x8 160000x9 20000xa 110250xb 80000xc reserved0xd reserved0xe reserv

39、ed0xf reservedchannel_configuration 表示声道数frame_length 一个adts帧的长度包括adts头和raw data block.adts_buffer_fullness 0x7ff 说明是码率可变的码流number_of_raw_data_blocks_in_frame表示adts帧中有number_of_raw_data_blocks_in_frame + 1个aac原始帧.所以说number_of_raw_data_blocks_in_frame = 0 表示说adts帧中有一个aac数据块并不是说没有。(一个aac原始帧包含一段时间内1024

40、个采样及相关数据)aac的rtp推送过程、播放过程及sdp的例子参见1.2节，rtp数据传输参考devicehandler/streamsender/streamaac.cpp和devicehandler/aachandler/aachander.cpp。sdp中有个参数config，它对应audiospecificconfig结构：audiospecificconfig两字节，它的16进制的字符串就是config。3.2.2 视频参考rfc 3016、live555 mpeg4和iso-iec-14496-2_2001_mpeg4_visual，mpeg4视频采用mp4v-es方式。mpeg

41、4编码的主要概念是对象(object)和背景，还有vs、vo、vol、vop、gop等。使用rtp推送mpeg4视频主要考虑两个关键点：1、参数：profile_level_id: 000001b0f3,其中f3就是profile_level_id，它紧跟在vs起始码之后config: 000001b6(vop)之前的数值的16进制表示。2、rtp封包：直接封包：mpeg4直接封包分片：mpeg4可以直接分片，防止包丢失可以copy vop头。3.2.2.1 推送过程2010-5-7 10:22:13:625announce rtsp:/202.102.126.10/0123456789abc

42、def.sdp rtsp/1.0cseq:3452816846content-type:application/sdpcontent-length:351v=0o=streamsender 0 0 in ip4 0.0.0.0c=in ip4 202.102.126.10a=tool:sdr v2.4a6m=video 554 rtp/avp 96a=control:trackid=0a=rtpmap:96 mp4v-es/90000a=fmtp:96 profile-level-id=224;config=000001b0f4000001b50900000100000001200086c40

43、0670c161048518f000001b2587669443030a=mpeg4-esid:201a=width:integer;352a=height:integer;288a=framerate:252010-5-7 10:22:13:656rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 34528168462010-5-7 10:22:13:656setup rtsp:/202.102.126.10/0123456789abcdef.s

44、dp/trackid=0 rtsp/1.0cseq:3452816847transport:rtp/avp/tcp;unicast;mode=record;interleaved=0-12010-5-7 10:22:13:671rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 3452816847cache-control: no-cachesession: 7902739850292date: fri, 07 may 2010 02:22:12

45、gmtexpires: fri, 07 may 2010 02:22:12 gmttransport: rtp/avp/tcp;unicast;mode=record;interleaved=0-12010-5-7 10:22:13:671record rtsp:/202.102.126.10/0123456789abcdef.sdp rtsp/1.0cseq: 3452816848session: 79027398502922010-5-7 10:22:13:687rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32;

46、release/darwin; state/beta; )cseq: 3452816848session: 7902739850292rtp-info: url=rtsp:/202.102.126.10/0123456789abcdef.sdp/trackid=03.2.2.2 播放过程2010-5-7 10:22:21:562options rtsp:/202.102.126.10:1554/0123456789abcdef rtsp/1.0cseq: 39user-agent: vlc media player (live555 streaming media v2009.04.20)20

47、10-5-7 10:22:21:562rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 39public: describe, setup, teardown, play, pause, options, announce, record2010-5-7 10:22:21:578describe rtsp:/202.102.126.10:1554/0123456789abcdef rtsp/1.0cseq: 40accept: applicatio

48、n/sdpuser-agent: vlc media player (live555 streaming media v2009.04.20)2010-5-7 10:22:21:578rtsp/1.0 200 okserver: dss/5.5.5 (build/489.16; platform/win32; release/darwin; state/beta; )cseq: 40cache-control: no-cachecontent-length: 387date: fri, 07 may 2010 02:22:20 gmtexpires: fri, 07 may 2010 02:2

49、2:20 gmtcontent-type: application/sdpx-accept-retransmit: our-retransmitx-accept-dynamic-rate: 1content-base: rtsp:/202.102.126.10:1554/0123456789abcdef/v=0o=streamsender 0 0 in ip4 0.0.0.0s=0123456789abcdef.sdpc=in ip4 0.0.0.0t=0 0a=tool:sdr v2.4a6a=control:*m=video 0 rtp/avp 96a=control:trackid=0a

50、=rtpmap:96 mp4v-es/90000a=fmtp:96 profile-level-id=224;config=000001b0f4000001b50900000100000001200086c400670c161048518f000001b2587669443030a=mpeg4-esid:201a=width:integer;352a=height:integer;288a=framerate:252010-5-7 10:22:21:609setup rtsp:/202.102.126.10:1554/0123456789abcdef/trackid=0 rtsp/1.0cseq: 41transport: rtp/avp;unicast;client_port=2774-2775user-agent: vlc media player (live555 streaming media v2009.04.20)2010-5-7 1