AWS-VPC深入探讨

1、AWS VPC深入探讨,主要议题,VPC设计原理, ,L2 L3,VPC新/高级特性, ,VPC S3 Endpoint VPC Flow Logs VPC Peering和其他连接方式,VPC实践问题, ,Enhanced Networking Linux System Tuning,VPC设计原理,AWS云服务,EBS,RDS,ElastiCache,Amazon Redshift,EC2,Elastic Load Balancing,客户自有数据中心,白板工程化,EBS,RDS,ElastiCache,Amazon Redshift,EC2,Elastic Load Balancing,

2、EC2 曾经是这样, 7, 7,,为什么不工作,/16,路由表, /16: /32: 7/32: /32:,本地 AWS AWS AWS,/16,, 7,7,,需求, ,客户指定的IP地址（段） 外部连接的路由聚合 与现有网络设计的一致性,虚拟私有云,/1

3、6,路由表 /16: /18:,本地 AWS,/24 /18,/24, ,,2 1,这就是virtual networking!, ,子网= VLAN VPC = VRF (虚拟路由转发) 但是,扩展的挑战,VLAN ID数量上的限制,12位 = 4096个VLANs,VRF支持上的限制,大型路由器 = 1K - 2K 个VRF表,VLAN:VRF间的固定比率,路由器和容量纬度,Big Rout

4、er Control Plane Data Plane,Big Router Control Plane Data Plane,一个例子, ,路由配置平均每行: 每个VPC的配置数: 每个VPC的子网数: 每个子网的配置数: 总VPC数: 配置大小:,50 个字符 10 行 4 个 5 行 2,000 3 MB,但是,无法扩展, ,12 位 VLAN ID = 4096 VLANs (远远不够) 大型路由器最多支持到4000 VRFs ($200k+), ,大量的VLAN导致网络工程师崩溃 受到供应商Bug修复速度约束(6个月+) 需要日用品型的, 可替换型的网络设备, ,少数几家公司生产大型

5、虚拟路由器 高级特性等卖点并不包含好的互操作性,容量库,A C G,C E G,0,1324/4,0 1,32/4,C A G,A C G,A A G,10 9 0 3 7,B D F,B D F,D D F,D D F,D F F,18/40,15/40,40 0 2 9,F B B B B,F B B B B,F B B B B,B B B B B,B B B B B,实现需求, ,缩放至 百万个 A规模的环境 一个region中任何位置的任何服务器 都能够 创建位于任意VPC的任意子网的实例,概念,Server Server 1

6、 ,Server Server ,VPC: ： ID: ： 实例 映射 VPC,客户创建的服务. 马逊EC2实例 位于式查找 亚 分布所创建符， 的标识,类似于 VPC + 实Cloud 映射 Private 例IP,映射服务 物理服务器: VPC亚马逊数据中心的物理主机 Virtualvpc-1a2B3c4d 到物理服务器,L2 - Ethernet,,,?,Ethernet Switch,收

7、到ARP 响应 交换 Src: MAC() 对所有端口广,播ARP请 具体地址为 求 L2 Dst: ff:ff:ff:ff:ff:ff MAC(),的端口 L3 Src:,L3 Who hasis at,L2 机会 MAC() 并了解 MAC() MAC() . ARP Dst: MAC() ICMP/TCP/UDP/,L2 - VPC,,Server Server 192

8、.168.1.4 ,映射服务,L2 Src: Mapping Service MAC(),L2 Dst: Mapping Service ff:ff:ff:ff:ff:ff,ARP Who has Reply:,Host: Blue ?,Src: MAC() Dst: MAC() Query: is at MAC() MAC: MAC(),Server

9、 Server ,Src: ,Dst: Mapping Service,,Server Server ,Server Server ,Src: Mapping Service,Dst: ,L3 Dst: Mapping valid:,映射服务 Src: 192

10、.168.0.3 Dst: VPC: Blue L2 MAC() L2 MAC() L3 Src: Validate: Blue isat ICMP/TCP/UDP/,L2 - VPC,VPC 隔离,,Server Server ,映射服务,Server Server ,Gr

11、ey,L2 Src: ,L2 Dst: Mapping Service,ARP,Src: MAC() Dst: ff:ff:ff:ff:ff:ff Query: Who has ? ,VPC 隔离,,Server Server ,映射服务,Server Server ,ARP,L2 Src: Src: is no

12、t,Dst: Dst: any L2 Mapping instances Service,Mapping Blue Denied, MAC() hosting ff:ff:ff:ff:ff:ff in VPC Blue. Query: Who has ? Alarm Raised,the instance.,VPC 隔离,Server Server ,Server

13、Server ,L3 Src: Mapping invalid!,Src: Mapping does not,L2 Src:the packet to Dst: Mapping Service ,映射服务 Src: Dst: VPC: Blue Service deliver MAC() L2 Dst: MAC() Validate: Alarm is at 192.168.0

14、.4 ICMP/TCP/UDP/,L3 Dst: Raised. Blue ,L3 IP 路由,,,?,Ethernet Switch,ff:ff:ff:ff:ff:ff MAC(),L3 hasis Who ,L2 Src: MAC() L2 Dst: MAC() L3 Src: ARP Dst: at MAC() ICMP/TCP/UDP/,Router,Ethernet,Switch,L2 Src: MAC(1

15、) L2 Dst: MAC() L3 Src: L3 Dst: ICMP/TCP/UDP/,L3 - VPC,,Server Server ,映射服务,L2 Src: Mapping Service,L2 Dst: Mapping Service ff:ff:ff:ff:ff:ff ,Reply: ARP Who has,? Host: Gateway

16、Blue,Src: MAC() Dst: MAC() Query: is at MAC() MAC: MAC(),Server Server ,L3 - VPC,Server Server ,Server Server 10.0.0.

17、4 ,映射服务 Src: Dst: VPC: Blue Src: MAC() Dst: MAC() L3 Src: Validate: Host: ICMP/TCP/UDP/, Blue is at, MAC: MAC(),Src: L2 L2 Src: Mapping Service Mapping Service,L2

18、 MAC() L2Dst: Mapping Service ,L3 Dst: valid: Query: L3 Dst: Mapping Reply:,缓存,Server Server ,,Server Server ,映射服务,L2 Src: MAC() L2 Dst: MAC() L3

19、 Src: L3 Dst: ICMP/TCP/UDP/,/18, /24,2 1 /24,与AWS之外互联 /16 Src: Dst: ? VPC: Blue L3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Edge Edge ,边界（Edges） Server

20、 Server ,L3 Src: ,L3 Dst: ,/16 Edge,映射服务 Src: Dst: VPC: Blue Host Host 7 ICMP/TCP/UDP/ ,Edges (三种情况),Src: Dst: VPC: Blue L

21、3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Src: 45 Dst: 4 IPSEC Stuff L3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Edge VPN,Edges (三种情况),Src: Dst: VPC: Blue L3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Src:

22、45 Dst: 4 802.1Q VLAN Tag L3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Edge 直连DX,Edges (三种情况),Edge ,Src: Dst: VPC: Blue,L3 Src: L3 Dst: 90 ICMP/TCP/UDP/,,Internet,L3 Src: L3 Dst: 90

23、 ICMP/TCP/UDP/,Edges (三种情况),VPN DX Internet,Edge Edge Edge ,Src: Dst: VPC: Blue L3 Src: L3 Dst: 7 ICMP/TCP/UDP/ Src: Dst: VPC: Blue L3 Src: L3 Dst: 7 ICMP/TCP/UDP/ Src: 192.168.

24、0.3 Dst: VPC: Blue L3 Src: L3 Dst: 90 ICMP/TCP/UDP/,Src: 45 Dst: 4 IPSEC Stuff L3 Src: L3 Dst: 7 ICMP/TCP/UDP/ Src: 45 Dst: 4 802.1QVLAN Tag L3 Src: L3 Dst: 7 ICMP/TCP/UDP/ L3 Src:

25、 6 L3 Dst: 90 ICMP/TCP/UDP/,Image credit: Wikipedia /wiki/1918_Eighth_Avenue,有趣的,VPC 价格 每个VPC的价格: 每个suBnet的价格: 之上实例间流量价格:,$0.00 $0.00 $0.00,VPC的新特性和高级特性,私有子网对S3的高效访问:,VPC S3 Endpoint,/0,VPC suBnet,VPC suBnet,NAT,/0,基于安全的VPC设计 Amazon-provide

26、d NAT instance image: amzn-ami-vpc-nat,, /24 /18,2,1 /24,为什么需要VPC S3 Endpoint,需要EMR运行在 私有子网吗,S3的VPC终端节点,/24 /18,/24,,2,S3的VPC终端节点,/24 /18,/24,172.31.1

27、.7,2,Server ,Edge ,Edge , ,,Server ,Src: Dst: ,边界（Edges）,映射服务,L3 Src: ,L3 Dst: ,/16 ,VPC: Blue Host Host 9 TCP/HTTP/ Ed

28、ge S3.us-east-1 Edge ,第四种Edge,Src: Dst: VPC: Blue L3 Src: L3 Dst: 9 TCP/HTTP/,Src: 45 Dst: 9 VPC Endpoint 1a2B3c4d L3 Src: L3 Dst: 9 TCP/HTTP/,Edge S3 endpoint, 172.31.

29、2.12,Statement: ,安全策略,/18,/24,/24,Statement: Sid: Access-to-specific-VPC-only, Principal: *, Action: s3:*, Effect: Deny, Resource: arn:aws:s3:my_secure_bucket, arn:aws:s3:my_secure_bucket/*, Condition: StringNotEquals: aws:sourceVpc: vpc-111bbb22 Sid: Access-to-specific

30、-bucket-only, Principal: *, Action: s3:GetObject, s3:PutObject , Effect: Allow, Resource: arn:aws:s3:my_secure_bucket, arn:aws:s3:my_secure_bucket/*, ,VPC Flow Logs,VPC Flow Logs: 了解网络通信状况, ,安全组规则效果立显 网络连通性排错 大数据分析网络流量 助力安全审计,可以在ENI，Subnet或者VPC级别开启,创建Flow Log,CloudWatch中查看Log,10分钟间隔累计和更新到CloudWatch

31、Logs,字段含义,VPC与VPC之间的互联:,VPC Peering以及其他连接方式,VPC对等的使用场景 通用的/核心服务, ,认证/AD 监控 日志 远程管理 扫描,VPC peering,VPC Peering,/16,/16,步骤一：发起请求,/16,/16,步骤1 发起对等请求,步骤二：接受请求,/16,/16,步骤一 发起对等请求,步骤二 接受对等请求,步骤三：创建路由,/16,/16,步骤一,步骤三 创建路由,目标地址为 发起对

32、等请求 到我的碗里来 步骤二 接受对等请求,VPC的,连接您的公司网络和AWS,VPN,DX,CGW,VGW,HA IPSec隧道,VPN：基础知识 /16,/16,192.168/16,您的网络设备,VPC实践问题,Enhanced Networking, HVM, SR-IO-V 提高性能 缩短延迟 降低抖动, 机型, ,C3 C4 D2 I2 M4 R3,第三方iperf测试（不代表AWS,仅供参考）,http:/B,Linux system Tuning,去掉nf_conntrack*相关的限制对小包性能提升显著：,之前： PacketSize

33、,THROUGHPUT,Packet(W)/s,MEAN_LATENCY(ms),P99_LATENCY(ms),MAX_LATENCY(ms),Retrans/s,R eransRate,78B, 61.715,9.891, 1.28251, 2.888, 648.408, 272.35, .2753%,之后： PacketSize,THROUGHPUT,Packet(W)/s,MEAN_LATENCY(ms),P99_LATENCY(ms),MAX_LATENCY(ms),Retrans/s,R eransRate,78B, 155.957,24.995, 0.512579, 0.76,

34、612.417, 223.10, .0892%,Linux system Tuning,1. 接收方的滑动窗口,Linux system Tuning,2. 发送方的拥塞窗口和慢启动 $ ip route list default via dev eth0 /24 dev eth0 proto kernel scope link 54 dev eth0 scope link,1448,1448,1448,= 4344 Bytes,Linux system Tuning # ip route change /

35、24 dev eth0 proto kernel scope link initcwnd 16 $ ip route list default via dev eth0 /24 dev eth0 proto kernel scope link initcwnd 16 54 dev eth0 scope link,1448,1448,1448,1448, + 12 ,= 23168 Bytes,Linux system Tuning,3. 调整拥塞算法 $ sysctl net.ipv4.tcp_available_conges

36、tion_control net.ipv4.tcp_available_congestion_control = cubic reno $ find /lib/modules -name tcp_* # modprobe tcp_illinois $ sysctl net.ipv4.tcp_available_congestion_control net.ipv4.tcp_available_congestion_control = cubic reno illinois # sysctl net.ipv4.tcp_congestion_control=illinois net.ipv4.tc

37、p_congestion_control = illinois,Socket 层分析, ,$ ss -ite State,Recv-Q Send-Q Local Address:Port Peer, ,Address:Port ESTAB 0 3829960 8:https 5:52008 timer:(on,012ms,0) uid:498 ino:7116021 sk:0001c286 ts sack cubic wscale:7,7 rto:204 rtt:1.423/0.14 ato:40 mss:1448 cwnd:138 ssthresh:8

38、0 send 1123.4Mbps unacked:138 retrans:0/11737 rcv_space:26847 TCP State,Socket 层分析, ,$ ss -ite State,Recv-Q Send-Q Local Address:Port Peer, ,Address:Port ESTAB 0 3829960 8:https 5:52008 timer:(on,012ms,0) uid:498 ino:7116021 sk:0001c286 ts sack cubic wscale:7,7 rto:204 rtt:1.423/

39、0.14 ato:40 mss:1448 cwnd:138 ssthresh:80 send 1123.4Mbps unacked:138 retrans:0/11737 rcv_space:26847 Bytes queued for transmission,Socket 层分析, ,$ ss -ite State,Recv-Q Send-Q Local Address:Port Peer, ,Address:Port ESTAB 0 3829960 8:https 5:52008 timer:(on,012ms,0) uid:498 ino:7116021 sk:0001c286 ts sack cubic