ArcSight事件关联分析课件

1、ArcSight CorrelationFabian LibeauSuperpan 翻译hongliangpanQQ:28797575ArcSight ESMArcSight ESM作为一款应对安全风险、合规要求和内部威胁的企业安全管理系统，ArcSight ESM（Enterprise Security Management）能够集中展示企业信息安全各方面的概况，同时还提供有实时监视和事件关联、风险分析、深入调查功能、报告、通知以及其他安全管理功能，可在企业范围内全面管理、审计安全事务。 2005 ArcSight Confidential2ArcSight ESM强大的事件收集能力和跨设备

2、的事件分类能力ArcSight ESM实现了实时数据格式标准化，超过260种默认支持的设备，对每一种事件都进行了详尽的分类，以帮助管理员理解事件的含义，并进行跨设备的分析。最为智能和灵活的关联分析ArcSight ESM提供实时的、内存内(In-Memory) 关联分析，具有106种预置关联规则，图形化规则编辑，支持资产分类、漏洞状态与企业策略与风险管理目标的关联。直观的调查分析和合规性报表ArcSight ESM具有169个可重用、图形化数据监视模块，自由定义的仪表板（预置41个），灵活的报表格式，提供图形化报表编辑器，提供预先打包的合规解决方案。 2005 ArcSight Confide

3、ntial3ArcSight ESM完善的自动安全响应能力ArcSight ESM可与安全设备共同协作来关闭威胁通信，以阻止正在进行的攻击，提供威胁升级和工单处理功能。智能存储ArcSight ESM集成了数据监控、备份脚本、分区管理等等一系列的数据库维护工具，提供综合安全生命周期信息管理（SLIM）策略，利用自动的高度压缩、存档和恢复系统以减少存储长期安全事件所需费用。 2005 ArcSight Confidential4ArcSight ESM 2005 ArcSight Confidential5SOC中日志关联分析的核心技术SIM/SEM/SIEM/SOC的日志关联分析核心技术主要集

4、中在：日志收集、格式化、事件映射、关联四个方面。日志收集：一个SIM产品是否有优势，就要看日志收集能否支持更多的设备日志类型，能否容易扩展，自动识别支持未知设备日志。例如需要支持的协议有syslog、snmp trap、windows log、checkpoint opsec、database、file、xml、soap等等。格式化：日志收集来了，需要格式化统一标准，为后面的关联，事件映射做准备，如果格式化不够标准，后面也不好做。 事件映射：将日志需要统一映射成一个标准，提供统一的解决方案，这个难度也比较大，各个厂家设备的日志名称，类型，含义都不相同，如果统一映射，是个难题。 关联分析：这个是

5、SIM的核心部分，例如ArcSight提供了简单的事件关联、上下文关联、攻击场景关联、低慢攻击关联、位置关联、身份关联、角色关联等等。关联分析还有脆弱性信息关联、因果关联、推理关联等等。 关键问题是如何利用这些技术，给用户提供一个很好的SIM/SEM/SIEM/SOC系统，也是一个难题。 2005 ArcSight Confidential6 2005 ArcSight Confidential7AgendaArchitectural Overview概述ArcSight Risk Prioritization风险的优先顺序ArcSight different ways of correlat

6、ing information不同的关联分析方法Rule based correlation基于规则Statistical correlation统计相关性分析Pattern discovery (advanced predictive DataMining)模式发现（先进的预测数据挖掘）ArcSight Key Concepts 2005 ArcSight Confidential8VulnerabilityAssessment漏洞评估Architectural Overview架构概述ConsoleDatabaseArcSightManagerAsset Management资产管理XML

7、Windows SystemsUnix/Linux/AIX/SolarisSecurityDevice安全设备SecurityDeviceDatabaseManagementSystemsSyslogConcentrator集中器Mainframe& Apps主机和应用SecurityDeviceData Flows数据流 2005 ArcSight Confidential9ArcSight SmartAgent Overview智能代理Largest number of supported devices 150+100% Data CaptureIntelligent Event Cap

8、ture智能事件捕获Normalization One format规范化 - 统一格式化Categorization Grouping similar events分类 - 分组类似事件Aggregation Event redundancy (50-80% for firewalls and routers)聚集 - 事件冗余（50-80的防火墙和路由器）Filtering Transfer and store only what you need过滤转移和存储您所需要的Secure, configurable and governed安全，配置和管辖的FlexAgents new Sma

9、rtAgents in hours在几个小时FlexAgents 新CounterAct Agents automated remediation抵制代理 - 自动修复Flexible Data Collection Centralized or Distributed灵活的数据收集 - 集中式或分布式Flexible Collection灵活采集CounterActSmartAgentFlexAgent 2005 ArcSight Confidential10ArcSight SmartAgent - Event Normalization and Categorization事件规范化和分

10、类Jun 01 2005 00:00:12: %PIX-3-106011: Deny inbound (No xlate) udp src outside:7/6346 dst outside:54/6346Jun 01 2005 00:00:12: %PIX-6-305011: Built dynamic TCP translation from isp:1/1967 to outside:54/62013Jun 01 2005 00:00:12: %PIX-6-302013: Built outbound TCP connection 2044303174 for outside:7/80

11、 (7/80) to isp:1/1967 (54/62013)Jun 02 2005 12:16:03: %PIX-6-106015: Deny TCP (no connection) from 02/15605 to 6/443 flags FIN ACK on interface outsideSample Raw Pix Events:Jun 02 2005 12:16:03: %PIX-6-106015: Deny TCP (no connection) from 02/15605 to 6/443 flags FIN ACK on interface outsideArcsight

12、 Categorization:Arcsight Normalization: 2005 ArcSight Confidential11ArcSight SmartAgent Guaranteed Delivery智能代理保证交付AnalystArcSightManagerPort 8443Cache缓存FailoverManager(optional)故障转移管理器（可选）ArcSightEventArcSightEventCompressedEventSSLContentUpdates 2005 ArcSight Confidential12The ArcSight Manager - O

13、verviewReal-Time, In-Memory Correlation实时内存关联Real-time Dashboards实时仪表盘Anomaly Detection异常检测Correlation Rules - known behaviors关联规则 已知行为Pattern Discovery undiscovered patterns模式发现 -未被发现的模式Flow Rates deviations from the norm流量速率-标准差 基线偏差Asset Linkage资产联动Priority Scoring优先评分Vulnerability漏洞Asset Value资产

14、价值Severity严重性Alerts, among other configurable actions其他配置的行动警告Scalability and High Availability Options可扩展性和高可用性选项Intelligent Processing智能处理ManagerLINUX, Windows,UNIX, Macintosh 2005 ArcSight Confidential13AgendaArchitectural OverviewArcSight Risk Prioritization风险的优先ArcSight different ways of correl

15、ating informationRule based correlationStatistical correlationPattern discovery (advanced predictive DataMining)ArcSight Key Concepts 2005 ArcSight Confidential14ArcSight Risk Correlation风险相关性EventsScansCorrelationDevicesPrioritizationWhats happening?Whatstargeted?Whatmatters?Whats vulnerable?漏洞、 脆弱

16、= False Alarm or Normal虚假报警或普通事件= Prioritized Red Alarm优先红色警报Dynamic Threat Severity Index动态威胁的严重程度指数Profiled Asset异常资产Confirmed Vulnerability已确认的漏洞Weighting Algorithms加权算法+Detected Event检测事件ArcSight fuses all key event sources and related inputs to rank event significance on multiple variables 所有关键

17、的事件源和多变量等级事件 2005 ArcSight Confidential15Asset Linkage and Priority Scoring - Overview资产联动和优先评分 - 概述Windows SystemsUnix/Linux/AIX/SolarisSecurityDeviceSecurityDeviceMainframe& AppsSecurityDevicePrioritization and Imported Scanned Assets资产的优先顺序和导入扫描的资产SmartAgentsArcSightEventArcSightManagerTMArcSight

18、 Prioritized Event事件优先权VulnerabilityScanner漏洞扫描SmartAgentsAsset Information建模的程度（信心）Model ConfidenceHas asset been scanned for open ports and vulnerabilities?关联RelevanceAre ports open on asset? Is it vulnerable?Severity严重性Is there a history withthis attacker or target (active lists)?资产重要性Asset Criti

19、calityHow important is thisasset to the business?代理严重性Agent SeverityMapping of reportingdevice severity toArcSight severity 2005 ArcSight Confidential16Asset Linkage and Priority Scoring Information Flow资产联动和优先评分 - 信息流Vulnerability Assessment漏洞评估 Three dimensional correlation of assets, events and v

20、ulnerabilities Allows organizations to apply SIM to risk management Minimizes dead end investigations Information seamlessly linked within the ArcSight system三二维相关的资产，事件和漏洞允许企业申请SIM卡风险管理最大限度地减少死胡同调查无缝链接的信息系统内的ArcSightArcSight ManagerAssets Compliance Requirement Business RoleApplicationOperating Sys

21、temData roleCriticality资产重要性Vulnerabilities- Zones区ArcSightEventEvent CVEEvent Severity事件等级Priority Score Relevance 2005 ArcSight Confidential17Threat Priority Variables Considered威胁优先 多种关系组合考虑Model Confidence:How well does ArcSight know this asset?Has it been scanned?Options: 0 = Asset is not model

22、ed没有建模 4 = Asset has not been scanned for open ports or vulnerabilities 没有扫描端口或漏洞 8 = Asset has been scanned for open ports or vulnerabilities, but not for both扫描端口或漏洞其一10 = Asset is scanned for both open ports and vulnerabilities扫描端口和漏洞Relevance:Is the port open, and has a vulnerability been exploi

23、ted利用?Options: 5 = Assets target port is open. 5 = Event will exploit a know asset vulnerabilitySeverity:Is there a history with this attacker or target (Active Lists)?Options: 5 = Hostile List 3 = Compromised 3 = Suspicious List 1 = Reconnaissance List 5 =敌对目录3 =不受影响 折中3 =可疑名单1 =侦察名单The Priority of

24、 an event is theAgent Severity adjusted by: Model Confidence Relevance、 Severity、 Asset Criticality一个事件的优先事项是代理严重性调整：模式的信心、关联、严重性、资产重要性Asset Criticality:资产重要性How critical have I rated this asset within my organization.Options: 10 = Very High Criticality Assets非常高 8 = High Criticality Assets高 6 = Med

25、ium Criticality Assets中 4 = Low Criticality Assets低 2 = Very Low Criticality Assets非常低 0 = Unknown Criticality Assets未知Agent Severity:Mapping of reporting device severity to ArcSight severity.代理严重性：报告设备严重性到ArcSight的严重性的映射。 2005 ArcSight Confidential18Relevance drags down the Agent Severity. 相关性Examp

26、le:If Relevance = 0, the Priority = 0If Relevance = 10, the Priority = Agent SeverityModel Confidence tempers the effect of relevance on priority.建模程度Example: If Model Confidence = 0, Relevance has no effect on PriorityIf Model Confidence = 10, Priority acts the way specified above3.Formulae for the

27、 multiplication factor contributed by Model Confidence (M) and Relevance (R) R = ( R + M - R * M / 10 )If Severity (S) = 10 it adds up to 30% to Agent Severity to provide Priority: (1 + S * 3 / 100)Criticality applies a boost to Agent Severity by 20% if = (Very High)10;does nothing if Criticality =

28、(High) 8; and applies a decrement/drag if the Criticality is Medium/Low/Unknown (6/4/2): (1 + (Criticality - 8) / 10)Threat Priority The Formula威胁优先级的公式 2005 ArcSight Confidential19Heuristic: Formula-Based启发式：按公式计算Threat level formulaPrioritizes incident investigation and responseSums up complex inf

29、ormation from the network model 威胁级别的公式事故调查和应对的优先顺序汇总了网络模型的复杂信息C:arcsightManagerconfigserverThreatLevelFormula.xml 2005 ArcSight Confidential20Priority Calculation Exercise优先级的计算练习StepsDevice Severity - Agent Severity - Calculation Exercise Agent Severity = Low Priority = 4 Asset Criticality is 0 =

30、20% decrease in priority. Priority = 3.2Severity = 0, no effect on priority. 2005 ArcSight Confidential21Priority is adjusted by Criticality通过重要性调整优先级Combined factor for model confidence and relevance, lets call it MCR = MCR is calculated using the formula R * 10 MCR = = ( R + M - R * M / 10 )where

31、R (Relavance) = 5, M (Model Confidence ) = 4MCR = 7 = 30% drop in priority again. New Priority = 3.2 * 0.7 = 2.24 rounded off gives a 2. The Final Priority is - because of low values for criticality and relevance your final priority of the event came down from 4 to 2. 2005 ArcSight Confidential22Age

32、ndaArchitectural OverviewArcSight Risk PrioritizationArcSight different ways of correlating informationRule based correlation基于规则的关联Statistical correlationPattern discovery (advanced predictive DataMining)ArcSight Key Concepts 2005 ArcSight Confidential23Rule based correlation基于规则的关联Fast memory base

33、d algorithm, based on RETE 2 (/rete2.htm)快速的内存算法Incorporates in Correlation:整合的相关性Events事件Vulnerability Information漏洞信息Active Lists (dynamic list with e.g. Asset/ User information)活动列表（如与动态列表资产/用户信息Asset Categories (see later slides)资产类别（见稍后幻灯片）Asset Zones (IP ranges)资产区（IP范围）Asset Networks (IP netw

34、orks/ groups of Asset Zones)资产网络（IP网络资产区/组）Results earlier rule based correlation早期规则为基础的相关性Results earlier statistical correlation早期统计（静态）为基础的相关性 2005 ArcSight Confidential24Rules Theory规则理论1. Simple Aggregation Single event type or categoryBasic conditionsDe-duplication简单 - 聚合单事件类型或类别基本条件重复数据删除 ta

35、rgetspinge.g., any source repetitively profiling targetsarcsight_category startsWith /recontarget_address inSubnetgroupBy source_address2 or more matching events in 1 minutesource2. Complex Correlation Multi-Event JoinMultiple event types or categoriesBoolean conditionsComplete session or “round tri

36、p”复杂的关系 - 多事件加入多个事件类型或类别布尔条件完整会话或“来回”targetse.g., any source successfully engaging a targetarcsight_category startsWith /attacktarget_address inSubnet groupBy source_address, target_address1+ matching events in 1 minutejoin events across IDS, firewall, and host3. Complex Long SequenceMultiple sessio

37、nsPre-attack probes,attack formation/progression, and attack conclusionHandles long-term memory need using active lists 复杂鈥长序列多个会话、预探测攻击，攻击编队/进程，攻击结束处理长期记忆需要使用活动列表attackFWIDSe.g., low&slow attack pattern across multiple rules/recon rule records source_address suspicious/attack rule upgrades source_a

38、ddress to hostileand records target_address as compromisedFinal rule looks for evidence of successrule1activelistactivelistrule2rule3sourceRule Types By Complexity复杂规则类型Example例子Approach方法途径Catch and accumulate events in real-time in memory- - - - - - - -Good for event bursts在内存中捕获和累积事件良好的突发事件Catch

39、and correlate events in real-time in memory until the rule chain is complete- - - - - - - -Good for cross-event matching that occurs in a single session在内存中捕获和累积事件直到完成该规则链 - - - - - - - - 良好的交叉配对活动，在单个会话发生Break up sequences in logical segments and maintain active lists in the database that tie toget

40、her multiple rules - - - - - - - -Good for long elapsed time attack sequences that start and stop across multiple sessions打破序列逻辑段，保持积极的数据库列出了多个规则联系在一起 - - - - 经过好长的时间序列，开始攻击和跨多个会话停止 2005 ArcSight Confidential25Simple Correlation: Event Aggregation简单的相关性：事件聚集Most basic correlation最基础的关联De-duplicates

41、events (many-to-one)去重Single source, single target单一源单一目标Flatten event bursts压扁事件爆发ArcSight SmartAgents do this too!CorrelationSingle EventMultiple Events(same base event)As above plusDistributed attack sources分布攻击源Multiple attack targets多攻击目标Any field or combination of event fields (types of event)

42、人行事件领域（事件类型的组合）Interrelates diverse events不同的事件相互联系CorrelationSingle EventMultiple Events(multiple event types, sources and/or targets) 2005 ArcSight Confidential26Simple Correlation: Event Aggregation简单的相关性：事件聚集Most basic correlation最基础的关联De-duplicates events (many-to-one)去重Single source, single ta

43、rget单一源单一目标Flatten event bursts压扁事件爆发ArcSight SmartAgents do this too!CorrelationSingle EventMultiple Events(same base event)As above plusDistributed attack sources分布攻击源Multiple attack targets多攻击目标Any field or combination of event fields (types of event)人行事件领域（事件类型的组合）Interrelates diverse events不同的事

44、件相互联系CorrelationSingle EventMultiple Events(multiple event types, sources and/or targets) 2005 ArcSight Confidential27Advanced Correlation: Multi-event Joins高级的相关性：多事件加人Inter-relates (joins) diverse events with any combination of common field values e.g., source IP, target IP, port, protocol, userna

45、me, domain, location, zone etc分析不同事件的相互联系，with事件通用属性：例如，源IP，目标IP，端口，协议，用户名，域，位置，区域等Compare any event fields using flexible boolean logic (AND, OR, NOT)比较任意事件字段采用比较灵活的布尔逻辑（与，或，非）Good for cross-event matching of complete end-to-end sessions良好的跨事件的完整的端至端会话匹配E.g. correlating an attacker detected by NIDS

46、, crossing the firewall, compromising a host, creating a back connection to steal confidential dataCorrelationSingle EventMultiple Events with Common Event Fields (different base events)在事件通用属性上分析多事件 2005 ArcSight Confidential28Complex Correlation: Attack State Monitoring复杂的相关性：攻击状态监测Inter-relates e

47、vents across sessions using Active Lists使用活动列表分析跨多会话事件Any field or combination of event fields may be persisted from base events任何字段或字段组合的事件可能会从基本事件提炼Long & short -term state machines长期与短期的状态机Good for tracking logical sequences of events良好的跟踪事件的逻辑顺序E.g. Reconnaissance, attack formation, progression

48、& conclusion例如侦察，攻击形成，进展及结论CorrelationEvent Sequence 1(multi-event joins)Record on Active List(state 1)CorrelationEvent Sequence 2Event Sequence 3CorrelationRecord on Active List(state 2)Single Event 2005 ArcSight Confidential29 ( 2 ) ( 1 )Rule based Cross-Correlation基于规则的交叉关联分析Scenario 1 The attack

49、er is unsuccessful and alarms are false positives方案1 - 攻击不成功和报警器误报HackerN-IDSIDS reports WEB-IIS ISAPI .printer access to 48ArcSight categorizes the signature as /Attack/ and recognizes thatthe target is hosting Mission Critical ApplicationsArcSight correlates and fires the 1st rule Yellow Alarm: /

50、Attack Started / Perimeter Alarm / Mission Critical Asset ( Warning_Display )The source IP address is quietly recorded as suspiciousFWFirewall reports a “drop” from that source IP to that target IP address ArcSight correlates and fires the 2nd and final rule Green Status: / Attack Blocked / Dropped

51、at Firewall / Mission Critical Asset ( Information_Display )The target host is never touchedArcSight records the event for an audit trail, alarms are suppressed and the source IP address remains on the suspicious list48 Host 2005 ArcSight Confidential30Hacker19 Scenario 2 The attacker is successful

52、方案2 - 攻击者成功N-IDSIDS reports WEB-IIS ISAPI .printer access to 48ArcSight categorizes the signature as /Attack/ and recognizes thatthe target is hosting Mission Critical Applications ( 1 )ArcSight correlates and fires the 1st rule Yellow Alarm: / Attack Started / Perimeter Alarm / Mission Critical Ass

53、et ( Warning_Display )The source IP address is quietly recorded as suspiciousFWFirewall reports an “accept” from that source IP to that target IP ( 2 )ArcSight correlates and fires the 2nd rule Red Alarm: / Attack Progressing / Crossed Firewall / Mission Critical Asset ( Threat_Display )The source I

54、P address is upgraded onto the hostile list48 The target gets “back doored”, indicated when thefirewall reports an FTP back out from the target to the attack sourceArcSight looks for FTP out signatures across different devicesHost ( 3 )ArcSight correlates and fires 3nd and final rule Double Red Alar

55、m: / Attack Succeeds / Compromised Target / Mission Critical Asset ( Confirmation_Display )The Target IP address is recorded as compromised, and an automated notification is sentRule based Cross-Correlation基于规则的交叉关联分析 2005 ArcSight Confidential31ArcSight Rule Editor规则编辑JoinCondition 1Condition 2Incl

56、ude predefined Filters/ ConditionsAdd Asset InformationAdd Vulnerability InformationAdd Information from Active List Explanation: This rule looks for a correlation event that triggers from an attack against a system and the attacked system begins attacking other systems说明：此规则寻找一个相关事件触发从一个对一个系统的攻击和被攻

57、击的系统开始攻击其它系统 2005 ArcSight Confidential32AgendaArchitectural OverviewArcSight Risk PrioritizationArcSight different ways of correlating informationRule based correlationStatistical correlation 统计关联Pattern discovery (advanced predictive DataMining)ArcSight Key Concepts 2005 ArcSight Confidential33Arc

58、Sight Statistical Correlation统计相关性ArcSight offers the following statistic types:Moving Average、Average、Identity、Kurtosis、SkewStandard DeviationVariance统计的ArcSight提供以下类型：移动平均线、平均、身分峰度、斜、标准差、方差Alarm from those can be used for rule based correlations. 34Statistical Correlation: Event Rates统计相关性：事件发生率Ch

59、oice of statistical function moving average, standard deviation, skew, variance or Kurtosis统计功能选择移动平均，标准偏差，偏差，方差或峰度Configurable sample period & interval配置的抽样周期和间隔CorrelationSteady Stream of Events(same base event)源源不断的事件流（相同的基础事件）Controllable event frequency可控事件频率Multiple attack dimensions多种攻击尺度Any

60、field or combination of event fields (types of event)任意事件字段或组合字段Much more sophisticated than simply considering the rate of all events directed at a single targetCorrelationSingle EventChange in Base Event Rate(statistical function)变化中的事件发生率（统计功能）No Events更为复杂的不仅仅是考虑到在一个单一的目标指示所有事件发生率 2005 ArcSight