OpenTSDB的简单入门.doc

此文档的定位：便于JAVA程序员快速了解OpenTSDB,以及利用OpenTSDB的API进行简单的开发，至于具体的搭建以及调优之类不作讨论。结构如下：一. 是什么？有什么作用二、结构，怎么干三. 主要API四、在开发中如何运用这些API1、 opentsdb是什么基于Hbase的分布式的，可伸缩的时间序列数据库（本质就是一个数据库，通过TCollector收集监控对象的各个指标，按时间的序列存入hbase中。通过查询在一段时间内某个指标的参数，经过处理展示给用户，用户可以看到各个时间点的指标值和这段时间内的变化，达到监控的目的）主要用途，就是做监控系统；譬如收集大规模集群（包括网络设备、操作系统、应用程序）的监控数据并进行存储，查询。2、 OpenTSDB的结构 Opentsdb本质上是一个数据库，因此须了解它的存储结构，包括最小的存储单位以及存储单元等 存储到OpenTSDB的数据，是以metric为单位的，metric就是1个监控项，譬如服务器的话，会有CPU使用率、内存使用率这些metric； OpenTSDB使用HBase作为存储，由于有良好的设计，因此对metric的数据存储支持到秒级别；OpenTSDB支持数据永久存储，即保存的数据不会主动删除；并且原始数据会一直保存（有些监控系统会将较久之前的数据聚合之后保存）在了解整个结构之前先学习些基础概念2.1、OpenTSDB存储的相关概念metric：proc.loadavg.1mtimestamp：1234567890value：0.42tags：host=web42，pool=static DataPoint：即在某个时间点的数值。包括以下： Metric：指标项 Value：一个Value表示一个metric的实际数值 Timestamp：即时间戳，用来描述Value是什么时候的 Tags:标签，由tagk和tagv组成 tagk=takv2.2、tsdb和tsdb-uidOpenTSDB使用Hbase存储，核心的存储是两张表 ，tsdb和tsdb-uid （1） tsdb 结构如下1）RowKey的设计RowKey其实和上面的metric|timestamp|value|host=web42|pool=static类似；但是区别是，OpenTSDB为了节省存储空间，将每个部分都做了映射。在OpenTSDB里面有这样的映射，metric-3字节整数、tagk-3字节整数、tagv-3字节整数上图的映射关系为，proc.loadavg.1m-052、host-001、web42-028、pool-047、static-0012）column的设计为了方便后期更进一步的节省空间。OpenTSDB将一个小时的数据，保存在一行里面。所以上面的timestamp1234567890，会先模一下小时，得出1234566000，然后得到的余数为1890，表示的是它是在这个小时里面的第1890秒；然后将1890作为column name，而0.42即为column value（2）tsdb-uid保存的就是tsdb中rowkey的映射关系 2.3、OpenTSDB的总体结构名词解释：Servers：服务器C：Collector，收集数据，推送数据TSD：TSD是对外通信的无状态的服务器，Collector可以通过TSD简单的RPC协议推送监控数据；另外TSD还提供了一个web UI页面供数据查询；另外也可以通过脚本查询监控数据，对监控数据做报警。HBase：TSD收到监控数据后，是通过AsyncHbase这个库来将数据写入到HBase；AsyncHbase是完全异步、非阻塞、线程安全的Hbase客户端，使用更少的线程、锁以及内存，可以提供更高的吞吐量，特别对于大量的写操作。2.4、OPENTSDB的界面在浏览器中输入对应的地址即可显示三. 主要APIOpenTSDB的API为HTTP API所以只要直接在浏览器中输入对应的URL就可调用，格式为OpenTSDB的域名+API名称。如04:4242/api/suggest?type=metrics&q=sys&max=10,结果如下OpenTSDB的API如下 /s /api/aggregators /api/annotation /api/config /api/dropcaches /api/put /api/query /api/search /api/serializers /api/stats /api/suggest /api/tree /api/uid /api/version3.1 /api/suggest这个API提供一个自动填充的功能。它可以查询到RequestsNameData TypeRequiredDescriptionDefaultQSRWExampletypeStringRequiredThe type of data to auto complete on. Must be one of the following:metrics,tagkortagvtypemetricsqStringOptionalA string to match on for the given typeqwebmaxIntegerOptionalThe maximum number of suggested results to return. Must be greater than 025max10请求示例：04:4242/suggest?type=metrics&q=p返回的是JSON格式的数据3.2 /api/query对于使用者来说，可能最有用处的就是/api/query，所以着重学习/api/query/api/query的请求方式有3种：get post delete。其中2.2版本或的数据匹配查询可以发送delete请求，在tsd.http.query.allow_delete配置元素允许删除deletions的时候。 （1） 请求元素包括以下NameData TypeRequiredDescriptionDefaultQSRWExamplestartString, IntegerRequiredThe start time for the query. This can be a relative or absolute timestamp. SeeQuerying or Reading Datafor details.start1h-agoendString, IntegerOptionalAn end time for the query. If not supplied, the TSD will assume the local system time on the server. This may be a relative or absolute timestamp. SeeQuerying or Reading Datafor details.current timeend1s-agoqueriesArrayRequiredOne or more sub queries used to select the time series to return. These may be metricmor TSUIDtsuidsqueriesm or tsuidsSee belownoAnnotationsBooleanOptionalWhether or not to return annotations with a query. The default is to return annotations for the requested timespan but this flag can disable the return. This affects both local and global notes and overridesglobalAnnotationsfalseno_annotationsfalseglobalAnnotationsBooleanOptionalWhether or not the query should retrieve global annotations for the requested timespanfalseglobal_annotationstruemsResolutionBooleanOptionalWhether or not to output data point timestamps in milliseconds or seconds. If this flag is not provided and there are multiple data points within a second, those data points will be down sampled using the querys aggregation function.falsemstrueshowTSUIDsBooleanOptionalWhether or not to output the TSUIDs associated with timeseries in the results. If multiple time series were aggregated into one set, multiple TSUIDs will be returned in a sorted mannerfalseshow_tsuidstrueshowSummaryBooleanOptionalWhether or not to show a summary of timings surrounding the query in the results. This creates another object in the map that is unlike the data point objects.falseshow_summarytrueshowQueryBooleanOptionalWhether or not to return the original sub query with the query results. If the request contains many sub queries then this is a good way to determine which results belong to which sub query. Note that in the case of a*or wildcard query, this can produce a lot of duplicate output.falseshow_querytruedeleteBooleanOptionalCan be passed to the JSON with a POST to delete any data points that match the given query.falseW （2） 子查询 OpenTSDB的查询语句需要至少一条子查询，并且时间序列应该在请求的结果集里面，有两种查询方式： Metric Query- The full name of a metric is supplied along with an optional list of tags. This is optimized for aggregating multiple time series into one result. TSUID Query- A list of one or more TSUIDs that share a common metric. This is optimized for fetching individual time series where aggregation is not required. 在一条查询语句中可以包含任意多个这两种查询方式。当提交一个查询请求时，如果一系列的TSUID都提供的话，另外提供的metric和tags将会被忽略。每个子查询可以检索一条或一组时间序列对应的指标项，也可以对查询的结果集进行聚合函数处理和分组运算。子查询字段包括：NameData TypeRequiredDescriptionDefaultExampleaggregatorStringRequiredThe name of an aggregation function to use. See/api/aggregatorssummetricStringRequiredThe name of a metric stored in the systemsys.cpu.0rateBooleanOptionalWhether or not the data should be converted into deltas before returning. This is useful if the metric is a continuously incrementing counter and you want to view the rate of change between data points.falsetruerateOptionsMapOptionalMonotonically increasing counter handling optionsSee below不列举downsampleStringOptionalAn optional downsampling function to reduce the amount of data returned.5m-avgtagsMapOptionalTo drill down to specific timeseries or group results by tag, supply one or more map values in the same format as the query string. Tags are converted to filters in 2.2. See the notes below about conversions. Note that if no tags are specified, all metrics in the system will be aggregated into the results.Deprecated in 2.2不列举filters(2.2)ListOptionalFilters the time series emitted in the results. Note that if no filters are specified, all time series for the given metric will be aggregated into the results.不列举（3） 指标查询的书写格式 完整的Metric Querym=:ratecounter,:=,.=,.= 打“”为子查询的元素，打“ ”的为非必要条件。除了aggregator和metric元素为必要元素，其他元素都是有选择性的。It can be a little daunting at first but you can break it down into components. If youre ever confused, try using the built-in GUI to plot a graph the way you want it, then look at the URL to see how the query is formatted. Changes to any of the form fields will update the URL (which you can actually copy and paste to share with other users). For examples, please see Query Examples.TSUID的查询格式TSUID queries are simpler than Metric queries. Simply pass a list of one or more hexadecimal encoded TSUIDs separated by commas:tsuid=:,.查询请求示例（GET请求）http:/localhost:4242/api/query?start=1h-ago&m=sum:rate:proc.stat.cpuhost=foo,type=idlehttp:/localhost:4242/api/query?start=1h-ago&tsuid=sum:000001000002000042,000001000002000043提交内容的请求示例（POST请求）Please see the serializer documentation for request information:HTTP Serializers. The following examples pertain to the default JSON serializer. start: 1356998400, end: 1356998460, queries: aggregator: sum, metric: sys.cpu.0, rate: true, tags: host: *, dc: lga , aggregator: sum, tsuids: 000001000002000042, 000001000002000043 2.2 query with filters start: 1356998400, end: 1356998460, queries: aggregator: sum, metric: sys.cpu.0, rate: true, filters: type:wildcard, tagk:host, filter:*, groupBy:true , type:literal_or, tagk:dc, filter:lga|lga1|lga2, groupBy:false , , aggregator: sum, tsuids: 000001000002000042, 000001000002000043 （4） 返回数据类型查询得到的数据类型取决于系统所选择的serializer（默认HTTP Serializers,它会返回Jason类型的数据）。在OpenTSDB中，一个查询请求可能会返回带有多组指标的数据集合，特别是在请求中包含多个子查询的情况下。每条返回的数据都包含如下元素：NameDescriptionmetricName of the metric retrieved for the time seriestagsA list of tags only returned when the results are for a single time series. If results are aggregated, this value may be null or an empty mapaggregatedTagsIf more than one timeseries were included in the result set, i.e. they were aggregated, this will display a list of tag names that were found in common across all time series.dpsRetrieved data points after being processed by the aggregators. Each data point consists of a timestamp and a value, the format determined by the serializer.annotationsIf the query retrieved annotations for timeseries over the requested timespan, they will be returned in this group. Annotations for every timeseries will be merged into one set and sorted bystart_time. Aggregator functions do not affect annotations, all annotations will be returned for the span.globalAnnotationsIf requested by the user, the query will scan for global annotations during the timespan and the results returned in this group当查询的过程中没有发生异常，我们将会收到一个200的正常执行的状态码。如果查询条件对应的条数为0，系统会返回一个空的集合。对于JSON的串行器来说，时间戳可以为一个整数或浮点数Example Aggregated Default Response metric: tsd.hbase.puts, tags: , aggregatedTags: host , annotations: tsuid: 00001C0000FB0000FB, description: Testing Annotations, notes: These would be details about the event, the description is just a summary, custom: owner: jdoe, dept: ops , endTime: 0, startTime: 1365966062 , globalAnnotations: description: Notice, notes: DAL was down during this period, custom: null, endTime: 1365966164, startTime: 1365966064 , tsuids: 0023E3000002000008000006000001 , dps: 1365966001: 25595461080, 1365966061: 25595542522, 1365966062: 25595543979,. 1365973801: 25717417859 Example Aggregated Array Response metric: tsd.hbase.puts, tags: , aggregatedTags: host , dps: 1365966001, 25595461080 , 1365966061, 25595542522 ,. 1365974221, 25722266376 Example Multi-Set ResponseFor the following example, two TSDs were running and the query was:http:/localhost:4242/api/query?start=1h-ago&m=sum:tsd.hbase.putshost=*. This returns two explicit time series. metric: tsd.hbase.puts, tags: host: , aggregatedTags: , dps: 1365966001: 3758788892, 1365966061: 3758804070,. 1365974281: 3778141673 , metric: tsd.hbase.puts, tags: host: , aggregatedTags: , dps: 1365966001: 3902179270, 1365966062: 3902197769,. 1365974281: 3922266478 Example With Show Summary and Query metric: tsd.hbase.puts, tags: , aggregatedTags: host , query: aggregator: sum, metric: tsd.hbase.puts, tsuids: null, downsample: null, rate: true, filters: tagk: host, filter: *, group_by: true, type: wildcard , rateOptions: null, tags: , dps: 1365966001: 25595461080, 1365966061: 25595542522, 1365966062: 25595543979,. 1365973801: 25717417859 , statsSummary: datapoints: 0, rawDatapoints: 56, aggregationTime: 0, serializationTime: 20, storageTime: 6, timeTotal: 26 Get请求示例：04:4242/api/query?start=1h-ago&m=sum:rate:proc.stat.cpuhost=foo,type=idle四、利用JAVA调用OpenTSDB的API 由于OpeTSDB的API为HTTP形式，所以在JAVA后端调用的话需要利用HttpClient，由它绑定数据、发出请求、接收返回数据。下面是简单用法。 (1) GET方法使用 HttpClient 需要以下 6 个步骤：1. 创建 HttpClient 的实例2. 创建某种连接方法的实例，在这里是GetMethod。在 GetMethod 的构造函数中传入待连接的地址3. 调用第一步中创建好的实例的 execute 方法来执行第二步中创建好的 method 实例4. 读 response5. 释放连接。无论执行方法是否成功，都必须释放连接6. 对得到后的内容进行处理根据以上步骤，我们来编写用GET方法来取得某网页内容的代码。大部分情况下 HttpClient 默认的构造函数已经足够使用。 HttpClient httpClient = new DefaultHttpClient();创建GET方法的实例。在GET方法的构造函数中传入待连接的地址即可。用GetMethod将会自动处理转发过程，如果想要把自动处理转发过程去掉的话，可以调用方法setFollowRedirects(false)。 GetMethod getMethod = new GetMethod(.);调用实例httpClient的executeMethod方法来执行getMethod。由于是执行在网络上的程序，在运行executeMethod方法的时候，需要处理两个异常，分别是HttpException和IOException。引起第一种异常的原因主要可能是在构造getMethod的时候传入的协议不对，比如不小心将http写成htp，或者服务器端返回的内容不正常等，并且该异常发生是不可恢复的；第二种异常一般是由于网络原因引起的异常，对于这种异常（IOException），HttpClient会根据你指定的恢复策略自动试着重新执行executeMethod方法。HttpClient的恢复策略可以自定义（通过实现接口HttpMethodRetryHandler来实现）。通过httpClient的方法setParameter设置你实现的恢复策略，本文中使用的是系统提供的默认恢复策略，该策略在碰到第二类异常的时候将自动重试3次。executeMethod返回值是一个整数，表示了执行该方法后服务器返回的状态码，该状态码能表示出该方法执行是否成功、需要认证或者页面发生了跳转（默认状态下GetMethod的实例是自动处理跳转的）等。 /设置成了默认的恢复策略，在发生异常时候将自动重试3次，在这里你也可以设置成自定义的恢复策略getMethod.getParams( ).setParameter(HttpMethodParams.RETRY_HANDLER,new DefaultHttpMethodRetryHandler();/执行getMethodint statusCode = client.executeMethod(getMethod);if (statusCode != HttpStatus. SC_OK) System.err.println(Method failed: + getMethod.getStatusLine();在返回的状态码正确后，即可取得内容。取得目标地址的内容有三种方法：第一种，getResponseBody，该方法返回的是目标的二进制的byte流；第二种，getResponseBodyAsString，这个方法返回的是String类型，值得注意的是该方法返回的String的编码是根据系统默认的编码方式，所以返回的String值可能编码类型有误，在本文的字符编码部分中将对此做详细介绍；第三种，getResponseBodyAsStream，这个方法对于目标地址中有大量数据需要传输是最佳的。在这里我们使用了最简单的getResponseBody方法。 byte responseBody = method.getResponseBody();释放连接。无论执行方法是否成功，都必须释放连接。 method.releaseConnection();处理内容。在这一步中根据你的需要处理内容，在例子中只是简单的将内容打印到控制台。 System.out. println(new String(responseBody);下面是程序的完整代码：package test;import java.io.IOException;import mons.httpclient.*;import mons.httpclient.methods.GetMethod;import mons.httpclient.params.HttpMethodParams;public class GetSamplepublic static void main(String args) /构造HttpClient的实例HttpClient httpClient = new HttpClient();/创建GET方法的实例GetMethod getMethod = new GetMethod(.);/使用系统提供的默认的恢复策略getMethod.getParams().setParameter( HttpMethodParams.RETRY_HANDLER,new DefaultHttpMethodRetryHandler( );try /执行getMethodint statusCode = httpClient.executeMethod(getMethod);if (statusCode != HttpStatus. SC_OK) System.err.println(Method failed: + getMethod.getStatusLine();/读取内容byte responseBody = getMethod.getResponseBody();/处理内容System.out.println (new String(responseBody); catch (HttpException e) /发生致命的异常，可能是协议不对或者返回的内容有问题System.out.println(Please check your provided http address!);e.printStackTrac