008 VAILAdvancedVantiq资源

1、?#?VAIL AdvancedVantiq Training Module 008Course AgendaBuildingand Implementing VantiqFundamentalsDeployment and Management?#?Vantiq ArchitectureCourse IntroductionVantiq OverviewVantiq PortalDemonstrationVantiq ArchitectureBuilding a Vantiq solutionLab 1 Source TutorialVAIL BasicsVAIL AdvancedLab 2

2、 Introduction and Autopsy123456789XXXXXXXXXXXXXXXXXAdvanced TopicsSolution DesignModule NumberDAY ONEModule Objectives?#?By the end of this module, you should be able to:Understand the concept of “series” in VantiqDefine the usage of time-series in Vantiq solutionsUnderstand basic geospatial capabil

3、ities with VantiqUnderstand the Vantiq Recommendation ServiceUnderstand Vantiqs approach to version managementWhat is “Time Series Data”?#?Time Series DataDefined as sequences of data elements that occur over time. A series is a set of time-based records that fall within a defined rolling windowVant

4、iq assumes the series is “streamed” in real time with records arriving continuously from the source of the data. Examples include:Tracking a users position as they move from location to location within a storeReadings from a temperature sensor over time.Series Creation and ManipulationIn the coming

5、slides we will describe How a time series can be constructed from the inbound stream How it should be analyzed. Functions are available to compute aggregate values for a time series,incrementally reducing the time required to detect and respond to a new situation. Example: moving averages can be com

6、puted incrementally as each data item is read from the stream. Functions are available to transform a series represented in one timescale to a series in another timescale. Example - a relative series consisting of business hours each week may be “joined” with a series of sensor readings to determine

7、the sensor readings that occurred during business hours across the entire set of sensor readingsHow Series Data Is Managed in Vantiq?#?A seriesIs a non-persistent (in-memory) collection of objects that is ordered in timeDefines computed properties representing aggregation operations on the seriesSer

8、ies often process event data (filtering/aggregation) into a more usable/compact form?Once processed, the data can be persisted or published to a downstream system.?Given in-memory aspects, records within the series may be lost during a system restart. ?Given a single series may want to track multipl

9、e streams, series supports “keys”?For example, a factory may have 10 identical machines all with voltage sensors. ?By defining a key, the system can store/process data for all the machines in a single series.Dates and Intervals?#?Time is an important characteristic of a time series. Vantiq support D

10、ate and IntervalsTime series window can be defined by a date or can by number of messagesDate -A date represents a single point in time with up to microsecond precisionActual precision available below a millisecond may depend on your platform.Implemented as java.time.Instantobjects that represent ti

11、me as seconds and nanoseconds relative to epoch, January 1st, 1970 00:00:00 GMT.Dates can be accepted and converted into any of the following forms: ISO 8601 Date/Time String -SimpleDateFormatterformat yyyy-MM- ddTHH:mm:ss.SSSZ”, epochDays, epochHours, epochMinutes, epochSeconds, epochMilliseconds,

12、epochMicrosecondsConversion from a higher precision to a lower precision representation truncates higher precision.Interval-An interval represents a length of time such as one minute, three days or 25 hours. An interval is a relative duration with no anchor point on a calendar Internally represented

13、 in milliseconds.Intervals can be specified the following units: milliseconds, seconds, minutes, hours, days, weeks, yearsFunctions for Data/Interval Manipulation?#?now() -A date representing the current date and time.date(value, sourceRepresentation, destinationRepresentation)-Converts the date sup

14、plied asvalue from its current representation to the specified destination representation.For example, to convert a date represented as a Java Instant to an ISO date string: date(value, date, ISO). To convert a date from epochMillisecondsto epochDaystruncates the time from the date leaving the day c

15、omponent:date(value, epochMillisecond, epochDays).interval(value, sourceRepresentation, destinationRepresentation)-Converts the interval supplied via sourceRepresentationto the specified destinationRepresentation.Extract procedures:millisecond(date)-Milliseconds component of the date (0-999)second(d

16、ate)-Seconds component of the date (0-59)minute(date) -Minutes component of the date (0-59)hour(date) -Hour component of the date (0-23)day(date)-Day component of the date returned as the day within the month (1-31)week(date)-Week within the current year (1-52)month(date) -Month within the year (1-1

17、2)year(date) -YearDoW(date) -Day of the week numbered 1 - Monday through 7 -SundayHow(date) -Hour within the week numbered from 0MoW(date)-Minute within the week numbered from 0DoY(date) -Day within the year numbered from 1MoD(date) -Minute of the day (0-1439)SoD(date) -Second of the day (0-86399)So

18、H(date) -Second of the hour (0-3599)CREATE SERIES (Advanced)?#?Object RepresentationCREATE SERIES () PROCESSED BY WITHIN SECONDS STATUS TO TIMEOUT ENDSQL RepresentationCREATE SERIES ( = , .) WITH seriesUnits = , startingAt = , seriesLength = , maxCount = groupBy = , window = , documentation = PROCES

19、SED BY WITHIN SECONDS STATUS TO TIMEOUT ENDExample:CREATE SERIES voltages ( sensorId String, timestamp DateTime, voltage Real, avgVoltage Real = avg(voltage) ) WITH window: timestamp, maxCount: 5, groupBy: sensorId Creates voltage series defining sensorIdas the key property, timestampproperty to def

20、ine the time window, and computes the average voltage in theavgVoltage computed aggregation property. The window is defined to be no more than the last 5 records received.SELECT FROM SERIES?#?The time series type is identified by its name and an optional alias name. Example:SELECT FROM SERIES Veloci

21、ty AS v This will retrieve the aggregate data from the series Velocity:The records contain the group by property and aggregate properties defined by the typeThere will be one record for each distinct value of the group by propertyGROUP BY is not supported since series contain an implicit group in th

22、eir definition. Use of WHERE clauseWHERE clauses on Series select statements can operate on either the group by property or any of the aggregate propertiesExample:SELECT FROM SERIES Velocity WHERE speed 100 orSELECT FROM SERIES Velocity AS v WHERE v.id = MyId INSERT SERIES?#?Syntax:INSERT INTO SERIE

23、S (: , : .)INSERT INTO SERIES ( = , = .) The window property and key properties defined in the series declaration MUST be assigned values in the INSERTINSERT supports the incremental computation of new series values based on the values being inserted. This enables you to update a series with data ar

24、riving from a streaming source. Following INSERT you can execute a SELECT to return the computed aggregateTypically, the series contains aggregate values that are incrementally computed and stored in the series. Aggregate operations include count, sum, avg, rise, min, max, first, lastExample:If we h

25、ave a Message/Event Stream: sensorId: Machine ABC, timestamp: 2016-03-02T12:34:23.432Z, voltage: 2.34 The below INSERT inserts a record into the series from the message:INSERT INTO SERIES voltages ( sensorId: m.sensorId, timestamp: m.timestamp, voltage: m.voltage ) UPDATE/DELETE/ALTER/DROP SERIES?#?

26、The UPDATE statement does not support series types. ?The DELETE statement does not support series types?ALTER SERIES and DROP SERIES are supportedAn Example of Series Data Background ?#?A sensor has the following properties:Produces readings of voltage between 0 and 5 volts Produces one reading ever

27、y second. The arriving sensor reading is placed on an MQTT queue represented as a JSON objectFormat of message is id: , ts: , vl: So we want to develop a solution that Needs to read these values and creating a series that consists of the last 5 readings. When a reading arrives, the average of last 5

28、 is the new reading e.g. moving average.If the moving average ever exceeds 2.0 volts, then produce a notification. For the purposes of this example the notification is delivered to a service. Notification are delivered to an application or a user via SMS, email or other notification system.An Exampl

29、e of Series Data Type Definition?#?Defining the Type:?Define a type that will hold the current reading from the sensor. ?It is not essential to create a type but it is convenient to keep a persistent copy of the current state of the sensor for use by external applications. ?CREATE SERIES voltages ?(

30、 sensorId String, timestamp DateTime, voltage Real, avgVoltage Real = avg(voltage) ) ?WITH window: timestamp, maxCount: 5, groupBy: sensorId”?This can be loaded directly to Vantiq via the CLI or via the console.An Example of Series Data Source Definition?#?Defining the Source:The source is the chann

31、el through which data is ingested into Vantiq. The source must be defined to obtain the voltage readings from MQTT, a commonly used connectivity protocol that is directly supported by Vantiq.Sources are not needed to drive the series data (e.g. they could arise via a procedure)The Source definition

32、in JSON format is as follows: name: sensorMqtt, type: MQTT, direction: SOURCE, config: serverURIs: tcp: localhost:1883 , topics: com.accessg2.mqtt.sensor.rdg The source is defined to connect to an MQTT server running locallyTheserverURIs property will need to reflect the address of your MQTT serverT

33、hetopic property will need to reflect the topic where you will publish the voltage readingsThis can be loaded directly to Vantiq either via the CLI or via the console. An Example of Series Data Series Definition?#?Defining the Series:A series must be defined that stores the last 5 records. We will n

34、eed a setup()procedure to accomplish thisThe PROCEDURE will look like the followingPROCEDURE setup() Create a series that stores the last 5 records CREATE SERIES voltages ( sensorId String, timestamp: DateTime, avgVoltage Real = avg(voltage) ) WITH window: timestamp, maxCount: 5, groupBy: sensorId R

35、ETURN status: success This procedure can be loaded through the CLI or via the console. To execute the procedure, you can use EXECUTE or click the play button in the consoleAn Example of Series Data Rule Definition?Defining the Rule:?A rule is needed to process the voltage readings. ?We are just show

36、ing the rule here in the notes, there are detailed comments to explain the processing flow for the inbound data.?The RULE will look like the following:?RULE voltageReading?WHEN MESSAGE ARRIVES FROM sensorMqtt AS message ?INSERT INTO SERIES voltages ( sensorId: message.id, timestamp: message.ts, volt

37、age: message.vl ) ?var avgVoltage = 0.0 ?SELECT FROM SERIES voltages ?for (v in voltages) UNTIL avgVoltage 0 ?if (v.sensorId = message.id) ?avgVoltage = v.avgVoltage ?SELECT UNIQUE* Sensor:s WHERE sensorId = message.id ?if(s = null) ?INSERT Sensor(sensorId : message.id, timestamp : message.ts, volta

38、ge : message.vl, avgVoltage : avgVoltage) ? else ?UPDATE Sensor(timestamp : message.ts, voltage : message.vl, avgVoltage : avgVoltage) ?WHERE sensorId = message.id ? ?#?An Example of Series Data Notification?#?Finally we need a notification if the moving average ever exceeds 2.0 voltsRecall Vantiq i

39、s ingesting the sensor data and computing the moving average over the voltage readings. So we will need to define a rule to identify when the moving average is over 2.0 voltsThe Rule will look like the followingRULE voltageTooHighWHEN UPDATE OCCURS ON Sensor AS sTHRESHOLD = 2.0if(s.avgVoltage THRESH

40、OLD) PUBLISH body: sensor: Sensor.sensorId, message: voltage for last five readings 2.0 TO SOURCE alertServiceThe rule is triggered each time theavgVoltage is updated for a Sensor. The rule will determine if the new average is exceeds 2.0 volts. If the value does exceed 2.0 volts a notification is s

41、ent to the REST source namedalertService. The notification contains the sensorId that exceeded the voltage limit and a message identifying the condition detected.Geospatial Capabilities in Vantiq?#?GeoJSON Vantiq type representing a location as a GeoJSON objectSupport for Point, LineString, and Poly

42、gon type: Point, coordinates: 40, 5 type: LineString, coordinates: 40, 5 , 41, 6 type: Polygon, coordinates: 0 , 0 , 3 , 6 , 6 , 1 , 0 , 0 GeoJSON features:May be referenced in an ORDER BY clauseCurrently you can only create in JSON (e.g. not via VAIL or Portal)Maybe used in an indexSpecification IN

43、DEX UNIQUEGEO ASC | DESC , ASC | DESC When GEO modifier is included, a single property containing a GeoJSON type must be specified as the indexed propertyGeospatial Functions?#?geoNear geoJSON value must be within the distance specified of the object identified in the right hand argument to geoNear.

44、Example:myLocation geoNear geometry: type: point, coordinates: 90.834, 34.1987 minDistance: 100 maxDistance: 1000 geoIntersectsgeoJSON value must intersect the object identified in the right hand argument to geoIntersects.Example:myLocation geoIntersects geometry: type: point, coordinates: 90.834, 3

45、4.1987 geoWithingeoJSON value must be contained within the object identified in the right hand argument to geoWithin.Example:myLocation geoWithin geometry: type: polygon, coordinates: 0, 0, 0, 10, 10, 10, 10, 0, 0, 0 geoNear and geoWithin require a Geo indexRecommendations Overview?#?“Recommendation

46、s”Provides ranked list of results based on a given inputExamples:Engagement (ad-tech): Dynamically indicates the user action (e.g. user just clicked the buy drink button on slot machine) and wants to identify ad inventory placement identifier or provide private content?Advanced Web Search e.g. Scrat

47、chd2: Dynamically indicates currently viewed product and user and wants top N products to recommend?Recommendations can be thought of as a “request- response” model for situation identification and response.?Response time performance requires preparing as “knowledge” in advance to minimize decision

48、work in real-time (analogous to how scoring works with predictive analytics algorithms)How Recommendations Work?#?FILTER and MATCH are the VAIL constructs for RecommendationsA recommendation is produced by applying MATCH conditions to a taxonomy (catalog) of potential recommendations to identify the

49、 recommendations that are most relevant to the current situation. The catalog is defined as a type in the model.The taxonomy is represented by a type in the model. The type MUST define a property that contains the taxonomy category assigned to each item in the taxonomy. Typically, the taxonomy value

50、s are hierarchically organized and can use any common notation for a hierarchy as long as components of the taxonomy name proceed from most general to most specific in a left to right manner. Example:/books/english/mysteries or books.english.mysteries might represent a taxonomy for books.The recomme

51、ndation process begins with the user providing a single example item from the catalog and requesting a recommendation for the most relevant recommendations given the example item provided.The engine applies the recommendation rules to the catalog to find the most relevant recommendations.See NOTES f

52、or code example?FILTER and MATCH?#?FILTER is similar to the FOR statement but specialized to perform filtering/finding relevant recommendations.?SyntaxFILTER IN INTO UNTIL MATCH RECOMMEND WITH relevance = , reason = . The FILTER statement may be applied to items of any type (though most typically it

53、 is applied to objects from the automation model).Example:var user = SELECT * From MyUser WHERE name = paul ?FILTER r in PossibleActions INTO actions UNTIL actions.size() 5 MATCH if (r.location = user.location) RECOMMEND r WITH relevance = 1, reason = close by MATCH if r.type = user.type RECOMMEND r

54、 WITH relevance = 1, reason = matching type The Filter will iterate over all objects in PossibleActions for the first MATCH condition producing an intermediate result consisting of all possible actions that are appropriate in the users location, along with the associated relevance and reason (if pro

55、vided). The filter will terminate if the intermediate result contains less than 5 objects as specified in the UNTIL clause. If more thanfive objects remain, the second MATCH condition will be applied to further reduce the set of possible actions.RECOMMEND?#?RECOMMEND places a candidate in the recomm

56、endation set The relevance and reason are optional.The members of the final set will have the following properties:candidate - Normally a value from the iteration or a constructed value relevance - optional numeric value (highest relevance assigned value of 1)reason - optional string or object -provides information on relevancy decision?The final set of recommendations are those remaining in recommendationSet once the FILTER statement has completed?Analytics models can be us