翻译原文-外文翻译-一种日志优化算法在地震前兆观测系统数据同步中的应用-A log optimization algorithm’s application in the earthquake precursor system data synchronization

本地得到 XXXXXXXXXXXXXXXX 科研基金资助,项目编号:XXXXXXXXXXXXXXXA log optimization algorithms application in the earthquake precursor system data synchronization Liu Qingjie1, Wang Chen2, Wang Maofa11) Department Of Disaster Information Engineering, Institute Of Disaster Prevention Science And Technology,San He,He BeiMrovince, China()2) Institute of Geophysics, China Earthquake Administration, Beijing, ChinaAbstractIn based on user-defined operation database synchronization logs earthquake precursor network observation system, stations and regional centers databases operation log table exists the same data multiple update, delete records, because of data pre-processing and data re-collection. The same data with multiple log operations affect the efficiency of the precursor data exchange. For this problem, we propose an optimization algorithm for the operation log, By calculating and merging record in the operation log table and leaving the last operation log of data records, We can greatly reduce the network traffic and improve the efficiency of database synchronization. KeywordsData exchange; Operation log; Log optimization; Data Transaction Isolation1. IntroductionIn the earthquake precursor network observation system now running, the data synchronization between the databases relies on the Oracle original database synchronization mechanism and Oracles DB-link plus trigger. Due to the complexity of the paths of data exchange, the huge amount of data sources, and the intricacies of data source types, together with the station registration and cancellation mechanism, trigger deadlocks caused by inconsistencies of data often happen, and account for obstructions of the DB-link, so that the data exchange fails. In the latest precursor data management system based on N-tier architecture, we use a new data exchange strategy-the data exchange strategy which is based on user-defined operation database log, and whose whole process is in control. This log data based on user operation synchronization has been widely used, traditional log just saves the update action and related data in chronological order, but this log stores the data which is updated involved globally unique primary key. As precursor data may have the business need of same data multiple pretreatments, and multiple update actions on the same record for the higher level data, only the last update is effective. So we use the log compression by computing and merging the logs, retaining only data exchange log information and related logs, thereby we can reduce network data traffic and improve the efficiency of data exchange. We know this way will popularize.2. Data exchange and data operation log2.1 Path of data synchronizationIn the earthquake precursor network observation systems based on distribution, the synchronization between the databases is a core technology of the observation system. In earthquake precursor observation system, the data is synchronized from the non-node station database to the node station database, from the node station databases synchronization to the regional databases, from the regional central 本地得到 XXXXXXXXXXXXXXXX 科研基金资助,项目编号:XXXXXXXXXXXXXXXdatabases to the national central database, and then distributed to the academic databases from the national central database. In academic central database, it formed the packaged data and backflow to the national central database. The following figure shows part of paths of synchronization data flow in precursor network observation systems.R e g i o n a l D BD e v i c eN a t i o n a l D BR e g i o n a lA c a d e m i c D BS p e c i f i e d E x c h a n g eI n c r e m e n t D a t aI n c r e m e n tI n c r e m e n t （ a c a d e m i c D B o n l y ）C o l l e c tI n c r e m e n tS p e c i f i e d D a t aI n c r e m e n tR e g i o n a l E x t e n s i o n a l D BS p e c i f i e d E x c h a n g eS p e c i f i e d E x c h a n g eS p e c i f i e d E x c h a n g eA c a d e m i c E x t e n s i o n a l D BI n c r e m e n tS p e c i f i e d E x c h a n g eS t a t i o n E x t e n s i o n a l D BD e v i c eC o l l e c tI n c r e m e n tS p e c i f i e d E x c h a n g eS t a t i o n E x t e n s i o n a l D BI n c r e m e n tS p e c i f i e d E x c h a n g eD e v i c eC o l l e c tN a t i o n a l E x t e n s i o n a l D BI n c r e m e n tS p e c i f i e d E x c h a n g eI n c r e m e n t D a t aS p e c i f i e d D a t aN a t i o n a l D BFigure 1 Path of Synchronization in precursor network observation systemsFrom the figure, we can know, the earthquake precursor network observation system is a typical two-way synchronization database system.2.2 Time of data synchronizationThe database Synchronization is divided into real-time data synchronization and asynchronous data synchronization based on synchronization time. Real-time data synchronization is a technology which means once data source is updated; in real time date is synchronized to the target database. Real-time data synchronization can ensure real-time data consistency between the data source and data destination, and protect the integrity of the applications in a distributed environment, reduce the technical difficulty of the development, and decrease the complexity of the application. When the data source data is updated, real-time data synchronization will instantly capture the change data, and updates the changes broadly to the data destination. Because of high real-time requirements, real-time data synchronization technology should be used when the data update frequency is low, the data change is little, and the data synchronization in the high-bandwidth, highly reliable LAN or private network environment.Asynchronous data synchronization is data database synchronization technology which is a delay distribution of data update and data access. It possesses high reliability and flexibility with on-demand synchronization, but it is complex when compared to real-time data synchronization technology. With asynchronous data synchronization, incremental data log stored in the synchronization source database, users can be flexible to run data synchronization according to network conditions, data priority, path, user needs and other conditions.Earthquake precursor network observation system is a large-scale distributed application system with low bandwidth, high data throughput, multi-path, its synchronous content changes frequently and its nodes lie across the whole country. So the using asynchronous data synchronization technology meets the needs of earthquake precursor observation and actual network conditions better.2.3 THE APPLICATION OF LOG OPTIMIZATION FOR DATABASE SYSTEMDatabase operation log data synchronization among the distributed databases is broadly used in the major mainstream database products: customized copy from Sybase, the DB2 data connection bridge, DBLink snapshot refresh by Oracle, replication / subscribe services in Microsoft SQLServer. These technologies all get incremental data by capturing the incremental 本地得到 XXXXXXXXXXXXXXXX 科研基金资助,项目编号:XXXXXXXXXXXXXXXlogs, and then synchronize the two terminals or multi-terminal, one-way or two-way data synchronization. Widely used in major mainstream databases, database synchronization based on the log has become a database synchronization standard technique, and the key technology about reliability, flexibility, scalability of relation data synchronization.Data synchronization efficiency plays a key role on the availability of a variety of application systems. The factors of the database synchronization efficiency are various: network bandwidth and degree of obstruction, the data update frequency, the size of data. Routine databases logs based on the database manufacturers are mainly developed for transaction security, and save the various versions of the data related to each change data. However, in asynchronous data synchronization mode, when a record of the source database has been frequently modified, logs will record each change and produce a lot of log entries, and then database logs spread to the destination databases. That leads to the same record being repeatedly modified, while only the last modification is valid, resulting in waste of resources. As only the last data update log is valid, so if we can optimize the database log, we will reduce the data synchronization data, reduce network bandwidth load, reduce unnecessary data update, thereby enhancing the efficiency of data synchronization.In the earthquake precursor network observation system, database synchronization is characterized by vast paths of demand, frequent synchronization content adjustment, general network conditions, synchronization frequent registration and cancellation in source and destination, unidirectional and bidirectional synchronization coexist, point-to-point synchronization and point-to-multipoint synchronization coexist, partially synchronization and intact synchronization coexist. But the log user interface provided by the database vendor is a black box, we cant operate log accurately to meet application demands. So practically we use user-defined operation database synchronization log to capture the data changes, use user-defined log interface and data operation interface to achieve the high flexibility, scalability, reliability of data synchronization program. This kind of user-defined database operations log is also known as virtual logs or simulated logs.3. Log optimization algorithm and steps3.1 Database logUser-defined operation log is the basic of precursor data exchange; it is a collection of incremental data update activities record in the entire database. Any update operation for any exchange of data (such as the insert, update, delete) will log the primary key, table name, action, the number of stations, measuring point number and other information of the corresponding data into the virtual log table.We define an access component of the unified access platform subsystem in earthquake precursor network observation system to achieve the interceptor function. Checking the synchronization flag of the data access interface, database operation logs can be saved to the virtual log table. We define modification sequence of the record with primary key id which is in table P from data operation log table as Sequence (T, v).With the compression of Sequence (T, v ), we can effectively reduce the data transmission amount in data synchronization process.Tv is defined as the record of the modification of v in table T. Only the insert, update, delete actions are possible changes to tv, so we have a State diagram to show all the states and processes in Figure 2.本地得到 XXXXXXXXXXXXXXXX 科研基金资助,项目编号:XXXXXXXXXXXXXXXFigure2 operation log state diagramAmong them, the nodes mean all possible modification operations to tv (the insert, update, delete); vectors with e head mean all possible transitions. The vector e4 stand for that tv data is deleted; the vector e2 stand for that tv has experienced several updates. We define set M m to express state transitions collection in the above figure recorded.Mm= ID,I UU,I UU D,D I UU,UU,UU D,UU DI. means N times data operation to tv; m is a set of possible modifications to record tv. So, regardless of what state is the Sequence (T, v) can be hashed a set with N ms in it.3.2 Principles of optimizationAssigning insert, delete, update three operating node that I = 1, U = 0, D = -1, we define Value (m) as the sum of the node value in a path. So, here come the following conclusions:(1) Value (m) = 1, before m operation occurs, tv does not exist in the data table, the log is added to the data table in the process of m operations.(2) Value (m) = -1, before m action occurs, the data table tv, the log is deleted in the process of m operation.(3) Value (m) = 0, although we cant determine if tv exists in the data table, but we are able to confirm if before m occurs the tv exists in the table, then after m occurs, the data still exists.Sequence (T, k) is a sequence of N ms, so the above conclusion is the same with the sequence set Sequence (T, v).3.3 Log optimization algorithmWhen Update operations need to be done, database operations log should run first. And for each Sequence (T, v), we compute Value (Sequence (T, k) to run strategies.(1) Value (Sequence (T, k) = 1, first retrieve the operation log table to tell whether there is a log containing this record. If the result is true, then we retain the last operation log, write nothing; otherwise, we will add an insert record of the corresponding record to log.(2) Value (Sequence (T, k) = -1, first retrieve the operation log table to tell whether there is a log containing this record. If the result is true, then we retain the last operation log, write nothing; otherwise, we will add a delete record of the corresponding record to log.(3) Value (Sequence (T, k) = 0, first retrieve the operation log table to tell whether there is a log containing this record. If the result is true, then we retain the last operation log, write nothing; otherwise, we will add an update record of the corresponding record to log.By the virtual optimization log, in the system that needs to be updated frequently, we are able to reduced amount of data transmission obviously during the data synchronization network. Virtual log table only stores table name, keyword, and method of operation of the update data, does not record the specific operation records. Comparing to traditional database log, its storage space can be neglected. 本地得到 XXXXXXXXXXXXXXXX 科研基金资助,项目编号:XXXXXXXXXXXXXXXBased on the above log optimization method, we can merge multiple tv operations through the calculation of the Value (Sequence (T, k), and only run the last work to avoid waste of resources, and increase the data synchronization efficiency.4. Realization of log optimization algorithm in the earthquake precursor observation system4.1 Operation log designIn the earthquake precursor observation system, we have to synchronize data from and to appointed devices. Because data operation log need the field of appointed devices, we should pick up the operation log of appointed devices from virtual operation log during the incremental data synchronization process. According to this requirement, the precursor data exchange needs an operation log table with structure as shown in Table 1:Field name Definition Data typeRecordIndex Record Index Number(38)TableName Table Name Varchar2(50)Operation Operation Varchar2(20)UUID Universal unique idVarchar2(255)StationID StationID Varchar2(5)PointID PointID of Device Varchar(1)Table1 data exchange operation log structure4.2 Pseudo code of log optimization algorithm（1）optimization pseudo code of Insert actionRead all trace and parameter initialization Loop from begin currentLogId-maxIdfind current Datas pk if find identical logkeep the current log endifelse insert current Datas log insert action end elseend loop（2）optimization pseudo code of Update actionRead all trace and parameter initialization Loop from begin currentLogId-maxIdfind current Datas pk if find identical logkeep the current log endifelse insert current Datas log update actionend else end loop（3）optimization pseudo code of Delete actionRead all trace and parameter initialization Loop from begin currentLogId-maxIdfind current Datas pk if find identical logkeep the current log endifelse insert current Datas log delete action end elseend loop4.3 Realization of log optimization algorithm in the projectIn precursor data management system, data I/O operations are realized by data unified access platform, and data unified access platform provides distributed DAO components based on view of data objects to access data CRUD operations. The object data access interface in DAO components has the sign of data exchange, and control the exchange of data according to the business layer. Log optimization algorithm code need to be availed to be called by all interfaces of DAO components. Considering code reuse and software decoupled, we packed log optimization code to three external interceptors; each one with a switching flag in DAO interface methods, call log optimization code by annotation to Interceptors.(1)Insert interceptor codeAroundInvoke public Object save(InvocationContext ctx) throws Exception Object o=ceed();Object params=ctx.getParameters();boolean isExchange =(Boolean)params1;if(isExchange)Class c=params0.getClass();Method method=c.getMethod(getId, new Class);String id=(String) method.invoke(params0, new Object);String stationid=;tryMethod method2=c.getMethod(getStationid, new Class);stationid=(String)method2.invoke(params0, new Object);catch(NoSuchMethodException ex)stationid=xxxxx;String pointid=;本地得到 XXXXXXXXXXXXXXXX 科研基金资助,项目编号:XXXXXXXXXXXXXXXtryMethod method3=c.getMethod(getPointid, new Class);pointid=(String)method3.invoke(params0, new Object);catch(NoSuchMethodException ex)pointid=x; String tableName=(c.getAnnotation(Table.class).name();GlExchangeOperations log=new GlExchangeOperations();log.setTablename(tableName);log.setUuid(id);log.setOperation(insert);log.setStationid(stationid);log.setPointid(pointid);em.persist(log);return o; (2)Update interceptor codeAroundInvokepublic String getTableName(InvocationContext ctx,String uuid) String results=new String3;String tableName = ;try String beanClassName = ctx.getTarget().getClass().getName();String EntityClassName = beanClassName.substring(0,beanClassName.indexOf(DAOImpl).replace(impl, entity);System.out.println(EntityClassName);Class c=Class.forN