OSIsoft PI 时序数据库核心技术

OSIsoftOverview DevelopmentsinthePISystemFamilyofProductsGreggLeBlancPISystemProductManager WhyScalability MoredatacanbetrackedeverydayPhoneshaveGPSlocationdataReal timemonitoringinthehomeElectricalusageSecuritysystemsFire FlooddetectionITApplicationsEverymachineonanetworkneedsmonitoringExample 700 000homesx10datapoints 7 000 000points Benefits ITMonitoringcandetectIntrusionsImpendinghardwareproblemsDiagnosesoftwareproblemsVirustrafficExample BecauseWiredCitycoulddetectandrecordtheamountofinboundtrafficfromtheNimdaworm theygotarefundfromMCIWorldcom WhereisPIToday 100 000 DataStreams100 sofsimultaneousclients300 Interfaces6 500 Customers1 000 sofeventsrecorded sec IssuesWithinScalability AscalablesystemhasseveralaspectsUsersservedCalculationsandanalysisIntegrationissuesDatacapacityDatathroughputSecurityInfrastructure UserLoad Integration UsersneedfastaccesstothedataClient ServerusersStatelessUsersClient ServerusersconnectdirectlytoPIStatelessUsersBroaderaudiencePIICEWebservicesandintegration ServerManager PISDK SafelyexposesPIdataThePITrustTableretainspointbypointsecurityConnectionmanagementbalancesuserrequests InternetInformationServer IIS Browser SecurityinICE PIWebService PISDK SOAPMessages PIEnterpriseServer PIPointSecurity WindowsAuthentication AggregatedQueries PITrustTable ConnectionsinICE Elvispidemo Nortonpiadmin Alicepialice Rickypiadmin PISystem Lucypidemo ICE Norton Ricky Alice Elvis Lucy PISDKServerObjects1peruniquePIuser 1TCP IPConnectionperserver PISDK SessionManager UnderlyingWebServiceStructure PIICEWebServicesLayer ICEWebParts ICEWebParts ICEWebParts ICEWebParts PISDK PIServer OtherHistorians RelationalDatabases DCSSCADALIMS LooselyCoupledApplications TightlyCoupled WebServicesLayer LooselyCoupledComputing TheideaofWebservicesAllowsforcomputinginanenvironmentthatis AsynchronousStatelessPlatformindependentGeographicallyindependentWillreplacesome tightlycoupled computing TightlyCoupledComputing Infrastructure leveltiesbetweensystemsUsesAPIcallsUsuallyproprietaryNotasflexibleExamples PIInterfacestootherinformationsystemsOLEDB ODBCconnectorsPoint to Point Middleware andotherintegration WebServiceIntegration LooselycoupledapplicationsbuiltonWebServiceIntegrationSpace TraditionalTightlyCoupledIntegrationSpace ICE KPI Reports Auditing ERP Quality WebServices PIBatchWebservices RelationalDBWebservices PIAuditingWebservices ERPWebservices PIICEWebservices WebServicesIntegrationSpace Infrastructure OSIsoftismovingto NETOlderapplicationswillmigrateNewapplicationswillleveragethe NETFrameworkManagementtoolswilluse NETNewtoolscomingforSMT SystemsManagementTools PartsofthePISystemwillbeaccessiblethrough NET DistributedAnalysis ServerBasedPerformanceEquationsDesktopBasedSpreadsheetsbasedonDataLinkDistributedCalculationsPIACEServerloadreducedCalculationscanbereused SecurityConcerns Isn tMicrosoftavulnerability Overthelastyear26vulnerabilitiesinApache22inMicrosoft sIISKeyissue systemmaintenanceWhataboutNimda 250 000systemsin9hoursAtleast 2 4BindamagesInfectionstartedJuly13 2001PatchwasavailableJune18 2001 MS01 033 SecurityResponse Moreaccountability MicrosoftNIPC NationalInfrastructureProtectionCenter workswithNERCandMicrosofttodevelopsecurityproceduresandstandardsKeyprecautionsstillremainGuardagainstsocialengineeringSystemmaintenanceUnknownvulnerabilitiesWiFi wardrivingUnprotectedmodemlinesusedforsupport OSIsoftSecurity PIPointSecurityTrustTableSingledirectionPItoPItransfersAuditingAnewdatabasethatrecordschangestoPIPointsValuesModulechanges What sBeenCovered HandlinguserloadCalculationsIntegrationdemandSecurityInfrastructureMillionpointPISystems HistoryoftheHistorian PI11983HP 10001985Vax VMS1988PINetClient ServerArchitecturePI21992Vax VMS1994Alpha OpenVMSPI31993DesignWork HistoryoftheHistorian PI3Big4UnixHP UXIBM AIXDec OSF 1Sun SolarisWindowsNT3 51DevelopmentLanguageC HistoryoftheHistorian PI3KeyconceptsofPI2wereusedSnapshotCompressionArchivecacheArchivenavigationPINet HistoryoftheHistorian PI3KeynewconceptsMulti processRPCBasedInter processcommunicationPISDKHistorizationofmanydifferentdatatypesDoublesStringsBLOBsCommoncodebaseforallplatforms64bitsupport PI3OriginalRelease PI3 0ReleasedNovember19951 000to100 000 PointsUpto2000eventsperseconds PI3 0toPI3 3 Howmuchcanyoudoin7years ConcentratedonfeaturesAlarmACETotalizerPISDKBatchDatabaseModuleDatabaseAuditNTSecurityCOMConnectorsDevelopmentInfra structureAutomatedbuildsandtestingBugfixes justafew Scaling 1996Typicalsystem 100KpointsDatarates 2000Events Second2002Manysystemsapproaching150KpointsUpto10 000Events Second Scaling 1995Intel Pentium ProProcessor2P200MHz2002Intel Xeon Processor4P1 6GHz Scaling ThePIdataratesreflectincreaseinprocessorspeed PointCountdoesnot Limitationisnotprocessorspeed LimitationsinPI3 3 Memory2GBofvirtualmemoryperprocessArchivecacheInter processcommunicationCompressedeventsfromsnapshottoarchiveSerializationofRPCsArchivesub systemhandlesonecallatatime Memory PIDatabasesarememoryresidentPointdatabaseModuledatabaseSnapshotPIArchiveCache PI3 3CacheMemoryIssues Cacherecordcanbequitelarge AddingasingleeventrequiresentirecacherecordSystemsreceivingdataformostpointswillhavealargememoryfootprintLotsofcacheactivityPushingrecordsoutofmemoryReadingrecordsintomemory PI3 4ArchiveCache TypicalPISystemMajorityofpointsreceivedataregularlyAddeventsNeweventsreceivedinorder andnearcurrenttime ThePI3 4Cache AddressesMemoryIssuesSmallermemoryfootprintMoreefficienthandlingofneweventsSameefficiencyreadingdataDesignTradeoffsMorecomplicated 3 3Snapshot ArchiveInter processCommunication CurrentMechanismEventreceivedisprocessedbythecompressionalgorithm Ifpassescompressioneventisaddedtoin memoryeventqueueEventsinqueuearepackagedandsentviaanRPCtothearchiveprocess 3 3SnapshotBuffering ArchiveunabletoprocesscompressedeventsBackupsExtremelyhighdataratesSystemproblemsSnapshotmustwriteto eventqueue fileArchivemustreadfromtheSnapshoteventqueueResult2physicaldiskI O shappenonsamefile 3 4Snapshot ArchiveInter processCommunication ChangedphysicalfiletomemorymappedfileFilesystemfeatureMappedbySnapshotandArchiveSnapshotwritescompressedeventsto file Archivereadscompressedeventsfrom file SynchronizationtechniquesrequiredtopreventcorruptionBufferingcasehandledbyfilesystem Limitation3 RPCSerialization PI3 3allsub systemsaresinglethreadedRequeststosub systemsareserializedArchiveexample 3simultaneouscallsArchiveSummaryPlotvaluesCompressedvalues RPCSerializationSolution MultipleThreadsThreadsaretheobviousapproachUnixandNThaveverypowerfulandeasytousethreadingmodelsConceptuallySimpleBut easytowritebugs Multi ThreadinginPI3 4 PI3hadsomethreadingsincetheoriginalreleasePINetManagerSub systemsuseathreadforreadingmessagesPI3 4Sub systemLevelthreadmodelMain housekeeping threadReadthreadMessagepumpPoolofworkerthreads Multi ThreadinginPI3 4 PI3 4Sub systemLevelthreadmodelEverysub systemismulti threadedSub systemmustimplementlocksforoptimalperformanceWorkerthreadpoolisconfigurableRuntimeKillthreadsSuspendthreadsChangepriorityAddworkerthreadsDeleteworkerthreads Multi ThreadinginPI3 4 Whatdoesthisgetus Thearchivecallfrom Won tkillyoursystem Whatwill10archivecallsform dotomysystem Longtime1 100Longtime2 1134289piar summary 1023453 time1 time2 rval pctgood ARCTOTAL PIon64BitWindows Oneadvantage largeraddressablememory32bit 2GB64bit 16 128GBXP 64 16GB NetEnterpriseServer 64GB NetDataCenter 128GB PIon64BitWindows Disadvantages SlowerVerysensitivetoproperlyoptimizedcodeServersonly64bitWindowsisnotintendedtoreplace32bitdesktops O