Data Mining.ppt

1、made by Radmilo Pesic its continuous-media data Heterogeneus Databases and Legacy Databases The World Wide Web mining path traversal patterns,made by Radmilo Pesic generalized, primitive-level or knowledge at multiple levels; regularities or irregularities) according to the kinds of techniques utili

2、zed (autonomous, interactive exploratory or query-driven systems; data warehouse oriented, statistics) according to the applications adapted (for finance, DNA, etc.),Data Mining,Database technology,Information science,Machine learning,Statistics,Visualization,Other disciplines,made by Radmilo Pesic

3、Choosing the grain of the business proces; Choosing the dimensions; Choosing the measures.,made by Radmilo Pesic the partial materialization of cuboids shoul consider three factors:,identify the subset of cuboids to materialize, exploit the materialized cuboids during query processing, and efficient

4、ly update the materialized cuboids during load and refresh.,made by Radmilo Pesic & Branko Golubovic,45/74,Multiway Array Aggregation in the Computation of Data Cubes,ROLAP: Sorting, hashing, and grouping operations are applied to the dimension attributes in order to reorder and cluster related tupl

5、es. Grouping is performed on some subaggregates as a “partial grouping step”. These “partial groupings” may be used to speed up the computation of other subaggregates. Aggregates may be computed from previously computed aggregates, rather than from the base fact tables. MOLAP: Partitition the array

6、into chunks. Compute aggregates by visiting cube cells.,made by Radmilo Pesic & Branko Golubovic,46/74,a0 a1 a2 a3,1 2 3 4,13 14 15 16,b3,b2,b1,b0,5,9,30,29,31,32,45,46,47,48,44,40,36,28,24,20,60,56,52,61,62,63,64,c0,c1,c2,c3,A,C,B,A 3-D array for the dimensions A, B, and C, organized into 64 chunks

7、,made by Radmilo Pesic & Branko Golubovic,47/74,Indexing OLAP Data,Bitmap indexing Join indexing,made by Radmilo Pesic & Branko Golubovic,48/74,Bitmap Indexing,Base table,Item bitmap index table,City bitmap index table,Indexing OLAP data using bitmap indices,made by Radmilo Pesic & Branko Golubovic,

8、49/74,Join Indexing,location,Linkages between a sales fact table and dimension tables for location and item,Main Street,Sony-TV,T459,T884,T238,T57,item,sales,Join index table for location/sales,Join index table for item/sales,Join index table linking two dimensions location/item/sales,Join index tab

9、les based on the linkages between the sales fact table and dimension tables for location and item,made by Radmilo Pesic & Branko Golubovic,50/74,Efficient Processing of OLAP Queries,Determine which operations should be performed on the available cuboids Determine to which materialized cuboid(s) the

10、relevant operations should be applied,made by Radmilo Pesic & Branko Golubovic,51/74,Metadata Repository,A description of the structure of the data warehouse Operational metadata The algorythms used for summarization The mapping from the operational environment to the data warehouse Data related to

11、system performance Business metadata,made by Radmilo Pesic & Branko Golubovic,52/74,Data Warehouse Back-End Tools and Utilities,Data extraction Data cleaning Data transformation Load Refresh,made by Radmilo Pesic & Branko Golubovic,53/74,Further Development of Data Cube TechnologyDiscovery-Driven Ex

12、ploration of Data Cubes,SelfExp InExp PathExp,made by Radmilo Pesic & Branko Golubovic,54/74,Change in sales over time,Change in sales for each item-time combination,made by Radmilo Pesic & Branko Golubovic,55/74,Change in sales for the item IBM desktop computer per region,made by Radmilo Pesic & Br

13、anko Golubovic,56/74,Complex Aggregation at Multiple Granularities: Multifeature Cubes,Example 1: Query 1: A simple data cube query. Find the total sales in 2000, broken down by item, region, and month, with subtotals for each dimension. Example 2: Query 2: A complex query. Grouping by all subsets o

14、f item, region, month, find the maximum price in 2000 for each group, and the total sales among all maximum price tuples. selectitem, region, month, MAX(price), SUM(R.sales) fromPurchases whereyear=2000 cube byitem, region, month: R such thatR.price=MAX(price),made by Radmilo Pesic & Branko Golubovi

15、c,57/74,Example 3: Query 3: An even more complex query. Grouping by all subsets of item,region,month, find the maximum price in 2000 for each group. Among the maximum price tuples, find the minimum and maximum item shelf life. Also find the fraction of the total sales due to tuples that have minimum

16、 shelf life within the set of all maximum price tuples, and the fraction of the total sales due to tuples that have maximum shelf life within the set of all maximum price tuples. selectitem, region, month, MAX(price), MIN(R1.shelf), MAX(R1.shelf), SUM(R1.sales), SUM(R2.sales), SUM(R3.sales) fromPurc

17、hases whereyear=2000 cube byitem, region, month: R1, R2, R3 such thatR1.price=MAX(price) and R2 in R1 and R2.shelf=MIN(R1.shelf) and R3 in R1 and R3.shelf=MAX(R1.shelf),made by Radmilo Pesic & Branko Golubovic,58/74,From Data Warehousing to Data MiningData Warehouse Usage,Information processing Anal

18、ytical processing Data mining,made by Radmilo Pesic & Branko Golubovic,59/74,From On-Line Analytical Processing to On-Line Analytical Mining,High quality of data in data warehouses Available information processing infrastructure surrounding data warehouses OLAP-based exploratory data analysis On-lin

19、e selection of data mining functions,made by Radmilo Pesic & Branko Golubovic,60/74,Architecture for On-Line Analytical Mining,Graphical user interface API,Cube API,Database API,OLAM engine,OLAP engine,Databases,Data warehouse,Meta data,MDDB,Databases,Data filtering, data integration,Data cleaning D

20、ata integration,Filtering,Constraint-based mining query,Mining result,Layer 1 data repository,Layer 2 multidimensional database,Layer 3 OLAP/OLAM,Layer 4 user interface,An integrated OLAM and OLAP architecture,made by Radmilo Pesic & Branko Golubovic,61/74,3,Data Preprocessing,made by Radmilo Pesic

21、& Branko Golubovic,62/74,-2, 32, 100, 59, 48,Data integration,Data transformation,Data cleaning,-0.02, 0.32, 1.00, 0.59, 0.48,transactions,attributes,transactions,attributes,Data reduction,Format of data preprocesing,made by Radmilo Pesic & Branko Golubovic,63/74,Data CleaningMissing values,Ignore t

22、he tuple Fill in the missing value manualy Use a global constant to fill in the missing value Use the attribute mean to fill in the missing value Use the attribute mean for all samples belonging to the same class as the given tuple Use the most probable value to fill in the missing value,made by Rad

23、milo Pesic & Branko Golubovic,64/74,Inconsistent dataNoisy data,Bining Sorted data for price (in dollars): 4, 8, 15, 21, 21, 24, 25, 28, 34 Partition info (equidepth) bins: Bin 1: 4, 8, 15 Bin 2: 21, 21, 24 Bin 3: 25, 28, 34 Smoothing by bin means: Bin 1: 9, 9, 9 Bin 2: 22, 22, 22 Bin 3: 29, 29, 29

24、Smoothing by bin boundaries: Bin 1: 4, 4, 15 Bin 2: 21, 21, 24 Bin 3: 25, 25, 34 Clustering Combined computer and human inspection Regression,made by Radmilo Pesic & Branko Golubovic,65/74,Data Integration and TransformationData IntegrationData Transformation,Smoothing Aggregation Generalization Nor

25、malization Attribute construction,made by Radmilo Pesic & Branko Golubovic,66/74,Data Reduction,Data cube aggregation Dimension reduction Data compression Numerosity reduction Discretization and concept hierarchy generation,made by Radmilo Pesic & Branko Golubovic,67/74,Dimensionality reduction,Step

26、wise forward selection Stepwise backward elimination Combination of forward selection and backward elimination Decision tree induction Example:,Greedy (heuristic)methods for attribute subset selection.,made by Radmilo Pesic & Branko Golubovic,68/74,Data Compression,Wavelet transforms Principal compo

27、nents analysis,made by Radmilo Pesic & Branko Golubovic,69/74,Numerosity Reduction,Regression and log-linear models Histograms Clustering Sampling,made by Radmilo Pesic & Branko Golubovic,70/74,Histogram Examples,A histogram for price using singleton buckets each bucket represent one price-value/frequency pair.,An equiwidth histogram for price, where values are aggregated so that each bucket has a uniform width of $10.,made by Radmilo Pesic & Branko Golubovic,71/74,Discretization And Concept Hierarchy Generation,A concept hierarchy for the attribute price.,made by Radmil