big-data-Application.doc

Application of big dataAbstract: Since the mid 1980s, the world has experienced an unprecedented explosion in the capacity to produce, store, and communicate data, primarily in digital formats. Simultaneously, access to computing technologies in the form of the personal PC, smartphone, and other handheld devices has mirrored this growth. With these enhanced capabilities of data storage and rapid computation as well as real-time delivery of information via the internet, the average daily consumption of data by an individual has grown exponentially. Key Words:Big data; education; application;Universe Discovery:Use Big data to analyze huge amount of data stored in the Glass plates in the traditional way and provide us a better picture of the origin and evolvement of the universeEnvironmental researches:An Irish company called Tree Metrics with eleven million trees indexed on three continents develops a system using big data to keep track of all the trees in order to meet the log market demand while at the same time minimizing the effect on the forest.Education Support:Big data technology is also used as an assistance of computer algorithm to transform large amount of books simultaneously into native languages to better the education in underdeveloped regions.Energy Saving :Big data can help power companies and their customers conserve energy by giving feedback messages from a malfunctioning meter to the power company or alarming customers when they are wasting energy.Crime PredictionBig Data analytics is helping police departments to develop prediction algorithms to prevent probable crime commitments.TechnologyHadoop EcosystemVolume: large amount of dataVariety: heterogeneity of data types, representation, and semantic interpretation.Velocity: both the rate at which data arrive and the time in which it must be acted upon.Veracity: refers to the biases, noise and abnormality in data. Is the data that is being stored, and mined meaningful to the problem being analyzed.( In scoping out your big data strategy you need to have your team and partners work to help keep your data clean and processes to keep dirty data from accumulating in your systems.)Challenges:Privacy:Big Data profoundly changes the problem of privacy another reason why the USs data-driven companies are lobbying so hard to use the review of data protection law to weaken it.A big data society seems to be inevitable, andpromises much, but privacy (properly understood) must be an important part of any such society.First, we need to think broadly about privacy as more than just the keeping of a secret，but as the rules that must govern personal information.Second, we need to realize that information does not lose legal protection just because it is held by another person.Confidentiality rules will be essential in any big data future.Third，we need to develop somebig data ethicsas a society.Big data in the field of Education:v Students using online systemv System to collect and record the studentsonline learning behavior ,saving into databasev For data analysis and processing, predicting the students academic performancev The result of the forecast and feedback for visual processing v To meet the requirements of the students individual learning materialsv Teachers,administrators, and developers timely to give students guidance and helpBig data has changed the way financial institutions operate in traditional data from four different ways， achieving great business value。These four areas（4C） include：Compatibility，Connectedness，Cost，Capitalization. The application of large data in financial industry is gradually expanding , Overseas, big data have been fully tried in the financial industry, such as risk control, operation management, sales support and business model innovation. In China, the application of large data in financial institutions is still basically at the initial stage. Key challenges such as data consolidation and departmental coordination remain the main bottlenecks that hinder financial institutions from translating data into value. Example:An online store has an existing SQL Database that manages its authorization system and storefront. SQL is necessary in order to ensure no lost transactions and full ACID compliance.The store also tracks its social activity using a KEY/Value database. This database provides speed and can easily store large amounts of data but provides less consistency than a SQL solution. However, as social data is not as critical as transactional, this solution is appropriate.The store also provides recommendations to customers about products they might like. They use a Column database for this purpose. SQL is not necessary because the information is not relational. The Ministry of Education is responsible for managing data about students around the province. They have over 125 million records (students), 241 million rows and over 15 years worth of data. The data is stored using 450 tables, but the only variety is at attribute level(i.e. Standardized testing,performance,etc). They receive the data from individual schools in giant chunks four times a year.The Ministry decided against using big data tools to manage their data and is still using a RDBMS. Although they hava large amounts of data, the type of their data and the velocity at which they receive it means that a RDBMS is still best. They also have compliance and privacy issues that make NoSQL an unsuitable choice.Each school and school board was sending the data to the Ministry using different schemas and this inconsistency made data management difficult and decreased the value of the data for BI purposes.The Ministry took a master data management approach. Over the course of 4 years, they created common methodologies and a data usage framework that ensured the data