Home   About   Contact 
 E-Business Overview 
 ERP 
 CRM 
 Supply Chain 
 Sales Automation 
 Knowledge Management 
 Technology 
 Implementation 
 Project Management 
 Vendors 
 Resources 

Data mining

2004-11-01
 

Data mining, also known as knowledge-discovery in databases (KDD), is the practice of automatically searching large stores of data for patterns. To do this, data mining uses computational techniques from Statistics and Pattern recognition.

Data mining has been defined as "The nontrivial extraction of implicit, previously unknown, and potentially useful information from data" [1] and "The science of extracting useful information from large data sets or databases" [2]. Although it is usually used in relation to analysis of data, data mining, like artificial intelligence, is an umbrella term and is used with varied meaning in a wide range of contexts.

Used in the technical context of data warehousing and analysis data mining is neutral. However, it sometimes has a more pejorative usage that implies imposing patterns (and particularly causal relationships) on data where none exist. This imposition of irrelevant, misleading or trivial attribute correlation is more properly criticized as "data dredging" in the statistical literature.

Used in this latter sense, data dredging implies scanning the data for any relationships, and then when one is found coming up with an interesting explanation. (This is also referred to as "overfitting the model".) The problem is that large data sets invariably happen to have some exciting relationships peculiar to that data. Therefore any conclusions reached are likely to be highly suspect. In spite of this, some exploratory data work is always required in any applied statistical analysis to get a feel for the data, so sometimes the line between good statistical practice and data dredging is less than clear.

A more significant danger is finding correlations that do not really exist. Investment analysts appear to be particularly vulnerable to this. In his book Where Are the Customers' Yachts? ISBN 0471119792 (1940), Fred Schwed, Jr, wrote: "There have always been a considerable number of pathetic people who busy themselves examining the last thousand numbers which have appeared on a roulette wheel, in search of some repeating pattern. Sadly enough, they have usually found it."

Most data mining efforts are focused on developing a finely-grained, highly detailed model of some large data set. In Data Mining For Very Busy People [3], researchers at West Virginia University and the University of British Columbia discuss an alternate method that involves finding the minimal differences between elements in a data set, with the goal of developing simpler models that represent relevant data.

There are also privacy concerns associated with data mining. For example, if an employer has access to medical records, they may screen out people with diabetes or have had a heart attack. Screening out such employees will cut costs for insurance, but it creates ethical and legal problems.

Data mining government or commercial data sets for national security or law enforcement purposes has also raised privacy concerns. [4]

There are many legitimate uses of data mining. For example, a database of prescription drugs taken by a group of people could be used to find combinations of drugs with an adverse reactions. Since the combination may occur in only 1 out of 1000 people, a single case may not be apparent. A project involving pharmacies could reduce the number of drug reactions and potentially save lives. Unfortunately, there is also a huge potential for abuse of such a database.

Basically, data mining gives information that wouldn't be available otherwise. It must be properly interpreted to be useful. When the data collected involves individual people, there are many questions concerning privacy, legality, and ethics.

The a priori algorithm is the most fundamental algorithm used in data mining.



Related Topics
Business performance management - Overview
Expert System
What is Knowledge Representation
Data warehouse
Text mining
Information retrieval
Learning Management Systems

 


This article is from Wikipedia.org. All text is available under the terms of the GNU Free Documentation License.