Data mining: past results as a forecast for the future

Data mining is searching large amounts of data for patterns: relationships, groups and trends you hadn’t thought of in advance. You use those patterns to understand what is happening, and to predict what will happen. Where data analysis in an improvement project mainly tests a suspicion, data mining looks for what you didn’t yet know.

Where does it come from?

Statistics has existed for centuries, and in the 1920s Walter Shewhart already used it to monitor processes, with what later became known as statistical process control (SPC). Data mining is younger. The term emerged in the 1990s, when companies stored more and more data in databases and computers became powerful enough to search it. Data mining combines statistics, database technology and machine learning.

Around 1999 a group of companies drew up the CRISP-DM model, a step-by-step plan still widely used. It is shown in the image at the top of this article:

1. Business understanding. Which question does the organisation want answered?
2. Data understanding. Which data is available, and how reliable is it?
3. Data preparation. Cleaning, combining, dealing with missing values. This often takes the most time.
4. Modelling. Applying techniques that look for patterns.
5. Evaluation. Is the model right, and does it answer the question?
6. Deployment. Putting the model to work, and keeping track of it.

It is not a straight line: after each phase you may find you need to go back.

What kinds of pattern do you look for?

Broadly, data mining looks for four kinds of pattern, as shown below.

This is one of the tools you learn to apply in the Green Belt training.

• Classification: which group does something belong to? For example: will this complaint escalate, is this transaction suspicious?
• Clustering: which groups are in the data, without defining them in advance? For example types of customers with similar behaviour.
• Association: what often occurs together? The best-known example is the shopping basket: people who buy A often buy B too.
• Prediction: what value will it be? How many requests will come in next week, how long will this delivery take?

Predicting with the past

The idea behind data mining is that patterns from the past say something about the future. That only works if the data is good enough, and that is about more than quantity.

Take an election poll. A large sample helps, but if you only ask people over sixty, you will still predict the outcome wrongly. The sample must be representative: characteristics that influence voting, such as age, income or place of residence, must be represented in the right proportions. In Six Sigma such influencing factors are called the Xs. The same applies to any dataset: does it actually contain the situations you want to predict?

Pitfalls

• A relationship is not a cause. Data mining finds association, not explanation. Read more under correlation.
• Flukes. Search long enough and you will always find a pattern. So test a model on data that wasn’t used to build it.
• The past is not the future. If the situation changes, the pattern no longer holds. People are more changeable than the weather.
• Privacy. Where personal data is involved, data protection rules such as the GDPR apply.

Data mining and process improvement

In process improvement, data mining and the Six Sigma approach go well together. Data mining points to where to look; with DMAIC and a targeted test you establish whether a cause is real, and improve the process. A special form is process mining, which derives from IT system data how a process really runs.