Data Leakage and How to Avoid It

Machine learning models are built to learn patterns from historical data and use those patterns to make predictions on new, unseen data. But sometimes a model performs exceptionally well during training and evaluation—and then performs poorly after deployment.

One of the most common reasons for this unexpected behavior is data leakage.

Data leakage occurs when information that should not be available to a machine learning model during training accidentally becomes part of the training process. This gives the model an unfair advantage and produces misleadingly high evaluation scores.

Understanding data leakage is essential for anyone building reliable machine learning systems. In this article, we will explore what data leakage is, why it happens, different types of leakage, practical Python examples, and techniques for preventing it.

1. What Is Data Leakage?

Data leakage occurs when information from outside the training dataset—or information that would not actually be available at prediction time—is used while training a machine learning model.

As a result, the model learns patterns that it would not have access to in a real-world environment.

Consider a loan approval model.

Suppose we want to predict whether a customer will default on a loan.

Our dataset contains:

At first glance, these features may look reasonable.

But imagine that Missed Payments represents payments missed after the loan was issued.

If we are trying to predict default before approving the loan, this information would not yet exist.

Using it during training is data leakage.

The model may achieve extremely high accuracy because customers who later miss several payments are naturally more likely to default.

But at the actual prediction time, the model will not know how many future payments the customer will miss.

Therefore:

A feature can be highly predictive and still be completely invalid.

2. Why Is Data Leakage Dangerous?

Data leakage can make a poor machine learning model appear excellent.

Suppose two models produce the following results:

Model A initially looks much better.

But Model B is actually the more reliable model because its evaluation performance reflects real-world performance.

Leakage can cause:

• Unrealistically high accuracy

• Misleading precision, recall, F1-score, or ROC-AUC

• Poor production performance

• Incorrect model selection

• Overconfidence in the model

• Invalid business decisions

• Difficulty reproducing evaluation results

A model with leakage is essentially cheating during the exam.

It has access to information it should not know.

3. Major Types of Data Leakage

Data leakage can occur in several ways. Two of the most important categories are:

1. Target Leakage

2. Train-Test Contamination

Let’s understand each one.

4. Target Leakage

Target leakage happens when one or more input features contain information that directly or indirectly reveals the target variable.

Suppose we are building a model to predict:

Will the customer default on the loan?

Our features are:

credit_score
annual_income
loan_amount
default_notice_sent
loan_default

Here:

loan_default

is the target.

But default_notice_sent may only become Yes after the customer has already defaulted or is known to be in default.

Therefore, it contains information about the outcome.

Using this feature creates target leakage.

Another Example: Employee Attrition

Suppose we want to predict whether an employee will leave a company.

Features:

age
salary
years_at_company
performance_rating
exit_interview_completed
resigned

Target:

resigned

The feature:

exit_interview_completed

is suspicious.

An exit interview usually happens after an employee has decided to leave.

Therefore, the model effectively receives information about the future.

5. Train-Test Contamination

Another common type of leakage occurs when information from the test dataset influences the training process.

Remember the fundamental rule:

The test dataset should simulate completely unseen future data.

If information from the test set influences preprocessing, feature engineering, feature selection, or model training, the evaluation becomes unreliable.

A very common example occurs during feature scaling.

6. Data Leakage During Feature Scaling

Consider the following code:

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X_scaled = StandardScaler().fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled,
    y,
    test_size=0.2,
    random_state=42
)

This looks reasonable.

But there is a problem.

We scaled the entire dataset before splitting it.

StandardScaler calculates statistics such as the mean and standard deviation.

Conceptually:

mean = mean(all data)
standard deviation = std(all data)

Since the test observations contributed to those statistics, information about the test set influenced the transformation applied to the training data.

This is leakage.

7. Correct Way to Perform Feature Scaling

First split the dataset.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

Then create the scaler.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

Fit it only on the training data.

X_train_scaled = scaler.fit_transform(X_train)

Then use the same learned transformation on the test data.

X_test_scaled = scaler.transform(X_test)

Notice the difference:

scaler.fit_transform(X_train)
scaler.transform(X_test)

We do not write:

scaler.fit_transform(X_test)

because the test set should not teach the preprocessing algorithm anything.

8. Understanding fit(), transform(), and fit_transform()

This distinction is extremely important in Scikit-learn.

fit()

Learns information from the dataset.

For StandardScaler, it learns values such as:

mean
standard deviation

For an imputer, it might learn:

median
mean
most frequent value

transform()

Uses previously learned information to transform data.

fit_transform()

Essentially performs:

fit()
transform()

Therefore, the common pattern is:

preprocessor.fit_transform(X_train)
preprocessor.transform(X_test)

Think of it this way:

Training Data
      ↓
Learn preprocessing rules
      ↓
Transform Training Data
      ↓
Train Model


Test Data
      ↓
Apply existing preprocessing rules
      ↓
Evaluate Model

The test data is never allowed to influence the rules.

9. Leakage During Missing Value Imputation

Suppose our dataset contains missing values.

A common approach is median imputation.

Incorrect:

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy="median")

X_imputed = imputer.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X_imputed,
    y,
    test_size=0.2,
    random_state=42
)

The median was calculated using both training and test observations.

That creates leakage.

Correct:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

imputer = SimpleImputer(strategy="median")

X_train = imputer.fit_transform(X_train)
X_test = imputer.transform(X_test)

The median is learned only from the training data.

10. Leakage During Normalization

The same principle applies to MinMaxScaler.

Incorrect:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2
)

Correct:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

scaler = MinMaxScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Again, the principle is:

Split first. Learn preprocessing parameters second.

11. Data Leakage During Feature Selection

Feature selection can also introduce leakage.

Suppose we perform feature selection using the entire dataset:

from sklearn.feature_selection import SelectKBest

selector = SelectKBest(k=5)

X_selected = selector.fit_transform(X, y)

Then:

X_train, X_test, y_train, y_test = train_test_split(
    X_selected,
    y,
    test_size=0.2
)

This is problematic.

Why?

Because feature selection used:

X
y

from the entire dataset, including observations that later become part of the test set.

The selected features therefore benefited from information about the test labels.

Correct approach:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

selector = SelectKBest(k=5)

X_train_selected = selector.fit_transform(
    X_train,
    y_train
)

X_test_selected = selector.transform(X_test)

12. Data Leakage with SMOTE

SMOTE is used to handle imbalanced datasets by generating synthetic examples of the minority class.

A dangerous mistake is applying SMOTE before splitting the dataset.

Incorrect:

from imblearn.over_sampling import SMOTE

smote = SMOTE(random_state=42)

X_resampled, y_resampled = smote.fit_resample(X, y)

X_train, X_test, y_train, y_test = train_test_split(
    X_resampled,
    y_resampled,
    test_size=0.2
)

Why is this problematic?

SMOTE creates synthetic observations based on neighboring minority-class samples.

If SMOTE is applied before splitting, synthetic training examples may be influenced by observations that later appear in the test set.

The test dataset is no longer truly independent.

Correct:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

smote = SMOTE(random_state=42)

X_train_resampled, y_train_resampled = smote.fit_resample(
    X_train,
    y_train
)

Leave the test data untouched.

model.fit(X_train_resampled, y_train_resampled)

predictions = model.predict(X_test)

A useful rule is:

SMOTE belongs to the training process—not to the entire dataset.

13. Time-Based Data Leakage

Time-series and forecasting problems require extra care.

Suppose we want to predict tomorrow’s stock price.

Our dataset contains:

Today's price
Today's volume
Tomorrow's price
Tomorrow's trading volume

If Tomorrow's trading volume is used to predict Tomorrow's price, we have future information.

That information would not be available at the moment the prediction is made.

This is called look-ahead bias or temporal leakage.

14. Random Splitting Can Cause Leakage in Time-Series Data

Consider:

train_test_split(
    X,
    y,
    test_size=0.2,
    shuffle=True
)

For ordinary independent datasets, random splitting is often appropriate.

But for time-dependent datasets, it can create problems.

Suppose we have observations from:

2022
2023
2024
2025
2026

A random split might create:

Training:
2022
2024
2026

Testing:
2023
2025

The model effectively learns from the future and is evaluated on the past.

Instead, preserve chronological order:

Training:
2022
2023
2024
2025

Testing:
2026

Scikit-learn provides:

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)

for time-aware cross-validation.

15. Duplicate Data Leakage

Suppose the same customer record appears multiple times.

If one copy goes into the training set and another goes into the test set, the model may effectively see the test observation during training.

Example:

Customer 101 → Training
Customer 101 → Test

The resulting evaluation score may be misleadingly high.

Before training, investigate duplicates:

df.duplicated().sum()

Remove exact duplicates when appropriate:

df = df.drop_duplicates()

However, duplicate handling should depend on what each row represents. Repeated rows are not automatically invalid—for example, legitimate repeated transactions may be meaningful.

16. Group Leakage

Sometimes rows are different but belong to the same underlying entity.

Suppose a medical dataset contains several records for each patient.

Patient 101 → Record 1
Patient 101 → Record 2
Patient 101 → Record 3

A random split could produce:

Training:
Patient 101 → Record 1
Patient 101 → Record 2

Testing:
Patient 101 → Record 3

The model has already learned information specific to Patient 101.

A similar problem occurs with:

• Multiple images from the same person

• Transactions from the same customer

• Sessions from the same user

• Multiple documents from the same organization

• Measurements from the same machine

• Multiple samples from the same household

In these cases, we should consider group-aware splitting.

For example:

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(
    n_splits=1,
    test_size=0.2,
    random_state=42
)

train_idx, test_idx = next(
    splitter.split(X, y, groups=customer_ids)
)

This helps keep the same entity from appearing in both datasets.

17. Leakage Through IDs

Consider a dataset containing:

customer_id
age
income
credit_score
default

Should customer_id be used as a feature?

Usually, no.

An identifier generally does not represent meaningful predictive information.

Even worse, some IDs may encode hidden information.

For example:

DEFAULT_10023
GOOD_28391

The ID may indirectly reveal the target.

Always investigate columns such as:

customer_id
transaction_id
employee_id
application_number
record_number

before including them in the model.

18. Leakage from Post-Outcome Features

One of the easiest ways to detect target leakage is to ask:

When exactly does this information become available?

Suppose we predict whether a patient will be readmitted to a hospital.

Features might include:

age
previous_admissions
diagnosis
treatment
discharge_status
readmitted

If prediction happens at admission time, discharge_status is not available yet.

It is therefore an invalid feature even though it may be strongly correlated with readmission.

This principle applies to many domains.

Fraud Detection

Potential leaked feature:

fraud_investigation_result

Employee Attrition

Potential leaked feature:

exit_interview_score

Loan Default

Potential leaked feature:

collection_notice_sent

Customer Churn

Potential leaked feature:

account_closed_date

These variables may reveal information that occurred after the outcome was already known or after the intended prediction point.

19. Data Leakage During Cross-Validation

Cross-validation can also leak information if preprocessing is performed before cross-validation.

Consider:

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

scores = cross_val_score(
    model,
    X_scaled,
    y,
    cv=5
)

The scaler learned statistics from the complete dataset before the folds were created.

Each validation fold has therefore influenced preprocessing.

A better approach is to use a Pipeline.

20. Using Scikit-Learn Pipeline to Prevent Leakage

A pipeline combines preprocessing and model training into a single workflow.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

Now:

pipeline.fit(X_train, y_train)

The pipeline automatically fits the scaler using the training data and then trains the model.

For cross-validation:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(
    pipeline,
    X,
    y,
    cv=5
)

During each fold, Scikit-learn performs the equivalent of:

Training Fold
     ↓
Fit Scaler
     ↓
Transform Training Fold
     ↓
Train Model
     ↓
Transform Validation Fold
     ↓
Evaluate

The validation fold does not influence the scaler.

This is one of the major reasons pipelines are strongly recommended in machine learning projects.

21. Pipeline with Missing Value Handling and Scaling

Real-world projects often require several preprocessing steps.

For example:

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

Then:

pipeline.fit(X_train, y_train)

predictions = pipeline.predict(X_test)

The pipeline ensures that the imputer and scaler learn only from the training data.

22. Pipeline with SMOTE

For imbalanced datasets, imblearn provides its own pipeline implementation.

from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("smote", SMOTE(random_state=42)),
    ("model", LogisticRegression())
])

Then cross-validation can be performed more safely:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(
    pipeline,
    X,
    y,
    cv=5,
    scoring="f1"
)

SMOTE is applied to the training portion of each fold rather than to the entire dataset before cross-validation.

23. Feature Engineering Can Also Leak Information

Imagine we have transaction data:

customer_id
transaction_date
transaction_amount
fraud

We create:

average_customer_transaction

But suppose this average is calculated using all transactions, including transactions occurring after the prediction date.

Now future information has entered the feature.

A safer feature would be:

average transaction amount before prediction time

This distinction is extremely important in production machine learning.

Feature engineering must respect the exact point in time when the prediction would have been made.

24. A Simple End-to-End Example

Let’s create a small classification problem.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

X, y = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=6,
    n_redundant=2,
    random_state=42
)

Split the data first:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

Create a pipeline:

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

Train:

pipeline.fit(X_train, y_train)

Predict:

y_pred = pipeline.predict(X_test)

Evaluate:

accuracy = accuracy_score(y_test, y_pred)

print("Accuracy:", accuracy)

The important point is not the final accuracy.

The important point is the workflow:

Raw Dataset
     ↓
Train-Test Split
     ↓
Training Data
     ↓
Preprocessing learned from training data
     ↓
Model Training
     ↓
Test Data
     ↓
Apply learned preprocessing
     ↓
Prediction
     ↓
Evaluation

25. Incorrect vs Correct Machine Learning Workflow

Incorrect

Entire Dataset
      ↓
Missing Value Imputation
      ↓
Scaling
      ↓
Feature Selection
      ↓
SMOTE
      ↓
Train-Test Split
      ↓
Model Training

Information can leak from the eventual test set into several preprocessing operations.

Better

Entire Dataset
      ↓
Train-Test Split
      ↓
Training Data
      ↓
Fit Preprocessing
      ↓
Feature Selection
      ↓
Resampling if needed
      ↓
Train Model

Meanwhile:

Test Data
      ↓
Apply previously learned transformations
      ↓
Model Prediction
      ↓
Final Evaluation

In practice, pipelines are often the safest way to enforce this separation.

26. How to Detect Data Leakage

Data leakage is not always obvious.

Here are several warning signs.

1. Suspiciously High Accuracy

If a difficult prediction problem suddenly produces:

Accuracy = 99.8%

investigate before celebrating.

High performance does not automatically mean leakage, but unexpectedly high performance should trigger additional checks.

2. Huge Gap Between Validation and Production

Example:

Validation accuracy: 97%
Production accuracy: 71%

Possible causes include:

• Data leakage

• Distribution shift

• Poor sampling

• Overfitting

• Changes in production behavior

3. One Feature Is Extremely Predictive

Suppose feature importance shows:

account_closed = 0.87
age = 0.03
income = 0.02
usage = 0.04
tenure = 0.04

If account_closed dominates a churn prediction model, ask whether the feature becomes available only after churn.

4. Features Contain Future Information

Always ask:

Would this feature actually exist at prediction time?

5. Unexpected Difference After Removing a Feature

If removing one suspicious column changes accuracy from:

99%

to:

82%

investigate that feature carefully.

The original performance may have depended heavily on leaked information.

27. Questions to Ask Before Using Any Feature

For every feature, ask:

Question 1

When is this value generated?

Question 2

Would this information be available when the prediction is made?

Question 3

Does this feature directly or indirectly contain information about the target?

Question 4

Was this value calculated using future observations?

Question 5

Was this feature generated using the entire dataset?

These questions can prevent many leakage problems before modeling begins.

28. Best Practices to Avoid Data Leakage

Follow these principles in real machine learning projects.

1. Define the Prediction Point

Before building the model, clearly define:

What are we predicting?
When are we predicting it?
What information exists at that moment?

2. Split Before Learning from the Data

Operations that learn parameters should generally happen after splitting.

Examples include:

Scaling
Normalization
Imputation
Feature selection
PCA
Encoding methods that learn categories/statistics
SMOTE

3. Fit Preprocessors Only on Training Data

Use:

fit_transform(X_train)
transform(X_test)

4. Use Pipelines

Whenever possible:

Pipeline([
    ("preprocessing", ...),
    ("model", ...)
])

This reduces the chance of accidentally contaminating validation or test data.

5. Keep the Test Set Truly Unseen

Avoid repeatedly evaluating different models against the final test set and then using those results to make modeling decisions.

Once you repeatedly optimize against the test set, it gradually becomes part of the development process.

Use:

Training Set → Model training

Validation Set / Cross-Validation → Model selection and tuning

Test Set → Final unbiased evaluation

6. Respect Time

For temporal data:

Past → Training
Future → Validation/Test

not:

Random Past + Future → Training
Random Past + Future → Testing

7. Handle Groups Properly

Keep related records together when appropriate.

Examples:

Same customer
Same patient
Same device
Same household
Same user
Same company

8. Review Feature Origins

Understanding where a feature came from can be more important than understanding its correlation with the target.

29. Data Leakage vs Overfitting

Students often confuse data leakage with overfitting.

They are related but different.

Example of overfitting:

Training Accuracy: 99%
Test Accuracy: 78%

Example of leakage:

Training Accuracy: 99%
Test Accuracy: 98%
Production Accuracy: 70%

The second pattern is particularly dangerous because even the test score may look excellent if the test data was contaminated.

30. Data Leakage vs Data Contamination

These terms are sometimes used interchangeably, but a useful distinction is:

Data leakage is the broad problem where forbidden information influences model development or evaluation.

Train-test contamination is one specific form of leakage where information crosses the intended boundary between training and evaluation data.

For example:

Scaling before splitting

can cause train-test contamination.

Whereas:

Using account_closed_date to predict customer churn

is target/temporal leakage even if the train-test split itself was performed correctly.

31. Practical Leakage Checklist

Before considering your model ready, verify the following:

• The target variable is clearly defined.

• The exact prediction time is defined.

• Every feature would exist at prediction time.

• No feature directly reveals the target.

• The test set was isolated before learning preprocessing parameters.

• Scaling was fitted only on training data.

• Imputation was fitted only on training data.

• Feature selection was performed only within training data.

• PCA or other dimensionality reduction was fitted only on training data.

• SMOTE or other resampling was applied only to training data.

• Cross-validation preprocessing is handled through pipelines.

• Duplicate observations have been investigated.

• Related entities do not improperly cross train/test boundaries.

• Time-series data preserves chronological order.

• Future information is not included in engineered features.

• The final test set has not been repeatedly used for model tuning.

32. Golden Rule of Data Leakage

A simple rule can prevent most leakage problems:

The model must never learn from information that would not be available at the exact moment a real-world prediction is made.

And for preprocessing:

Learn from training data; apply what was learned to validation and test data.

Think of the workflow as:

TRAIN
Learn preprocessing
Learn feature relationships
Learn model parameters

        ↓

TEST
Apply learned preprocessing
Make predictions
Measure performance

The arrow should move from training to testing.

Information should never flow backward from the test set into training.

Data leakage is one of the most important—and sometimes overlooked—problems in machine learning.

A sophisticated algorithm cannot compensate for a contaminated evaluation process.

You might build an advanced model using:

XGBoost
LightGBM
CatBoost
Random Forest
Neural Networks

and achieve:

Accuracy = 99%

But if future information, test-set statistics, target-derived variables, or duplicated entities have leaked into training, that number may have little real-world meaning.

Reliable machine learning therefore requires more than choosing the right algorithm.

It requires building the right data pipeline.

Always remember:

Split carefully. Fit preprocessing on training data. Protect validation and test data. Respect time. Inspect feature origins. Use pipelines whenever possible.

A trustworthy 85% model is far more valuable than a leaked 99% model.

Happy Learning!