Exploratory Data Analysis Beyond the Basics

Exploratory Data Analysis Beyond the Basics, often treated as a routine step before building a machine learning model. Load the dataset, check for missing values, calculate a few statistics, draw some charts, and move on.

But good EDA can do much more than that.

It can reveal relationships that are easy to miss in a conventional summary, expose differences between customer or business segments, identify suspicious data patterns, and even highlight potential problems before a model ever reaches production.

You also do not necessarily need expensive enterprise analytics software to get there. With Python, Pandas, and a few open-source libraries, you can build surprisingly powerful exploratory workflows.

Three particularly useful techniques are:

• pd.crosstab()
• pd.pivot_table()
• ProfileReport

Using the California housing dataset as an example, let’s see how each technique answers a different type of analytical question.

1. Use pd.crosstab() to Find Hidden Relationships Between Categories

A crosstab, or cross-tabulation, is one of the simplest ways to investigate how two categorical variables interact.

Imagine asking:

Are expensive homes concentrated in particular geographical areas?

A simple frequency table can answer part of the question. But converting the results into percentages makes the relationship much easier to interpret.

First, load the California housing dataset and convert the continuous median_house_value variable into three categories.

import pandas as pd

df_housing = pd.read_csv(
    "https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv"
)

df_housing["value_tier"] = pd.qcut(
    df_housing["median_house_value"],
    q=3,
    labels=["Low", "Medium", "High"]
)

crosstab_pct = pd.crosstab(
    df_housing["ocean_proximity"],
    df_housing["value_tier"],
    normalize="index"
) * 100

print(crosstab_pct.round(2))

The important part is normalize="index".

Instead of simply counting observations, Pandas calculates the percentage distribution of value tiers within each ocean_proximity category.

This makes the output much easier to compare.

For example, you can ask whether inland districts have a substantially different house-value distribution from areas near the ocean or bay.

You can also turn the table into a heatmap:

import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(10, 6))

sns.heatmap(
    crosstab_pct,
    annot=True,
    fmt=".2f",
    cmap="YlGnBu"
)

plt.title("Ocean Proximity vs. House Value Tier")
plt.xlabel("House Value Tier")
plt.ylabel("Ocean Proximity")

plt.tight_layout()
plt.show()

The heatmap provides an immediate visual summary of the categorical relationship.

This technique is useful well beyond housing data. In business analytics, you could use a crosstab to examine:

• Customer segment vs. churn status
• Marketing channel vs. conversion category
• Product category vs. return status
• Region vs. sales tier
• Credit risk category vs. loan default
• Subscription plan vs. cancellation status

The major advantage is speed. A relationship that might require several lines of custom analysis can often be explored with one pd.crosstab() call.

2. Use pd.pivot_table() for Multi-Dimensional Analysis

Crosstabs are excellent for categorical relationships, but business questions often involve more than two dimensions.

For example:

How does the median house value change across geographical location and house age?

This is where pd.pivot_table() becomes particularly useful.

Create an age category first:

import pandas as pd

df_housing = pd.read_csv(
    "https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv"
)

df_housing["age_tier"] = pd.qcut(
    df_housing["housing_median_age"],
    q=2,
    labels=["Newer Homes", "Older Homes"]
)

pivot = pd.pivot_table(
    df_housing,
    values="median_house_value",
    index="ocean_proximity",
    columns="age_tier",
    aggfunc="median",
    margins=True
)

print(pivot.round())

The resulting table gives you a matrix of median house values for combinations of geographical location and house-age category.

This is much more informative than looking at the overall median house value.

You might discover, for example, that the relationship between property age and value is not consistent across geographical locations.

That is an important analytical insight.

A variable that appears weakly related to price at the overall dataset level could behave very differently within individual segments.

Pivot tables are especially valuable in business intelligence and analytics because they allow analysts to perform segmentation without writing complicated SQL queries or manually grouping the dataset.

You can also calculate several statistics simultaneously:

pivot = pd.pivot_table(
    df_housing,
    values="median_house_value",
    index="ocean_proximity",
    columns="age_tier",
    aggfunc=["mean", "median"]
)

print(pivot.round())

Now you can compare both mean and median values.

That distinction matters because the mean can be influenced by extreme observations, while the median is generally more resistant to outliers.

3. Use ProfileReport() When You Want a Full EDA Audit

The first two techniques are targeted tools. You ask a specific question and build an analysis around it.

But what if you receive a completely unfamiliar dataset and want to understand its overall structure quickly?

This is where automated profiling can save substantial time.

The ydata-profiling package extends the Pandas workflow by automatically generating a detailed exploratory report.

Install it with:

pip install ydata-profiling

Then generate a report:

import pandas as pd
from ydata_profiling import ProfileReport

df_housing = pd.read_csv(
    "https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv"
)

profile = ProfileReport(
    df_housing,
    title="Housing Data Automated Profiling",
    minimal=True
)

profile.to_file("housing_eda_report.html")

Open the resulting HTML file in your browser and you get an interactive overview of the dataset.

Depending on the configuration and dataset, the report can provide information about:

• Missing values
• Variable distributions
• Data types
• Unique values
• Duplicate records
• Correlations
• Skewness
• Potentially problematic variables
• Statistical summaries
• Distribution visualizations

This is particularly useful during the initial data-audit stage.

Instead of manually inspecting dozens or hundreds of columns, you can generate an initial report and use it to decide where deeper investigation is required.

For very large datasets, the minimal=True option can reduce the computational workload. However, automated profiling should be treated as a starting point rather than a replacement for analytical judgment.

The Real Power Comes From Combining the Three

These three techniques solve different EDA problems.

pd.crosstab() is excellent when you want to understand categorical relationships.

pd.pivot_table() is better when you need multi-dimensional aggregation and segmentation.

ProfileReport() is useful when you want a broad automated audit of an unfamiliar dataset.

A practical workflow could therefore look like this:

Step 1: Start with ProfileReport() to understand the overall structure.

Step 2: Identify interesting categorical relationships and investigate them with pd.crosstab().

Step 3: Drill deeper into important segments using pd.pivot_table().

This creates a much more effective EDA process than simply generating a collection of charts.

Why These Techniques Matter for Real-World Data Science

The value of EDA is not the number of charts you produce.

The real value is discovering something that changes what you do next.

A customer analytics team might discover that churn is concentrated in one combination of customer segment and subscription plan.

A fintech company might find that default rates change dramatically across risk categories and geographic regions.

A SaaS company might discover that customers acquired through one channel have very different retention patterns depending on their plan type.

These are the kinds of insights that can influence pricing, marketing, product development, risk management, and machine learning strategy.

And the tools required to begin investigating these questions are remarkably accessible.

Final Takeaway

Pandas is much more than a library for loading CSV files and calculating averages.

With pd.crosstab(), you can quickly expose relationships between categorical variables. With pd.pivot_table(), you can explore multiple dimensions and compare meaningful segments. With ProfileReport(), you can automate much of the initial data-quality and exploratory analysis process.

Used together, they provide a practical EDA toolkit that can take you from “What is in this dataset?” to “What patterns should I investigate further?”

That is where exploratory data analysis becomes genuinely useful: not just describing the data, but helping you decide what deserves your attention next.