Powerful Pandas Techniques to Take Basic EDA

Powerful Pandas Techniques to Take Basic EDA, Exploratory Data Analysis (EDA) is often treated as the first step before statistical modeling or machine learning. But a basic describe() output, a few histograms, and a correlation matrix rarely reveal the deeper relationships hidden inside a large dataset.

You don’t always need expensive enterprise analytics software to perform more sophisticated exploratory analysis. Python’s Pandas ecosystem provides several powerful tools that can uncover relationships between categorical variables, compare multiple dimensions, and automatically audit an entire dataset.

In this article, we’ll explore three practical techniques that can take a Pandas-based EDA workflow to the next level:

• pd.crosstab() for analyzing categorical relationships and conditional distributions
• pd.pivot_table() for multidimensional aggregation
• ProfileReport from ydata-profiling for automated dataset auditing

We’ll use the California Housing dataset to demonstrate each technique with executable Python examples.

1. Use pd.crosstab() to Discover Relationships Between Categorical Variables

A cross-tabulation, commonly called a crosstab, summarizes the relationship between two or more categorical variables.

Instead of simply counting observations, Pandas can normalize a crosstab to calculate percentages. This makes it possible to examine the distribution of one categorical variable conditional on another.

In practical terms, a normalized crosstab can answer questions such as:

• What percentage of inland properties fall into each price category?
• Which locations contain the largest proportion of high-value properties?
• How does customer behavior vary between demographic groups?
• What proportion of transactions in each region belong to different risk categories?

This makes pd.crosstab() particularly useful for categorical EDA.

Create price categories from a continuous variable

The California Housing dataset contains median_house_value as a continuous variable. We can convert it into three approximately equal-sized groups using pd.qcut().

import pandas as pd

# Load the California Housing dataset
url = "https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv"
df_housing = pd.read_csv(url)

# Divide house values into three approximately equal-sized groups
df_housing["value_tier"] = pd.qcut(
    df_housing["median_house_value"],
    q=3,
    labels=["Low", "Medium", "High"]
)

# Create a percentage-based cross-tabulation
crosstab_pct = pd.crosstab(
    df_housing["ocean_proximity"],
    df_housing["value_tier"],
    normalize="index"
) * 100

print(crosstab_pct.round(2))

The normalize="index" argument is important here.

It calculates the percentage distribution within each ocean-proximity category, rather than calculating percentages across the entire dataset.

The result can therefore be interpreted as:

Given a particular ocean-proximity category, what percentage of properties belong to each price tier?

This is closely related to conditional probability.

Turn the crosstab into a heatmap

A heatmap makes these differences much easier to identify visually.

import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(10, 6))

sns.heatmap(
    crosstab_pct,
    annot=True,
    fmt=".2f",
    cmap="YlGnBu"
)

plt.title("Ocean Proximity vs. House Value Tier")
plt.xlabel("Value Tier")
plt.ylabel("Ocean Proximity")

plt.tight_layout()
plt.show()

The resulting visualization allows you to quickly compare the distribution of low-, medium-, and high-value properties across geographical categories.

For example, the analysis can reveal whether inland properties have a different price distribution from properties near the ocean or bay.

The same technique can be applied to many business datasets.

A marketing analyst could cross-tabulate customer segment vs. conversion category. A fraud analyst could compare transaction region vs. fraud status. A SaaS company could examine subscription plan vs. churn category.

The important point is that pd.crosstab() converts categorical relationships into a format that is easy to quantify and visualize.

2. Use pd.pivot_table() for Multidimensional Analysis

A crosstab is excellent for categorical frequency analysis, but what happens when you want to aggregate a numerical variable across multiple dimensions?

This is where Pandas pivot tables become extremely useful.

pd.pivot_table() lets you summarize numerical data according to combinations of categorical variables.

For example, suppose you want to compare median house values according to:

• Ocean proximity
• Property age

Rather than analyzing each variable separately, a pivot table allows both dimensions to be examined simultaneously.

Create an age category

The original dataset contains housing_median_age, which is numerical. We can divide properties into two groups using pd.qcut().

import pandas as pd

# Load the dataset
url = "https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv"
df_housing = pd.read_csv(url)

# Divide properties into two age groups
df_housing["age_tier"] = pd.qcut(
    df_housing["housing_median_age"],
    q=2,
    labels=["Newer Homes", "Older Homes"]
)

# Create a multidimensional pivot table
pivot = pd.pivot_table(
    df_housing,
    values="median_house_value",
    index="ocean_proximity",
    columns="age_tier",
    aggfunc="median",
    margins=True
)

print(pivot.round())

The resulting table summarizes the median house value for every combination of ocean proximity and property-age category.

A simplified structure looks like this:

The index parameter determines the row dimension, while columns defines the column dimension.

Here:

index="ocean_proximity"
columns="age_tier"

creates a two-dimensional analytical structure.

Meanwhile:

values="median_house_value"
aggfunc="median"

tells Pandas to calculate the median house value for each combination.

The margins=True argument adds the All row and column, giving you overall summaries alongside the individual groups.

Use multiple aggregation functions

You aren’t restricted to a single statistic.

For example:

pivot = pd.pivot_table(
    df_housing,
    values="median_house_value",
    index="ocean_proximity",
    columns="age_tier",
    aggfunc=["mean", "median"]
)

print(pivot.round())

This produces separate sections for the mean and median.

You can also use functions such as:

aggfunc=["mean", "median", "min", "max"]

This makes pivot tables particularly valuable when investigating distributions across multiple business dimensions.

For example, an e-commerce analyst could build a table containing:

Product Category × Customer Segment → Revenue

A financial analyst could examine:

Region × Risk Category → Average Loss

A SaaS analyst could investigate:

Subscription Plan × Customer Segment → Average Monthly Revenue

The underlying concept remains the same: use categorical dimensions to organize and summarize numerical measurements.

3. Automate EDA with ydata-profiling

Manually creating dozens of plots and summary tables isn’t always practical, particularly during the initial investigation of an unfamiliar dataset.

For this situation, ydata-profiling provides automated exploratory data analysis through ProfileReport.

It can generate an interactive HTML report containing information such as:

• Dataset dimensions
• Variable types
• Missing values
• Unique values
• Descriptive statistics
• Distributions
• Correlations
• Duplicate observations
• Potential data-quality problems
• Statistical warnings and alerts

This can be particularly useful during the data-audit stage of a machine learning or analytics project.

Install the package

Install the library with:

pip install ydata-profiling

Then generate a profile report:

import pandas as pd
from ydata_profiling import ProfileReport

# Load the California Housing dataset
url = "https://raw.githubusercontent.com/gakudo-ai/open-datasets/main/housing.csv"
df_housing = pd.read_csv(url)

# Generate an automated profiling report
profile = ProfileReport(
    df_housing,
    title="California Housing EDA Report",
    minimal=True
)

# Save the report as an interactive HTML file
profile.to_file("housing_eda_report.html")

After execution, the HTML report can be opened in a web browser.

The minimal=True option can reduce computation for larger datasets by limiting some of the more expensive analyses.

For smaller datasets, you can generate a more comprehensive report:

profile = ProfileReport(
    df_housing,
    title="California Housing EDA Report"
)

profile.to_file("housing_eda_report.html")

This approach can save considerable time during the first stage of a data science project.

Instead of manually writing separate code for dozens of descriptive statistics and visualizations, you can generate a structured overview and then focus your analysis on the areas that require deeper investigation.

How These Three Pandas Techniques Complement Each Other

These tools solve different parts of the EDA problem.

A practical EDA workflow could therefore look like this:

Step 1 — Profile the dataset

Start with ProfileReport to understand the overall structure, identify missing values, inspect distributions, and detect potential problems.

Step 2 — Investigate categorical relationships

Use pd.crosstab() when you discover interesting categorical variables that may be associated with the target variable.

Step 3 — Perform multidimensional aggregation

Use pd.pivot_table() to investigate numerical outcomes across combinations of business dimensions.

Step 4 — Visualize important relationships

Convert important crosstabs and pivot tables into heatmaps, bar charts, or other visualizations where appropriate.

Step 5 — Validate the findings statistically

EDA is designed to generate hypotheses and identify patterns. Important relationships should be investigated further using appropriate statistical tests, domain knowledge, and, where relevant, predictive modeling.

Why These Techniques Matter for Data Science Projects

The biggest advantage of these methods isn’t simply that they produce more tables.

They encourage a more structured way of thinking about data.

A basic EDA workflow might tell you that a dataset contains 20,000 observations and that one variable has missing values.

A multidimensional workflow can go further and ask:

• Which groups contain those missing values?
• Does the target variable behave differently across categories?
• Are certain combinations of features associated with unusually high or low outcomes?
• Are apparent relationships driven by a particular subgroup?
• Does the distribution change across geographic, demographic, temporal, or business segments?

These questions are often more valuable than a collection of isolated summary statistics.

For machine learning projects, this type of analysis can also help identify potential feature-engineering opportunities, data-quality issues, class imbalance, and relationships that deserve further investigation before model development.

Important Considerations Before Automating EDA

Automated profiling is useful, but it shouldn’t replace analytical judgment.

A profiling report can identify a correlation, but correlation alone does not establish causation.

Similarly, a heatmap can reveal a strong difference between two groups, but you may still need statistical testing to determine whether the observed difference is meaningful.

Large datasets can also make automated profiling computationally expensive. In those situations, sampling, minimal=True, or targeted analysis may be more appropriate.

Finally, always consider the business or scientific context. A statistically interesting relationship isn’t necessarily operationally useful.

The strongest EDA workflows combine automated tools with domain knowledge and targeted statistical analysis.

Conclusion

Effective exploratory data analysis doesn’t require expensive analytics platforms. The Pandas ecosystem already provides powerful capabilities for investigating complex datasets.

pd.crosstab() is particularly useful for understanding categorical relationships and conditional distributions. pd.pivot_table() makes it easy to analyze numerical measures across multiple dimensions. And ydata-profiling can automate much of the initial data-quality and exploratory analysis process.

Used together, these techniques provide a practical foundation for moving from basic dataset inspection to deeper, multidimensional analysis.