Exploratory Data Analysis (EDA)

Exploratory Data Analysis is the most important step in any data project. Before building models or drawing conclusions, you need to understand your data — its structure, quality, patterns, and anomalies.

For sensor data, EDA typically answers questions like:

• Are there gaps or outliers in the sensor readings?

• What are the daily and seasonal patterns?

• How do different variables relate to each other (e.g., outdoor temperature vs. heating energy)?

• Has something changed over time (e.g., HVAC system replacement, occupancy changes)?

This chapter walks through a systematic EDA workflow using a real temperature and humidity dataset from four flats.

EDA Workflow Overview

The first four steps are shown here; the deeper analyses each have their own recipe in Part III, linked in the checklist at the end.

Step 1

Load & Inspect

shape, info(), describe()

Step 1: Load and Inspect

Step 2

Data Quality

Missing values, duplicates, gaps

Step 2: Data Quality

Step 3

Time Series Overview

Plot raw data, zoom in

Step 3: Time Series Overview

Step 4

Distribution Analysis

Histograms

Step 4: Distribution Analysis

Step 1: Load and Inspect

Always start by loading the data, parsing dates, and getting a first overview. If you are unfamiliar with loading data in pandas, see the Loading Data chapter first.

import pandas as pd
import numpy as np

url = "https://raw.githubusercontent.com/retomarek/edap/main/edap/sampleData/flatTempHum.csv"

df = pd.read_csv(url, sep=";")
df["time"] = pd.to_datetime(df["time"])
df = df.set_index("time")

print(f"Shape: {df.shape}")
print(f"Period: {df.index.min()} to {df.index.max()}")
print(f"Duration: {df.index.max() - df.index.min()}")
df.head()

Shape: (17181, 8)
Period: 2018-10-03 00:00:00 to 2020-09-17 20:00:00
Duration: 715 days 20:00:00

	FlatA_Hum	FlatA_Temp	FlatB_Hum	FlatB_Temp	FlatC_Hum	FlatC_Temp	FlatD_Hum	FlatD_Temp
time
2018-10-03 00:00:00	53.0	24.43	38.8	22.40	44.0	24.5	49.0	24.43
2018-10-03 01:00:00	53.0	24.40	38.8	22.40	44.0	24.5	49.0	24.40
2018-10-03 02:00:00	53.0	24.40	39.3	22.40	44.7	24.5	48.3	24.38
2018-10-03 03:00:00	53.0	24.40	40.3	22.40	45.0	24.5	48.0	24.33
2018-10-03 04:00:00	53.3	24.40	41.0	22.37	45.2	24.5	47.7	24.30

df.describe()

	FlatA_Hum	FlatA_Temp	FlatB_Hum	FlatB_Temp	FlatC_Hum	FlatC_Temp	FlatD_Hum	FlatD_Temp
count	16484.000000	16484.000000	16470.000000	16470.000000	16497.000000	16497.000000	16354.000000	16354.000000
mean	54.336599	23.786531	49.666697	23.404749	48.895339	23.964698	49.299156	24.134841
std	6.300389	1.273444	6.903822	1.470354	6.487768	1.360014	6.524641	1.308097
min	23.500000	19.100000	25.000000	20.530000	22.500000	21.380000	20.500000	18.120000
25%	51.000000	22.800000	44.700000	22.320000	44.300000	22.880000	44.300000	23.030000
50%	54.800000	23.480000	50.000000	23.000000	49.200000	23.600000	49.000000	23.700000
75%	58.300000	24.600000	54.500000	24.100000	53.800000	24.800000	54.000000	25.200000
max	76.500000	28.880000	73.300000	29.120000	69.800000	28.800000	69.800000	28.030000

What to look for in describe()

• count differs between columns? → some columns have more missing values

• min/max outside physically plausible range? → possible sensor error (e.g., temperature of -999 or 99)

• mean vs. 50% (median) very different? → skewed data, possibly caused by outliers

• std very large relative to mean? → high variability or mixed data

Step 2: Data Quality

Real-world sensor data almost always has quality issues: missing values, duplicate timestamps, or irregular intervals. Identifying these early prevents wrong conclusions later.

# Missing values per column
missing = pd.DataFrame({
    "missing": df.isna().sum(),
    "% missing": (df.isna().mean() * 100).round(1)
})
missing

	missing	% missing
FlatA_Hum	697	4.1
FlatA_Temp	697	4.1
FlatB_Hum	711	4.1
FlatB_Temp	711	4.1
FlatC_Hum	684	4.0
FlatC_Temp	684	4.0
FlatD_Hum	827	4.8
FlatD_Temp	827	4.8

from pyedautils.data_quality import plot_missing_values_heatmap

fig = plot_missing_values_heatmap(df)
fig.show()

Interpreting the missing values heatmap

Red bands indicate periods where sensors reported no data. Common causes:

• Vertical red band across all sensors → system-wide outage (network, gateway, power)

• Single sensor red band → individual sensor failure or battery empty

• Red at the end → sensors stopped reporting (decommissioned or data export cutoff)

# Check for duplicate timestamps and irregular intervals
print(f"Duplicate timestamps: {df.index.duplicated().sum()}")
print(f"Index is monotonic:  {df.index.is_monotonic_increasing}")

# Most common interval
time_diffs = df.index.to_series().diff().dropna()
print(f"Most common interval: {time_diffs.mode()[0]}")
print(f"Min interval: {time_diffs.min()}")
print(f"Max interval: {time_diffs.max()}")

Duplicate timestamps: 0
Index is monotonic:  True
Most common interval: 0 days 01:00:00
Min interval: 0 days 01:00:00
Max interval: 0 days 01:00:00

Step 3: Time Series Overview

Plot the raw data to get a visual overview. Interactive plots allow zooming into specific periods.

from pyedautils.plots import plot_timeseries

fig = plot_timeseries(df, columns=["FlatA_Temp"], title="FlatA Temperature", ylab="Temperature [°C]")
fig.show()

# Column list reused in later cells
temp_cols = [col for col in df.columns if "Temp" in col]

fig = plot_timeseries(df, columns=temp_cols, title="All Flats — Temperature", ylab="Temperature [°C]")
fig.show()

Step 4: Distribution Analysis

A histogram reveals the distribution of values — how often each temperature occurs and whether there are unusual clusters. Box plots per month or hour show how the distribution changes over time (see Boxplot).

from pyedautils.plots import plot_distribution

fig = plot_distribution(df, columns=temp_cols, title="Temperature Distribution by Flat", xlab="Temperature [°C]")
fig.show()

What distribution patterns tell you

• Two peaks (bimodal) → different operating modes (e.g., heating season and summer, occupied/unoccupied)

• Long tail on one side → occasional events (window openings, sun) or sensor problems

• One flat with a very narrow distribution → heavily controlled room or a stuck sensor

Steps 5 to 8 (outliers, correlation, decomposition, autocorrelation) each have their own recipe in Part III; the checklist below links them.

EDA Checklist

Use this checklist as a starting point for every new dataset:

Step	Question	Where in this book
1. Inspect	How big? What columns? What types?	shape, dtypes, info(), describe() (above)
2. Quality	Missing values? Duplicates? Gaps?	Data Quality
3. Visualize	What does the raw data look like?	plot_timeseries() (above)
4. Distribute	How are values distributed?	Histogram, Boxplot
5. Outliers	Are there suspicious values?	Outlier Detection
6. Correlate	How do variables relate?	Correlation & Scatter, Cross-Correlation
7. Decompose	What are the trend and seasonal patterns?	Time Series Decomposition
8. Cycles	What are the dominant cycles?	Autocorrelation