Outlier Detection

Outlier Detection#

Goal

Detect and visualize outliers in sensor data using the Interquartile Range (IQR) method. Outliers can indicate sensor errors, unusual events (e.g. window openings), or data quality issues.

Data Basis

Dataset: flatTempHum.csv — Hourly indoor temperature and humidity measurements from four residential flats (A–D).
Column used: FlatA_Temp
Time range: October 2018 – September 2020

Data Preparation#

import pandas as pd

url = "https://raw.githubusercontent.com/retomarek/edap/main/edap/sampleData/flatTempHum.csv"
df = pd.read_csv(url, sep=";")
df["time"] = pd.to_datetime(df["time"])
df = df.set_index("time", drop=True)
df.head()

	FlatA_Hum	FlatA_Temp	FlatB_Hum	FlatB_Temp	FlatC_Hum	FlatC_Temp	FlatD_Hum	FlatD_Temp
time
2018-10-03 00:00:00	53.0	24.43	38.8	22.40	44.0	24.5	49.0	24.43
2018-10-03 01:00:00	53.0	24.40	38.8	22.40	44.0	24.5	49.0	24.40
2018-10-03 02:00:00	53.0	24.40	39.3	22.40	44.7	24.5	48.3	24.38
2018-10-03 03:00:00	53.0	24.40	40.3	22.40	45.0	24.5	48.0	24.33
2018-10-03 04:00:00	53.3	24.40	41.0	22.37	45.2	24.5	47.7	24.30

Outlier Calculation#

The IQR method flags values outside [Q1 − k×IQR, Q3 + k×IQR] as outliers, where k is typically 1.5.

from pyedautils.data_quality import calc_outliers

result = calc_outliers(df, column="FlatA_Temp", multiplier=1.5)

print(f"IQR fences: [{result['lower']:.1f}, {result['upper']:.1f}] °C")
print(f"Outliers:   {result['count']} of {len(df)} ({result['percentage']}%)")

IQR fences: [20.1, 27.3] °C
Outliers:   148 of 17181 (0.86%)

Visualization#

from pyedautils.plots import plot_outliers

fig = plot_outliers(
    df,
    column="FlatA_Temp",
    title="FlatA Temperature — Outlier Detection (IQR)",
    ylab="Temperature [°C]",
)
fig.show()

Adjusting Sensitivity#

The multiplier parameter controls how aggressive the detection is. A lower value flags more outliers.

for k in [1.0, 1.5, 2.0, 3.0]:
    r = calc_outliers(df, column="FlatA_Temp", multiplier=k)
    print(f"k={k:.1f}  →  fences [{r['lower']:.1f}, {r['upper']:.1f}]  →  {r['count']:>4d} outliers ({r['percentage']}%)")

k=1.0  →  fences [21.0, 26.4]  →   639 outliers (3.72%)
k=1.5  →  fences [20.1, 27.3]  →   148 outliers (0.86%)
k=2.0  →  fences [19.2, 28.2]  →     5 outliers (0.03%)
k=3.0  →  fences [17.4, 30.0]  →     0 outliers (0.0%)

fig = plot_outliers(
    df,
    column="FlatA_Temp",
    multiplier=3.0,
    title="FlatA Temperature — Outlier Detection (IQR × 3.0)",
    ylab="Temperature [°C]",
)
fig.show()