data_quality#
Energy time series often have gaps caused by sensor outages or communication
failures. The data_quality module provides functions to detect, fill,
and visualize these gaps.
This example uses hourly indoor temperature data bundled with the package.
import plotly.io as pio
pio.renderers.default = "notebook_connected"
Load sample data#
The bundled flat_temp.csv contains hourly indoor temperature readings
with real gaps from sensor outages.
import pandas as pd
from importlib import resources
# Load the bundled sample data
data_path = resources.files("pyedautils") / "data" / "flat_temp.csv"
df = pd.read_csv(data_path, sep=";", index_col="timestamp", parse_dates=True)
df.columns = ["temperature"]
print(f"Rows: {len(df)}, Date range: {df.index[0]} – {df.index[-1]}")
df.head()
Rows: 16484, Date range: 2018-10-03 00:00:00 – 2020-09-15 21:00:00
| temperature | |
|---|---|
| timestamp | |
| 2018-10-03 00:00:00 | 24.43 |
| 2018-10-03 01:00:00 | 24.40 |
| 2018-10-03 02:00:00 | 24.40 |
| 2018-10-03 03:00:00 | 24.40 |
| 2018-10-03 04:00:00 | 24.40 |
Detect gap durations#
calc_gap_duration computes the time difference between consecutive rows
and a rolling median for comparison.
from pyedautils.data_quality import calc_gap_duration
gaps = calc_gap_duration(df)
gaps[gaps["gapDuration"] > 60].head()
| gapDuration | gapDurationRollMedian | |
|---|---|---|
| timestamp | ||
| 2018-10-03 01:00:00 | 3600.0 | 3600.0 |
| 2018-10-03 02:00:00 | 3600.0 | 3600.0 |
| 2018-10-03 03:00:00 | 3600.0 | 3600.0 |
| 2018-10-03 04:00:00 | 3600.0 | 3600.0 |
| 2018-10-03 05:00:00 | 3600.0 | 3600.0 |
Histogram of gap durations#
A log-scaled histogram of the rolling median gap duration reveals the dominant sampling rate and any outlier gaps.
import plotly.express as px
fig = px.histogram(
gaps,
x="gapDuration",
nbins=100,
log_y=True,
labels={"gapDuration": "Gap duration [s]", "count": "Count"},
title="<b>Histogram of gap durations</b>",
)
fig.update_layout(
template="plotly_white",
title_x=0.5,
yaxis_title="Count (log scale)",
)
fig.show()
Fill gaps with NaN#
fill_missing_values_with_na inserts NaN rows at the expected timestamps
where data is missing.
from pyedautils.data_quality import fill_missing_values_with_na
df_filled = fill_missing_values_with_na(df)
print(f"Before fill: {len(df)} rows, After fill: {len(df_filled)} rows")
print(f"NaN rows added: {df_filled['temperature'].isna().sum()}")
Before fill: 16484 rows, After fill: 17134 rows
NaN rows added: 650
Check NaN percentage#
calc_isna_percentage gives a quick summary of data completeness.
from pyedautils.data_quality import calc_isna_percentage
pct = calc_isna_percentage(df_filled, column="temperature")
print(f"Missing: {pct}%")
Missing: 3.794%
Visualize missing values#
plot_missing_values creates an interactive Plotly chart with red bands
highlighting the NaN regions.
from pyedautils.data_quality import plot_missing_values
fig = plot_missing_values(df_filled, column="temperature", ylab="Temperature [°C]")
fig.show()
Detect gaps explicitly#
detect_gaps returns an explicit table of gap intervals (start, end, duration). The sampling interval is inferred with infer_interval.
from pyedautils.data_quality import infer_interval, detect_gaps
interval = infer_interval(df.index.to_series())
print("Inferred interval:", interval)
detect_gaps(df, interval).head()
Inferred interval: 0 days 01:00:00
| gap_start | gap_end | gap_duration_h | |
|---|---|---|---|
| 0 | 2018-10-20 00:00:00 | 2018-10-20 03:00:00 | 3.0 |
| 1 | 2018-10-22 22:00:00 | 2018-10-23 01:00:00 | 3.0 |
| 2 | 2018-10-23 12:00:00 | 2018-10-23 18:00:00 | 6.0 |
| 3 | 2018-11-19 04:00:00 | 2018-11-19 07:00:00 | 3.0 |
| 4 | 2018-12-02 22:00:00 | 2018-12-03 04:00:00 | 6.0 |
Detect stuck values#
detect_stuck flags periods where the sensor repeatedly reports the same value (run-length encoding).
from pyedautils.data_quality import detect_stuck
detect_stuck(df, min_repeats=10, min_duration_h=3.0).head()
| stuck_start | stuck_end | stuck_value | stuck_duration_h | n_repeats | |
|---|---|---|---|---|---|
| 0 | 2018-12-23 07:00:00 | 2018-12-23 17:00:00 | 22.9 | 10.0 | 11 |
| 1 | 2018-12-24 14:00:00 | 2018-12-24 23:00:00 | 22.8 | 9.0 | 10 |
| 2 | 2019-01-03 13:00:00 | 2019-01-03 23:00:00 | 23.3 | 10.0 | 11 |
| 3 | 2019-04-21 03:00:00 | 2019-04-21 12:00:00 | 23.0 | 9.0 | 10 |
| 4 | 2019-07-08 02:00:00 | 2019-07-08 11:00:00 | 26.5 | 9.0 | 10 |
Quality flags#
classify_quality_flags turns a per-sensor quality summary into an ok / warning / critical flag using configurable thresholds.
from pyedautils.data_quality import classify_quality_flags
summary = pd.DataFrame({
"coverage_pct": [99, 85, 60],
"longest_gap_h": [1, 30, 200],
"outlier_pct": [0, 2, 10],
"n_stuck_periods": [0, 2, 9],
})
summary["quality_flag"] = classify_quality_flags(summary)
summary
| coverage_pct | longest_gap_h | outlier_pct | n_stuck_periods | quality_flag | |
|---|---|---|---|---|---|
| 0 | 99 | 1 | 0 | 0 | ok |
| 1 | 85 | 30 | 2 | 2 | warning |
| 2 | 60 | 200 | 10 | 9 | critical |
Time series with gaps shaded#
plot_data_quality plots the series and shades the detected gaps.
from pyedautils.data_quality import plot_data_quality
fig = plot_data_quality(df, ylab="Temperature [°C]")
fig.show()