data_quality

data_quality#

Detect and visualize data-quality issues in time series: gaps (missing values), stuck values and range outliers, interval inference, ok/warning/critical quality flags, and missing-value plots.

See data_quality for usage examples.

API Reference#

Functions for detecting and visualizing gaps in time series data.

pyedautils.data_quality.calc_gap_duration(df: DataFrame, window: int = 20) → DataFrame[source]#

Calculate the time difference between consecutive index entries.

Computes the gap duration in seconds between rows using the DatetimeIndex, plus a rolling median for comparison.

Parameters:

  • df – DataFrame with a DatetimeIndex.

  • window – Rolling median window size. Default 20.

Returns:

DataFrame with columns gapDuration (seconds between rows) and gapDurationRollMedian (rolling median of gap durations).

pyedautils.data_quality.fill_missing_values_with_na(df: DataFrame, window: int = 20) → DataFrame[source]#

Detect gaps in a time series and fill them with NaN rows.

A gap is detected when the time difference between consecutive rows exceeds the rolling median gap duration. For each detected gap, rows are inserted at the expected timestamps with NaN values.

Parameters:

  • df – DataFrame with a DatetimeIndex.

  • window – Rolling median window size passed to calc_gap_duration(). Default 20.

Returns:

DataFrame with original data plus NaN-filled rows at missing timestamps. The result is sorted by index.

pyedautils.data_quality.calc_isna_percentage(df: DataFrame, column: str | None = None, decimals: int = 3) → float[source]#

Calculate the percentage of NaN values in a DataFrame or column.

Parameters:

  • df – DataFrame to check.

  • column – If given, only check this column. Otherwise check the entire DataFrame.

  • decimals – Number of decimal places for rounding. Default 3.

Returns:

Percentage of NaN values (0–100).

pyedautils.data_quality.plot_missing_values(df: DataFrame, column: str | None = None, title: str | None = None, xlab: str = 'Time', ylab: str = 'Value', missing_color: str = 'rgba(255,0,0,0.2)', line_color: str = 'green') → Figure[source]#

Plot a time series and highlight regions with missing (NaN) values.

Creates a Plotly step plot of the data with red vertical rectangles marking NaN regions.

Parameters:

  • df – DataFrame with a DatetimeIndex.

  • column – Column to plot. If None, uses the first column.

  • title – Plot title. Auto-generated with NaN percentage if None.

  • xlab – X-axis label. Default "Time".

  • ylab – Y-axis label. Default "Value".

  • missing_color – Fill color for NaN regions. Default "rgba(255,0,0,0.2)".

  • line_color – Line color for the data trace. Default "green".

Returns:

Plotly figure with the time series and NaN highlights.

Return type:

go.Figure

pyedautils.data_quality.plot_missing_values_heatmap(df: DataFrame, title: str = 'Missing Values Over Time', height: int = 300, color_scale: list | None = None) → Figure[source]#

Show a heatmap of missing values across all columns.

Each row represents a column, each position along the x-axis a timestamp. Red cells indicate missing (NaN) values, white cells indicate present values.

Parameters:

  • df – DataFrame with a DatetimeIndex and one or more value columns.

  • title – Plot title.

  • height – Figure height in pixels. Default 300.

  • color_scale – Two-element color scale list. Default ["white", "red"].

Returns:

Plotly heatmap figure.

Return type:

go.Figure

pyedautils.data_quality.calc_outliers(df: DataFrame, column: str | None = None, multiplier: float = 1.5) → Dict[source]#

Detect outliers using the IQR method.

Computes the interquartile range and flags values outside [Q1 - multiplier*IQR, Q3 + multiplier*IQR].

Parameters:

  • df – DataFrame with a DatetimeIndex.

  • column – Column to analyse. If None, uses the first column.

  • multiplier – IQR multiplier for the fence. Default 1.5.

Returns:

  • lower (float): Lower fence value.

  • upper (float): Upper fence value.

  • outliers (pd.DataFrame): Rows flagged as outliers.

  • count (int): Number of outliers.

  • percentage (float): Percentage of outliers.

Return type:

Dict with keys

pyedautils.data_quality.infer_interval(timestamps: Series) → Timedelta[source]#

Estimate a sensor’s sampling interval from the median time difference.

Parameters:

timestamps – Series (or DatetimeIndex) of timestamps.

Returns:

Median positive interval; falls back to 10 minutes if there are fewer than two usable timestamps.

Return type:

pd.Timedelta

pyedautils.data_quality.detect_gaps(data: Series | DataFrame, expected_interval: Timedelta | None = None, factor: float = 2.0, min_floor: Timedelta = Timedelta('0 days 00:30:00')) → DataFrame[source]#

Find gaps (interruptions) in a single sensor time series.

A gap is an interval larger than factor × expected_interval but at least min_floor. Unlike calc_gap_duration()/ fill_missing_values_with_na() (rolling-median based, NaN-filling), this returns an explicit table of gap intervals.

Parameters:

  • data – Series (DatetimeIndex), or DataFrame with a timestamp column or DatetimeIndex.

  • expected_interval – Expected sampling interval. Inferred via infer_interval() when None.

  • factor – Multiplier on the expected interval. Default 2.0.

  • min_floor – Minimum gap length to report. Default 30 minutes.

Returns:

Columns [gap_start, gap_end, gap_duration_h].

Return type:

pd.DataFrame

pyedautils.data_quality.detect_stuck(data: Series | DataFrame, min_repeats: int = 20, min_duration_h: float = 6.0, column: str | None = None) → DataFrame[source]#

Find periods where a sensor repeatedly reports the same value.

Uses run-length encoding: consecutive identical values form a run, and runs that are both long enough (n_repeats >= min_repeats) and last long enough (stuck_duration_h >= min_duration_h) are flagged.

Parameters:

  • data – Series (DatetimeIndex), or DataFrame with a timestamp column / DatetimeIndex plus a value column.

  • min_repeats – Minimum number of identical consecutive samples. Default 20.

  • min_duration_h – Minimum duration of the run in hours. Default 6.0.

  • column – Value column name (DataFrame input only); first non- timestamp column by default.

Returns:

Columns [stuck_start, stuck_end, stuck_value, stuck_duration_h, n_repeats].

Return type:

pd.DataFrame

pyedautils.data_quality.detect_outliers(data: Series | DataFrame, lo: float, hi: float, column: str | None = None) → DataFrame[source]#

Find values outside a plausibility range [lo, hi].

Complements the IQR-based calc_outliers() with a fixed, physically motivated range (e.g. relative humidity 0–100 %).

Parameters:

  • data – Series (DatetimeIndex), or DataFrame with a timestamp column / DatetimeIndex plus a value column.

  • lo – Lower plausibility bound (inclusive).

  • hi – Upper plausibility bound (inclusive).

  • column – Value column name (DataFrame input only).

Returns:

Columns [timestamp, value, reason] sorted by timestamp; empty if there are no out-of-range values.

Return type:

pd.DataFrame

pyedautils.data_quality.DEFAULT_QUALITY_THRESHOLDS: Dict[str, float] = {'cov_crit': 70.0, 'cov_warn': 90.0, 'gap_crit': 168.0, 'gap_warn': 24.0, 'out_crit': 5.0, 'out_warn': 1.0, 'stuck_crit': 5.0, 'stuck_warn': 1.0}#

Default thresholds for classify_quality_flags().

pyedautils.data_quality.classify_quality_flags(summary: DataFrame, thresholds: Dict[str, float] | None = None) → Series[source]#

Classify each row of a quality summary as ok / warning / critical.

A row is critical if any metric crosses its critical threshold, else warning if any crosses its warning threshold, else ok.

Parameters:

  • summary – DataFrame with (any subset of) columns coverage_pct, longest_gap_h, outlier_pct, n_stuck_periods.

  • thresholds – Overrides merged onto DEFAULT_QUALITY_THRESHOLDS.

Returns:

"ok" / "warning" / "critical" per row, named quality_flag.

Return type:

pd.Series

pyedautils.data_quality.plot_data_quality(data: Series | DataFrame, expected_interval: Timedelta | None = None, column: str | None = None, title: str = 'Data Quality', ylab: str = 'Value', line_color: str = '#0D7377', gap_color: str = 'rgba(239,68,68,0.25)', height: int = 300) → Figure[source]#

Plot a sensor series with detected gaps shaded.

Draws the time series as points (WebGL) with red background rectangles over the gaps found by detect_gaps().

Parameters:

  • data – Series (DatetimeIndex), or DataFrame with a timestamp column / DatetimeIndex plus a value column.

  • expected_interval – Passed to detect_gaps(); inferred if None.

  • column – Value column name (DataFrame input only).

  • title – Plot title.

  • ylab – Y-axis label.

  • line_color – Marker/line color for the data trace.

  • gap_color – Fill color for gap rectangles.

  • height – Figure height in pixels. Default 300.

Returns:

go.Figure