arraystats

arraystats#

Fast, NumPy-only statistics for large time series (millions of samples): descriptive statistics with coverage (describe()), sampling interval, gap and stuck-value detection, and outlier detection (IQR, z-score, Hampel filter). No pandas required; designed for interactive tools where every millisecond counts. The pandas-based functions in pyedautils.data_quality use these as their core.

import numpy as np
from pyedautils.arraystats import describe, outliers_hampel

ts = np.arange(525_600, dtype=np.int64) * 60_000      # one year, 1-min, epoch ms
values = 20 + 5 * np.sin(ts / 8.64e7 * 2 * np.pi)
stats = describe(values, ts)                          # n, coverage, min, max, mean, median, ...
mask = outliers_hampel(values, window=30)             # boolean outlier mask

API Reference#

Fast statistics for (large) time series on plain NumPy arrays.

All functions operate on plain NumPy arrays and avoid Python loops over data points, so they are suitable for interactive tools with millions of samples per series. Timestamps are numeric (e.g. int64 epoch milliseconds) or datetime64; intervals and durations are returned in the same unit as the timestamps (datetime64 input is treated as its underlying int64 ticks).

Missing values are represented as NaN and ignored by all statistics.

class pyedautils.arraystats.Description[source]#

Bases: TypedDict

Result of describe(). Statistics are nan if n == 0.

n: int#

n_missing: int#

coverage: float#

min: float#

max: float#

mean: float#

median: float#

std: float#

p05: float#

p95: float#

ts_min: int | None#

ts_max: int | None#

ts_first: int | None#

ts_last: int | None#

Descriptive statistics of a series, ignoring NaN values.

Parameters:

values – Sample values.
ts – Timestamps belonging to values (same length, sorted). Needed for coverage and the ts_* fields.
expected_interval – Expected sampling interval in timestamp units. Inferred via infer_interval() when None.
start – Start of the analysed range (inclusive) for coverage. Defaults to the first timestamp.
end – End of the analysed range (exclusive) for coverage. Defaults to the last timestamp plus one interval.
hold – One of HOLD_MODES: mean, std, median and the percentiles are then weighted by the time each sample stands for (hold_weights()), i.e. they describe the signal over time, not the samples. None: plain sample statistics.
max_hold – Cap on a hold, see hold_weights().
prior – For hold="step": the value holding at start from before the range (the sample before the first one). It weighs in from start to the first timestamp but is not counted in n, min, max.

Returns:

Description with n, missing count, coverage (0..1, nan without timestamps), min, max, mean, median, std (population), 5th/95th percentile and the timestamps of min, max, first and last valid sample.

pyedautils.arraystats.HOLD_MODES = ('step', 'step-back', 'linear')#

step holds until the next sample (setpoints, states, change-of-value data), step-back holds until its own timestamp (counter readings), linear is a continuous signal.

Type:: How a value is read between two samples

The time each sample stands for, as a weight per sample.

Parameters:

ts – Sorted timestamps.
hold – step: a sample holds until the next one, the last one until end (else one expected_interval). step-back: a sample holds from the previous one, the first one from start. linear: a sample stands for half the interval to each neighbour (trapezoid rule).
start – Bounds of the analysed range for the first / last hold.
end – Bounds of the analysed range for the first / last hold.
expected_interval – Fallback for the open end; inferred when None.
max_hold – Cap on a hold (a number or one cap per sample, inf = none): a value does not hold across an outage.

Returns:

Float weights, same length as ts (zeros for a single sample without bounds).

pyedautils.arraystats.weighted_stats(values: ndarray, weights: ndarray) → dict[source]#

Mean, population std, median and the 5th / 95th percentile of values weighted by weights (e.g. hold_weights()): the statistics of the signal over time rather than of the samples. Non-finite values and non-positive weights are ignored; all nan without any weight.

Percentiles are the inverse of the weighted cumulative distribution (the smallest value the signal is at or below for the share p of the weight): exact for a step signal, within one sample of numpy.percentile for equal weights.

pyedautils.arraystats.held_at(ts: ndarray, values: ndarray, at: ndarray, max_hold: None | float | ndarray = None) → ndarray[source]#: The value a step series holds at each time in at: the last sample at or before it, nan before the first sample or once the hold is capped by max_hold (a number or one cap per sample). Non-finite samples do not hold.

pyedautils.arraystats.step_bucket_mean(ts: ndarray, values: ndarray, edges: ndarray, max_hold: None | float | ndarray = None) → ndarray[source]#: Time-weighted mean of a step series (each sample holds until the next one) in the buckets [edges[i], edges[i + 1]): the integral of the held value over the time the bucket is covered, divided by that time; nan where nothing holds. max_hold caps a hold (a number or one cap per sample), the last sample holds until the last edge at most.

pyedautils.arraystats.bucket_reduce(index: ndarray, values: ndarray, n: int, fn: str) → ndarray[source]#: One value per bucket 0 .. n-1 from samples with their bucket index (sorted): mean, min, max, first or last of the finite samples in the bucket, nan for a bucket without one. Samples outside 0 .. n-1 are ignored.

pyedautils.arraystats.step_bucket_reduce(ts: ndarray, values: ndarray, edges: ndarray, fn: str, max_hold: None | float | ndarray = None) → ndarray[source]#: One value per bucket [edges[i], edges[i + 1]) of a step series (each sample holds until the next one, for max_hold at most): mean is time-weighted (step_bucket_mean()); min / max take the samples in the bucket and the value entering it; first is the entering value, else the first sample; last the last sample, else the value held at the bucket’s end. A bucket is nan only when nothing holds in it.

pyedautils.arraystats.infer_interval(ts: ndarray) → float | None[source]#

Estimate the sampling interval as the median positive time difference.

Parameters:: ts – Sorted timestamps (numeric or datetime64).
Returns:: The median positive interval in timestamp units, or None if there are fewer than two distinct timestamps.

pyedautils.arraystats.gap_indices(ts: ndarray, expected_interval: float | None = None, factor: float = 2.0, min_gap: float | None = None, adaptive: bool = False, block: int = 16) → ndarray[source]#

Indices i where a gap lies between ts[i] and ts[i + 1].

A gap is a time difference larger than max(factor * expected_interval, min_gap); the interval is inferred via infer_interval() when None. With adaptive=True (and no expected_interval) the expected interval is the local one (local_interval()), so a change of the sampling rate is not a gap. Returns an int64 array (empty if no gaps).

pyedautils.arraystats.detect_gaps(ts: ndarray, expected_interval: float | None = None, factor: float = 2.0, min_gap: float | None = None, adaptive: bool = False, block: int = 16) → tuple[ndarray, ndarray][source]#

Find gaps between consecutive timestamps.

A gap is a time difference larger than max(factor * expected_interval, min_gap).

Parameters:

ts – Sorted timestamps (numeric or datetime64).
expected_interval – Expected sampling interval in timestamp units; inferred via infer_interval() when None.
factor – Multiplier on the expected interval. Default 2.0.
min_gap – Minimum gap length in timestamp units. Default 0.
adaptive – Compare against the local interval (local_interval()) instead of one interval for the whole series. Default False.
block – Block size of the local interval. Default 16.

Returns:

arrays with the last timestamp before and the first timestamp after each gap (same dtype as ts).

Return type:

(gap_start, gap_end)

pyedautils.arraystats.local_interval(ts: ndarray, block: int = 16) → ndarray[source]#

Expected sampling interval before each step, robust to rate changes.

The positive time differences are split into blocks of block steps; the expected interval of a step is the largest median of its own block and the two neighbouring blocks. A switch from 10 to 60 minutes (e.g. a LoRaWAN downlink) therefore does not make the first 60-minute steps look like gaps, while a long outage between regular samples stands out. Fewer than 2 * block steps fall back to the global median. Vectorised, O(n).

Limitation: if more than half of the packets of a block are lost, the local interval reads too long and those losses are not flagged.

Parameters:

ts – Sorted timestamps (numeric or datetime64).
block – Steps per block. Default 16.

Returns:

float64 array of length len(ts) - 1; nan for duplicate timestamps.

Raises:

ValueError – if ts is not sorted.

class pyedautils.arraystats.Gaps(index: np.ndarray, start: np.ndarray, end: np.ndarray, expected: np.ndarray)[source]#

Bases: NamedTuple

Gaps of a series: index i means between ts[i] and ts[i + 1] (len(ts) - 1 for a trailing gap up to end).

index: ndarray#: Alias for field number 0

start: ndarray#: Alias for field number 1

end: ndarray#: Alias for field number 2

expected: ndarray#: Alias for field number 3

property duration: ndarray#

without(exclude: ndarray | Sequence[Sequence[float]]) → Gaps[source]#: Drop gaps whose missing part ([start + expected, end)) lies completely inside the excluded ranges ([[start, end], ...], e.g. removed test data).

pyedautils.arraystats.find_gaps(ts: ndarray, factor: float = 3.0, min_gap: float | None = None, block: int = 16, expected_interval: float | None = None, end: float | None = None) → Gaps[source]#

Gaps longer than max(factor * expected interval, min_gap).

The expected interval is the local one (local_interval()) unless expected_interval is given. With end (e.g. “now”), the time after the last sample counts as a trailing gap if it is that long.

Parameters:

ts – Sorted timestamps (numeric or datetime64).
factor – Multiplier on the expected interval. Default 3.0.
min_gap – Minimum gap length in timestamp units. Default 0.
block – Block size of local_interval(). Default 16.
expected_interval – Fixed expected interval (disables the local estimate).
end – Timestamp up to which data is expected (ticks).

Returns:

Gaps (start and end in ticks).

class pyedautils.arraystats.IntervalSegments(start: np.ndarray, end: np.ndarray, interval: np.ndarray, n: np.ndarray)[source]#

Bases: NamedTuple

Periods with one sampling interval (e.g. 10 min, then 60 min).

start: ndarray#: Alias for field number 0

end: ndarray#: Alias for field number 1

interval: ndarray#: Alias for field number 2

n: ndarray#: Alias for field number 3

pyedautils.arraystats.interval_segments(ts: ndarray, block: int = 16, tolerance: float = 0.25, min_blocks: int = 3) → IntervalSegments[source]#

Split a series into periods with a stable sampling interval.

Block medians (see local_interval()) are put into logarithmic classes of relative width tolerance; runs of the same class form segments, and runs shorter than min_blocks blocks (flicker, outages) join the previous segment. The interval of a segment is the median of its positive time differences.

Parameters:

ts – Sorted timestamps (numeric or datetime64).
block – Steps per block. Default 16.
tolerance – Relative difference treated as the same interval. Default 0.25.
min_blocks – Minimum length of a segment in blocks. Default 3.

Returns:

IntervalSegments (empty for fewer than two distinct timestamps).

pyedautils.arraystats.interval_overlap(starts: ndarray, ends: ndarray, edges: ndarray) → ndarray[source]#: Total length of sorted, non-overlapping intervals [starts, ends) inside each bucket [edges[k], edges[k + 1]). Vectorised, O((n + k) log n).

class pyedautils.arraystats.BucketCoverage(observed: np.ndarray, missing: np.ndarray, samples: np.ndarray, n_gaps: np.ndarray)[source]#

Bases: NamedTuple

Per bucket: observed time, missing time (in gaps), samples and gap count. All fields are additive, e.g. over the series of a building.

observed: ndarray#: Alias for field number 0

missing: ndarray#: Alias for field number 1

samples: ndarray#: Alias for field number 2

n_gaps: ndarray#: Alias for field number 3

property coverage: ndarray#: Share of observed time without gap (nan where nothing was observed).

pyedautils.arraystats.bucket_coverage(edges: ndarray, gaps: Gaps, first: float, obs_end: float, ts: ndarray | None = None, exclude: ndarray | Sequence[Sequence[float]] | None = None) → BucketCoverage[source]#

Time-based coverage per bucket [edges[k], edges[k + 1]).

The observed period is [first, obs_end) (e.g. first sample to now). Of each gap, [start + expected, end) is missing: the sample before a gap still covers one interval.

Parameters:

edges – Sorted bucket edges (ticks).
gaps – Result of find_gaps().
first – Start of the observed period.
obs_end – End of the observed period.
ts – Timestamps for the sample counts (optional).
exclude – Ranges [[start, end], ...] that are neither observed nor missing (e.g. excluded test data). Optional.

Returns:

BucketCoverage.

pyedautils.arraystats.detect_stuck(values: ndarray, ts: ndarray, min_repeats: int = 20, min_duration: float = 0) → tuple[ndarray, ndarray, ndarray, ndarray][source]#

Find runs of identical consecutive values (run-length encoding).

NaN values are dropped before the runs are formed.

Parameters:

values – Sample values.
ts – Sorted timestamps (numeric or datetime64), same length.
min_repeats – Minimum number of identical consecutive samples.
min_duration – Minimum run duration (last - first timestamp) in timestamp units.

Returns:

(run_start, run_end, run_value, n_repeats) arrays.

pyedautils.arraystats.outliers_iqr(values: ndarray, multiplier: float = 1.5) → ndarray[source]#

Flag values outside [Q1 - k*IQR, Q3 + k*IQR] (Tukey fences).

Returns:: Boolean mask, False for NaN values.

pyedautils.arraystats.outliers_zscore(values: ndarray, threshold: float = 3.0) → ndarray[source]#

Flag values whose absolute z-score exceeds threshold.

Returns:: Boolean mask, False for NaN values or zero variance.

pyedautils.arraystats.outliers_hampel(values: ndarray, window: int = 15, n_sigma: float = 3.0) → ndarray[source]#

Hampel filter: flag values deviating from the rolling median.

A value is an outlier if |x - median| > n_sigma * 1.4826 * MAD within a centred window of 2 * window + 1 samples. Robust against level changes, unlike global IQR/z-score methods. NaN values are never flagged; for the window medians they are replaced by the previous valid value (or the first valid value at the start).

Returns:: Boolean mask.

arraystats

Contents

arraystats#

API Reference#