sampling

sampling#

How a series was sampled and stored, estimated from the data alone: interval percentiles, change-of-value (COV) detection, dead band, heartbeat, quantisation step, compression factor (Thornhill et al. 2004) and how long values stood unchanged. Building automation archives often store only changes, which biases spread, spectra and oscillation indices; the profile shows which analyses a series supports. NumPy only.

import numpy as np
from pyedautils.sampling import sampling_profile, interval_histogram

p = sampling_profile(ts, values, long_hold=86_400_000)   # ts: epoch ms
p.change_of_value, p.deadband, p.interval_min, p.compression
interval_histogram(ts, [1_000, 10_000, 60_000, 600_000, 3_600_000, 86_400_000])

API Reference#

How a time series was sampled and stored, estimated from the data alone.

Building automation archives store values either at a fixed interval or only on a change of value (COV: a new sample when the value moved by more than a dead band, sometimes with a periodic “heartbeat” of an unchanged value). Which of the two, and with which settings, decides which analyses are meaningful (e.g. oscillation indices need samples well below the oscillation period; a compressed archive biases spread and spectra, see Thornhill et al. 2004, doi:10.1016/j.jprocont.2003.06.003).

sampling_profile() estimates these settings from the timestamps and values. Timestamps are numeric (e.g. int64 epoch milliseconds) or datetime64; durations are returned in the same unit. NaN values are dropped.

Bases: NamedTuple

Result of sampling_profile(). Durations are in timestamp units; an estimate that the data does not support is None.

n: int#: samples (without NaN)

span: float#: last - first timestamp

interval_min: float | None#: 1st percentile of the steps (minimum send interval)

interval_p10: float | None#: Alias for field number 3

interval_median: float | None#: Alias for field number 4

interval_p90: float | None#: Alias for field number 5

interval_max: float | None#: Alias for field number 6

change_of_value: bool#: see is_change_of_value()

repeat_share: float#: share of steps that repeat the previous value

deadband: float | None#: smallest typical value change (change of value only)

resolution: float | None#: every value change is a multiple of this

heartbeat: float | None#: refresh interval of an unchanged value (COV)

compression: float | None#: N / m, m = non-zero second differences (Thornhill)

longest_hold: float#: longest time a value stood unchanged

hold_share: float#: share of the span in holds longer than long_hold

pyedautils.sampling.interval_histogram(ts: ndarray, edges: Sequence[float]) → ndarray[source]#

Number of sampling intervals (positive time steps) per bin.

Parameters:

ts – Sorted timestamps (numeric or datetime64).
edges – Increasing bin edges in timestamp units; steps below the first edge count in the first bin, steps above the last edge in the last bin.

Returns:

len(edges) - 1 counts.

pyedautils.sampling.is_change_of_value(ts: ndarray, ratio: float = 10.0, share: float = 0.5, min_samples: int = 64, min_long: int = 8) → bool[source]#

Change-of-value sampling: most of the time passes in steps far longer than the typical one (a regular series with a few outages does not qualify), or the steps follow no interval at all - few lie near a common length and they spread widely (a slow signal with a large dead band: long steps, but none of them regular).

Parameters:

ts – Sorted timestamps (numeric or datetime64).
ratio – A step longer than ratio x the median step is “long”.
share – Long steps must cover more than this share of the time.
min_samples – Fewer (positive) steps are never change of value.
min_long – At least this many long steps.

Returns:

True for change-of-value sampling.

pyedautils.sampling.on_grid_share(steps: ndarray, modes: int = 3) → float[source]#: Share of the (positive) steps near one of the modes most common step lengths (GRID_BIN wide on a log scale): about 1 for sampling at one or a few intervals, small for change-of-value data.

pyedautils.sampling.sampling_profile(ts: ndarray, values: ndarray, long_hold: float | None = None) → SamplingProfile[source]#

Estimate how a series was sampled and stored.

Intervals: percentiles of the positive time steps; interval_min (1st percentile) is the shortest interval the sender typically uses.
heartbeat: the median interval between two samples with the same value, when such repeats are common and regular while changes come much faster (a change-of-value sender refreshing an unchanged value; periodic sampling of a mostly constant value repeats at its sampling interval and has no heartbeat).
Change of value: is_change_of_value() or a heartbeat. Then deadband is the 5th percentile of the non-zero value changes, i.e. what a value must move before a new sample is stored.
resolution: a quantisation step, if (nearly) all value changes are whole multiples of it (e.g. 0.1 for values stored with one decimal).
compression: N / m with m the number of non-zero second differences of the values (Thornhill et al. 2004): about 1 for uncompressed data, larger when an archive stores straight segments or holds. Change-of-value data is about 1 by construction; there, the dead band describes the compression.
longest_hold, hold_share: how long values stood unchanged (from a sample to the next sample with a different value); the share counts holds longer than long_hold (None: no share, 0).

Parameters:

ts – Sorted timestamps (numeric or datetime64).
values – Sample values, same length.
long_hold – Hold duration from which hold_share counts.

Returns:

SamplingProfile.

sampling

Contents

sampling#

API Reference#