Data resolution and smoothning

Every week we do a full scan of all the time series and how they have been behaving in the past 7 days. We identify the typical rate at which the data come in (frequency), and and how often they change (activity). Then each time series is classified according to frequency and activity into three main groups:

High Frequency High activity: Time series with data points registered more frequently than every 15 minutes and that change values more frequently than every 15 minutes.

High Frequency Low Activity: Time series with data points registered more frequently than every 15 minutes and that change values every 15 minutes or more slowly.

Low Frequency: Times series with data points coming in with a typical interval bigger than 15 minutes.

This can be time consuming and computationally intensive, therefore it is done once per week. The rest of the week we use the values already calculated.

Anomaly detection frequency and Smoothning

Anomaly detection runs every minute for high frequency metrics and every hour for low frequency.

The data for anomaly detection are resampled (smoothned) to reduce noise.

For High frequency Eyer uses rolling resample. For High Frequency High Activity metrics the resampling window is determined by the activity for aggregation average and maximum, while is 15 minutes for aggregation total. The reason for the 15 minutes in the total case is because this meatrics have often the tendency of oscillating more and to better distinguish the case of occasional zero from the metric going to zero for a prolonged time. For High Frequency Low Activity metrics the resampling window is determined by the frequency.

For Low Frequency the resampling window is 60 minutes.