Field stats monitor

Field stats monitors automatically track key statistical properties of one or more fields in a table and learn a time-series pattern for each metric. These monitors help you detect unexpected changes in distribution, null rates, or value ranges without writing any SQL.

Examples of what you can detect with field stats monitors:

• Sudden spikes in null or empty values after a pipeline change
• Drops in the number of unique IDs, indicating upstream duplication issues
• Unusual shifts in a numeric metric such as fare amount, order value, lead time, etc.
• Outliers in mean, max, or standard deviation that may indicate data corruption

​

Metrics tracked automatically

By default, each selected field generates a time-series for every metric available for its data type. Metrics can be narrowed per column; for monitors defined as code, see monitors as code.

Metric	ID	Description
Number of rows	NUM_ROWS	Total number of rows in the dataset
Number of null	NUM_NULL	Count of null values
Percentage of Null	PCT_NULL	Percentage of null values
Number of empty	NUM_EMPTY	Count of empty string values
Percentage of Empty	PCT_EMPTY	Percentage of empty string values
Number of unique	NUM_UNIQUE	Count of unique values
Percentage of Unique	PCT_UNIQUE	Percentage of unique values relative to total rows
Mean	MEAN	Average value of the field
Median	MEDIAN	Median value
Min	MIN	Minimum value
Max	MAX	Maximum value
Standard deviation	STDDEV	Statistical standard deviation

Metrics available per data type:

Field data type	Metrics
String	NUM_ROWS, NUM_UNIQUE, NUM_EMPTY, NUM_NULL, PCT_NULL, PCT_UNIQUE, PCT_EMPTY
Numeric, integer, float	the string metrics plus MEAN, MIN, MAX, MEDIAN, STDDEV
Date, datetime	NUM_ROWS, NUM_NULL, NUM_UNIQUE, MIN, MAX, PCT_NULL, PCT_UNIQUE

Redshift does not compute NUM_UNIQUE or PCT_UNIQUE on numeric columns, so they are skipped there. A column whose whole selection is skipped produces no predictions.

​

Setting up a field stats monitor

1. Navigate to the table where you want to create the monitor and click Add Monitor

   

2. Select Field stats and choose the fields you want to monitor
   You can monitor one or multiple fields at once. For each selected field, choose the metrics to track; all are on by default.

3. Configure time partitioning. A time-series will be learned per metric per field, using the timestamp column you select.

   

4. Set sensitivity, and severity (optional)

   

Once confirmed, the monitor will appear under the table and on the

On its first run, a field stats monitor reads the recent history of the table. After that it queries only new partitions, to keep query costs low. Anomaly detection starts right after setup.

​

Resetting a monitor

If you changed historical data and want the monitor to learn it again, open the monitor, go to Settings, and click Reset monitor. A reset cannot be undone, so only reset if you are sure the historical values have changed.