KeyError: 'date' on synthetic data

[marrrcin]

Hi,
I'm exploring the use of your library and I've stumped across an error when working with my data.

Popmon version: 1.4.5
Error:

 in <lambda>(plot)
    157             # filter out potential empty plots
    158             plots = [e for e in plots if len(e)]
--> 159             plots = sorted(plots, key=lambda plot: plot["date"])
    160 
    161             # basic checks for histograms

KeyError: 'date'

Full stack trace: ⬇️

KeyError                                  Traceback (most recent call last)
[<ipython-input-39-c55c117796f8>](https://localhost:8080/#) in <cell line: 1>()
----> 1 report = popmon.df_stability_report(
      2     df,
      3     time_axis="time",
      4     time_width="1w",
      5 )

7 frames
[/usr/local/lib/python3.10/dist-packages/popmon/pipeline/report.py](https://localhost:8080/#) in df_stability_report(df, settings, time_width, time_offset, var_dtype, reference, split, **kwargs)
    196 
    197     # generate data stability report
--> 198     return stability_report(
    199         hists=hists,
    200         settings=settings,

[/usr/local/lib/python3.10/dist-packages/popmon/pipeline/report.py](https://localhost:8080/#) in stability_report(hists, settings, reference, **kwargs)
     73     # execute reporting pipeline
     74     pipeline = get_report_pipeline_class(settings.reference_type, reference)(**cfg)
---> 75     result = pipeline.transform(datastore)
     76 
     77     stability_report_result = StabilityReport(datastore=result)

[/usr/local/lib/python3.10/dist-packages/popmon/base/pipeline.py](https://localhost:8080/#) in transform(self, datastore)
     65         for module in self.modules:
     66             self.logger.debug(f"transform {module.__class__.__name__}")
---> 67             datastore = module.transform(datastore)
     68         return datastore
     69 

[/usr/local/lib/python3.10/dist-packages/popmon/pipeline/report_pipelines.py](https://localhost:8080/#) in transform(self, datastore)
    255     def transform(self, datastore):
    256         self.logger.info(f'Generating report "{self.store_key}".')
--> 257         return super().transform(datastore)

[/usr/local/lib/python3.10/dist-packages/popmon/base/pipeline.py](https://localhost:8080/#) in transform(self, datastore)
     65         for module in self.modules:
     66             self.logger.debug(f"transform {module.__class__.__name__}")
---> 67             datastore = module.transform(datastore)
     68         return datastore
     69 

[/usr/local/lib/python3.10/dist-packages/popmon/base/module.py](https://localhost:8080/#) in _transform(self, datastore)
     49 
     50         # transformation
---> 51         outputs = func(self, *list(inputs.values()))
     52 
     53         # transform returns None if no update needs to be made

[/usr/local/lib/python3.10/dist-packages/popmon/visualization/histogram_section.py](https://localhost:8080/#) in transform(self, data_obj, sections)
    157             # filter out potential empty plots
    158             plots = [e for e in plots if len(e)]
--> 159             plots = sorted(plots, key=lambda plot: plot["date"])
    160 
    161             # basic checks for histograms

[/usr/local/lib/python3.10/dist-packages/popmon/visualization/histogram_section.py](https://localhost:8080/#) in <lambda>(plot)
    157             # filter out potential empty plots
    158             plots = [e for e in plots if len(e)]
--> 159             plots = sorted(plots, key=lambda plot: plot["date"])
    160 
    161             # basic checks for histograms

KeyError: 'date'

Reproduction steps:
https://colab.research.google.com/drive/1N59kn7C9LN6W9AJkfz9SougiZoOMM0bn?usp=sharing

Additional information:
I'm using a function to generate synthetic data (see colab). When I generate "less" data - e.g. for 200 days, the code works fine, but after some unknown threshold (like 360 days), it breaks.
I've also tried changing the time_width parameter - sometimes it starts to work with 2w, sometimes it works with 1d but I haven't figured out any pattern.

Also note that it happens both for self-referencing data as well as data with a reference set (see second part of the colab).

Expected result:
Monitoring report generates properly.

[sbrugman]

Thanks for reporting Marcin, will look into it

[marrrcin]

@sbrugman an update from my side:
It seems like the following lines in the data generator are causing the popmon to break:

    feature_anomalies = np.random.normal(loc=0.5, scale=0.05, size=num_days)
    anomaly_indices = np.random.choice(num_days, num_anomalies, replace=False)
    feature_anomalies[anomaly_indices] = np.random.uniform(low=-5, high=0.1, size=num_anomalies)
    feature_out_of_range = np.random.uniform(low=0, high=1, size=num_days)
    out_of_range_indices = np.random.choice(num_days, num_out_of_range, replace=False)
    feature_out_of_range[out_of_range_indices] = np.random.uniform(low=2, high=3, size=num_out_of_range)

Initially, I thought it has something to do with the memory allocation / assignments, but it it seems like the range of values is a problem. If I increase the num_anomalies to something in closer to at half of my examples (which means - generating more examples that are e.g. out of range), the code proceeds normally. It should work in both cases though.

[sbrugman]

@marrrcin Could you please provide the minimum reproducible code here as a snippet? Policy doesn't allow us to use colab...

[marrrcin]

Absolutely!

import pandas as pd
import popmon
import numpy as np

def generate_mock_data(num_days, num_anomalies, num_out_of_range, random_state=666, start_date='1/1/2022'):
    np.random.seed(random_state)
    time = pd.date_range(start=start_date, periods=num_days, freq='D')
    feature_increasing = np.arange(1, num_days+1)
    feature_decreasing = np.arange(1000000, 1000000-num_days, -1)
    feature_stable = np.random.normal(loc=0.5, scale=0.05, size=num_days)
    feature_unstable = np.random.normal(loc=0.5, scale=2.0, size=num_days)
    feature_anomalies = np.random.normal(loc=0.5, scale=0.05, size=num_days)
    anomaly_indices = np.random.choice(num_days, num_anomalies, replace=False)
    feature_anomalies[anomaly_indices] = np.random.uniform(low=-5, high=0.1, size=num_anomalies)
    feature_out_of_range = np.random.uniform(low=0, high=1, size=num_days)
    out_of_range_indices = np.random.choice(num_days, num_out_of_range, replace=False)
    feature_out_of_range[out_of_range_indices] = np.random.uniform(low=2, high=3, size=num_out_of_range)
    trend_change = np.concatenate([np.linspace(0, 3.0, num_days//2+(num_days % 2)), np.linspace(3.0, 0, num_days//2)]) + np.random.normal(loc=0, scale=0.01, size=num_days)
    cyclic_feature = np.sin(np.linspace(0, 4*np.pi, num_days)) + np.random.normal(loc=0, scale=0.1, size=num_days)
    data = {'time': time, 'feature_increasing': feature_increasing, 'feature_decreasing': feature_decreasing, 'feature_stable': feature_stable, 'feature_unstable': feature_unstable, 'feature_anomalies': feature_anomalies, 'feature_out_of_range': feature_out_of_range, 'trend_change': trend_change, 'cyclic_feature': cyclic_feature}
    df = pd.DataFrame(data)
    return df


df = generate_mock_data(num_days=300, num_anomalies=10, num_out_of_range=13)


report = popmon.df_stability_report(
    df,
    time_axis="time",
    time_width="1w",
)

[sbrugman]

Can confirm this is a bug with the histogram plotting with outliers, will release a patch soon!

[sbrugman]

@marrrcin Release is out, feel free to open up another issue if you encounter other problems. Thanks a lot!

[marrrcin]

Thanks for a quick fix, I confirm that it works now!