Memory concerns with large `l_max` values

[kenteross]

The Issue:

I'm running a model using a logistic saturation transformation followed by a delayed adstock transformation for a PYMC model. I have a burn-in period of 30 days, and my issue appears when I add the code to slice the burn-in period out. After adding the slice the model will run out of RAM even with very few samples. Reversing the order of the transformations (i.e., adstock before saturation) seems to work without running into RAM issues though.

Reproducible code example

Error Message:

KernelInterrupted: Execution interrupted by the Jupyter kernel.

PyMC version information:

The version of PYMC I'm using is '5.13.1'.

Context for the issue:

I think the issue is coming from the convolution in the adstock transformation, but I'm not sure why my code works when I do the adstock transformation before the saturation transformation.

[juanitorduz]

Hi @kenteross! Thanks for opening this issue! As the code is long and has text, can you share it in a GitHub gist or a Google Colab notebook, please 🙏 ?

[kenteross]

Hi @juanitorduz! Thanks for the quick reply! Sorry, I didn't realize it was in text format. Here's a GitHub gist of my code!

[williambdean]

Hi @kenteross , I see that that the l_max is 50
Have you tried a smaller value? Does it cause issues with l_max = 5 or 10?
Weekly datasets tend to be common so 50 days = l_max of 7

[williambdean]

Are you also able to import the function from the package as well instead of copy and paste. I see the functions have changed and maybe there has been a bug fix. Module for reference

from pymc_marketing.mmm.transformers import delayed_adstock, geometric_adstock, logistic_saturation

[kenteross]

@wd60622 thanks for your suggestions! I tried reducing the l_max and importing the packages independently and it seems like the issue was coming from the l_max. It seems like the maximum threshold for an l_max is around 20 in this case. I'll stick with weekly data for now. Thanks for the help!

[williambdean]

@wd60622 thanks for your suggestions! I tried reducing the l_max and importing the packages independently and it seems like the issue was coming from the l_max. It seems like the maximum threshold for an l_max is around 20 in this case. I'll stick with weekly data for now. Thanks for the help!

Glad that it helped.

And just to be clear, did the model start sampling? (progress bar, etc)?
And it only successful with which ordering?

1. adstock -> saturation
2. saturation -> adstock

If you just perform the adstock, does the large l_max cause issues?

[kenteross]

Yeah, thanks!

The model did start sampling even when it ran into an issue. It would just run my machine out of RAM, almost like it was caught in a loop.

When I reduce the l_max it's successful in either order, but with a large l_max it's only successful in the adstock -> saturation order.

If I remember correctly just performing adstock with a large l_max would result in an issue. I can test it again tomorrow if you like.

[williambdean]

If I remember correctly just performing adstock with a large l_max would result in an issue. I can test it again tomorrow if you like.

All good! Thank you for summarizing your findings!

My only suggestion is to model weekly data in order to reduce the l_max by factor of 7. Maybe @juanitorduz has some additional tips?

[kenteross]

@wd60622, yeah, that makes sense to model weekly data and reduce l_max so much. Thanks for the tips!