Using Incremental Matrix Profiles for discords, possible ?

[udf2457]

The docs seem to strongly hint that the incremental method can only be used for motifs and not discords, e.g. the following wording:

However, note that any change in the matrix profile can only move downward (toward zero)

Is my understanding correct ? Or are the docs just unclear and need updating with a discord example ?

[seanlaw]

Hi @udf2457, thank you for your question and welcome to the STUMPY community

The docs seem to strongly hint that the incremental method can only be used for motifs and not discords, e.g. the following wording:

However, note that any change in the matrix profile can only move downward (toward zero)

No, I don't think that it suggests that. First, we must remember what information is stored inside of the matrix profile. Essentially, for each subsequence in your time series, the matrix profile always stores the distance (and index location) to its one-nearest-neighbor. This is true even for a discord (i.e., the matrix profile itself doesn't actually distinguish between motifs/discords)! The only difference between a "motif" and a "discord" is that the "motif" has a one-nearest-neighbor that is very "close", while the "discord" has a one-nearest-neighbor that is, sadly, very far away. However, in both cases, the distance is measured as the z-normalized Euclidean distance.

By definition, since the matrix profile only captures the one-nearest-neighbor information, the definition of a "discord" is merely the subsequence in your time series that, currently, has a one-nearest-neighbor that is far away. However, in stumpi, since we are dealing with "updating" the one-nearest-neighbor distance, by definition, the captured matrix profile distance(s) can NEVER get larger. Note that this is a matrix profile definition (as published by the original authors) and not a limitation of STUMPY. So, when a new subsequence arrives in your stream, that subsequence may indeed have a larger distance from your existing discord subsequence. However, that new distance is ignored or thrown away because we only keep the one-nearest-neighbor distance for matrix profiles.

Or are the docs just unclear and need updating with a discord example ?

So, I think the docs are actually very clear. stumpi works perfectly fine for discords without any change. However, feel free to ask any clarifying questions and I'll try my best to provide more details. I've certainly not thought through everything so please let me know if I've missed anything that needs to be considered further.