Skip to content

Retention of S3 object once message consumed by queue. #40

@james-bowen-au

Description

@james-bowen-au

One part of the outstanding pull request #6 addresses this.

If a queue is one of many subscribers to a topic, then we want all the queues to have a 'fair go' at processing a message before the max retention period of the topic lapses.

With the current setup, the deletion of a message from the queue using the extended lib deletes the underlying S3 bucket. This is fine when the queue is the only consumer of the message (and implicitly the S3 bucket) but with a topic and queue setup the first consumer will stop any other subscriber from getting the contents of the S3 bucket.

We can currently work around this by writing custom code to override the behaviour of the client when deleting a message, but this seems flaky given we don't control the library itself.
Would be great to get a configuration option that leaves the message in S3 when a consumer sends a delete message request . This way we could let the topic drive when an S3 bucket is finally cleaned up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions