Upgrade of a QnABot stack in an aws account with multiple QnABot stacks deployed, brought down a separate QnABot Admin UI and Bot.

[bobpskier]

Describe the bug
After a successful upgrade of one QnABot stack (Stack A) to a 7.x version, we noticed that an independent deployment of QnABot via a separate stack (Stack B - 6.1.5 version of a QnABot) in the same account was affected. The Admin UI and Bot for Stack B no longer worked correctly and the Bot was not functioning. Tracking down the Admin UI errors, it turned out that the API get request on /questions was returning a 404 error. Looking in the log files for the responsible lambda, the CFN-DefaultQnABotSettings SSM parameter for this instance was not available and the lambda returned an error to the Admin UI. I checked cloudtrail for who deleted this SSM parameter and the user name associated with this delete was the CFN lambda for the first stack that was upgraded. Both stacks still exists. The upgrade of stack A effected stack B to the point that stack B is not usable due to the missing parameter. I don't know yet if anything else was deleted.

To Reproduce
I image, install stack A and B using QnABot version 6.1.5. Both stacks should operate fine. Upgrade stack A to 7.0.2. Verify that Stack B still functions and that the DefaultQnABotSettings SSM parameter still exists.

Expected behavior
Stack updates should operate independently of each other.

Please complete the following information about the solution:

• Version:6.1.5 and Version 7.0.2

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0189) QnABot [...] v0.0.1".

• [ us-east-1] Region: us-east-1
• [ No] Was the solution modified from the version published on this repository? No
• If the answer to the previous question was yes, are the changes available on GitHub?
• Have you checked your service quotas for the services this solution uses?
• [Yes ] Were there any errors in the CloudWatch Logs?

Additional context
Still looking to find the DeleteParameter logic on the SSM parameter in the CFN lambda function. Something got confused and deleted multiple SSM parameters owned by different stacks.

[ihmaws]

Thanks @bobpskier for the report. We will look into this.

Looking at the changelog, v7.0.0 moved from SSM to to DynamoDB and may be affecting the upgrade path. It is concerning to see this affect across independent stacks. We are treating this with priority.

[bobpskier]

@ihmaws In addition to the DefaultQnABotSettings, the CustomQnABotSettings SSM parameter was also removed. I recovered by manually creating these two parameters using the specific names referenced in the environment parameters from the lambdas servicing the API for the second stack (B). I restored the parameters based on the values stored for the first stack (A) which incidentally had not be removed by the original upgrade to stack A. We then confirm the recovered settings were valid. Then we upgraded the second stack (B) to 7.0.2 which succeeded. Both stacks are now on 7.0.2 and are functioning correctly.

[ihmaws]

Thanks @bobpskier for all the details. We have identified the root cause and are working on a fix. We are hoping to have a patch release up over the next few days.

We are glad to hear you were able to recover and aren't seeing any data loss. Thank you for the bug report and sorry for the inconvenience.

⚠ For now, I would recommend not upgrading to v7.x.x from a version <v7.0.0 if you have multiple QnABot stacks in the same account and region. ⚠

Issue Summary

In v7.0.0, the solution moved from storing these values in an SSM parameter to storing them in DynamoDB (to help alleviate sizing constraints especially as customers start using larger prompts).

A piece of code was added in that release to help customers migrate from older versions. It would copy over existing SSM parameters and store the value in DDB. Once migrated over into DDB, it would go in and delete the SSM parameter.

This logic however was flawed (assumed the solution had a single tenancy), and selected too broad a set of parameters to migrate/delete leading to all QnABot SSMs to be deleted.

[tmekari]

Hi @bobpskier, thank you again for bringing this issue to our attention. It has been resolved in our recent v7.0.3 release.

While @ihmaws was working on this issue, we also looked for a way for you to restore your settings. Your settings should be pretty well detailed within the fulfillment lambda logs. If you search for "Merged Settings" using all of your log streams, you should get a full accounting. Please feel free to reach out if you have any issue restoring your settings. Thanks!