feat: Allow experiments from override files to be used

[SarahFrench]

PR illustrating an approach to fixing #38012, but what the solution is is still up for debate.

Terraform allows the use of override files to change configuration while leaving the original configuration unedited. Overrides are compatible with the terraform block, but each setting inside that block has slightly different behaviour. For example required_providers is merged between the original config and the override config, whereas a backend or cloud block in the override config completely replaces the original config's backend or cloud block.

This PR allows the experiments field from the terraform block to be used from override files. The merging behaviour is additive, so experiments present in both the original config and the override config will be active in the final parsed configuration.

Currently any experiments configuration in the override file is silently ignored.

Target Release

1.15.x

Rollback Plan

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

• This change is user-facing and I added a changelog entry.
• This change is not user-facing.

[mildwonkey]

As a former infra person, I don't know that I like this functionality. I would not want to risk experimental features making their way into production via sneaky override file - I won't die on this hill, I'm not going to block this PR, but I think we want to talk through the implications a bit?

[SarahFrench]

As a former infra person, I don't know that I like this functionality. I would not want to risk experimental features making their way into production via sneaky override file - I won't die on this hill, I'm not going to block this PR, but I think we want to talk through the implications a bit?

Yeah sure - I was planning on raising it in the team channel once this PR was ready

[SarahFrench]

Ok, PR is now green and I have raised the discussion in the team channel. It could either be we go with this PR in its current draft state or we swap the PR over to explicitly disallow experiments in override files. For now this PR is just to illustrate the problem and potential solution 👍🏻

[SarahFrench]

Closing in favour of discussion happening in #38012