terraform test expected_failure not recognizing missing data resource as failure

[mgrubb]

Terraform Version

Actual code
Terraform v1.9.4
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.63.0

Minimal Example
Terraform v1.9.4
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.74.0

Terraform Configuration Files

main.tf:

variable "batch_exec_role_name" {
  type = string
 }

data "aws_iam_role" "batch_exec_role" {
    name = var.batch_exec_role_name
}
# ...

main.tftest.hcl:

variables {
  some_other_var = "some_val"
}

run "invalid_role_name" {
  variables {
    batch_exec_role_name = "does-not-exist"
  }
  expect_failures = [
    data.aws_iam_role.batch_exec_role,
  ]
  command = plan
}

Debug Output

https://gist.github.com/mgrubb/ba584b36173af380a9c0c8ea191e7beb

Expected Behavior

Test should pass by noting the data resource was not found and caused an error

Actual Behavior

Test fails by missing not finding the failure caused by the missing data resource.

Steps to Reproduce

1. terraform init
2. terraform test

Additional Context

using expect_failures = [var.some_var] for detecting variables that do not pass validation rules works as expected.
I also tried adding a postcondition to the data resource with a condition of self.name == var.batch_exec_role_name but that didn't seem to have any impact.

References

No response

[liamcervante]

Hi @mgrubb, the expect_failures attribute only validates against Custom Conditions within the configuration. The custom conditions are a part of the configuration that configuration authors have control over and hence is something they could write tests against. The behaviour you've described seems more about testing the behaviour of Terraform itself rather than the configuration that you've written - you can trust that Terraform will error if the provider for a given data source returns an error when it is retrieved. That's not something you should have to test.

I've updated this to an enhancement request, but I think it's unlikely we'll make this change. This is because, as mentioned, the test command is to enable authors to test the configuration they've written rather than validating the behaviour of Terraform internals. There's a similar feature request in #34871 which effectively has the same response as here, configuration authors should be able to trust that Terraform will behave as expected and we have other tests for that.

[bebold-jhr]

As my issue ( #37799 ) got closed as a duplicate of this one, I just wanted to post my use case here as well.

What I wanted to achieve was integration tests for my SCPs in AWS.
A negative test would mean that as a team I try to perform an action which is restricted by SCPs. So my assumption was to use expect_failure which would catch the resulting error and proof that the SCP works as designed.
However if it's a permission problem, the error is not being catched by expect_failure.

I woud also be fine with a new assertion like expect_exception or expect_error to clarify that this is not about validation problem.