aws-stepfunctions-tasks: CallAwsServiceCrossRegion credentials property does not work as expected for cross-account scenarios

[t3yamoto]

Describe the bug

When using CallAwsServiceCrossRegion with the credentials property for cross-account and cross-region AWS service calls, the StateMachine execution role assumes the specified credentials role and attempts to execute the auto-generated Lambda function (created internally by CallAwsServiceCrossRegion) with that role. This behavior prevents cross-account scenarios where the credentials role is from another account, as the cross-account role lacks Lambda execution permissions and cannot invoke the auto-generated Lambda function in the original account's region.

Architecture Context:

• StateMachine in Account A (XXXXXXXXXXXX)
• Cross-account role in Account B (YYYYYYYYYYYY)
• Lambda function auto-generated by CallAwsServiceCrossRegion and deployed in Account A's ap-northeast-1 region
• Target AWS service calls need to be made in Account B using Account B's permissions

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

2.202.0

Expected Behavior

• The Lambda function should execute with its normal execution role (Account A)
• The credentials role should only be used for AWS SDK calls to the target service in Account B
• Cross-account AWS service calls should work seamlessly

Current Behavior

• StateMachine execution role assumes the credentials role from Account B
• The assumed cross-account role attempts to execute the Lambda function in Account A
• Lambda execution fails with lambda:InvokeFunction permission error
• AWS SDK calls to target services cannot be performed

The current implementation appears to assume the credentials role at the StateMachine level rather than passing it to the Lambda function for AWS SDK calls only. This causes the cross-account role to attempt Lambda execution, which it lacks permissions for. This may be the intended design, but it creates challenges for cross-account use cases.

Error message:

User: arn:aws:sts::YYYYYYYYYYYY:assumed-role/AccountBRole/ArHOvqMjmgomDQtEhSPnYAJQkCjBRGuQ is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:function:AccountAFunction because no resource-based policy allows the lambda:InvokeFunction action (Service: Lambda, Status Code: 403, Request ID: zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz) (SDK Attempt Count: 1)

Reproduction Steps

1. Create a CallAwsServiceCrossRegion task with cross-account credentials:

new tasks.CallAwsServiceCrossRegion(this, 'Foo', {
  service: 'ec2',
  action: 'describeNetworkInsightsAccessScopes',
  region: sfn.JsonPath.stringAt('$region'),
  iamResources: ['*'],
  credentials: {
    role: sfn.TaskRole.fromRoleArnJsonPath('$anotherAccountRoleArn'),
  },
});

2. Set up cross-account scenario:

   • Account A (XXXXXXXXXXXX): Contains StateMachine and auto-generated Lambda function (created by CallAwsServiceCrossRegion)
   • Account B (YYYYYYYYYYYY): Contains IAM Role that trusts Account A's StateMachine and has Network Access Analyzer execution policies

3. Execute the StateMachine

4. Observe the Lambda execution failure

Possible Solution

If the current behavior is not intended, the credentials property could be modified to be used only for AWS SDK calls within the auto-generated Lambda function, not for Lambda function execution itself. The Lambda function would always execute with its normal execution role from Account A, and only assume the cross-account role when making the actual AWS service API calls.

Alternatively, if this is the intended behavior, it would be helpful to have documentation clarifying this limitation and providing guidance for cross-account scenarios, or perhaps a separate property/option for cross-account AWS SDK calls that doesn't affect Lambda execution.

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.202.0

AWS CDK CLI version

2.1019.2 (build c29855e)

Node.js Version

v24.4.1

OS

macOS

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Hi @t3yamoto,

Thank you for reporting this issue with CallAwsServiceCrossRegion and cross-account credentials. I've analyzed the problem and can confirm this is a valid architectural limitation that affects cross-account use cases.

Issue Analysis

The problem occurs because CallAwsServiceCrossRegion inherits the credentials property from TaskStateBaseOptions, which is designed for Step Functions service integrations. When you specify credentials, Step Functions assumes that role before invoking the Lambda function, rather than passing the credentials to the Lambda function for internal
AWS SDK calls.

This creates a fundamental conflict:
• Step Functions behavior: Assumes the cross-account role, then tries to invoke Lambda with that role
• Expected behavior: Lambda should execute with its normal role, then assume the cross-account role for AWS SDK calls

Root Cause

The CallAwsServiceCrossRegion implementation doesn't handle the credentials
property specially - it relies on the base class behavior. The Lambda handler creates AWS SDK clients without any credential configuration.

Proposed Solutions

There are several approaches to fix this:

Option 1: Override credentials behavior (Breaking Change)

Modify CallAwsServiceCrossRegion so that credentials only affects the Lambda's internal AWS SDK calls, not Step Functions invocation.

Option 2: Add new property (Non-breaking)

Add a new property like awsSdkCredentials specifically for the Lambda's AWS SDK calls:

new tasks.CallAwsServiceCrossRegion(this, 'Foo', {
  service: 'ec2',
  action: 'describeNetworkInsightsAccessScopes',
  region: sfn.JsonPath.stringAt('$region'),
  iamResources: ['*'],
  awsSdkCredentials: {
    role: sfn.TaskRole.fromRoleArnJsonPath('$anotherAccountRoleArn'),
  },
});

Option 3: Smart detection

Detect when credentials is used with CallAwsServiceCrossRegion and automatically pass it to the Lambda handler instead of using it for Step Functions invocation.

Implementation Requirements

Any solution would need to:

1. Modify the CallAwsServiceCrossRegion class to handle credentials
2. Update the Lambda handler to accept and use credentials for AWS SDK calls
3. Add appropriate IAM permissions for the Lambda execution role to assume the specified credentials

This is a valuable enhancement for cross-account scenarios. Would you be interested in contributing a PR to implement this functionality? I'd recommend Option 2 (new property) as it's non-breaking and makes the intent clear.

Let me know your thoughts on the preferred approach!

[t3yamoto]

@pahud Thank you for the detailed analysis and clear explanation of the root cause!

I agree that Option 2 (adding a new awsSdkCredentials property) is the best approach. It's simple and won't confuse existing users since it's a non-breaking change.

I'd be happy to take on the challenge of contributing a PR to implement this functionality since I need this feature for my work.

[pahud]

I believe CallAwsServiceCrossRegion is designed for cross-region same account, and cross-account support is indeed a feature request.

Let me try to explain what's happening here using ascii diagrams. Correct me as I might be wrong.

Implementation of CallAwsServiceCrossRegion()

aws-cdk/packages/aws-cdk-lib/aws-stepfunctions-tasks/lib/lambda/call-aws-service-cross-region.ts

Line 114 in f95f7ac

export class CallAwsServiceCrossRegion extends sfn.TaskStateBase {

Execution Flow for cross-account calls

Step Function Execution ->(invokes)-> Lambda Function->(makes AWS API calls)->Target AWS Service on Account B

The problem is lambda is not allowed to make cross-account API calls unless

1. Lambda assumes to another role from Account B internally
2. Account B grant this lambda role permissions.

Now, if we look at CallAwsServiceCrossRegion() current implementation, it does NOT have any options that allow users to specify any roles or credentials(this makes sense as it's not designed for cross-account calls).

Current Implementations

It should just work but in fact lambda execution role can't make cross-account API calls.

Proposed Solution - Role Assumpton

I think for cross-account API calls, always assuming the target account role for API calls makes sense to me. CallAwsServiceCrossRegion() currently just doesn't allow that.

[t3yamoto]

@pahud

Thank you for the detailed explanation and diagrams! Your understanding is correct.

In my PR (#35472), I implemented what you described in the "Proposed Solution - Role Assumption" diagram. The Lambda handler internally assumes the role specified in the awsSdkCredentials property before making AWS API calls.