Bug: Invoke failed error=ErrAgentNameCollision

[exodus-justinz]

Description:

sam local invoke fails with the following error Invoke failed error=ErrAgentNameCollision when using Cloudwatch Synthetic lambda layer, but works fine without it.

Steps to reproduce:

Run sam build then sam local invoke with the following files

index.js

exports.handler = async (event, context) => {
  console.log("Success");
};

template.yaml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Sample SAM Template for testing Synthetics NodeJS canary locally

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      Description: Local Synthetics NodeJS Dev Canary
      Handler: index.handler
      Runtime: nodejs18.x
      Layers:
        - arn:aws:lambda:us-east-1:378653112637:layer:Synthetics:44

Observed result:

> sam local invoke --debug
2024-06-03 10:52:35,246 | No config file found in this directory.                                                                                                    
2024-06-03 10:52:35,248 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/Users/<USER>/github/node-test/samconfig.toml'           
2024-06-03 10:52:35,249 | Config file location: /Users/<USER>/github/node-test/samconfig.toml                                                                      
2024-06-03 10:52:35,249 | Config file '/Users/<USER>/github/node-test/samconfig.toml' does not exist                                                               
2024-06-03 10:52:35,250 | Using SAM Template at /Users/<USER>/github/node-test/.aws-sam/build/template.yaml                                                        
2024-06-03 10:52:35,267 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/Users/<USER>/github/node-test/samconfig.toml'           
2024-06-03 10:52:35,268 | Using config file: samconfig.toml, config environment: default                                                                             
2024-06-03 10:52:35,268 | Expand command line arguments to:                                                                                                          
2024-06-03 10:52:35,268 | --template_file=/Users/<USER>/github/node-test/.aws-sam/build/template.yaml --no_event                                                   
--layer_cache_basedir=/Users/<USER>/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1                                            
2024-06-03 10:52:35,269 | local invoke command is called                                                                                                             
2024-06-03 10:52:35,270 | No Parameters detected in the template                                                                                                     
2024-06-03 10:52:35,282 | Sam customer defined id is more priority than other IDs. Customer defined id for resource Function is Function                             
2024-06-03 10:52:35,282 | 0 stacks found in the template                                                                                                             
2024-06-03 10:52:35,283 | No Parameters detected in the template                                                                                                     
2024-06-03 10:52:35,291 | Sam customer defined id is more priority than other IDs. Customer defined id for resource Function is Function                             
2024-06-03 10:52:35,291 | 1 resources found in the stack                                                                                                             
2024-06-03 10:52:35,292 | Found Serverless function with name='Function' and CodeUri='Function'                                                                      
2024-06-03 10:52:35,292 | --base-dir is not presented, adjusting uri Function relative to /Users/<USER>/github/node-test/.aws-sam/build/template.yaml              
2024-06-03 10:52:35,302 | Found one Lambda function with name 'Function'                                                                                             
2024-06-03 10:52:35,302 | Invoking index.handler (nodejs18.x)                                                                                                        
2024-06-03 10:52:35,303 | No environment variables found for function 'Function'                                                                                     
2024-06-03 10:52:35,303 | Loading AWS credentials from session with profile 'None'                                                                                   
2024-06-03 10:52:37,319 | Resolving code path. Cwd=/Users/<USER>/github/node-test/.aws-sam/build, CodeUri=/Users/<USER>/github/node-test/.aws-sam/build/Function 
2024-06-03 10:52:37,320 | Resolved absolute path to code is /Users/<USER>/github/node-test/.aws-sam/build/Function                                                 
2024-06-03 10:52:37,321 | Resolving code path. Cwd=/Users/<USER>/github/node-test/.aws-sam/build, CodeUri=/Users/<USER>/github/node-test/.aws-sam/build/Function 
2024-06-03 10:52:37,321 | Resolved real code path to /Users/<USER>/github/node-test/.aws-sam/build/Function                                                        
2024-06-03 10:52:37,322 | Code /Users/<USER>/github/node-test/.aws-sam/build/Function is not a zip/jar file                                                        
2024-06-03 10:52:37,322 | arn:aws:lambda:us-east-1:378653112637:layer:Synthetics:44 is already cached. Skipping download                                             
2024-06-03 10:52:41,510 | Local image is up-to-date                                                                                                                  
2024-06-03 10:52:41,535 | Checking free port on 127.0.0.1:8346                                                                                                       
2024-06-03 10:52:41,547 | Using local image: samcli/lambda-nodejs:18-x86_64-f71ec5643fe0df391046c2029.                                                               
                                                                                                                                                                     
2024-06-03 10:52:41,548 | Mounting /Users/<USER>/github/node-test/.aws-sam/build/Function as /var/task:ro,delegated, inside runtime container                      
2024-06-03 10:52:41,821 | Starting a timer for 3 seconds for function 'Function'                                                                                     
2024-06-03 10:52:41,824 | Getting lock for the key localhost-8346                                                                                                    
2024-06-03 10:52:41,826 | Waiting to retrieve the lock (localhost-8346) to start invocation                                                                          
START RequestId: 1acade56-67bd-4275-bccc-6eb08a6a7e0e Version: $LATEST
03 Jun 2024 02:52:41,965 [ERROR] (rapid) Init failed error=signal: segmentation fault InvokeID=
03 Jun 2024 02:52:41,969 [ERROR] (rapid) Invoke failed error=ErrAgentNameCollision InvokeID=c969ec12-b3a9-44ec-9c74-b0365b2d1212
03 Jun 2024 02:52:41,971 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-06-03 10:52:41,974 | Failed to deserialize response from RIE, returning the raw response as is                                                                  

2024-06-03 10:52:42,988 | [Container state] OOMKilled False                                                                                                          
2024-06-03 10:52:43,106 | Cleaning all decompressed code dirs                                                                                                        
2024-06-03 10:52:43,107 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                 
2024-06-03 10:52:43,172 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                 
2024-06-03 10:52:43,173 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'f8499d24-991b-41e5-a3c6-43cfd2a4ea63', 'installationId':                     
'ce33ef12-8739-473b-8579-37f993e80fae', 'sessionId': 'e03186b6-3442-424c-b1bf-f574d1eb33a5', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.12.3',      
'samcliVersion': '1.116.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes':     
{'projectType': 'CFN', 'gitOrigin': None, 'projectName': 'ae9f89a5516b52de1ce5b3595bf68531c9737250c891d8f618e517db18487386', 'initialCommit': None}, 'duration':     
7839, 'exitReason': 'success', 'exitCode': 0}}]}                                                                                                                     
2024-06-03 10:52:43,173 | Unable to find Click Context for getting session_id.                                                                                       
2024-06-03 10:52:43,177 | Sending Telemetry: {'metrics': [{'events': {'requestId': '48844173-9b5e-48f4-8b5f-3d667ec8a021', 'installationId':                         
'ce33ef12-8739-473b-8579-37f993e80fae', 'sessionId': 'e03186b6-3442-424c-b1bf-f574d1eb33a5', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.12.3',      
'samcliVersion': '1.116.0', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value':       
'.toml', 'thread_id': 'c3b2bcdafa85498fb06726c5b1535eef', 'time_stamp': '2024-06-03 02:52:35.248', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 
'event_value': '.toml', 'thread_id': 'a373f5c179504c6583cbd7a293fb38d8', 'time_stamp': '2024-06-03 02:52:35.268', 'exception_name': None}]}}}]}                      
2024-06-03 10:52:43,907 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)           
2024-06-03 10:52:43,913 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)           
2024-06-03 10:52:43,915 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                 
2024-06-03 10:52:43,916 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId': 'b6527113-a8ba-4362-9b36-55ce5dc422ef', 'installationId':                  
'ce33ef12-8739-473b-8579-37f993e80fae', 'sessionId': 'e03186b6-3442-424c-b1bf-f574d1eb33a5', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.12.3',      
'samcliVersion': '1.116.0', 'runtimes': ['nodejs18.x']}}]}                                                                                                           
2024-06-03 10:52:44,419 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)   

Expected result:

> sam local invoke
Invoking index.handler (nodejs18.x)                                                                                                                                  
Local image is up-to-date                                                                                                                                            
Using local image: public.ecr.aws/lambda/nodejs:18-rapid-x86_64.                                                                                                     
                                                                                                                                                                     
Mounting /Users/<USER>/github/node-test/.aws-sam/build/Function as /var/task:ro,delegated, inside runtime container                                                
START RequestId: d6129caa-891d-4702-a75d-dcb1f16a3a87 Version: $LATEST
2024-06-03T02:55:34.246Z        08f97fda-1155-4e35-be11-0658da88d6e0    INFO    Success
END RequestId: 08f97fda-1155-4e35-be11-0658da88d6e0
REPORT RequestId: 08f97fda-1155-4e35-be11-0658da88d6e0  Init Duration: 0.35 ms  Duration: 992.94 ms     Billed Duration: 993 ms Memory Size: 128 MB     Max Memory Used: 128 MB
null

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac (M1 arm64)
2. sam --version: SAM CLI, version 1.116.0
3. AWS region: us-east-1

# Paste the output of `sam --info` here
{
  "version": "1.116.0",
  "system": {
    "python": "3.12.3",
    "os": "macOS-14.5-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "26.1.1",
    "aws_cdk": "Not available",
    "terraform": "1.3.9"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Add --debug flag to command you are running

[hnnasit]

Hi @exodus-justinz, thanks for reporting the issue. I am able to reproduce the error. The error seems to be coming from lambda RIE which emulates the lambda invocation. Let me check with the lambda RIE team about this and get back.

[hnnasit]

I checked with the RIE team and it seems like the issue is possibly coming from registration of the extension in the layer itself. The extension in theory is able to tell if it's running inside a lambda or an extension but it's possible the layer is registering the extension multiple times causing the AgentNameCollision error. I also noticed the previous version of the layer arn:aws:lambda:us-east-1:378653112637:layer:Synthetics:24did not give this issue and started from the version 25. I tried deploying the project using sam sync and invoking the lambda using sam remote invoke and it was working fine. You could try testing it on the cloud as well. Not sure if we can do anything on SAM CLI side as the issue is originating from the layer during emulation.

[exodus-justinz]

thanks for looking into this. I tried updating the layer and it was able to run successfully. closing this issue 🙏

> sam local invoke
Invoking index.handler (nodejs16.x)                                                                                                                           
arn:aws:lambda:us-east-1:378653112637:layer:Synthetics:24 is already cached. Skipping download                                                                
Local image is up-to-date                                                                                                                                     
Using local image: samcli/lambda-nodejs:16-x86_64-f9fd7a7a463f7b9752563cbd6.                                                                                  
                                                                                                                                                              
Mounting /Users/<USER>/github/node-test/.aws-sam/build/Function as /var/task:ro,delegated, inside runtime container                                         
START RequestId: 9ae3952c-6420-448a-9ecd-f2dfdb19364d Version: $LATEST
2024-06-10T02:29:56.509Z        a43dbaba-f562-4165-9a7b-7ab1b046e884    INFO    Success
END RequestId: a43dbaba-f562-4165-9a7b-7ab1b046e884
REPORT RequestId: a43dbaba-f562-4165-9a7b-7ab1b046e884  Init Duration: 0.50 ms  Duration: 939.05 ms     Billed Duration: 940 ms Memory Size: 128 MB     Max Memory Used: 128 MB
null

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[billyjbryant]

I am getting this error when using a Lambda Function with the arn:aws:lambda:us-west-2:345057560386:layer:AWS-Parameters-and-Secrets-Lambda-Extension:4 Lambda Layer.

2024-09-27 11:29:35,822 | Starting a timer for 3 seconds for function 'TokenRotateLambda'                                                                                                               
2024-09-27 11:29:35,823 | Getting lock for the key localhost-8578                                                                                                                                       
2024-09-27 11:29:35,823 | Waiting to retrieve the lock (localhost-8578) to start invocation                                                                                                             
START RequestId: 896f2ed8-7156-42b2-a99f-d911102dac31 Version: $LATEST
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 PARAMETERS_SECRETS_EXTENSION_LOG_LEVEL is debug. Log level set to debug.
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG PARAMETERS_SECRETS_EXTENSION_CACHE_ENABLED is true.
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG PARAMETERS_SECRETS_EXTENSION_CACHE_SIZE is 1000 objects.
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG SECRETS_MANAGER_TTL is 5m0s
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG SSM_PARAMETER_STORE_TTL is 5m0s.
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG SECRETS_MANAGER_TIMEOUT_MILLIS is 0s.
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG SSM_PARAMETER_STORE_TIMEOUT_MILLIS is 0s.
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG PARAMETERS_SECRETS_EXTENSION_MAX_CONNECTIONS is 3.
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG PARAMETERS_SECRETS_EXTENSION_HTTP_PORT is 2773.
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 INFO Systems Manager Parameter Store and Secrets Manager Lambda Extension 1.0.103
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 DEBUG Creating a new cache with size 1000
[AWS Parameters and Secrets Lambda Extension] 2024/09/27 18:29:36 INFO Serving on port 2773
qemu-x86_64-static: QEMU internal SIGSEGV {code=MAPERR, addr=0x20}
27 Sep 2024 18:29:37,634 [ERROR] (rapid) Init failed InvokeID= error=Runtime exited with error: signal: segmentation fault
27 Sep 2024 18:29:37,656 [ERROR] (rapid) Invoke failed error=ErrAgentNameCollision InvokeID=73a23bf3-065d-4eda-be06-997a4006babb
27 Sep 2024 18:29:37,659 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-09-27 11:29:37,502 | Failed to deserialize response from RIE, returning the raw response as is                                                                                                     

2024-09-27 11:29:38,523 | [Container state] OOMKilled False                                                                                                                                             
2024-09-27 11:29:38,784 | Cleaning all decompressed code dirs                                                                                                                                           
2024-09-27 11:29:38,787 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                                    
2024-09-27 11:29:38,789 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                                    
2024-09-27 11:29:38,790 | Unable to find Click Context for getting session_id.                                                                                                                          
2024-09-27 11:29:38,792 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics   

sam --info
{
  "version": "1.125.0",
  "system": {
    "python": "3.8.20",
    "os": "macOS-14.6.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "5.0.3",
    "aws_cdk": "2.160.0 (build 7a8ae02)",
    "terraform": "1.5.7"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}