Spark integration fails due to web sockets not being supported

[mkjois]

I have a working hello-world-ish local Spark server, but when trying to run it on Lambda it throws the UnsupportedOperationException when the embedded server tries to configure web sockets.

Here's the exact error I'm getting:

START RequestId: <request_id_redacted> Version: $LATEST
[main] INFO <class_redacted> - Initializing routes
[Thread-0] ERROR spark.Spark - ignite failed
java.lang.UnsupportedOperationException
	at com.amazonaws.serverless.proxy.spark.embeddedserver.LambdaEmbeddedServer.configureWebSockets(LambdaEmbeddedServer.java:65)
	at spark.Service.lambda$init$2(Service.java:502)
	at java.lang.Thread.run(Thread.java:745)
END RequestId: <request_id_redacted>
REPORT RequestId: <request_id_redacted>	Duration: 1501.41 ms	Billed Duration: 1600 ms 	Memory Size: 512 MB	Max Memory Used: 54 MB	
RequestId: <request_id_redacted> Process exited before completing request

I can't see a current workaround other than some crazy reflection magic. I guess there are some potential fixes like:

• change LambdaEmbeddedServer.configureWebSockets() to not throw an exception
• use a custom initExceptionHandler on the Spark.Service instance to ignore the exception

Thoughts?

[sapessi]

API Gateway and Lambda do not support web sockets so you wouldn't be able to receive requests from clients. What are you trying to accomplish with web sockets?

[mkjois]

I'm not using web sockets. But the Spark service always calls configureWebSockets() on startup, which throws an exception, which kills the process.

[sapessi]

Can you show me the code for your LambdaHandler class?

[sapessi]

The one catch with spark is that you need to define the resources after you have initialized the handler: https://github.com/awslabs/aws-serverless-java-container/blob/master/samples/spark/pet-store/src/main/java/com/amazonaws/serverless/sample/spark/LambdaHandler.java#L45

[mkjois]

Sure. Thanks for the quick reply btw.

public class LambdaHandler
        implements RequestHandler<AwsProxyRequest, AwsProxyResponse> {

    private SparkLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;

    @Override
    public AwsProxyResponse handleRequest(AwsProxyRequest input, Context context) {

        if (handler == null) {
            try {
                handler = SparkLambdaContainerHandler.getAwsProxyHandler();
                HelloSparkServer.start();
            } catch (ContainerInitializationException e) {
                throw new RuntimeException("Failed to initialize server container", e);
            }
        }

        return handler.proxy(input, context);
    }
}

public class HelloSparkServer {

    private static final Logger LOG = LoggerFactory.getLogger(HelloSparkServer.class);

    // not used on AWS Lambda
    public static void main(String[] args) {

        int port = getPort();
        if (port >= 0) {
            port(port);
        }

        start();
        LOG.info(format("Listening on port %d", port()));
    }

    public static void start() {

        LOG.info("Initializing routes");

        path("/api", () -> {

            path("/v1", () -> {

                before("/*", (req, res) -> {
                    LOG.info(format("%s %s %s",
                            req.ip(),
                            req.requestMethod(),
                            req.url()));
                });

                get("/things/:id/prop",
                        (req, res) -> format("Hello thing %s prop", req.params(":id")));
                post("/things/:id/prop/update",
                        (req, res) -> format("Hello thing %s prop update", req.params(":id")));

                notFound((req, res) -> "Not found");
                internalServerError((req, res) -> "Internal server error");

                after("/*", (req, res) -> res.type("application/json"));
            });
        });
    }

    private static int getPort() {
        String portFromEnv = System.getenv("FM_API_PORT");
        return portFromEnv == null ? -1 : Integer.parseInt(portFromEnv);
    }
}

[sapessi]

Thanks, I will test this. Which version of spark are you using?

[mkjois]

2.6.0

I'm looking right now to see if that's the issue. Their changelog is here: http://sparkjava.com/news#spark-kotlin-released

[sapessi]

Thanks. I'll test this and publish a fix in the next few days. Flying around right now.

[sapessi]

Meanwhile, could you try with spark 2.5.3 - the library is tested with this version. If you want to join the Gitter chat room I'll probably post updates there as I work on this.

[mkjois]

Getting a strange NPE with Spark 2.5.3:

Error while handling request: nulljava.lang.NullPointerException
	at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.doFilter(AwsLambdaServletContainerHandler.java:179)
	at com.amazonaws.serverless.proxy.spark.SparkLambdaContainerHandler.handleRequest(SparkLambdaContainerHandler.java:158)
	at com.amazonaws.serverless.proxy.spark.SparkLambdaContainerHandler.handleRequest(SparkLambdaContainerHandler.java:57)
	at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:119)
	at <package_redacted>.LambdaHandler.handleRequest(LambdaHandler.java:28)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:456)
	at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:375)
	at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:1139)
	at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:278)
	at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:62)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)

It's hard to tell but it looks like maybe filterChainManager isn't being set? Looking through the code, it looks like the implementation of SparkLambdaContainerHandler.handleRequest() doesn't set the httpServletContext with the setter here, which sets filterChainManager. I'm not certain though...have you seen this before?

[sapessi]

Yes, it looks like it's a bug with the latest release. I think FilterChainManager was introduced in 0.5. Try and use 0.4 of the library - I'll include this fix in the TT. Sorry about the churn on this.

[sapessi]

I've found the issue and have a fix - I will create some unit tests for this over the weekend and push it. I'll let you know when it's there and you can start using 0.6-SNAPSHOT.

The fix is very simple. I needed to call the

super.handleRequest(httpServletRequest, httpServletResponse, lambdaContext);

in the spark handler. If you want, you can add it yourself here.