Pygls crashes with Vim LSP plugin

[itsmeknt]

Hello! I am testing the my Pygls server on various plugins. It seems to work find for Emacs and Helix, but when I tried it with Vim it crashes.

For Pygls, I am using the branch tagged v2.0.0a3
For Vim, I am using this library for LSP plugin: https://github.com/yegappan/lsp?tab=readme-ov-file

My .vimrc file looks like this:

packadd lsp

call LspAddServer([#{
        \    name: 'toxidious',
        \    filetype: ['python', 'toxchat'],
	\    path: 'nc',
	\    args: ['localhost', '8282']
	\  }])

(my language server is called toxidious and runs on a local TCP port 8282.)

I initiated a ls.worksplace_apply_edit_async() request to the editor, which crashes with the following error message:

Unable to deserialize message                                                                                                                                                                                                                                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                         
  File "/home/kevin/workspace/toxidious/venv/lib/python3.13/site-packages/pygls/protocol/json_rpc.py", line 434, in structure_message                                                                                                                                                                                                                                      
    self._result_types.pop(data["id"]) or JsonRPCResponseMessage                                                                                                                                                                                                                                                                                                           
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                     
KeyError: 15036         

For convenience, the link to the code is here

Poking around the code in the debugger, I notice that:
self._result_types = {'434205ff-56f1-4ca1-9b16-662bf9e1e233': <class 'lsprotocol.types.ApplyWorkspaceEditResponse'>}

But Vim responded with this message:

{                                                                                                                                                                                                                                                                                                                                                                          
  "id": 434205,                                                                                                                                                                                                                                                                                                                                                            
  "jsonrpc": "2.0",                                                                                                                                                                                                                                                                                                                                                        
  "result": {                                                                                                                                                                                                                                                                                                                                                              
    "applied": true                                                                                                                                                                                                                                                                                                                                                        
  }                                                                                                                                                                                                                                                                                                                                                                        
}     

It seems like Vim parses the ID incorrectly, keeping only the first 6 digits and removing the first non-digit character it sees. Additionally, the ID data type is an integer, not a string.

It seems like Vim only expects integer IDs.

When I changed the ID to use integer IDs here and typecasted the data["id"] to str(data["id"]) here, Vim correctly applies the workspace edit. However, pygls complains that it doesn't recognize the response id, so this probably broke something else.

I'd like to have my language server support Vim with any of their plug ins. Is there anything Pygls can do to fix this, or is this simply a Vim LSP plugin implementation error?

[itsmeknt]

It seems like it is indeed a Vim LSP plugin implementation error. The official specs allow id to be either int or strings but the LSP plugins seems to only handle int.

I have raised an issue in their github.

I can close this issue, but I wonder if there is any discussion to be had for pygls to somehow be flexible enough to cooperate with clients that only work with int IDs? Would it make sense to have a flag force_int_ids_in_requests that we can turn on or off in Pygls?

[tombh]

Nice debugging. Could you link to the issue you made on the Vim Github? And have you tried yout code with Neovim too?

[itsmeknt]

Here is the issue I reported: yegappan/lsp#635

I switched to a different LSP plug-in for Vim (CoC.nvim) and that one worked fine. I also tried other editors like Emacs (with Eglot plugin) and Helix and they worked fine too.

CoC.nvim also works for Neovim too. I think this is just a one-off bug from the yegappan/lsp plug-in since they're a smaller and newer project, but more mature LSP plug-ins will handle the id field correctly.

[tombh]

You write great issues 🤓

So when you say "Vim LSP plugin", you're referring to https://github.com/yegappan/lsp specifically? I ask because I was of the understanding that https://github.com/mattn/vim-lsp-settings is the current "blessed" plugin. Our very own Alex Carney develops a fantastic Pygls-based LSP server, esbonio, I often refer to his docs for current best practices, like setting up LSPs in various editors, etc. Here's esbonio's page for Vim for example: https://docs.esbon.io/en/esbonio-language-server-v0.16.4/lsp/getting-started.html?editor=vim-vim-lsp And Neovim actually has native support for LSP, even without a plugin.

So I'd be curious if any of these "blessed" methods failed with your LSP.

[itsmeknt]

Thanks! And yes I was referring to the yegappan/lsp plugin that specifically had this issue. I just tried prabirshrestha/vim-lsp and that one worked. I never use vim to be honest, I wanted to just quickly try out a plug in to test my server.

Thanks for the doc link! I'll bookmark it when I want to set up other editors to test. Getting Vim working was honestly a lot harder than I anticipated.

[alcarney]

I can close this issue, but I wonder if there is any discussion to be had for pygls

Definitely 😄

I initiated a ls.worksplace_apply_edit_async() request to the editor, which crashes with the following error message:

Unable to deserialize message                                                                                                                                                                                                                                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                         
  File "/home/kevin/workspace/toxidious/venv/lib/python3.13/site-packages/pygls/protocol/json_rpc.py", line 434, in structure_message                                                                                                                                                                                                                                      
    self._result_types.pop(data["id"]) or JsonRPCResponseMessage                                                                                                                                                                                                                                                                                                           
   ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                     
KeyError: 15036         

First of all, we shouldn't crash if we receive an unexpected id! I guess .pop(), unlike .get() throws by default! 😅

Would it make sense to have a flag force_int_ids_in_requests that we can >turn on or off in Pygls?

Maybe not a flag... but perhaps we should allow you to pass a function to override the default id generation method. The trick I guess, is figuring out what info we'd need to pass that function to ensure it can pick ids that don't conflict with previous ones (or do for that matter, if you wanted to simulate a misbehaving client/server in your tests)

[tombh]

First of all, we shouldn't crash if we receive an unexpected id! I guess .pop(), unlike .get() throws by default! 😅

Agreed! Although I'm not yet convinced by this case, as I don't see the reasoning for yegappan/lsp to truncate and cast the ID from string to integer. I assume it's trying to be flexible but I think it'd actually be better to just error if it receives something it can't handle. I'm following the issue there to see what they say.

I think for now it's okay that pygls only supports string IDs, as they are formally described in the spec.

[itsmeknt]

FYI crashing might not have been the right word choice by me. At first I thought the entire Pygls process died, but it actually continues to run... only the coroutine that serves the request is affected (all my endpoints are async). It raises an exception and terminates. But the process itself is still running.