-- Leo's gemini proxy
-- Connecting to tilde.pink:1965...
-- Connected
-- Sending request
-- Meta line: 20 text/gemini;
2024-02-29
A while back I was working a bit with the Neovim RPC, so this is an end to end review of the inner workings of RPC in Neovim. In particular I am focusing on what Neovim calls "hosts", the co-processes that execute code from various programming languages.
When Neovim forked Vim, authors decided to externalize certain features - namely the python/perl support. In Neovim these were moved to external processes, that communicate with the main Neovim process using the msgpack-rpc protocol. Communication happens via pipes between the two processes.
meta:
neovim git commit 6ab0876f51e8ff5debdff03b36508fe7279032b7
pynvim git commit 5f989dfc47d98bba9f98e5ea17bfbe4c995cb0b0
At a high level Neovim provides the ability to invoke programming languages from Vimscript code, or the command input. For example if you type the following command:
:python print('hello')
you should see some output (provided python and pynvim are installed).
The same applies to other languages. In my case perl is not installed so this gets me an error:
:perl print "hello"; E319: No "perl" provider found. Run ":checkhealth provider"
Writing plugins in python/perl is just an extension of these commands. In Neovim this causes the python host process to be spawned, and code is executed by passing the python code to remote process and getting a response back.
You can detect these processes in your system by checking the list of processes. For example in my system
1452 ? S 0:04 \_ nvim 1876 ? Ss 0:00 \_ python3 -c import sys; sys.path = [p for p in sys.path if p != ""]; import neovim; neovim.start_host() script_host.py
this shows that nvim has a child process (python3) which was invoked with arguments to execute the neovim host (part of the pynvim library).
For more documentation on how to configure these hosts check the Neovim docs (:h provider.txt). There are default providers for python/perl/nodejs/ruby.
The entry point for remote processes in Neovim code is a vimscript plugin which can be found at runtime/autoload/remote/host.vim. In there you will find various calls executed when the vimscript code is loaded. Here is the one for python3
194: call remote#host#Register('python3', '*',
195: \ function('provider#python3#Require'))
This triggers the registration of the python3 provider. The Register function is in the same file
5: " Register a host by associating it with a factory(funcref)
6: function! remote#host#Register(name, pattern, factory) abort
7: let s:hosts[a:name] = {'factory': a:factory, 'channel': 0, 'initialized': 0}
8: let s:plugin_patterns[a:name] = a:pattern
9: if type(a:factory) == type(1) && a:factory
10: " Passed a channel directly
11: let s:hosts[a:name].channel = a:factory
12: endif
13: endfunction
The first argument is the name of the provider (python3), the second is the wildcard used to identify plugins (*) and the third is a callback function which is called when the provider needs to be started. Triggering of the callback happens from the remote#host#Require function, but only when the provider is needed.
For the callback function code for python check runtime/autoload/provider/python3.vim
9: function! provider#python3#Require(host) abort 10: return v:lua.vim.provider.python.require(a:host) 11: endfunction
but this is only a vimscript shim that calls into Lua code (besides vimscript Neovim also runs lua internally). We can follow this up by looking inside runtime/lua/vim/provider/python.lua
101: function M.require(host)
102: -- Python host arguments
103: local prog = M.detect_by_module('neovim')
104: local args = {
105: prog,
106: '-c',
107: 'import sys; sys.path = [p for p in sys.path if p != ""]; import neovim; neovim.start_host()',
108: }
109:
110: -- Collect registered Python plugins into args
111: local python_plugins = vim.fn['remote#host#PluginsForHost'](host.name) ---@type any
112: ---@param plugin any
113: for _, plugin in ipairs(python_plugins) do
114: table.insert(args, plugin.path)
115: end
116:
117: return vim.fn['provider#Poll'](
118: args,
119: host.orig_name,
120: '$NVIM_PYTHON_LOG_FILE',
121: { ['overlapped'] = true }
122: )
123: end
In here we can see something that resembles a python process invocation and a snippet of python code that runs something called start_host.
The two remaining pieces in this function are
an invocation of remote#host#PluginsForHost
a call to provider#Poll
The call to Poll is the one that actually spawns the python process (see provider.vim)
1: " Common functions for providers
2:
3: " Start the provider and perform a 'poll' request
4: "
5: " Returns a valid channel on success
6: function! provider#Poll(argv, orig_name, log_env, ...) abort
7: let job = {'rpc': v:true, 'stderr_buffered': v:true}
8: if a:0
9: let job = extend(job, a:1)
10: endif
11: try
12: let channel_id = jobstart(a:argv, job)
13: if channel_id > 0 && rpcrequest(channel_id, 'poll') ==# 'ok'
14: return channel_id
15: endif
16: catch
after starting the process, it invokes an rpc request to a function called poll(). The poll() RPC works as a ping to check if the remote process is alive. If all works, the function returns a channel id which identifies the open RPC channel.
However this callback is only really called when needed, from the remote#host#Require function (back in host.vim):
32: " Get a host channel, bootstrapping it if necessary
33: function! remote#host#Require(name) abort
34: if !has_key(s:hosts, a:name)
35: throw 'No host named "'.a:name.'" is registered'
36: endif
37: let host = s:hosts[a:name]
38: if !host.channel && !host.initialized
39: let host_info = {
40: \ 'name': a:name,
41: \ 'orig_name': get(host, 'orig_name', a:name)
42: \ }
43: let host.channel = call(host.factory, [host_info])
44: let host.initialized = 1
45: endif
46: return host.channel
47: endfunction
Require function is called from different places, e.g. RegistrationCommands
112: function! s:RegistrationCommands(host) abort
113: " Register a temporary host clone for discovering specs
114: let host_id = a:host.'-registration-clone'
115: call remote#host#RegisterClone(host_id, a:host)
116: let pattern = s:plugin_patterns[a:host]
117: let paths = nvim_get_runtime_file('rplugin/'.a:host.'/'.pattern, 1)
118: let paths = map(paths, 'tr(resolve(v:val),"\\","/")') " Normalize slashes #4795
119: let paths = uniq(sort(paths))
120: if empty(paths)
121: return []
122: endif
123:
124: for path in paths
125: call remote#host#RegisterPlugin(host_id, path, [])
126: endfor
127: let channel = remote#host#Require(host_id)
128: let lines = []
129: let registered = []
130: for path in paths
131: unlet! specs
132: let specs = rpcrequest(channel, 'specs', path)
133: if type(specs) != type([])
134: " host didn't return a spec list, indicates a failure while loading a
135: " plugin
136: continue
137: endif
138: call add(lines, "call remote#host#RegisterPlugin('".a:host
139: \ ."', '".path."', [")
140: for spec in specs
141: call add(lines, " \\ ".string(spec).",")
142: endfor
143: call add(lines, " \\ ])")
144: call add(registered, path)
145: endfor
146: echomsg printf("remote/host: %s host registered plugins %s",
147: \ a:host, string(map(registered, "fnamemodify(v:val, ':t')")))
besides starting the host process, this function also handles individual plugin setup which we have not covered yet. Without going into further detail we can see there is an RPC call to a method called specs.
Another important location is the C function that calls providers in eval.c:
8755: typval_T eval_call_provider(char *provider, char *method, list_T *arguments, bool discard)
8756: {
8757: if (!eval_has_provider(provider)) {
8758: semsg("E319: No \"%s\" provider found. Run \":checkhealth provider\"",
8759: provider);
8760: return (typval_T){
8761: .v_type = VAR_NUMBER,
8762: .v_lock = VAR_UNLOCKED,
8763: .vval.v_number = 0
8764: };
8765: }
8766:
8767: char func[256];
8768: int name_len = snprintf(func, sizeof(func), "provider#%s#Call", provider);
and here is the python3 entry point C (funcs.c):
5707: /// "py3eval()" and "pyxeval()" functions (always python3)
5708: static void f_py3eval(typval_T *argvars, typval_T *rettv, EvalFuncData fptr)
5709: {
5710: script_host_eval("python3", argvars, rettv);
5711: }
This gives us an overview of how Neovim starts these processes and redirects calls to them. We should note here that this API is perfectly accessible to plugins, so you could write your provider with a bit of vimscript or lua to trigger registration.
For example you can call these as commands
remote#host#Require('python3') will start the python3 provider
remote#host#IsRunning('python3') will return 1 if it is already running
Meanwhile the python host side is handled by the pynvim library. This is the function being called when python is spawned. This is the same function we saw being passed to the python process:
33: def start_host(session: Optional[Session] = None) -> None: 34: """Promote the current process into python plugin host for Nvim. 35: 36: Start msgpack-rpc event loop for `session`, listening for Nvim requests 37: and notifications. It registers Nvim commands for loading/unloading 38: python plugins. 39: 40: The sys.stdout and sys.stderr streams are redirected to Nvim through 41: `session`. That means print statements probably won't work as expected 42: while this function doesn't return. 43: 44: This function is normally called at program startup and could have been 45: defined as a separate executable. It is exposed as a library function for 46: testing purposes only.
I wont list it all here, but after doing plugin initialization and nvim version checks it calls an event loop
80: def start(self, plugins): 81: """Start listening for msgpack-rpc requests and notifications.""" 82: self.nvim.run_loop(self._on_request, 83: self._on_notification, 84: lambda: self._load(plugins), 85: err_cb=self._on_async_err)
which continues in api/nvim.py
224: def run_loop( 225: self, 226: request_cb: Optional[Callable[[str, List[Any]], Any]], 227: notification_cb: Optional[Callable[[str, List[Any]], Any]], 228: setup_cb: Optional[Callable[[], None]] = None, 229: err_cb: Optional[Callable[[str], Any]] = None 230: ) -> None:
and eventually calls the msgpack-rpc loop
145: def run(self, 146: request_cb: Callable[[str, List[Any]], None], 147: notification_cb: Callable[[str, List[Any]], None], 148: setup_cb: Optional[Callable[[], None]] = None) -> None:
I wont detail how the msgpack-rpc protocols work. It is general purpose RPC protocol that supports both synchronous request-response as well as notifications.
Most hosts seem to provide the following RPC methods
specs
poll
Here is the python code that registers them
60: self._request_handlers = {
61: 'poll': lambda: 'ok',
62: 'specs': self._on_specs_request,
63: 'shutdown': self.shutdown
64: }
poll is used as a ping function (it just returns the string ok). specs is a function called by Neovim to get a list of specification i.e. all the plugins and the functions/commands they provide.
Each host program can provide individual plugins, loaded from files. The high level process is described in remote_plugin.txt.
For most remote hosts plugin setup works as follows
you place a file inside the runtimepath (under rplugin/host-name) e.g. rplugin/python/example.py.
you or Neovim calls UpdateRemotePlugins
the host process is started and the specs(path) RPC is used (based on the wildcard from Register) to get a plugin list
Neovim stores this spec in a file called rplugin.vim
plugins/hosts are only loaded on demand based on data from rplugin.vim
This means that the lifecycle of your host process can require it to be spawned once to search for plugins, but once rplugin.vim is in place it should only be started if needed.
I did not get far in understanding all the plugin semantics, but the best place to look seems to be the vimscript remote code. For example plugin registration, the format for rplugin.vim and the format for specs return value:
56: " Example of registering a Python plugin with two commands (one async), one
57: " autocmd (async) and one function (sync):
58: "
59: " let s:plugin_path = expand('<sfile>:p:h').'/nvim_plugin.py'
60: " call remote#host#RegisterPlugin('python', s:plugin_path, [
61: " \ {'type': 'command', 'name': 'PyCmd', 'sync': 1, 'opts': {}},
62: " \ {'type': 'command', 'name': 'PyAsyncCmd', 'sync': 0, 'opts': {'eval': 'cursor()'}},
63: " \ {'type': 'autocmd', 'name': 'BufEnter', 'sync': 0, 'opts': {'eval': 'expand("<afile>")'}},
64: " \ {'type': 'function', 'name': 'PyFunc', 'sync': 1, 'opts': {}}
65: " \ ])
66: "
67: " The third item in a declaration is a boolean: non zero means the command,
68: " autocommand or function will be executed synchronously with rpcrequest.
69: function! remote#host#RegisterPlugin(host, path, specs) abort
the rest is covered in the RegistrationCommands function (listed earlier). One aspect to node is that function returns code, which is then stored in rplugin.vim:
156: function! remote#host#UpdateRemotePlugins() abort
157: let commands = []
158: let hosts = keys(s:hosts)
159: for host in hosts
160: if has_key(s:plugin_patterns, host)
161: try
162: let commands +=
163: \ ['" '.host.' plugins']
164: \ + s:RegistrationCommands(host)
165: \ + ['', '']
166: catch
167: echomsg v:throwpoint
168: echomsg v:exception
169: endtry
170: endif
171: endfor
172: call writefile(commands, g:loaded_remote_plugins)
173: echomsg printf('remote/host: generated rplugin manifest: %s',
174: \ g:loaded_remote_plugins)
175: endfunction
while the setup of the Neovim commands/functions is done inside autoload/remote/define.vim.
-- Response ended
-- Page fetched on Sun May 19 09:26:40 2024