Since around Neovim 0.5, Lua was added as an alternative configuration language to Vimscript. Vimscript isn’t bad but I think anyone who has written Vimscript will tell you that Lua is an easier language to learn. Fennel is a Lisp that compiles to Lua which means you can also use it in Neovim (sort of). Every now and then, there’s a post that comes up on hn about Lisp. These posts were my original inspiration for learning Fennel. I’ve dabbled a bit with Fennel before and I’m back at it again. Specifically, I want to create my own Fennel adapter plugin for Neovim.
There’s currently three implementations of Fennel for Neovim that I know of.
All of these plugins basically do the same thing. They add Fennel as a first class language to Neovim which includes features such as writing config files in Fennel, a built in repl, and automatic compilation.
I’ve been exploring the source code for Hotpot and Tangerine. Both plugins use different implementations to achieve more or less the same feature set. Mostly, I’m interested in finding strategies for automatic compilation of Fennel to Lua.
Hotpot comes pretty close to how the Fennel docs recommend implementing Fennel support. Fennel provides some builtin functions that allow you to automatically source Fennel files. It does this by adding a package loader specifically for Fennel files.
When you require a module in Lua, the resolution algorithm uses package loaders
to look for the file you want.
There’s 5 of them
that are built-in to Neovim:
package.preload - predefined package locations stored in a tablevim._load_package - Neovim’s run-time pathpackage.path - default paths to look through for Lua modulespackage.cpath - default paths to look through for C modulesall-in-one - tries looking everywhere since everything else failedWhat Hotpot (and Fennel) do is they add a new loader that can resolve Fennel files.
For Hotpot, this loader is injected as the first loader
which means that it gives the highest priority to resolving Fennel files over Lua files
whenever require is called.
In order to speed up startup time, Hotpot implements a cache. This cache includes 2 parts:
The cache index is basically a map of module names to compiled file location. When a module is required, Hotpot will check the index to see if the module has already been compiled. Hotpot then diffs the write time of the original Fennel file against it’s compiled Lua counterpart to see if it should recompile the Fennel file.
The cache index is a bytecode file that is encoded using Neovim’s builtin message pack functionality. This means that it can be loaded very quickly.
So to summarize, Hotpot injects a package loader that can resolve Fennel files. This package loader implements a caching system that can automatically recompile Fennel to Lua anytime the source Fennel code has changed.
The main difference between Hotpot and Fennel’s default package loader implementation is that Fennel doesn’t implement this caching functionality. For Hotpot, having a cache means that Fennel files aren’t recompiled on every startup unless absolutely necessary, thereby improving overall startup time.
In comparison, Tangerine has a more simple approach to keeping compiled Lua
files up to date.
Tangerine implements a series of hooks
into the Neovim editing life cycle.
The hooks attach to the VimEnter and BufWritePost events.
These events will trigger an autocommand that diffs all Fennel source files against
their compiled Lua counterparts and recompiles the ones that are stale.
Every Fennel source file’s timestamp is checked whenever the hook is run.
On start, Tangerine injects the path that it uses for compiled Lua files
into the package.path.
When using require to load a module, Lua will also look through Tangerine’s injected paths to resolve the module.
To summarize, Tangerine doesn’t have a fancy caching system. It just diffs and re-compiles files whenever they’re edited in Neovim. This works pretty well since every file is checked whenever you start Neovim.
Both of these implementations use interesting tactics to keep Neovim’s startup fast and compiled Lua files up to date. The thing I don’t like about both of them though is that there’s always some startup time associated with checking the validity of compiled Lua files. Currently, I’m opting to use a Makefile that I have to manually run after editing Fennel files. While I don’t get the seamless experience of:
edit -> restart editor -> see changeswhen I edit my Neovim configs, I think it’s a small price to pay to save a few milliseconds on every startup where I’m not editing my configs.
One fun thing I was able to do was to use Neovim itself as a Fennel compiler.
I added this line at the top of my init.lua
that checks if I’m running Neovim in Fennel compiler mode.
if vim.env["FENNEL_COMPILE"] then
require "bulb"
return
endThe bulb module loads the fennel.lua compiler API.
I only load bulb and fennel.lua when Neovim is in Fennel compiler mode
so that I don’t slow down my startup times when I’m using Neovim normally.
bulb provides a user command called FnlCompile that compiles Fennel files.
It’s a simple function that just loads a file as a byte stream, compiles it,
then writes the output.
vim.api.nvim_create_user_command("FnlCompile", function(t)
local in_path, out_path = unpack(vim.fn.split(t.args, " "))
assert(in_path, "missing input path")
assert(out_path, "missing output path")
local stream = open_stream(in_path)
local out = fennel.compileStream(stream, { ["compiler-env"] = _G })
local file = assert(io.open(out_path, "w"))
file:write(out)
file:close()
end, { nargs = 1 })Finally, I can run Neovim in headless mode to compile Fennel files.
FENNEL_COMPILE=true nvim --headless -c 'FnlCompile fnl/enoch/helpers.fnl lua/enoch/helpers.lua' +qThere’s two main benefits of running the Fennel compiler through Neovim. First, it allows my configs to be fully portable. I don’t have to rely on the host computer having an installed copy of Fennel. In addition, the compiler is always run through Neovim’s builtin LuaJIT which is much faster than the standard Lua 5.1 implementation.
Second, and more importantly, it allows me to use Neovim builtin functions in macros.
Since Fennel version 1.0, the compiler is sandboxed by default meaning that
builtin functions such as io, os, or any Neovim provided functions such as vim.api can’t be used in macros.
But I couldn’t turn down the sweet sweet power of compile time madness
so I just turned off compiler sandboxing.
This isn’t very secure, but I’m the only person who will be editing my configs
so I’m not too concerned.
I’m not sure where I’ll go next with my Fennel adventures. I’ll probably start on implementing a repl of sorts that’s similar to how Vimscript and Lua can be executed in Neovim’s command line. Ideally, I want to package up my Fennel compilation implementation into a Neovim plugin so perhaps I will eventually ditch the Makefile. It was really interesting learning about Lua’s module resolution algorithm and how hackable it is.