Since around Neovim 0.5, Lua was added as an alternative configuration language to Vimscript. Vimscript isn’t bad but I think anyone who has written Vimscript will tell you that Lua is an easier language to learn. Fennel is a Lisp that compiles to Lua which means you can also use it in Neovim (sort of). Every now and then, there’s a post that comes up on hn about Lisp. These posts were my original inspiration for learning Fennel. I’ve dabbled a bit with Fennel before and I’m back at it again. Specifically, I want to create my own Fennel adapter plugin for Neovim.

There’s currently three implementations of Fennel for Neovim that I know of.

• Olical/aniseed
• rktjmp/hotpot.nvim
• udayvir-singh/tangerine.nvim

All of these plugins basically do the same thing. They add Fennel as a first class language to Neovim which includes features such as writing config files in Fennel, a built in repl, and automatic compilation.

I’ve been exploring the source code for Hotpot and Tangerine. Both plugins use different implementations to achieve more or less the same feature set. Mostly, I’m interested in finding strategies for automatic compilation of Fennel to Lua.

Hotpot

Hotpot comes pretty close to how the Fennel docs recommend implementing Fennel support. Fennel provides some builtin functions that allow you to automatically source Fennel files. It does this by adding a package loader specifically for Fennel files.

When you require a module in Lua, the resolution algorithm uses package loaders to look for the file you want. There’s 5 of them that are built-in to Neovim:

1. package.preload - predefined package locations stored in a table
2. vim._load_package - Neovim’s run-time path
3. package.path - default paths to look through for Lua modules
4. package.cpath - default paths to look through for C modules
5. all-in-one - tries looking everywhere since everything else failed

What Hotpot (and Fennel) do is they add a new loader that can resolve Fennel files. For Hotpot, this loader is injected as the first loader which means that it gives the highest priority to resolving Fennel files over Lua files whenever require is called.

In order to speed up startup time, Hotpot implements a cache. This cache includes 2 parts:

1. Compiled Lua files
2. The cache index which keeps track of when these Lua files should be updated

The cache index is basically a map of module names to compiled file location. When a module is required, Hotpot will check the index to see if the module has already been compiled. Hotpot then diffs the write time of the original Fennel file against it’s compiled Lua counterpart to see if it should recompile the Fennel file.

The cache index is a bytecode file that is encoded using Neovim’s builtin message pack functionality. This means that it can be loaded very quickly.

So to summarize, Hotpot injects a package loader that can resolve Fennel files. This package loader implements a caching system that can automatically recompile Fennel to Lua anytime the source Fennel code has changed.

The main difference between Hotpot and Fennel’s default package loader implementation is that Fennel doesn’t implement this caching functionality. For Hotpot, having a cache means that Fennel files aren’t recompiled on every startup unless absolutely necessary, thereby improving overall startup time.

Tangerine

In comparison, Tangerine has a more simple approach to keeping compiled Lua files up to date. Tangerine implements a series of hooks into the Neovim editing life cycle. The hooks attach to the VimEnter and BufWritePost events. These events will trigger an autocommand that diffs all Fennel source files against their compiled Lua counterparts and recompiles the ones that are stale. Every Fennel source file’s timestamp is checked whenever the hook is run.

On start, Tangerine injects the path that it uses for compiled Lua files into the package.path. When using require to load a module, Lua will also look through Tangerine’s injected paths to resolve the module.

To summarize, Tangerine doesn’t have a fancy caching system. It just diffs and re-compiles files whenever they’re edited in Neovim. This works pretty well since every file is checked whenever you start Neovim.

Brainstorming

Both of these implementations use interesting tactics to keep Neovim’s startup fast and compiled Lua files up to date. The thing I don’t like about both of them though is that there’s always some startup time associated with checking the validity of compiled Lua files. Currently, I’m opting to use a Makefile that I have to manually run after editing Fennel files. While I don’t get the seamless experience of:

edit -> restart editor -> see changes

when I edit my Neovim configs, I think it’s a small price to pay to save a few milliseconds on every startup where I’m not editing my configs.

One fun thing I was able to do was to use Neovim itself as a Fennel compiler. I added this line at the top of my init.lua that checks if I’m running Neovim in Fennel compiler mode.

if vim.env["FENNEL_COMPILE"] then
    require "bulb"
    return
end

The bulb module loads the fennel.lua compiler API. I only load bulb and fennel.lua when Neovim is in Fennel compiler mode so that I don’t slow down my startup times when I’m using Neovim normally. bulb provides a user command called FnlCompile that compiles Fennel files. It’s a simple function that just loads a file as a byte stream, compiles it, then writes the output.

vim.api.nvim_create_user_command("FnlCompile", function(t)
    local in_path, out_path = unpack(vim.fn.split(t.args, " "))
    assert(in_path, "missing input path")
    assert(out_path, "missing output path")

    local stream = open_stream(in_path)
    local out = fennel.compileStream(stream, { ["compiler-env"] = _G })

    local file = assert(io.open(out_path, "w"))
    file:write(out)
    file:close()
end, { nargs = 1 })

Finally, I can run Neovim in headless mode to compile Fennel files.

FENNEL_COMPILE=true nvim --headless -c 'FnlCompile fnl/enoch/helpers.fnl lua/enoch/helpers.lua' +q

There’s two main benefits of running the Fennel compiler through Neovim. First, it allows my configs to be fully portable. I don’t have to rely on the host computer having an installed copy of Fennel. In addition, the compiler is always run through Neovim’s builtin LuaJIT which is much faster than the standard Lua 5.1 implementation.

Second, and more importantly, it allows me to use Neovim builtin functions in macros. Since Fennel version 1.0, the compiler is sandboxed by default meaning that builtin functions such as io, os, or any Neovim provided functions such as vim.api can’t be used in macros. But I couldn’t turn down the sweet sweet power of compile time madness so I just turned off compiler sandboxing. This isn’t very secure, but I’m the only person who will be editing my configs so I’m not too concerned.

I’m not sure where I’ll go next with my Fennel adventures. I’ll probably start on implementing a repl of sorts that’s similar to how Vimscript and Lua can be executed in Neovim’s command line. Ideally, I want to package up my Fennel compilation implementation into a Neovim plugin so perhaps I will eventually ditch the Makefile. It was really interesting learning about Lua’s module resolution algorithm and how hackable it is.