nix: tinysearch

tinysearch is a static site-embedded search engine written in Rust by Matthias Endler. It consumes a fuse index (in my case, generated by Zola) and emits a WASM binary containing the index and search engine that can be loaded by the site and hooked up to an input element (see the search bar above).

The functionality is great, but it's incompatible with nix in its current state, as the tinysearch generator requires network access. In particular, it:

• generates a storage blob in its preferred format from the Zola-emitted search index
• generates an "engine" crate containing the search logic, which also embeds the storage blob
• calls wasm-pack to build the engine crate
  • which calls wasm-bindgen
    • which shells out to cargo
• calls wasm-opt on the result (optionally) to reduce size

The bolded step requires network access because cargo build has to download dependencies. Nix tooling normally solves for this by vendoring dependencies using the crate's Cargo.lock and then invoking cargo build --offline, but it's not possible to configure this through the tinysearch -> wasm-pack -> wasm-bindgen stack.

nix port

I ported tinysearch support to Nix in a pair of packages:

• tinysearch: search index generator — consumes a fuse index, outputs a storage blob (runs on the host, ported using crane)

# produces tinysearch index at ./store
$ tinysearch -m store search_index.en.json

• tinysearch_engine: wasm payload and js wrapper to be loaded by website
  • Base wasm payload built directly through cargo build --target wasm32-unknown-unknown, via crane
  • Invoke wasm_bindgen --target web on the result to produce the js wrapper
  • Invoke wasm-opt on that result to minimize wasm payload size

I include it in my blog build like this:

# flake.nix -- details elided for concision
{
    inputs = {
        # ...
        tinysearch_nix.url = "git+https://pub.npry.dev/tinysearch_nix";
    };

    outputs = { flake-utils, ... } @ inputs: (flake-utils.lib.eachDefaultSystem (system: let
        pkgs = import nixpkgs {
            inherit system;

            overlays = [
                inputs.tinysearch_nix.overlays.default
            ];
        };

    in {
        packages.default = pkgs.callPackage ./blog.nix {};
    }));
}

# blog.nix
{
    # ...
    runCommand,

    tinysearch,
    tinysearch_engine,
}: runCommand "my_blog" {} ''
    # blog build elided

    blog_generate_index | \
        ${tinysearch.generate_index}/bin/tinysearch_generate_index > $out/tinysearch_index

    cp ${tinysearch_engine} $out/tinysearch_engine
    cp ${./search.js} $out/search.js
''

engine changes

I changed tinysearch_engine to not embed the search index; rather, the wasm blob is now just the search engine, and you dynamically load (possibly multiple) indices into it before using it. The result is less "batteries included" and requires another roundtrip to the server, but is more straightforwardly compatible with the Nix build process.¹

Before:

import { search, default as init } from 'tinysearch';

// Init tinysearch engine.
await init();

// Submit a user query.
const results = search('user_search', 5);

After:

import { search, load_filters, default as init } from 'tinysearch';

const fetch_index = async (index_path) => {
    const resp = await fetch(index_path);
    const blob = await resp.blob();
    return await blob.bytes();
};

const [_, index_store] = await Promise.all([
    // Init tinysearch engine.
    init(),

    // Fetch the search index from the backend.
    fetch_index('/tinysearch_index'),
]);

// Load the index into tinysearch.
const filters = load_filters(index_store);

// Submit a user query.
const results = search(filters, 'user_search', 5);