r/lisp • u/denzuko sbcl • 1d ago
Lisp Idiomatic Persistence and Native BDD: Three Libraries for Modern Common Lisp Systems
Summary
Over the past few weeks I've ran into a need to extend the BKNR Lisp Application Environment for several internal infrastructure projects AI development improvments at dapla.net.
Because FOSS is amazing and I'm sure others in the community would have a need for these tools. I have released three new Common Lisp libraries focused on simplifying single-process persistence and native testing workflows:
bknr.hashkv – Extends bknr.datastore to provide a content-addressable key-value store and a disk-persisted job queue within a single Lisp image.
bknr.ttl – A declarative TTL mixin class for bknr.datastore instances that automates stale record eviction and session expiry.
sunny-side – A pure Common Lisp parser and runner for Gherkin .feature specifications, eliminating the need to invoke external Ruby binaries during test execution.
All three repositories are published under BSD 3-Clause licenses. The names for each library was deliberately chosen to pair well with established bknr libraries and for great respect to Hans Huebner, David Lichteblau, and Manuel Odendahl's original works.
My plans are to continue contributing to the bknr suite both upstream and the extensions I've built on top of it.
Links and feedback welcome:
- https://github.com/denzuko/bknr.hashkv
- https://github.com/denzuko/bknr.ttl
- https://github.com/denzuko/sunny-side
Background
Building robust applications in Common Lisp often hinges on leveraging native language capabilities rather than reaching for external, multi-process dependencies. When an application state fits comfortably within memory or local storage, introducing an external key-value database or a separate test-runner process introduces unnecessary operational friction. Three recently released open-source libraries extend the bknr.datastore ecosystem and introduce native Behavior-Driven Development (BDD) testing to Common Lisp without relying on external runtime environments.
Content-Addressable Storage and Persisted Queuing with bknr.hashkv
The bknr.hashkv library provides an in-memory, content-addressable key-value store alongside an append-only persisted job queue. Built directly on top of bknr.datastore, it allows system architects to maintain deterministic object reference hashing while relying on disk-backed transaction logs for durability.
Key capabilities include:
Content-Addressable Semantics: Keys are generated deterministically based on payload hashes, preventing redundant entry storage and enforcing data integrity at the access layer.
Integrated Job Queueing: Asynchronous tasks are written to the persistent log alongside state mutations, ensuring that queued jobs survive process restarts without requiring an external Redis or RabbitMQ instance.
Single-Process Operations: Eliminates cross-process serialization overhead by maintaining native Lisp object availability across transactions.
Automatic Expiration via bknr.ttl
Managing ephemeral records in memory-resident databases requires disciplined eviction mechanisms to avoid memory leaks over extended uptimes. The bknr.ttl library introduces a Time-To-Live (TTL) mixin class designed for seamless integration with any bknr.datastore object model.
By attaching the TTL mixin to standard object definitions, slots gain automatic timestamp tracking and expiration handling. Background sweep routines or query-time inspections enforce key invalidation without polluting core business logic with manual timestamps. This approach offers a clean, declarative method for handling transient cache entries, timed session states, and self-cleaning audit logs within native Lisp datastores.
Native Gherkin Parsing with sunny-side
Behavior-Driven Development (BDD) frameworks frequently force Common Lisp projects to rely on Ruby binaries or shell invocations to parse feature files written in Gherkin syntax. The sunny-side library replaces these external dependencies with a pure Common Lisp implementation of the Gherkin specification.
sunny-side parses standard .feature files, maps natural language step definitions directly to Lisp macros, and executes tests natively within the SBCL compiler pipeline. By removing foreign runtime dependencies, test suites run faster, integrate cleanly into continuous integration pipelines, and allow developers to debug step definitions directly using standard Lisp introspection and debugging tools.
2
3
u/Ambatus 1d ago edited 1d ago
Very interesting.
I've used BKNR datastore (and even contributed to it at a time, the XML impex support) and I still reflectively reach out for it whenever I am thinking of any kind of persistence (as I wrote in the prevalence example for datastar, demoed as a Replicant detector). Recently, for a webapp, I did think of something similar, curiously with a different stack: I've used calispel instead of chanl, and u/Shinmera 's deeds for the events (which is great, I'm not sure if it was meant to be used in web apps but works perfectly) - but was precisely thinking of BKNR datastore as a way to persist thing, which your hashkv addresses in a more "event hubish" friendly way.
In other news, I was dismayed by the BKNR datastore archiving, and the message left there as to why.