Lilith.
⌕
Editorial illustration: Datasette streams millions of rows straight into Arrow without a memory detour
Lilith illustration · editorial remix

The datasette-arrows 0.1a0 plugin adds a .arrows export to every table, view, row and SQL query page. With ?_stream=1, it fetches and sends data page by page, so a table with millions of rows does not require the full result to sit in memory.

Arrow became the reference test for Datasette’s new streaming interface

Datasette 1.0a43 opened streaming output renderers to plugins. Simon Willison released a separate plugin alongside that change because a new interface should have a concrete user. Claude Fable 5.2 suggested Arrow IPC and, according to Willison, built most of the plugin almost in one shot.

The export preserves filters, sorting and column selection. The server returns application/vnd.apache.arrow.stream, converting each fetched page into one Arrow record batch. Datasette’s allow_stream and max_stream_mb limits also apply.

Data teams get a shorter route from SQLite to analytical tools

Tools including pandas, Polars and DuckDB can read Arrow IPC without an intermediate CSV. The benefit is therefore broader than export size. A columnar format preserves types and removes another conversion step that can consume time or alter values in large datasets.

The plugin also shows why an extensible streaming interface matters more than one format. Another developer can add an exporter through the same path without changing Datasette core.

Mixed SQLite types can terminate a stream halfway through

SQLite permits different value types in one column, while Arrow declares a type before the first batch. The plugin infers it from the column declaration and first page. If a later page contains text in a column declared as INTEGER, for example, it drops the connection so an incomplete file cannot look complete. An explicit SQL cast is the workaround.

This remains an alpha integration. Willison also notes that the current release does not yet group Arrow with the CSV and TSV all rows options in the export dialog. That interface change is intended for the next alpha.

Long and messy exports will determine whether the plugin holds up

The useful signal will come from genuinely large databases with inconsistent types. Users need predictable behavior for size limits, interrupted connections and retries.

Compatibility with subsequent Datasette alphas matters just as much. This plugin is a credible reference client for the new API. Routine use will show whether that interface supports more than one fortunate demonstration format.

Lilith's verdict

Arrow now carries millions of Datasette rows in batches instead of cramming them into one overloaded lift. The first stalled car full of mixed types will reveal how well the mechanism was actually built.