Typed JSON Lines inputs and outputs, for a single file or a directory of them.
JSONL
The JSONL plugin adds two typed I/O types for working with JSON Lines data as task inputs and outputs: flyteplugins.jsonl.JsonlFile for a single JSONL file and flyteplugins.jsonl.JsonlDir for a directory of sharded JSONL files. Both are backed by orjson for fast serialization and stream records one at a time, so you can process datasets that don’t fit in memory.
JsonlFile and JsonlDir extend the built-in flyte.io.File and flyte.io.Dir types, so they inherit remote-storage, upload/download, and caching behavior. They simply add JSONL-aware streaming readers and writers on top. Every read/write method has a synchronous _sync counterpart (writer_sync(), iter_records_sync()) for use in non-async tasks.
When to use this plugin
- Passing line-delimited JSON datasets (LLM training/eval sets, event logs, model outputs) between tasks
- Streaming records without loading an entire file into memory
- Writing large outputs as automatically rotated, sharded directories
- Working with compressed JSONL (
.jsonl.zst) transparently
Installation
pip install flyteplugins-jsonlAdd the plugin to your task image. Installing it registers JsonlFile and JsonlDir with the Flyte type engine automatically. No explicit registration call is needed:
import flyte
from flyteplugins.jsonl import JsonlDir, JsonlFile
env = flyte.TaskEnvironment(
name="jsonl-examples",
image=flyte.Image.from_debian_base(name="jsonl").with_pip_packages(
"flyteplugins-jsonl"
),
)
Working with JsonlFile
Create a writable file reference with JsonlFile.new_remote(), then stream records through the writer() context manager without holding the whole dataset in memory:
@env.task
async def write_records() -> JsonlFile:
"""Write records to a single JSONL file."""
out = JsonlFile.new_remote("results.jsonl")
async with out.writer() as writer:
for i in range(500_000):
await writer.write({"id": i, "score": i * 0.1})
return out
Reading is equally streaming. iter_records() yields one parsed dict per line:
@env.task
async def read_records(data: JsonlFile) -> int:
"""Read records from a JsonlFile and return the count."""
count = 0
async for record in data.iter_records():
count += 1
return count
Bulk and pre-serialized writes
write_many() takes a whole list at once: every record lands in the writer’s buffer and the flush threshold is checked once instead of once per record. write_raw() goes a level lower and appends bytes verbatim, which is what you want when records are already serialized and a decode/re-encode round trip would be pure overhead:
@env.task
async def write_bulk() -> JsonlFile:
"""Hand the writer a whole list instead of one record at a time.
write_many() serializes every record into the writer's buffer and checks
the flush threshold once, instead of once per record.
"""
out = JsonlFile.new_remote("bulk.jsonl")
records = [{"id": i, "score": i * 0.1} for i in range(200_000)]
async with out.writer() as writer:
await writer.write_many(records)
return out
@env.task
async def passthrough(source: JsonlFile) -> JsonlFile:
"""Forward pre-serialized bytes without a re-encode round trip."""
import orjson
out = JsonlFile.new_remote("passthrough.jsonl")
async with out.writer() as writer:
async for record in source.iter_records():
# write_raw() takes bytes verbatim. Unlike write() and write_many(),
# it does not append the record separator, so include it yourself.
await writer.write_raw(orjson.dumps(record) + b"\n")
return out
write_raw() does not add the record separatorwrite() and write_many() append the trailing newline for you. write_raw() writes exactly the bytes it is given, so a payload without a trailing \n runs into the next record and corrupts the file.
The writer returned by writer_sync() exposes the same write(), write_many(), write_raw() and flush() methods synchronously. write_raw() is a JsonlFile writer method; the JsonlDir writer has write() and write_many() only, since it has to inspect each record’s size to decide when to rotate.
Working with JsonlDir
JsonlDir writes a directory of shard files (part-00000.jsonl, part-00001.jsonl, …) and reads them back transparently in sorted order. Pass max_records_per_shard (or max_bytes_per_shard) to control shard rotation. Both are optional: with neither set the writer still rotates at max_bytes_per_shard, which defaults to 256 MiB.
@env.task
async def write_large_dataset() -> JsonlDir:
"""Write a large dataset to a sharded JsonlDir.
JsonlDir automatically rotates to a new shard file once the
current shard reaches the record or byte limit. Shards are named
part-00000.jsonl, part-00001.jsonl, etc.
"""
out = JsonlDir.new_remote("dataset/")
async with out.writer(
max_records_per_shard=100_000,
max_bytes_per_shard=256 * 1024 * 1024, # 256 MB
) as writer:
for i in range(500_000):
await writer.write({"index": i, "value": i * i})
return out
Opening a writer on a directory that already holds shards appends rather than overwrites. The writer scans for existing part-NNNNN files and starts numbering at the next free index:
@env.task
async def append_more(dataset: JsonlDir) -> JsonlDir:
"""Add shards to a directory that already has some.
Opening a writer on a populated JsonlDir is safe: it scans for existing
part-NNNNN shards and starts numbering at the next free index, so nothing
already written is overwritten.
"""
async with dataset.writer(max_records_per_shard=100_000) as writer:
for i in range(50_000):
await writer.write({"index": i, "appended": True})
return dataset
Reading iterates across all shards transparently, prefetching the next shard in the background to overlap network I/O with processing:
@env.task
async def sum_values(dataset: JsonlDir) -> int:
"""Read all records across all shards and compute a sum.
Iteration is transparent across shards and handles mixed
compressed/uncompressed shards automatically. The next shard is
prefetched in the background for higher throughput.
"""
total = 0
async for record in dataset.iter_records():
total += record["value"]
return total
For bulk processing, iter_batches() yields lists of records at a time; JsonlDir also inherits all flyte.io.Dir capabilities (walk(), list_files(), download()):
@env.task
async def process_in_batches(dataset: JsonlDir) -> int:
"""Process records in batches of dicts for bulk operations."""
total = 0
async for batch in dataset.iter_batches(batch_size=1000):
# Each batch is a list[dict]
total += len(batch)
return total
iter_batches() is a JsonlDir method. A single JsonlFile has no list-of-dicts batching; batch it at the Arrow level with iter_arrow_batches() or group the records yourself.
Configuration and options
Compression
Give the file a .jsonl.zst (or .jsonl.zstd) extension and records are zstd-compressed transparently on write and decompressed on read. Tune the level via the writer:
@env.task
async def write_compressed() -> JsonlFile:
"""Write a zstd-compressed JSONL file.
Compression is activated by using a .jsonl.zst extension.
Both reading and writing handle compression transparently.
"""
out = JsonlFile.new_remote("results.jsonl.zst")
async with out.writer(compression_level=3) as writer:
for i in range(100_000):
await writer.write({"id": i, "compressed": True})
return out
For JsonlDir, set shard_extension=".jsonl.zst" on writer(). Mixed compressed and uncompressed shards within a directory are supported on read:
@env.task
async def write_compressed_dir() -> JsonlDir:
"""Write zstd-compressed shards by specifying the shard extension."""
out = JsonlDir.new_remote("compressed_dataset/")
async with out.writer(
shard_extension=".jsonl.zst",
max_records_per_shard=50_000,
) as writer:
for i in range(200_000):
await writer.write({"id": i, "data": f"payload-{i}"})
return out
Write buffering
Both writers hold serialized records in memory and flush once the buffer reaches flush_bytes, which defaults to 1 MiB. Raise it to trade memory for fewer round trips when records are small or lower it to cap the writer’s footprint:
async with out.writer(flush_bytes=8 << 20) as writer:
...JsonlDir.writer() forwards both flush_bytes and compression_level to the per-shard file writer underneath, so each applies to sharded output exactly as it does to a single file.
Prefetch
Reading a JsonlDir prefetches the next shard in the background so network I/O overlaps with processing. It is on by default. queue_size bounds the read-ahead buffer at 8192 records, and prefetch=False switches it off:
@env.task
async def read_with_prefetch_tuning(dataset: JsonlDir) -> int:
"""Tune or switch off the background shard prefetch.
queue_size bounds the read-ahead buffer in records; lower it when records
are large. prefetch=False reads strictly one shard at a time.
"""
count = 0
async for record in dataset.iter_records(prefetch=True, queue_size=1024):
count += 1
async for record in dataset.iter_records(prefetch=False):
count += 1
return count
Lower queue_size when records are large: the bound is a record count, not a byte count, so wide records make the default buffer much heavier than it looks.
Prefetch is async-only. iter_records_sync() and iter_batches_sync() accept no prefetch or queue_size and read one shard at a time, as does iter_arrow_batches() in both its async and sync forms.
Error handling on read
The record iterators accept an on_error argument: "raise" (default), "skip" to drop malformed lines, or a callable (line_number, raw_line, exception) -> None for custom handling:
@env.task
async def read_with_error_handling(data: JsonlFile) -> int:
"""Read records, skipping any corrupt lines instead of raising."""
count = 0
async for record in data.iter_records(on_error="skip"):
count += 1
return count
@env.task
async def read_with_custom_handler(data: JsonlFile) -> int:
"""Read records with a custom error handler that collects errors."""
errors: list[dict] = []
def on_error(line_number: int, raw_line: bytes, exc: Exception) -> None:
errors.append({"line": line_number, "error": str(exc)})
count = 0
async for record in data.iter_records(on_error=on_error):
count += 1
print(f"{count} valid records, {len(errors)} errors")
return count
Arrow batches
To hand JSONL data to columnar tooling, stream it as Arrow RecordBatches with iter_arrow_batches(batch_size=...). Memory usage stays bounded by the batch size. Arrow iteration requires the optional pyarrow dependency. Install it with pip install 'flyteplugins-jsonl[arrow]':
arrow_env = flyte.TaskEnvironment(
name="jsonl-arrow",
image=flyte.Image.from_debian_base(name="jsonl-arrow").with_pip_packages(
"flyteplugins-jsonl[arrow]"
),
)
@arrow_env.task
async def analyze_with_arrow(dataset: JsonlDir) -> float:
"""Stream records as Arrow RecordBatches for analytics.
Each batch holds at most batch_size records, but collecting every
batch into one table, as this example does, holds the whole dataset
in memory. To keep memory bounded, aggregate each batch as it arrives.
"""
import pyarrow as pa
batches = []
async for batch in dataset.iter_arrow_batches(batch_size=65_536):
batches.append(batch)
table = pa.Table.from_batches(batches)
mean_value = table.column("value").to_pylist()
return sum(mean_value) / len(mean_value)
Collecting every batch and combining them with pa.Table.from_batches(), as the example does, holds the whole dataset in memory. To keep memory bounded, aggregate each batch as it arrives instead.
Common use cases
- LLM dataset pipelines: stream prompt/completion or eval records between preprocessing, generation, and scoring tasks.
- Event and log processing: read large line-delimited logs shard by shard without buffering the whole file.
- Fan-out writes: produce a
JsonlDirof rotated shards from a task that emits millions of records, then consume it downstream.
API reference
See the JSONL API reference for the full JsonlFile and JsonlDir method listings.