Troubleshooting

Common errors and what to check first. For multi-GPU specifics, see Distributed → Common failures. CPU construction and limits are described in Pipeline composition.

InvalidArgument: no CUcontext is current

Config.device was not set and no CUDA context has been primed on the calling thread yet. Two fixes:

• Pass device=0 (or device=local_rank under torchrun) to Config. damacy will retain that device's primary context internally.
• Or prime a context implicitly before constructing the pipeline: torch.empty(1, device="cuda") is enough.

Pipeline construction fails during metadata I/O setup

On Linux, the CPU and CUDA metadata paths use io_uring for Zarr metadata and shard indexes. At construction damacy requires kernel support for IORING_OP_STATX, IORING_OP_OPENAT2, IORING_OP_READ, and IORING_OP_CLOSE. If ring creation or the operation probe fails, pipeline construction fails without a thread-pool fallback. Check the native log for the exact io_uring failure.

macOS uses POSIX metadata worker threads and has no io_uring dependency. An unusually high metadata_io_concurrency can exhaust thread resources on macOS. On either platform it can also stress process file-descriptor limits because each in-flight metadata read can hold an open fd. The default is 64; for much deeper settings, check ulimit -n and remember to multiply by ranks per node.

BudgetExceeded at Pipeline(cfg)

max_gpu_memory_bytes is too small for the requested batch geometry. The log message includes a byte-level breakdown of what damacy tried to reserve. See GPU memory budget → When the budget refuses for the full fix list; the usual answer is to raise the cap.

BudgetExceeded mid-stream

A chunk's actual uncompressed size exceeds PlanLimits.max_chunk_bytes or Config.max_chunk_uncompressed_bytes (default 2 MiB). Raise that cap to fit the dataset, and raise max_gpu_memory_bytes along with it if needed. CPU executors also enforce CpuLimits for encoded chunks, decoded chunks, and total executor buffers. See GPU memory budget.

NotFound or DtypeMismatch from pop() (not push())

push() only validates what's locally checkable — sample shape and rank against Config.sample_shape. Errors that depend on store contents — missing URIs, unsupported source dtypes, per-array rank mismatch — surface from pop(), since damacy fetches the zarr metadata asynchronously after push() returns.

with Pipeline(cfg) as d:
    d.push([Sample(uri="missing", aabb=[(0, 8), (0, 16)])])  # returns fine
    try:
        d.pop()                                              # raises NotFound
    except NotFound:
        ...

Once any pop-side error fires, the pipeline is terminal — subsequent pop() calls re-raise the same status. Build a fresh Pipeline to recover.

PoolStarved from pop()

The pool was empty for longer than Config.pop_timeout_s (default 30 s). The usual cause is holding references to tensors from previous batches — for example, stashing them in a list — which prevents damacy from reusing the underlying slot. Drop the references before the next pop(), or .clone() if you need to keep them. For a NumPy CPU view, use .copy().

pop() blocks forever

The push didn't reach the lookahead, or the consumer never frees a slot:

• Read pipeline.stats().batches_emitted — if it stays at 0, pushed samples are not draining into the native lookahead.
• Make sure each pop() is inside a with block so the batch releases its slot before the next pop().
• Verify you're popping at the rate you push (or push lazily via a generator so backpressure is automatic).
• Don't pop more batches than were produced: only len(pushed) // Config.samples_per_batch complete batches are emitted (drop_last=True), so a pop() past that waits on a partial tail that is never sealed. Ragged-batch support is planned — see issue #139.