Damacy

LOCAL STORAGE · COLD READS · 128–256 PIXEL CROPS

How should we store training samples?

Random crops and preassembled samples, measured on an H100 node and an EPYC node. Seven image sources, three crop sizes, and the same pixels in both layouts.

Loading measurements…

Explore all settings

Compression tradeoffs, file reads, and individual measurements

Reading the measurements

Samples/s
Completed single-plane crops per second on the measured system. Compare rates at the same crop size.
Image throughput
Requested image bytes delivered per second, counted in the source data type. Converting uint16 to float32 does not increase this number.
Compression ratio
Uncompressed image bytes in the stored arrays divided by their file bytes. Larger values mean less storage is needed.
File read requests / sample
Data-file read requests issued by Damacy divided by delivered crops. Excludes metadata. One request can serve multiple crops, so this can be below one.
Read amplification
File bytes requested divided by uncompressed bytes in the delivered crops, counted in the source data type. A value of 4× means four file bytes requested per image byte delivered; compression can bring this below 1×.

Image throughput uses GiB/s, where 1 GiB is 2³⁰ bytes. All rates use elapsed measurement time, including reading, decoding, and final GPU completion. Initialization, warmup, and cache clearing are excluded. The Storage system section identifies the hardware; Diagnostics retains device IOPS and storage bandwidth.

Measurement limits

File read counts and read amplification describe demand on storage. They depend on the layout and reader, including how it combines or aligns requests. They do not include the operating system’s extra reads or other processes, and do not by themselves predict performance on another storage system.

The observed frontier compares settings within the same input, crop size, read pattern, backend, machine session, and storage condition. Filters recompute it. Connections join measured points; they do not predict unmeasured settings. All compared panels use the same axes.

Ranges are observed minima and maxima, not confidence intervals. Use the measurements table or keyboard to select overlapping points. This is a reduced parameter sweep; unmeasured settings may perform better.

Compression ratio includes padding, shard indexes, and array metadata. Mixed studies use expanded, tiled source distributions. A ratio below one means the files are larger than the uncompressed image bytes. The compression axis is logarithmic.

Every timed pass starts with zero resident data-file pages; codecs and metadata are warmed. Overlapping reads and read-ahead within each pass are included. Storage counters include other activity on the same drives. Idle measurements are available in Diagnostics, and SSD/controller caches are not controlled.

Measurement protocol and reproduction · Design inspired by the chucky microscopy analysis.