LOCAL STORAGE · COLD READS · 128–256 PIXEL CROPS
How should we store training samples?
Random crops and preassembled samples, measured on an H100 node and an EPYC node. Seven image sources, three crop sizes, and the same pixels in both layouts.
Loading measurements…
01 / LAYOUT
Baked layouts deliver more image data per second
Best observed medians among tested settings. Baking combines one sample per chunk with sequential reads across 16 files; the selected codec can also change. This compares the complete layouts.
○ Random crops◆ Baked samplesSmall marks: other tested settingsRanges: observed min–max, n = 3
Values are image throughput in GiB/s, median [min–max]. Every panel uses the same scale. Select an encoding to inspect its measurements. Close medians with overlapping ranges are not established winners.
02 / CROP SIZE & READER
More image bytes per second can mean fewer samples
Follow a fixed encoding across crop sizes, and compare all three readers on matched settings. The fastest reader depends on the crop and encoding.
Different vertical scales; compare crop sizes and readers within each panel.
CPU output is in system RAM; CUDA output is in GPU memory. CPU results exclude the transfer needed for GPU training. The machines have separate storage arrays, so this does not isolate processor performance.
03 / CHUNKS & BLOCKS
Change one encoding parameter at a time
These comparisons keep the codec, crop, and read pattern fixed. Each pair shows image throughput, median [min–max] across three repetitions. The two figures have different scales.
What was measured: two machines, one storage setup per machine
The November 6, 2026 studies contain 93 settings and 387 executions. Every timed pass began with zero resident shard-data pages. Warmup, initialization, and cache clearing are outside the timer; reading, decoding, and final GPU completion are inside it.
H100 node: Intel Xeon Platinum 8462Y+ and one H100 80 GB. EPYC node: AMD EPYC 9655P. Each used 32 allowed logical CPUs, 16 file workers, and 16 CPU decode workers, with XFS on its own array of eight 7.68 TB Intel drives. Exact inventories and idle activity are in the explorer.
EPYC measurements were noisier: median min–max spread 11.3%, versus 1.7% on the H100 node. Ranges describe three observations, not confidence intervals. Random arrays have 16 spatial shards per source; baked samples use 16 files total. Shards target about 1 GiB uncompressed. Full protocol, sources, and reproduction.
Explore all settings
Compression tradeoffs, file reads, and individual measurements
Storage system
Filter settings
Select a setting to inspect its measurements.
View configuration detailsAll configuration measurements
Reading the measurements
- Samples/s
- Completed single-plane crops per second on the measured system. Compare rates at the same crop size.
- Image throughput
- Requested image bytes delivered per second, counted in the source data type. Converting uint16 to float32 does not increase this number.
- Compression ratio
- Uncompressed image bytes in the stored arrays divided by their file bytes. Larger values mean less storage is needed.
- File read requests / sample
- Data-file read requests issued by Damacy divided by delivered crops. Excludes metadata. One request can serve multiple crops, so this can be below one.
- Read amplification
- File bytes requested divided by uncompressed bytes in the delivered crops, counted in the source data type. A value of 4× means four file bytes requested per image byte delivered; compression can bring this below 1×.
Image throughput uses GiB/s, where 1 GiB is 2³⁰ bytes. All rates use elapsed measurement time, including reading, decoding, and final GPU completion. Initialization, warmup, and cache clearing are excluded. The Storage system section identifies the hardware; Diagnostics retains device IOPS and storage bandwidth.
Measurement limits
File read counts and read amplification describe demand on storage. They depend on the layout and reader, including how it combines or aligns requests. They do not include the operating system’s extra reads or other processes, and do not by themselves predict performance on another storage system.
The observed frontier compares settings within the same input, crop size, read pattern, backend, machine session, and storage condition. Filters recompute it. Connections join measured points; they do not predict unmeasured settings. All compared panels use the same axes.
Ranges are observed minima and maxima, not confidence intervals. Use the measurements table or keyboard to select overlapping points. This is a reduced parameter sweep; unmeasured settings may perform better.
Compression ratio includes padding, shard indexes, and array metadata. Mixed studies use expanded, tiled source distributions. A ratio below one means the files are larger than the uncompressed image bytes. The compression axis is logarithmic.
Every timed pass starts with zero resident data-file pages; codecs and metadata are warmed. Overlapping reads and read-ahead within each pass are included. Storage counters include other activity on the same drives. Idle measurements are available in Diagnostics, and SSD/controller caches are not controlled.
Measurement protocol and reproduction · Design inspired by the chucky microscopy analysis.