Image pulling much slower during run than regular pull.

[Hungerarray]

Hi, so I was performing some benchmarking with stargz-snapshotter. My test setup basically included, me creating regular images, an esgz image with optimizations disabled and one with optimizations enabled. For each of these three variants, I go through, pull, create, and start individually then measure the time taken.
I have been testing primarily with large AI models, as one of our use case is to see if stargz-snapshotter would be a viable alternative to get the start time for containers with AI models quicker.

During the experimentation I stumbled upon some measurements that I am having hard time explaining. For instance while running, llama 3.2, here are the timings I got, (Start basically measures, from when we issue a container start request until, the model is fully loaded and ready to serve)

Pull	Create	Start
2m9s	135ms	47s

While for esgz without optimization this is what I saw,

Pull	Create	Start
1.34s	210ms	4m27s

For sake of completion here is also the timing we received for esgz with optimization enabled,

Pull	Create	Start
2.54s	10.57s	2m5s

As can be seen from the measurements the timing for run phase always takes very long. I had been monitoring the network bandwidth usage during each of the runs, and it's quite apparent that the while the regular pull, nearly saturated the available bandwidth, the same wasn't true during run phase. I did find a blob, chunk_size property in config, set to default 50,000. When increased to say 50Mb, I do see a boost in run, for instance here are the measurements after setting blog, chunk_size.

Pull	Create	Start
2m26s	246ms	45s

for regular overlayfs snapshotter,

Pull	Create	Start
1.9s	129ms	2m26s

for esgz images without optimization and,

Pull	Create	Start
2s	10.28s	2m10s

for esgz images with optimization enabled.

This setting did also see increased number of fetched bytes per layer as per, /.stargz-snapshotter/*.json files.

It would be great if you could help explain the difference in pulling the bytes during "run phase", and if there is a way to increase it.

[polarathene]

Disclaimer, I've not yet used eStargz. This reply is just to share some insights to help you troubleshoot further. If it helps please reply with what solved the issue 👍

Some docs references:

• https://github.com/containerd/stargz-snapshotter/blob/main/docs/passthrough.md#advanced-configuration
• https://github.com/containerd/stargz-snapshotter/blob/main/docs/smaller-estargz.md#--estargz-min-chunk-size-usage
• https://github.com/containerd/stargz-snapshotter/blob/main/docs/ctr-remote.md

---

I did find a blob, chunk_size property in config, set to default 50,000. When increased to say 50Mb, I do see a boost in run, for instance here are the measurements after setting blog, chunk_size.

Did you try experimenting with larger chunks?

During the experimentation I stumbled upon some measurements that I am having hard time explaining. For instance while running, llama 3.2, here are the timings I got,

Was llama model part of the image? If not what were the large files in the image?

You did not explain much about the host, but assuming this is GPU workload I'm assuming CPU, memory and disk I/O wasn't under any heavy pressure with eStargz?

Models for example may not be very compressible binary files, so if there is time wasted in the decompression perhaps use a lower compression level?

Likewise large binary files, even when chunked into smaller pieces wouldn't be usable until the full file is reconstructed. So you're likely still blocked on pulling this content, and then depending on the parallelized operations going on memory usage may also be higher? If you're on a system with limited memory that has disk swap / paging available, that may be used as anonymous page cache during decompression and you may want to inspect how much memory is being used by reads.

If you're using Docker Desktop on Windows/macOS a VM involved can add additional overhead here. I have for example had image builds that take only 30s individually, but when doing 12 concurrently where hardware resources become bottlenecked this slowed completion down to 9 minutes. You could be experiencing similar perhaps with the 4 minute run time being notably slower than others.

When you are taking your measurements, you may need to be aware of warm cache from file buffer in memory may need to be flushed out too. Sequential I/O will have higher throughput, so large blocks of data (such as layers/files) will work better for sequential reads/writes via disk and network.

All of this aside, look at the project docs I linked, where you can optimize the run at expense of create time, which you did do, but there was no mention if you had done the profiling and that it actually included the larger files to prioritized. If you had done that, these should have been acquired before run which should better match your overlay run time, where I assume the bulk of this is related to reading in the model into system memory / vRAM.

I assume the image is gzip layers, was it decompressing them in parallel? (docs describe gzip helpers)

You could look at converting the image to zstd:chunked, where container engines with the chunked support have additional optimizations for retrieving and accessing the image content that is needed (you may also want to verify that the overlay image being compared is not already using zstd compression).

From what I have read some of the benefits (1, 2) that you'd get with eStargz is available with zstd:chunked compression, eStargz apparently supports this compression too, which may help, but my guess is the files being accessed such as the model are your blockers and that the need to be retrieved and extracted on-demand, you need to parallelize this to the pull/create stages without introducing pressure that negatively impacts the start (such as too much concurrency for the environment).