`render_list` not using all cores

[coogle]

I am building tiles with raster_list and meta2tile, and I have noticed a performance issue I can't figure out why is happening that maybe someone here can shed some light on.

I am using a c5d.12xlarge instance on AWS which has 48 cores and 96G of memory backed by a 2TB SSD drive. I am rendering the North America dataset as raster tiles, and this is the render_list command I am using:

render_list \
    --all \
    --force \
    --min-zoom=0 \
    --max-zoom=18 \
    --max-load=$(nproc) \
    --map=current \
    --num-threads=$(nproc) \
    --tile-dir=/osm-data/renderd/tiles

Things run fine, but I've noticed that my load never has gotten past 32, about 60% of my total CPU capacity. I know that, for example, if you don't specify --max-load then render_list will throttle itself pretty aggressively at 16 cores... but is there some other setting (compiled in or otherwise) that is limiting render_list to 32 cores? I've even tried increasing --num-threads to 2x my cores to absolutely no effect.

Other facts:

• I have plenty of free RAM (consuming 32G of 92G available)
• My Disk Writes seem to not impact at all... sometimes my writes are like 100-1000 K/s sometimes they are in the MB but neither has any impact on my load.

Unless I'm misunderstanding something (possible), it seems like I'm not disk bound, or memory bound -- yet render_list won't take advantage of the extra 16 vCPUs available to it.

Thoughts anyone?

[hummeltech]

What are you using for [renderd].num_threads in your renderd.conf file? (example here)

Also, are you building this yourself? There have been some recent merges which might be helpful, I.E.:

• Improve lock contention in render_list etc #335
• Increase queue limits to reflect modern machines #336

[coogle]

Ahh, I missed that this was set in renderd.conf (which was set, unsurprisingly, at 32). That was almost certainly the problem I was experiencing above. Yes, I am building it myself -- and I will certainly investigate the idea next time I do this of compiling from source given those merges ... for now I'm using whatever is prepackaged in Ubuntu 22.04. Thanks for the pointer!

[hummeltech]

Sounds good, feel free to close this issue if updating that value resolves the issue for you.

[coogle]

Yeah it appears to have. One more slightly unrelated question before I do: I was actively building tiles out and it was in the process of building zoom 15... I edited my renderd.conf to up my processes to 48, restarted the service, and render_list died because of the lost connection. So I re-ran the command with two differences:

• I upped --min-zoom to 15
• I removed the --force flag

My understanding here is that should basically pick up right where I left off building from 15, with no missing tiles in the final output, correct?

[coogle]

Oh and (sorry) one last question and then I think I'm really done... is the num_threads option in renderd.conf required, or if it's omitted will it "do the right thing" and use all available cores? The build process for the tiles machine is in ansible/packer, etc. and isn't ever serving tiles live -- so it'd be nice if I could just omit that configuration value and it'll figure out how to set itself up properly if I change instance types to something with less/more vCPUs

[hummeltech]

Oh and (sorry) one last question and then I think I'm really done... is the num_threads option in renderd.conf required, or if it's omitted will it "do the right thing" and use all available cores? The build process for the tiles machine is in ansible/packer, etc. and isn't ever serving tiles live -- so it'd be nice if I could just omit that configuration value and it'll figure out how to set itself up properly if I change instance types to something with less/more vCPUs

No, it defaults to NUM_THREADS (see here), although that might be a good idea for a pull request.

[hummeltech]

Yeah it appears to have. One more slightly unrelated question before I do: I was actively building tiles out and it was in the process of building zoom 15... I edited my renderd.conf to up my processes to 48, restarted the service, and render_list died because of the lost connection. So I re-ran the command with two differences:

* I upped `--min-zoom` to 15

* I removed the `--force` flag

My understanding here is that should basically pick up right where I left off building from 15, with no missing tiles in the final output, correct?

Yes, I believe that is how render_list should work, but as I do not generally use it, there might be some minor details I am missing.

[coogle]

Thanks @hummeltech for all the answers.

NUM_THREADS defaults to 4? lol.

When I revisit this down the road (right now I'm under a gun to get these tiles built), I will take a look at this issue -- maybe I can find the time to make a PR that dynamically sets the vCPUs since I'll be mucking about building from source anyway.

Thanks again - super helpful!!

[hummeltech]

When I revisit this down the road (right now I'm under a gun to get these tiles built), I will take a look at this issue -- maybe I can find the time to make a PR that dynamically sets the vCPUs since I'll be mucking about building from source anyway.

@coogle, I have a pull request in to implement the ability to use all available cores, feel free to try if out if you have the time.

• Add option to detect available cores when num_threads=-1 #380