Overpass API developpement

[yann Guillerm <>]

Hi Roland,

I manage to make a lot more request now but i have to slow down my process to much.... i will try again to reach the 22 concurent request you tell at the end of you answer.

the average request take 11 seconds to be resolved ( from 1 to 30 seconds, but mostly around 10/11 seconds)

cpu seems to be the bootleneck of my server .

here is the result of dispatcher --osm-base --status : 

$EXEC_DIR/bin/dispatcher --osm-base --status
Number of not yet opened connections: 0
Number of connected clients: 2
Rate limit: 0
Total available space: 12884901888
Total claimed space: 4294967296
Average claimed space: 7247757312
Total available time units: 262144
Total claimed time units: 0
Average claimed time units: 19800
Counter of started requests: 87276
Counter of finished requests: 87198
202 8399 0 4294967296 0 1523348264

i'm not sure to understand all those numbers ;-)

i run the command when my script was not running.

i will launch a test and use iotop to check disk usage.

thanks for your tips.

2018-04-09 17:04 GMT+02:00 Roland Olbricht <>:

Hi Yann,

I have already change the Apache settings du 3600 but that does not help.

I think that the number of concurrent call is the problem.
In may script if i limit the concurrent call to 20, my test programme work.
If i try the same script with a limit to 40, a got 504 errors.

        - processor is a 25% ( one full core)
        - memory stay flat at 24/30%

thank you for asking. This helps me to understand how people use non-public instances. The Overpass dispatcher uses two internal limits, "space" and "time" to determine whether it has load. Please run

dispatcher --osm-base --status

An example from z.overpass-api.de:

Number of not yet opened connections: 0
Number of connected clients: 19
Rate limit: 2
Total available space: 12884901888
Total claimed space: 4831838208
Average claimed space: 7416928460
Total available time units: 262144
Total claimed time units: 2415
Average claimed time units: 3387
Counter of started requests: 69180046
Counter of finished requests: 69171436
[...] [list of running and pending processes]

If the used space is close to the maximum space then raise the space:
dispatcher --osm-base --space=20000000000

If the used space is close to the maximum space then raise the space:
dispatcher --osm-base --space=20000000000

None of the two could work if the disk is the limiting factor. The best tool to check disk load I know is "iotop". It should suffice to cross-check whether disk performance matters.

I expect that separating for CPU kernels will not work. The queries are distinct processes, and virtually any operating system is good in distributing processes on CPUs. Opposed to that I have run into two other systematic bottlenecks:
- super slow motion of swap: The OS will not warn if the RAM is short and instead start to swap if too many processes perform actual work in parallel. Swap is horribly slow in such a setting.
- disk dead lock: Similarly, the OS lets the disk jump so often between many concurrent processes that most time is spent seeking. I have not systematically ivestigated whether the problem persists with a SSD.

Given that in addition on the public instance most requests have a very short runtime (90% of all successful queries take less than a second), I have designed the system to delay further requests if much more than the number of CPU kernels is already running, represented by the above mentioned limits. You can identify this phenomenon quite clearly if there are many requests with HTTP 504 and 15 seconds runtime in the log files. The number of parallel requests is essentially
(space - 1 GB)/512 MB, i.e. for 12 GiB, the default value, it is 22 concurrent requests.

Best regards,
Roland