Overpass API developpement

[mmd <>]

Hi,

if you've been around on this mailing list a bit, you probably remember
the performance tests I did last year. As more and more performance
improvements have accumulated in my test758 branch, it was time to
re-run a similar test. So, here's a short summary on those recent
activities. I will probably write a more detailed blog post later.

Hardware was very similar to last year (32GB, Intel Xeon E3-1270V3),
with the exception of a bigger SSD, which now holds a full attic
database (480GB DC SSD). Still commodity hardware for 50-60€/month.

First test was focused on ridiculously simple queries
[out:json];node(63174280);out; (with node ids increasing), but a larger
number of them in parallel.

Due to changes in the dispatcher processing, we're now at 2000
queries/s, or some 172 mio. requests/day.

Of course, I don't want to advocate this kind of usage, but it's good to
know, we can keep up with a large number of very small requests you
would typically see with interactive maps.

Second test was a simulation of a full day in the life of
overpass-api.de. That's 534'497 real queries in total, processed with 7
parallel processes in 10.75 hours, without any waiting time, though. I
found the response time distribution quite interesting:

Response times
--------------

50%     90%     95%     99%     99.5%   99.9%   99.99%  99.999%
0.062s  0.685s  1.58s   5.5s    11.06s  33.92s  168.37s 853.6s

(quantile percentages: 50% of all queries were processed in up to 62ms)

With those changes in place, I would even go as far and reduce the
default timeout down to 60s, and set a hard query timeout to maybe 300s
- 600s, even if people set the timeout value to, say 500000s.

You can find tons of munin charts and more details over at Github:

Full day simulation: https://github.com/mmd-osm/Overpass-API/issues/3
Scalability: https://github.com/mmd-osm/Overpass-API/issues/2
Source: https://github.com/mmd-osm/Overpass-API/tree/test758_lz4hash

Cheers
mmd