Overpass API developpement

[Arnold Szederjesi <>]

Got it, thank you for your help :)

With respect,

Arnold Szederjesi

On 18 May 2022, at 15:58, yann Guillerm <> wrote:



about updates : 

 sometimes the files /diffs/changes.osm is not correct  and then you will spot in the /changes.log the "XML parsing error" repeating again and again ...

You can see an example here : https://github.com/wiktorn/Overpass-API/issues/69. (the 2 first block of code ) 

i made a this small script to unstuck the updating process : 

#!/bin/sh

tail -n1 /data/var/db/overpass_db_0/changes.log |grep "XML parsing error" > /dev/null 2>&1

if [ -n "$?" ] ;

then

rm -rf /data/var/db/overpass_db_0/diffs/changes.osm

fi

which do : 

 if last line of changes.log contains "XML parsing error"  

then remove file /diffs/changes.osm

at next update the changes.osm will be reloaded and the update process will restart correctly

Not really sure but i think that there is no change lost. In our usecase, some missing changes are not so creepy ... 

sincerely,

Yann.

Le mer. 18 mai 2022 à 12:54, Arnold Szederjesi <> a écrit :

It helps of course yes. Thank you for the detailed explanation.

Can I although ask about those updates ? How does it get stuck ? What are the symptoms ? What do you do to unstuck ?

With respect,

Arnold Szederjesi

On 18 May 2022, at 09:30, yann Guillerm <> wrote:

hello, 

here is my experience with overpass server in production.

We are using overpass since 5 years. 

we use docker container  with two sort of extract : France and World.

on a Server with AMD Epyc 7351P - 16c/32t - 2.4 GHz/2.9 GHz , 128Go Ram and nvme SSD 512 Go   we launch 16 servers (docker) . load balancing is done with nginx.

SSD is only use for container data --> multiple servers are using the same disk storage.

It works without problems for years.

Why this choice ? 

We first try to work with one overpass server  serving all the request, but it create errors ( 5XX http errors) . 

With multiple docker containers we don't have this problem .... except if we mix request type :  

  we use two types of request : 
     - lot of small requests

     - some very big requests

if we use both request type against the same server, we create 5XX errors . So we have some server for the big request, and most server for the small requests.

 

the only little problem we have with the overpass stuff is the update part which sometimes get stuck (so we write a small script to unstuck it using a cron).

hope this will help you.

Sincerely

Yann Guillerm for coachaac.com

Le mar. 17 mai 2022 à 13:07, Arnold Szederjesi <> a écrit :

Hello all,

We have started to use overpass to query OSM. We have found out early that we will need our own server to be able to cope with the load needed for us.

I have been successful using the docker container described here: https://github.com/wiktorn/Overpass-API.

The next step is to deploy it to production, but I have some questions which are probably the easiest to be answered on this mailing list.

We will probably need only the Europe extract.

The questions are:

1. How much CPU and memory does a normal Overpass server need ?

2. Disk/volume and compute resources will be separated. Therefore, is it okay or does it make sense to have multiple servers using the same volume/mount/disk storage ?

3. Also are there any other performance suggestions ?

Thank you in advance.

With respect,

Arnold Szederjesi