Subject: Overpass API developpement
- From: Arnold Szederjesi <>
- To: "" <>
- Subject: Re: [overpass] Overpass server in production
- Date: Thu, 19 May 2022 08:59:09 +0000
- Accept-language: en-US
- Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=octoscreen.de; dmarc=pass action=none header.from=octoscreen.de; dkim=pass header.d=octoscreen.de; arc=none
- Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SkdPlrTyWms5OPejX4a0iRzrdNagAp8je0jIE1wNVy0=; b=YmTxlXn0r9IxokEgK49GuaUBjnFvnt5odf8ln9gc3Xc7fGXiIWwpvxLRC2UhPOH7N/9quua72hOc1HKxOvCbDHSofUT4STNGQvViP2x7zo0IFEerz7xmYez79xxLPb5xZemKXTUN0otoKOL81cMDqOWiewrVLSgQwUDWcUPPbLEII6fOIgCXpZ74oat/VVWGYgpDGi1mOJTHlIXMH84AcTfhjURaXHvUv9vIG7YJxj+oWEAhhDiv+TzVFn6HqdbnqDGKspenCNkxx24MYTSWok5znJ/UB1xpsceYzlrN5SqsEDL9Ee25I39Jl2XjChLmMaLYZQqbuLUZDwZmfipf8Q==
- Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ElEVADPQAn66W0nT+VXlPh8yYLuXTgG1ZeNWHBcW9llHRIk8PwQQBiV6E54LQMgxCZW+RTP+8+Ypi7ZG8uGa8Ph8ZVJ/5nAWu9R3+PWVvOMpNqze8mMwWVUWVfGxjGpM42hDGUccXGF/w3kUfXT9w5ODOageAwq4DLe8UYxSoIMJ4/RMMXkFAZD8wrEMbNv/SiKw96TQ8zoZ2syjTzhMO+ijhylbJY8imIh1uFIAIwPTeBuCOH0I4wdg+JibaS7fZ2x1QL+1DS9wme0nzgzSoIDESDdilajjT7tRL6a0y7G1mo9yy1agV30PXgwq7BtkfscPn07odna3Pql5qdTPQQ==
- Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=octoscreen.de;
Got it, thank you for your help :)
On 18 May 2022, at 15:58, yann Guillerm <> wrote:
about updates :sometimes the files /diffs/changes.osm is not correct and then you will spot in the /changes.log the "XML parsing error" repeating again and again ...
You can see an example here : https://github.com/wiktorn/Overpass-API/issues/69. (the 2 first block of code )
i made a this small script to unstuck the updating process :
tail -n1 /data/var/db/overpass_db_0/changes.log |grep "XML parsing error" > /dev/null 2>&1
if [ -n "$?" ] ;
rm -rf /data/var/db/overpass_db_0/diffs/changes.osm
which do :if last line of changes.log contains "XML parsing error"then remove file /diffs/changes.osm
at next update the changes.osm will be reloaded and the update process will restart correctly
Not really sure but i think that there is no change lost. In our usecase, some missing changes are not so creepy ...
Le mer. 18 mai 2022 à 12:54, Arnold Szederjesi <> a écrit :
It helps of course yes. Thank you for the detailed explanation.Can I although ask about those updates ? How does it get stuck ? What are the symptoms ? What do you do to unstuck ?
With respect,Arnold Szederjesi
On 18 May 2022, at 09:30, yann Guillerm <> wrote:
here is my experience with overpass server in production.We are using overpass since 5 years.we use docker container with two sort of extract : France and World.on a Server with AMD Epyc 7351P - 16c/32t - 2.4 GHz/2.9 GHz , 128Go Ram and nvme SSD 512 Go we launch 16 servers (docker) . load balancing is done with nginx.SSD is only use for container data --> multiple servers are using the same disk storage.
It works without problems for years.
Why this choice ?We first try to work with one overpass server serving all the request, but it create errors ( 5XX http errors) .With multiple docker containers we don't have this problem .... except if we mix request type :we use two types of request :
- lot of small requests- some very big requestsif we use both request type against the same server, we create 5XX errors . So we have some server for the big request, and most server for the small requests.
the only little problem we have with the overpass stuff is the update part which sometimes get stuck (so we write a small script to unstuck it using a cron).
hope this will help you.
Yann Guillerm for coachaac.com
Le mar. 17 mai 2022 à 13:07, Arnold Szederjesi <> a écrit :
We have started to use overpass to query OSM. We have found out early that we will need our own server to be able to cope with the load needed for us.I have been successful using the docker container described here: https://github.com/wiktorn/Overpass-API.The next step is to deploy it to production, but I have some questions which are probably the easiest to be answered on this mailing list.We will probably need only the Europe extract.The questions are:1. How much CPU and memory does a normal Overpass server need ?2. Disk/volume and compute resources will be separated. Therefore, is it okay or does it make sense to have multiple servers using the same volume/mount/disk storage ?3. Also are there any other performance suggestions ?
Thank you in advance.
With respect,Arnold Szederjesi
- [overpass] Overpass server in production, Arnold Szederjesi, 05/17/2022
Archive powered by MHonArc 2.6.19+.