Overpass API developpement

[mmd <>]

Hi,

I hope you don't mind if I answer in English, so that the post is of
interest to a wider audience. I will also try to translate some of your
questions.

First of all, it would be interesting to hear a bit about your use case
(what kind of queries you plan to run), and a bit more on your
environment (OS, storage, etc.).


Am 04.11.2016 um 16:40 schrieb Zecke:
> Hello,
> 
> 
> Ich bin dabei, eine Overpass Instanz für lokale Queries hochzuziehen.
> Einen Planet habe ich installiert (Version 27.10.). Das Einspielen hat
> ca. 36h gedauert. Aktuell spiele ich hourly diffs ein, das dauert mir
> irgendwie zu lange. Nach 12h ist er immer noch bei den ersten 22 diffs.
> 


36 hours processing time for a full planet import still sounds
reasonable. In my tests release 0.7.53 took about 20 hours on fast SSD
and using lz4 compression ("configure --enable-lz4").

As you've probably also noticed in the documentation, there's an option
to start with a clone db instead. This would save you some time for the
first import.

Regarding hourly diffs: do you want to continue processing hourly diffs
in the future as well? Did you set the replicate_id file to a value
which matches your planet file. It is very important that there's no gap
between the planet file's last edit (use osmconvert --out-statistics to
get that timestmap) and the first edit of your hourly diff.

btw: from previous experience, I recommend to do a backup after a first
successful full import and only then start with (minutely/hourly/daily)
diffs. That way you can recover easily in case things go wrong during
subsequent db updates.


> In den Logs fallen mir zwei Dinge auf:
> 
> - Es gibt unzählige Einträge der Art "Node 1909617360 used in way
> 180303878 not found.".

Lots of missing node messages is always a sign that something went wrong
somewhere before. I guess those messages were created when applying
hourly diffs, rather during your first full import? Most likely, there's
some issue with the replicate_id value you've chosen before starting db
updates.

> 
> - Des weiteren hat er im db-Verzeichnis eine Menge shadow-Dateien, die
> alle root gehören. (Das Einspielen läuft als normaler User). Könnte
> evtl. vom Dispatcher kommen, der als root läuft?

In any case, all Overpass processes should run as a dedicated non-root
user, and maybe www-data for the Apache CGI. The effects you see are all
caused by running the dispatcher as root.


> Es gibt halt auch im Console-log Meldungen:
> File error caught: 13 Permission denied
> /opt/osm3s/db/nodes.bin.idx.shadow File_Blocks_Index::~File_Blocks_Index::1
> die offensichtlich von diesen shadow-Dateien kommen.
> 
> War es vielleicht ein Fehler die Option "inkl. Metadaten" zu verwenden?

Using meta is really perfectly fine. Never seen an issue with it.

> Dauert es deshalb so lange?
> Sollte der init_osm3s.sh und apply_osc_to_db.sh auch unter root laufen
> (oder der dispatcher als normaler user)?

I guess most of the long processing time is really caused by writing
lots of "Node ... used in way ... not found" messages. But in any case,
I don't think it makes sense to continue this process without fixing
those messages in the first place.

> (Der dispatcher wurde über das Ubuntu-initscript und damit als root
> gestartet)


Yes, this script needs some rework. None of the overpass executables
should run as root user.


--