Overpass API developpement

[Igor Brejc <>]

Hi Roland,

Latest results: I installed the new versions of services yesterday, left them running for a few hours and just by checking the two log files produces, they seemed to be running fine (no error entries either in fetch_osc_and_apply.log or transactions.log).

BTW I used --meta=no option for fetch_osc_and_apply.sh (since it's now mandatory to specify the value of this argument).

However, today I decided to check the systemd journal since it looks like a lot of logging is done just to stdout and not the log file, and I found these entries (immediately after the first batch of diff files was downloaded):

sep 10 17:23:48 jazz fetch_osc_and_apply.sh[20388]: 2017-09-10 17:23:48 URL:http://planet.osm.org/replication/minute//002/599/947.state.txt [168/168] -> "/tmp/osm-3s_update_N5f8Ry/002599947.state.txt" [1]

sep 10 17:23:48 jazz fetch_osc_and_apply.sh[20388]: 2017-09-10 17:23:48 URL:http://planet.osm.org/replication/minute//002/599/947.osc.gz [51927/51927] -> "/tmp/osm-3s_update_N5f8Ry/002599947.osc.gz" [1]

sep 10 17:23:50 jazz fetch_osc_and_apply.sh[20388]: Reading XML file ... finished reading nodes. Version 0 has a later or equal timestamp (0000-00-00T00:00:00Z) than version 0 (0000-00-00T00:00:00Z) of Node 176084016

sep 10 17:23:50 jazz fetch_osc_and_apply.sh[20388]: Version 0 has a later or equal timestamp (0000-00-00T00:00:00Z) than version 0 (0000-00-00T00:00:00Z) of Node 176090219

sep 10 17:23:50 jazz fetch_osc_and_apply.sh[20388]: Version 0 has a later or equal timestamp (0000-00-00T00:00:00Z) than version 0 (0000-00-00T00:00:00Z) of Node 176101272

sep 10 17:23:50 jazz fetch_osc_and_apply.sh[20388]: Version 0 has a later or equal timestamp (0000-00-00T00:00:00Z) than version 0 (0000-00-00T00:00:00Z) of Node 176103018

sep 10 17:23:50 jazz fetch_osc_and_apply.sh[20388]: Version 0 has a later or equal timestamp (0000-00-00T00:00:00Z) than version 0 (0000-00-00T00:00:00Z) of Node 189122042

sep 10 17:23:50 jazz fetch_osc_and_apply.sh[20388]: Version 0 has a later or equal timestamp (0000-00-00T00:00:00Z) than version 0 (0000-00-00T00:00:00Z) of Node 269204750

This then results in following errors:

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 1411413259 used in way 7990122 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 3797985553 used in way 7990122 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 5068195841 used in way 8091970 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 30923267 used in way 8091970 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 765690030 used in way 8091970 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 288376716 used in way 8091970 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 765689879 used in way 8091970 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 4729032203 used in way 8091970 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 4729032204 used in way 8091970 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 1645753235 used in way 39671787 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 1645753228 used in way 39671787 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 1327871645 used in way 117968807 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 1348966463 used in way 120270858 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 1327871645 used in way 127964698 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 5068223552 used in way 182970606 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 1933333342 used in way 182970606 not found.

sep 10 17:36:51 jazz fetch_osc_and_apply.sh[20388]: Node 1933333328 used in way 182970606 not found.

On Sun, Sep 10, 2017 at 6:35 AM, Igor Brejc <> wrote:

Thank you Roland, I will try the new release and let you know. 

On Sat, Sep 9, 2017 at 12:26 PM, Roland Olbricht <> wrote:

Hi Daniel, hi Igor,

I'm pretty confident that there is a different way to solve the runaway problem.

As it turns out, fetch_osc_and_apply.sh employs update_database instead of update_from_dir. update_database does write error messages, but fetch_osc_and_apply.sh does discard these error messages.

I've ensured in a new release that
- update_database returns an error code on any kind of error
- fetch_osc_and_apply.sh does not suppress error messages any more

On closer inspection, fetch_osc_and_apply.sh has other issues, too. In particular, it processes every mintue diff individually, which makes catching-up with updates much (about five times) slower than catching-up with updates by batch processing.

A little bit of background: the script is the specific script for the French instance, and I added it to the main repo to avoid unnecessary fragmentation. I never checked it beyond a basic dry-run if it works, as I never expected that anybody else would use a script outside the standard installation instructions.

However, I agree, if a file is in my repositoy then you should can assume that it works.

Please download and try the fixed release v0.7.54.6:
https://dev.overpass-api.de/releases/osm-3s_v0.7.54.tar.gz

Best regards,

Roland