Overpass API developpement

[mmd <>]

Hi

in your Munin stats I noticed that Nginx reported 34 requests/s at the time when the update process stopped (which coincidentally matches the number I found out in Perf tests, so the dispatcher was maxed out for what is possible w/ 0.7.54).

It looks like you need to set up rate limiting for Overpass and maybe also check transaction.log to see what happened at that time. Can you provide more details on this?

Problem 3 below probably means you need to start with a fresh db, especially due to the node not found messages.

In any case more complete log files would be helpful.

Cheers 

marc marc <> schrieb am Mi. 13. Sep. 2017 um 16:52:

Hello,

1) root cause of the crash
we didn't find the root cause of the crash
log extract:
Sep 12 18:54:00 osm147 dispatcher[27374]: File_Error No such file or
directory 2 /ssd/overpass/database//osm3s_v0.7.54_osm_base Unix_Socket::7

but the file /ssd/overpass/database//osm3s_v0.7.54_osm_base exist

2) dirname missing in some log entry
Sep 12 19:17:26 osm147 rules_loop.sh[99]: runtime error: open64: 2 No
such file or directory /osm3s_v0.7.54_osm_base Dispatcher_Client::1
Sep 12 19:17:28 osm147 dispatcher[844]: File_Error Address already in
use 98 /ssd/overpass/database//osm3s_v0.7.54_osm_base Unix_Socket::4
Sep 12 19:17:28 osm147 dispatcher[846]: File_Error No such file or
directory 2 /osm3s_v0.7.54_osm_base Dispatcher_Client::1

3) diff problem : "node not found"
Rodolph remove the socket, restart the server, remove the lock
that prevent the process to start. request are working, update
are running but I see several lines like :
Sep 12 19:54:02 osm147 fetch_osc_and_apply.sh[10522]: 2017-09-12
19:54:02
URL:https://planet.osm.org/replication/minute/002/619/669.osc.gz
[197403/197403] -> "/tmp/osm-3s_update_pN9aiF/002619669.osc.gz" [1]
Sep 12 20:04:33 osm147 fetch_osc_and_apply.sh[10522]: Reading XML file
... finished reading nodes. Flushing to database ...... done.
Sep 12 20:04:34 osm147 fetch_osc_and_apply.sh[10522]: Reading XML file
... finished reading ways. Version 2 has a later or equal timestamp
(2017-09-12T19:00:51Z) than version 3 (2017-09-12T19:00:51Z) of Way 71962472
Sep 12 20:14:13 osm147 fetch_osc_and_apply.sh[10522]: Node 5101567155
used in way 6059604 not found.
<cut a lot of line with "node not found">
Sep 12 20:30:39 osm147 fetch_osc_and_apply.sh[10522]: Flushing to
database ...... done.

logs are using UTC time.

Regards,
Marc