Overpass API developpement

[marc marc <>]

Yes incoming rate was very hight.
but fastcgi was limited to 1 req during the peak (we raise it to 4)
Of course It 'll be better to add a true rate limite on ngix, we 'll
check it.

I sent you transaction.log
I am unfortunately unable to find what is abnormal in addition
to the extract paste in the previous email.

to fix the db issue, should the current db be deleted or can it be 
repaired with a diff id created before the problem starts ?

Regards,
Marc

Le 13. 09. 17 à 22:24, mmd a écrit :
> Hi
> 
> in your Munin stats I noticed that Nginx reported 34 requests/s at the 
> time when the update process stopped (which coincidentally matches the 
> number I found out in Perf tests, so the dispatcher was maxed out for 
> what is possible w/ 0.7.54).
> 
> It looks like you need to set up rate limiting for Overpass and maybe 
> also check transaction.log to see what happened at that time. Can you 
> provide more details on this?
> 
> Problem 3 below probably means you need to start with a fresh db, 
> especially due to the node not found messages.
> 
> In any case more complete log files would be helpful.
> 
> Cheers
> 
> marc marc 
> <
>  
> <mailto:>>
>  
> schrieb am Mi. 13. Sep. 2017 um 16:52:
> 
>     Hello,
> 
>     1) root cause of the crash
>     we didn't find the root cause of the crash
>     log extract:
>     Sep 12 18:54:00 osm147 dispatcher[27374]: File_Error No such file or
>     directory 2 /ssd/overpass/database//osm3s_v0.7.54_osm_base
>     Unix_Socket::7
> 
>     but the file /ssd/overpass/database//osm3s_v0.7.54_osm_base exist
> 
>     2) dirname missing in some log entry
>     Sep 12 19:17:26 osm147 rules_loop.sh[99]: runtime error: open64: 2 No
>     such file or directory /osm3s_v0.7.54_osm_base Dispatcher_Client::1
>     Sep 12 19:17:28 osm147 dispatcher[844]: File_Error Address already in
>     use 98 /ssd/overpass/database//osm3s_v0.7.54_osm_base Unix_Socket::4
>     Sep 12 19:17:28 osm147 dispatcher[846]: File_Error No such file or
>     directory 2 /osm3s_v0.7.54_osm_base Dispatcher_Client::1
> 
>     3) diff problem : "node not found"
>     Rodolph remove the socket, restart the server, remove the lock
>     that prevent the process to start. request are working, update
>     are running but I see several lines like :
>     Sep 12 19:54:02 osm147 fetch_osc_and_apply.sh[10522]: 2017-09-12
>     19:54:02
>     URL:https://planet.osm.org/replication/minute/002/619/669.osc.gz
>     [197403/197403] -> "/tmp/osm-3s_update_pN9aiF/002619669.osc.gz" [1]
>     Sep 12 20:04:33 osm147 fetch_osc_and_apply.sh[10522]: Reading XML file
>     ... finished reading nodes. Flushing to database ...... done.
>     Sep 12 20:04:34 osm147 fetch_osc_and_apply.sh[10522]: Reading XML file
>     ... finished reading ways. Version 2 has a later or equal timestamp
>     (2017-09-12T19:00:51Z) than version 3 (2017-09-12T19:00:51Z) of Way
>     71962472
>     Sep 12 20:14:13 osm147 fetch_osc_and_apply.sh[10522]: Node 5101567155
>     used in way 6059604 not found.
>     <cut a lot of line with "node not found">
>     Sep 12 20:30:39 osm147 fetch_osc_and_apply.sh[10522]: Flushing to
>     database ...... done.
> 
>     logs are using UTC time.
> 
>     Regards,
>     Marc
>