Overpass API developpement

[mmd <>]

Hi,

Overpass API has its own rate limiting mechanism which works independent of the web server in use. This can be activated via osm3s_query command with a rate limit command line parameter (see command usage for exact syntax, which I don't recall without a system around). It is based on a leaky bucket algo and should avoid such large amounts of requests in a very short timeframe.

Unfortunately deleting the db looks like the way to go, as it's probably inconsistent and we don't know to which extent.

I'll take a look at the log files in the next days, maybe send those files to Roland as well.

Cheers 

marc marc <> schrieb am Do. 14. Sep. 2017 um 00:21:

Yes incoming rate was very hight.
but fastcgi was limited to 1 req during the peak (we raise it to 4)
Of course It 'll be better to add a true rate limite on ngix, we 'll
check it.

I sent you transaction.log
I am unfortunately unable to find what is abnormal in addition
to the extract paste in the previous email.

to fix the db issue, should the current db be deleted or can it be
repaired with a diff id created before the problem starts ?

Regards,
Marc

Le 13. 09. 17 à 22:24, mmd a écrit :
> Hi
>
> in your Munin stats I noticed that Nginx reported 34 requests/s at the
> time when the update process stopped (which coincidentally matches the
> number I found out in Perf tests, so the dispatcher was maxed out for
> what is possible w/ 0.7.54).
>
> It looks like you need to set up rate limiting for Overpass and maybe
> also check transaction.log to see what happened at that time. Can you
> provide more details on this?
>
> Problem 3 below probably means you need to start with a fresh db,
> especially due to the node not found messages.
>
> In any case more complete log files would be helpful.
>
> Cheers
>
> marc marc < <mailto:>>
> schrieb am Mi. 13. Sep. 2017 um 16:52:
>
>     Hello,
>
>     1) root cause of the crash
>     we didn't find the root cause of the crash
>     log extract:
>     Sep 12 18:54:00 osm147 dispatcher[27374]: File_Error No such file or
>     directory 2 /ssd/overpass/database//osm3s_v0.7.54_osm_base
>     Unix_Socket::7
>
>     but the file /ssd/overpass/database//osm3s_v0.7.54_osm_base exist
>
>     2) dirname missing in some log entry
>     Sep 12 19:17:26 osm147 rules_loop.sh[99]: runtime error: open64: 2 No
>     such file or directory /osm3s_v0.7.54_osm_base Dispatcher_Client::1
>     Sep 12 19:17:28 osm147 dispatcher[844]: File_Error Address already in
>     use 98 /ssd/overpass/database//osm3s_v0.7.54_osm_base Unix_Socket::4
>     Sep 12 19:17:28 osm147 dispatcher[846]: File_Error No such file or
>     directory 2 /osm3s_v0.7.54_osm_base Dispatcher_Client::1
>
>     3) diff problem : "node not found"
>     Rodolph remove the socket, restart the server, remove the lock
>     that prevent the process to start. request are working, update
>     are running but I see several lines like :
>     Sep 12 19:54:02 osm147 fetch_osc_and_apply.sh[10522]: 2017-09-12
>     19:54:02
>     URL:https://planet.osm.org/replication/minute/002/619/669.osc.gz
>     [197403/197403] -> "/tmp/osm-3s_update_pN9aiF/002619669.osc.gz" [1]
>     Sep 12 20:04:33 osm147 fetch_osc_and_apply.sh[10522]: Reading XML file
>     ... finished reading nodes. Flushing to database ...... done.
>     Sep 12 20:04:34 osm147 fetch_osc_and_apply.sh[10522]: Reading XML file
>     ... finished reading ways. Version 2 has a later or equal timestamp
>     (2017-09-12T19:00:51Z) than version 3 (2017-09-12T19:00:51Z) of Way
>     71962472
>     Sep 12 20:14:13 osm147 fetch_osc_and_apply.sh[10522]: Node 5101567155
>     used in way 6059604 not found.
>     <cut a lot of line with "node not found">
>     Sep 12 20:30:39 osm147 fetch_osc_and_apply.sh[10522]: Flushing to
>     database ...... done.
>
>     logs are using UTC time.
>
>     Regards,
>     Marc
>