Overpass API developpement

[marc marc <>]

Hello,

I often find it counterproductive to launch too many simultaneous 
processes. having too many simultaneous processes by core makes the cpu 
spend more of its time switching between processes. With intensive 
tasks, the ideal is often one to 2 simultaneous processes per cpu core.

Regards,
Marc

Le 10. 04. 18 à 13:38, yann Guillerm a écrit :
> So,
> 
> i use iotop to check for disk usage ...
>   Nothing / Nada / Que dale  : disk usage is flat with some max usage 
> near 6% ...
> in the same time CPU is at 100% and RAM about 30%.
> 
> i have to find a better CPU server.
> 
> best regards,
> 
> 
> Yann.
> 
> 2018-04-10 12:56 GMT+02:00 yann Guillerm 
> <
>  
> <mailto:>>:
> 
>     Hi Roland,
> 
>     I manage to make a lot more request now but i have to slow down my
>     process to much.... i will try again to reach the 22 concurent
>     request you tell at the end of you answer.
> 
>     the average request take 11 seconds to be resolved ( from 1 to 30
>     seconds, but mostly around 10/11 seconds)
> 
>     cpu seems to be the bootleneck of my server .
> 
>     here is the result of dispatcher --osm-base --status :
> 
>     $EXEC_DIR/bin/dispatcher --osm-base --status
>     Number of not yet opened connections: 0
>     Number of connected clients: 2
>     Rate limit: 0
>     Total available space: 12884901888
>     Total claimed space: 4294967296
>     Average claimed space: 7247757312
>     Total available time units: 262144
>     Total claimed time units: 0
>     Average claimed time units: 19800
>     Counter of started requests: 87276
>     Counter of finished requests: 87198
>     202 8399 0 4294967296 0 1523348264
> 
> 
> 
>     i'm not sure to understand all those numbers ;-)
>     i run the command when my script was not running.
> 
>     i will launch a test and use iotop to check disk usage.
> 
>     thanks for your tips.
> 
> 
>     2018-04-09 17:04 GMT+02:00 Roland Olbricht 
> <
>     
> <mailto:>>:
> 
>         Hi Yann,
> 
>             I have already change the Apache settings du 3600 but that
>             does not help.
> 
>             I think that the number of concurrent call is the problem.
>             In may script if i limit the concurrent call to 20, my test
>             programme work.
>             If i try the same script with a limit to 40, a got 504 errors.
> 
>                      - processor is a 25% ( one full core)
>                      - memory stay flat at 24/30%
> 
> 
>         thank you for asking. This helps me to understand how people use
>         non-public instances. The Overpass dispatcher uses two internal
>         limits, "space" and "time" to determine whether it has load.
>         Please run
> 
>         dispatcher --osm-base --status
> 
>         An example from z.overpass-api.de <http://z.overpass-api.de>:
> 
>         Number of not yet opened connections: 0
>         Number of connected clients: 19
>         Rate limit: 2
>         Total available space: 12884901888
>         Total claimed space: 4831838208
>         Average claimed space: 7416928460
>         Total available time units: 262144
>         Total claimed time units: 2415
>         Average claimed time units: 3387
>         Counter of started requests: 69180046
>         Counter of finished requests: 69171436
>         [...] [list of running and pending processes]
> 
>         If the used space is close to the maximum space then raise the
>         space:
>         dispatcher --osm-base --space=20000000000
> 
>         If the used space is close to the maximum space then raise the
>         space:
>         dispatcher --osm-base --space=20000000000
> 
>         None of the two could work if the disk is the limiting factor.
>         The best tool to check disk load I know is "iotop". It should
>         suffice to cross-check whether disk performance matters.
> 
>         I expect that separating for CPU kernels will not work. The
>         queries are distinct processes, and virtually any operating
>         system is good in distributing processes on CPUs. Opposed to
>         that I have run into two other systematic bottlenecks:
>         - super slow motion of swap: The OS will not warn if the RAM is
>         short and instead start to swap if too many processes perform
>         actual work in parallel. Swap is horribly slow in such a setting.
>         - disk dead lock: Similarly, the OS lets the disk jump so often
>         between many concurrent processes that most time is spent
>         seeking. I have not systematically ivestigated whether the
>         problem persists with a SSD.
> 
>         Given that in addition on the public instance most requests have
>         a very short runtime (90% of all successful queries take less
>         than a second), I have designed the system to delay further
>         requests if much more than the number of CPU kernels is already
>         running, represented by the above mentioned limits. You can
>         identify this phenomenon quite clearly if there are many
>         requests with HTTP 504 and 15 seconds runtime in the log files.
>         The number of parallel requests is essentially
>         (space - 1 GB)/512 MB, i.e. for 12 GiB, the default value, it is
>         22 concurrent requests.
> 
>         Best regards,
>         Roland
> 
> 
>