Overpass API developpement

Text archives Help


RE: [overpass] How is the resource managment working


Chronological Thread 
  • From: QUILLET Jean-Charles <jean-charles.quillet@alyotech.fr>
  • To: "overpass@listes.openstreetmap.fr" <overpass@listes.openstreetmap.fr>
  • Subject: RE: [overpass] How is the resource managment working
  • Date: Mon, 29 Aug 2016 07:24:56 +0000
  • Accept-language: fr-FR, en-US

Hello,

Thanks for such a complete answer.

I did try extending the available size for a request using the argument
--space= but I never manage to make it works. The dispatcher ends up
returning:

File_Error No such file or directory 2 /osm3s_v0.7.52_osm_base
Dispatcher_Client::1

Then I tried to dig into the code source trying to understand how it works.
But after some time I thought that I had better asking. That was the good
idea, indeed !

As for updating the wiki. I can do that. Point me a page where you think it
would be appropriate and I'll do it.

Jean-Charles

-----Message d'origine-----
De : overpass-request@listes.openstreetmap.fr
[mailto:overpass-request@listes.openstreetmap.fr] De la part de Roland
Olbricht
Envoyé : dimanche 28 août 2016 12:36
À : overpass@listes.openstreetmap.fr
Objet : Re: [overpass] How is the resource managment working

Hello,

> I find it difficult to understand how the resource management is working.
>
> According to the wiki:
> http://wiki.openstreetmap.org/wiki/Overpass_API#Resource_management_op
> tions_.28osm-script.29

Basically, three scarce resources are managed by the resource management:
1. runtime
2. used RAM
3. number of concurrent requests

1. and 2. are managed similar, 3. is a little bit different. I start with 2.,
because you have asked for it.

The manager has a model of the amount of RAM that is admissible to use.
This is essentially a number. You can ask for it on the server by

bin/dispatcher --osm-base --status

and then look for the line

Total available space: X

You can set this number with

bin/dispatcher --osm-base --space=X

to any number you want.

The dispatcher keeps track of the earmarked RAM for all the currently running
requests. A request is granted its full RAM right from the beginning.
Otherwise, there could be a deadlock when all already started requests were
asking for more memory at the same time. You can see that number again with

bin/dispatcher --osm-base --status

and then look for

Total claimed space: X

When a new request arrives then the dispatcher checks if the requested RAM is
less than half of the remaining RAM (available minus claimed). If yes (and
none of the other preconditions fail, see below) then the request is accepted
and its requested RAM is added to the claimed space.
Otherwise the request waits for up to 15 seconds if it can get the required
RAM. If not then the request is ultimately rejected.

Runtime works the same way. Usually, only RAM is really scarce. But the
runtime does effect that there can run a large number of small requests in
parallel but the system cannot be clogged with long running requests.
The lines in question are:

Total available time units: X
Total claimed time units: Y

and the command to change the value is

bin/dispatcher --osm-base --time=

All in all the system is designed such that with increasing load it will
refuse large requests much earlier than small requests to ensure a
as-fair-as-possible resource sharing.

Now for the number of concurrent requests. This is also controlled by a
number, which is listed in the status output:

Rate limit: X

The server keeps track how many requests come from each IP address. For each
IP address there is the model of a set of slots:

- if a request is accepted then it will occupy a slot for this IP address
- after the request is completed the slot will remain closed for an extra
time to allow fair resource sharing. After that extra time it is open for
requests again.
- the rate limit is the number of slots available for each IP address.
- the API call /api/status (available on the development versions, not the
0.7.52 release) will tell you your IP address' slot statuses

If no slot is available then the request retries for 15 seconds. After that
it fails.

The extra time for the request is calculated as follows: the server reads the
average remaining capacity (that is available space minus average claimed
space divided by available space). It then divides the runtime of the request
by the remaining capacity and adds this as extra time. I.e., the higher the
load the longer the slot is occupied and the longer the request has run the
longer the slot is occupied.

What does this mean for you specific situation:

If your server isn't public then just set the space and time constraints to
high values and rate limit to zero:

../bin/dispatcher --osm-base --space=1000000000000 --time=1000000000
--rate-limit=0

In the end I would like to ask you a service:
- Please ask back further questions until you understand the explanation
- Then put the explanation to the wiki
The reason why I wouldn't like to do it myself is that I may take things as
granted that you are asking for. That means that a documentation solely from
me is likely to be incomplete and incomprehensible.

Thanks,

Roland




Archive powered by MHonArc 2.6.18.

Top of page