Overpass API developpement

[Roland Olbricht <>]

Hello,

> I find it difficult to understand how the resource management is working.
>
> According to the wiki:
> http://wiki.openstreetmap.org/wiki/Overpass_API#Resource_management_options_.28osm-script.29

Basically, three scarce resources are managed by the resource management:
1. runtime
2. used RAM
3. number of concurrent requests

1. and 2. are managed similar, 3. is a little bit different. I start 
with 2., because you have asked for it.

The manager has a model of the amount of RAM that is admissible to use.
This is essentially a number. You can ask for it on the server by

  bin/dispatcher --osm-base --status

and then look for the line

  Total available space: X

You can set this number with

  bin/dispatcher --osm-base --space=X

to any number you want.

The dispatcher keeps track of the earmarked RAM for all the currently 
running requests. A request is granted its full RAM right from the 
beginning. Otherwise, there could be a deadlock when all already started 
requests were asking for more memory at the same time. You can see that 
number again with

  bin/dispatcher --osm-base --status

and then look for

  Total claimed space: X

When a new request arrives then the dispatcher checks if the requested 
RAM is less than half of the remaining RAM (available minus claimed). If 
yes (and none of the other preconditions fail, see below) then the 
request is accepted and its requested RAM is added to the claimed space. 
Otherwise the request waits for up to 15 seconds if it can get the 
required RAM. If not then the request is ultimately rejected.

Runtime works the same way. Usually, only RAM is really scarce. But the 
runtime does effect that there can run a large number of small requests 
in parallel but the system cannot be clogged with long running requests. 
The lines in question are:

  Total available time units: X
  Total claimed time units: Y

and the command to change the value is

  bin/dispatcher --osm-base --time=

All in all the system is designed such that with increasing load it will 
refuse large requests much earlier than small requests to ensure a 
as-fair-as-possible resource sharing.

Now for the number of concurrent requests. This is also controlled by a 
number, which is listed in the status output:

  Rate limit: X

The server keeps track how many requests come from each IP address. For 
each IP address there is the model of a set of slots:

- if a request is accepted then it will occupy a slot for this IP address
- after the request is completed the slot will remain closed for an 
extra time to allow fair resource sharing. After that extra time it is 
open for requests again.
- the rate limit is the number of slots available for each IP address.
- the API call /api/status (available on the development versions, not 
the 0.7.52 release) will tell you your IP address' slot statuses

If no slot is available then the request retries for 15 seconds. After 
that it fails.

The extra time for the request is calculated as follows: the server 
reads the average remaining capacity (that is available space minus 
average claimed space divided by available space). It then divides the 
runtime of the request by the remaining capacity and adds this as extra 
time. I.e., the higher the load the longer the slot is occupied and the 
longer the request has run the longer the slot is occupied.

What does this mean for you specific situation:

If your server isn't public then just set the space and time constraints 
to high values and rate limit to zero:

  ../bin/dispatcher --osm-base --space=1000000000000 --time=1000000000 
--rate-limit=0

In the end I would like to ask you a service:
- Please ask back further questions until you understand the explanation
- Then put the explanation to the wiki
The reason why I wouldn't like to do it myself is that I may take things 
as granted that you are asking for. That means that a documentation 
solely from me is likely to be incomplete and incomprehensible.

Thanks,

Roland