Overpass API developpement

[QUILLET Jean-Charles <>]

Hello,

Thanks for such a complete answer. 

I did try extending the available size for a request using the argument 
--space= but I never manage to make it works. The dispatcher ends up 
returning:

        File_Error No such file or directory 2 /osm3s_v0.7.52_osm_base 
Dispatcher_Client::1

Then I tried to dig into the code source trying to understand how it works. 
But after some time I thought that I had better asking. That was the good 
idea, indeed ! 

As for updating the wiki. I can do that. Point me a page where you think it 
would be appropriate and I'll do it.

Jean-Charles

-----Message d'origine-----
De :  
[] De la part de Roland 
Olbricht
Envoyé : dimanche 28 août 2016 12:36
À : 
Objet : Re: [overpass] How is the resource managment working

Hello,

> I find it difficult to understand how the resource management is working.
>
> According to the wiki:
> http://wiki.openstreetmap.org/wiki/Overpass_API#Resource_management_op
> tions_.28osm-script.29

Basically, three scarce resources are managed by the resource management:
1. runtime
2. used RAM
3. number of concurrent requests

1. and 2. are managed similar, 3. is a little bit different. I start with 2., 
because you have asked for it.

The manager has a model of the amount of RAM that is admissible to use.
This is essentially a number. You can ask for it on the server by

   bin/dispatcher --osm-base --status

and then look for the line

   Total available space: X

You can set this number with

   bin/dispatcher --osm-base --space=X

to any number you want.

The dispatcher keeps track of the earmarked RAM for all the currently running 
requests. A request is granted its full RAM right from the beginning. 
Otherwise, there could be a deadlock when all already started requests were 
asking for more memory at the same time. You can see that number again with

   bin/dispatcher --osm-base --status

and then look for

   Total claimed space: X

When a new request arrives then the dispatcher checks if the requested RAM is 
less than half of the remaining RAM (available minus claimed). If yes (and 
none of the other preconditions fail, see below) then the request is accepted 
and its requested RAM is added to the claimed space. 
Otherwise the request waits for up to 15 seconds if it can get the required 
RAM. If not then the request is ultimately rejected.

Runtime works the same way. Usually, only RAM is really scarce. But the 
runtime does effect that there can run a large number of small requests in 
parallel but the system cannot be clogged with long running requests. 
The lines in question are:

   Total available time units: X
   Total claimed time units: Y

and the command to change the value is

   bin/dispatcher --osm-base --time=

All in all the system is designed such that with increasing load it will 
refuse large requests much earlier than small requests to ensure a 
as-fair-as-possible resource sharing.

Now for the number of concurrent requests. This is also controlled by a 
number, which is listed in the status output:

   Rate limit: X

The server keeps track how many requests come from each IP address. For each 
IP address there is the model of a set of slots:

- if a request is accepted then it will occupy a slot for this IP address
- after the request is completed the slot will remain closed for an extra 
time to allow fair resource sharing. After that extra time it is open for 
requests again.
- the rate limit is the number of slots available for each IP address.
- the API call /api/status (available on the development versions, not the 
0.7.52 release) will tell you your IP address' slot statuses

If no slot is available then the request retries for 15 seconds. After that 
it fails.

The extra time for the request is calculated as follows: the server reads the 
average remaining capacity (that is available space minus average claimed 
space divided by available space). It then divides the runtime of the request 
by the remaining capacity and adds this as extra time. I.e., the higher the 
load the longer the slot is occupied and the longer the request has run the 
longer the slot is occupied.

What does this mean for you specific situation:

If your server isn't public then just set the space and time constraints to 
high values and rate limit to zero:

   ../bin/dispatcher --osm-base --space=1000000000000 --time=1000000000
--rate-limit=0

In the end I would like to ask you a service:
- Please ask back further questions until you understand the explanation
- Then put the explanation to the wiki
The reason why I wouldn't like to do it myself is that I may take things as 
granted that you are asking for. That means that a documentation solely from 
me is likely to be incomplete and incomprehensible.

Thanks,

Roland