Overpass API developpement

Text archives Help


[overpass] Re: Overpass Quotas


Chronological Thread 
  • From: Roland Olbricht <roland.olbricht@gmx.de>
  • To: overpass@listes.openstreetmap.fr, osm-talk <talk@openstreetmap.org>
  • Subject: [overpass] Re: Overpass Quotas
  • Date: Tue, 05 May 2015 22:21:49 +0200

Dear all,

the last mail needs some more clarification. While I appreaciate that quite a lot of people care for not overusing the server, they are not the ones I have meant with "clumsy clients".

I fear it is quite a common problem and probably has a name: those who care about usage policies restrict themselves even more, but it doesn't help, because the problem comes from people that don't care or don't understand that they cause trouble.

In other words: if you read this message then you don't contribute to congestion.

Hence, if you have never seen a "HTTP 429 Too much requests" message, then your usage of overpass-api.de is perfectly fine. Please carry on and don't mind.

Even if you have seen every now and then (as a hint: in less than 20 percent of all requests) a HTTP 429 then just relax a little bit, try to move long running requests (beyond 180 seconds, if any) off peak times (the time from UTC 12:00 to UTC 22:00 is more busy than the rest). And still everything is fine, even if you cannot move out of peak times. The server will not keep any record after rejection, and you will be treated after a few minutes as if you never have been involved in a HTTP 429 request.

You should be only concerned if you find yourself on this short list of blocked IP addresses:

5.9.65.X
104.47.142.X
104.40.194.X
167.114.116.0/22
58.51.144.X
58.51.144.Y
54.78.255.X
54.247.44.X

The next risk candidates are:

188.166.8.X
194.33.69.X

A general misbehaving group of clients has been identified by sending the user request "Only a test!". While I appreciate to send a self-chosen user agent, I have difficulties to derive a point of contact from that specific user agent.

Once again: In all other cases, your usage of overpass-api.de is responsible and perfectly fine.

For the curious, some more data:

01/Apr/2015 111710 67894 700976 488641 38386
02/Apr/2015 125437 73602.1 774208 506222 36288
03/Apr/2015 111768 66832.3 821650 495482 35969
04/Apr/2015 119659 49824.3 906004 459878 32267
05/Apr/2015 106234 45427.5 881590 462305 33588
06/Apr/2015 83007.5 43324.7 930086 505258 32662
07/Apr/2015 100788 42118.3 936754 511573 33880
08/Apr/2015 87590.3 40828.3 788823 474827 33610
09/Apr/2015 112620 37723.3 663166 450491 31912
10/Apr/2015 114220 37376.3 665173 465647 30522
11/Apr/2015 69314.6 26770 661636 439874 28347
12/Apr/2015 113752 36699.7 631125 469711 30236
13/Apr/2015 67609.4 30192.6 742857 475160 33531
14/Apr/2015 47800.8 26262.2 732373 456884 31838
15/Apr/2015 86428.3 30434.3 733738 484012 32638
16/Apr/2015 94440.9 28273.7 783472 466555 32041
17/Apr/2015 74105.2 26220 829937 471074 30380
18/Apr/2015 63287.7 16449.5 851242 460553 27105
19/Apr/2015 87040.5 20821.3 775956 471591 30032
20/Apr/2015 71236.3 28174.4 775381 471193 31782
21/Apr/2015 74426.7 27312.3 805662 423813 30961
22/Apr/2015 76482.8 32474.2 776979 458964 30663
23/Apr/2015 63931 30986 779185 400018 29209
24/Apr/2015 65186.2 29074.9 713110 414088 30181
25/Apr/2015 63562 20196.7 604212 355809 28733
26/Apr/2015 61562.2 22193.2 600918 362236 29523
27/Apr/2015 95413.6 33839.2 627806 407742 32568
28/Apr/2015 71292.1 30761 678194 411622 32213
29/Apr/2015 108186 29082.4 692393 416049 30986
30/Apr/2015 126114 28901.1 857178 397983 29670

The second column is the total amount of data in megabytes by day. The third column is the total amount of data in megabytes per day for all except the 30 heaviest users. You can notice that on most days these 30 heaviest users consume 50% to 80% of all data.

In a similar manner, the fourth and fifth column list the number of requests. Again, heavy users are quite dominant.

The last column is the number of unique IP addresses the server has seen that day.

The full diagram has more columns to tell apart the heaviest user, the 3 heaviest users, and the 10 heaviest users. However, the tendence remains the same that very few (less than 1 in 1000 users) consume more than 50% of the resources.

So if you are not in this top-30-list, then your usage of overpass-api.de is perfectly fine.

Happy querying,

Roland




Archive powered by MHonArc 2.6.18.

Top of page