Overpass API developpement

[mmd <>]

Am 18.09.2016 um 15:27 schrieb Stefan Keller:

> 
> 0. The typical delay (augmented diff id) compared to current time is 1
> to 4 minutes.

Right, you can see the current delay in munin [1], and the current
timestamp via a dedicated API call [2].

[1]
http://overpass-api.de/munin/localdomain/localhost.localdomain/osm_db_lag.html
[2] http://overpass-api.de/api/timestamp

> 1. The id is always increasing but sometimes an id (or more) is/are
> left out between to consecutive deliveries.

At first sight, IDs may appear to be left out. But as the ID just
corresponds exactly to the database timestamp, there may be some
non-consecutive increases.

However, that's not really an issue. The reason for the larger gaps is
the way how minutely diffs are processed: if there's a backlog of a few
minutes, several diffs will be processed as one package to speed up
processing and the timestamp increases by more than a minute.

From the outside, augmented_diff_status appears to have jumped by
several numbers, but you still have to query every number in between,
even if it wasn't announced by augmented_diff_status.

> 2. The time interval for a new diff can be more than a minute
> (sometimes hours in blackouts).

Please remember that augmented diffs are calculated on the fly. Based on
the sequence id you provide, data from an interval of exactly one minute
is returned.

It also doesn't matter how the minutely diffs were originally posted to
the database. Augmented diffs just follow a fixed 60 second grid, with a
1:1 relationship between the id and the respective timeframe.

For reference, here's the respective code snippet. I think a few lines
of code tell a thousand words ;)

https://github.com/drolbr/Overpass-API/blob/master/src/cgi-bin/augmented_diff#L40-L48

> So there may be still missing diff files when fetching up (which means
> that an internal sequence id is not enough; there is also a need for a
> list of missing id's)?

There are really no missing ids, as mentioned, augmented_diff_status
just returns the maximum available number to be queried. You still need
to query every single id up to and including that number.

> 
> And the client can poll every minute (sometimes longer if client is
> busy or diff downloading takes time) without hitting load-limitations?
> 

Well, http://overpass-api.de/api/status should have a line containing
"slots available now" in there, otherwise, all slots are currently busy
and you will hit a HTTP 429 error. IIRC, Roland explained that idea
somewhere on the Overpass Dev list before...

--