Overpass API developpement

[Stefan Keller <>]

Thanks mmd: That was exactly what I tried to summarize after some 
observations:

0. The typical delay (augmented diff id) compared to current time is 1
to 4 minutes.
1. The id is always increasing but sometimes an id (or more) is/are
left out between to consecutive deliveries.
2. The time interval for a new diff can be more than a minute
(sometimes hours in blackouts).
3. The interval delivering new diffs can be less than a minute (esp.
also when previous left out's are handed in later).

So a client polls augmented_diff_status and
a. waits to download diffs if there's no new id number from
augmented_diff_status,
b. or it fetches diffs up to and including to the augmented_diff_status id

So there may be still missing diff files when fetching up (which means
that an internal sequence id is not enough; there is also a need for a
list of missing id's)?

And the client can poll every minute (sometimes longer if client is
busy or diff downloading takes time) without hitting load-limitations?

:Stefan

P.S. I'm at SoTM too this week!


2016-09-18 14:58 GMT+02:00 mmd <>:
> Hi,
>
> Am 18.09.2016 um 14:41 schrieb Michael Larsen:
>> Hi,
>>
>> Last time I tried consuming augmented diffs on a minutely basis, I hit the
>> load-limitations which meant that I could not consume augmented diffs for 
>> some
>> time afterwards, i.e. this will lead to black holes in your history.
>
> the value returned by augmented_diff_status corresponds to current
> database timestamp (as number of minutes since the license change). It
> does not need to increase one-by-one(!), e.g. the database may process
> several minutely diffs in one go, due to some backlog. If you always
> download the number returned by augmented_diff_stauts, you will indeed
> get some holes!
>
> That situation can be avoided, if you keep your own internal sequence
> id, and fetch augmented diffs up to and including to the value returned
> by augmented_diff_status.
>
> If augmented_diff_status does not return any value (due to overload),
> just wait some time and try again. The same applies to downloading
> augmented_diffs: you may get HTTP 429 or HTTP 504 in case of overload,
> or if you exceed your quota (see /api/status for details). In that case,
> don't increase your internal sequence id yet, but try downloading the
> same augmented diff again.
>
>>
>> Also, using timestamp start/end to fetch diffs for a given timestamp (like
>> avachi) is problematic with some changesets that stay open for > 1 hours 
>> (this
>> happens quite ofter). The live service running on osm.expandable.dk use the
>> API as described previously to get the augmented diff for a changeset. If
>> there where a better way I'm all ears!
>>
>
> I hope the situation will improve a bit once the database has moved to
> version 0.7.53 and a different compression. If you're at SotM next week,
> you could maybe ask Roland for a current status.
>
> --
>
>