Overpass API developpement

[mmd <>]

Am 05.09.2016 um 11:21 schrieb Donal Diamond:
> Here's 'ls' rather than 'du' output:
> 
> overpass/db$ ls -al area*
> -rw-rw-r-- 1 overpass overpass 17975148544 Sep  5 10:16 area_blocks.bin
> -rw-rw-r-- 1 overpass overpass        3720 Sep  5 10:12 area_blocks.bin.idx
> -rw-rw-r-- 1 overpass overpass        3720 Sep  5 10:12
> area_blocks.bin.idx.shadow
> -rw-rw-r-- 1 overpass overpass           8 Sep  5 10:12
> area_blocks.bin.shadow
> -rw-rw-r-- 1 overpass overpass   294125568 Sep  5 10:16 areas.bin
> -rw-rw-r-- 1 overpass overpass          24 Sep  5 10:12 areas.bin.idx
> -rw-rw-r-- 1 overpass overpass          24 Sep  5 10:12 areas.bin.idx.shadow
> -rw-rw-r-- 1 overpass overpass           8 Sep  5 10:12 areas.bin.shadow
> -rw-rw-r-- 1 overpass overpass           5 Sep  5 10:12 areas_shadow.lock
> -rw-rw-r-- 1 overpass overpass  2105344000 Sep  5 10:16 area_tags_global.bin
> -rw-rw-r-- 1 overpass overpass         922 Sep  5 10:12
> area_tags_global.bin.idx
> -rw-rw-r-- 1 overpass overpass         922 Sep  5 10:12
> area_tags_global.bin.idx.shadow
> -rw-rw-r-- 1 overpass overpass           8 Sep  5 10:12
> area_tags_global.bin.shadow
> -rw-rw-r-- 1 overpass overpass  2352840704 Sep  5 10:16 area_tags_local.bin
> -rw-rw-r-- 1 overpass overpass        2892 Sep  5 10:12
> area_tags_local.bin.idx
> -rw-rw-r-- 1 overpass overpass        2892 Sep  5 10:12
> area_tags_local.bin.idx.shadow
> -rw-rw-r-- 1 overpass overpass           8 Sep  5 10:12
> area_tags_local.bin.shadow
> -rw-rw-r-- 1 overpass overpass          21 Sep  5 10:05 area_version
> -rw-rw-r-- 1 overpass overpass          21 Sep  5 10:12 area_version.shadow
> 
> 


I'm having difficulties to reproduce this. On which platform, OS and
compiler did you experience this, also which compiler settings did you use?

I started on an empty db directory with initial DB load of
49728-ireland-and-northern-ireland.osm.pbf, then started dispatcher
processes for osm-base and areas, followed by continuous updates:

./fetch_osc_and_apply.sh
http://planet.openstreetmap.ie/replication/15minute/ --meta=yes

Verified that DB update worked ok by cross-checking osc.gz contents with
osm3s_query results.

In parallel to this, areas were updated:

while true ; do ./osm3s_query --rules --progress <
../../src/rules/areas.osm3s ; sleep 60 ; done ;

This is how the area processing looks like here:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API">
<note>The data included in this document is from www.openstreetmap.org.
The data is made available under ODbL.</note>
<meta osm_base="2016-09-05T10:15:02Z" areas="2016-09-05T10:15:02Z"/>

After 0h0m15s: in "recurse", part 0, on line 26. Stack: 7416 of 67359 0 of 0
After 0h0m30s: in "recurse", part 0, on line 27. Stack: 15842 of 67359 0
of 8
After 0h0m45s: in "recurse", part 0, on line 27. Stack: 24524 of 67359 0
of 6
After 0h1m2s: in "recurse", part 0, on line 27. Stack: 31509 of 67359 0 of 7
After 0h1m17s: in "make-area", part 0, on line 29. Stack: 39292 of 67359
After 0h1m32s: in "make-area", part 0, on line 29. Stack: 47566 of 67359
After 0h1m47s: in "recurse", part 0, on line 27. Stack: 55967 of 67359 0
of 9
After 0h2m4s: in "recurse", part 0, on line 26. Stack: 61823 of 67359 0 of 0
After 0h2m19s: in "make-area", part 0, on line 257. Stack: 1453 of 41706
After 0h2m34s: in "make-area", part 0, on line 257. Stack: 12618 of 41706
After 0h2m49s: in "make-area", part 0, on line 257. Stack: 26676 of 41706
After 0h3m4s: in "make-area", part 0, on line 257. Stack: 31618 of 41706
After 0h3m19s: in "make-area", part 0, on line 257. Stack: 33267 of 41706
After 0h3m34s: in "make-area", part 0, on line 257. Stack: 34911 of 41706
After 0h3m49s: in "make-area", part 0, on line 257. Stack: 41700 of 41706

</osm>

After a few iterations, area* file sizes are still very modest.

 172M Sep  5 12:21 area_blocks.bin
 3,7K Sep  5 12:21 area_blocks.bin.idx
 3,8M Sep  5 12:21 areas.bin
   24 Sep  5 12:21 areas.bin.idx
  28M Sep  5 12:21 area_tags_global.bin
  901 Sep  5 12:21 area_tags_global.bin.idx
  18M Sep  5 12:21 area_tags_local.bin
 2,8K Sep  5 12:21 area_tags_local.bin.idx
   21 Sep  5 12:17 area_version


--

Attachment: signature.asc
Description: OpenPGP digital signature