Overpass API developpement

[Donal Diamond <>]

Thanks for looking at this. My setup was the following:

Ubuntu 14.04.5 LTS
g++ (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4



####

#1  Download, create dirs

Download http://dev.overpass-api.de/releases/osm-3s_v0.7.53.tar.gz

( ~overpass = /home/roles/overpass )

mkdir -p ~overpass/log
mkdir -p ~overpass/db
mkdir -p ~overpass/app/osm3s

####

#2 build/install
 ./configure --enable-lz4  --prefix=/home/roles/overpass/app/osm3s/; make install

####

#3  cp rules file

DB_DIR=~overpass/db; cp -pR rules $DB_DIR

####

#4 import data
wget http://planet.openstreetmap.ie/$PLANET_FILE
wget http://planet.openstreetmap.ie/replication/15minute/state.txt

osmconvert ireland-and-northern-ireland.osm.pbf | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ --meta --compression-method=lz4 --map-compression-method=lz4

####

#5  start dispatcher

nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR --meta > dispatcher.out &

####

#6 import diffs

# dont need to run fetch_osc.sh as planets diff are already on server as it runs planet.openstreetmap.ie

# get diff timestamp
osmconvert --out-timestamp ireland-and-northern-ireland.osm.pbf

# check time is same as sequence

cat state.txt
nohup $EXEC_DIR/bin/apply_osc_to_db.sh ~planet/www/replication/15minute/ XXXXX --meta=yes --augmented_diffs=no > apply_osc_to_db.out &

####

# 7 Areas

nohup $EXEC_DIR/bin/dispatcher --areas --db-dir=$DB_DIR > dispatcher_areas.out &
nohup $EXEC_DIR/bin/rules_loop.sh $DB_DIR > rules_loop.out &

( do renice and ionice )

####

# 8 Xapi

nohup $EXEC_DIR/bin/cleanup_xapi_tmp.sh  > cleanup_xapi_tmp.out &
chown www-data /tmp/translate_xapi

D

On 5 September 2016 at 11:33, mmd <> wrote:

Am 05.09.2016 um 11:21 schrieb Donal Diamond:
> Here's 'ls' rather than 'du' output:
>
> overpass/db$ ls -al area*
> -rw-rw-r-- 1 overpass overpass 17975148544 Sep  5 10:16 area_blocks.bin
> -rw-rw-r-- 1 overpass overpass        3720 Sep  5 10:12 area_blocks.bin.idx
> -rw-rw-r-- 1 overpass overpass        3720 Sep  5 10:12
> area_blocks.bin.idx.shadow
> -rw-rw-r-- 1 overpass overpass           8 Sep  5 10:12
> area_blocks.bin.shadow
> -rw-rw-r-- 1 overpass overpass   294125568 Sep  5 10:16 areas.bin
> -rw-rw-r-- 1 overpass overpass          24 Sep  5 10:12 areas.bin.idx
> -rw-rw-r-- 1 overpass overpass          24 Sep  5 10:12 areas.bin.idx.shadow
> -rw-rw-r-- 1 overpass overpass           8 Sep  5 10:12 areas.bin.shadow
> -rw-rw-r-- 1 overpass overpass           5 Sep  5 10:12 areas_shadow.lock
> -rw-rw-r-- 1 overpass overpass  2105344000 Sep  5 10:16 area_tags_global.bin
> -rw-rw-r-- 1 overpass overpass         922 Sep  5 10:12
> area_tags_global.bin.idx
> -rw-rw-r-- 1 overpass overpass         922 Sep  5 10:12
> area_tags_global.bin.idx.shadow
> -rw-rw-r-- 1 overpass overpass           8 Sep  5 10:12
> area_tags_global.bin.shadow
> -rw-rw-r-- 1 overpass overpass  2352840704 Sep  5 10:16 area_tags_local.bin
> -rw-rw-r-- 1 overpass overpass        2892 Sep  5 10:12
> area_tags_local.bin.idx
> -rw-rw-r-- 1 overpass overpass        2892 Sep  5 10:12
> area_tags_local.bin.idx.shadow
> -rw-rw-r-- 1 overpass overpass           8 Sep  5 10:12
> area_tags_local.bin.shadow
> -rw-rw-r-- 1 overpass overpass          21 Sep  5 10:05 area_version
> -rw-rw-r-- 1 overpass overpass          21 Sep  5 10:12 area_version.shadow
>
>

I'm having difficulties to reproduce this. On which platform, OS and
compiler did you experience this, also which compiler settings did you use?

I started on an empty db directory with initial DB load of
49728-ireland-and-northern-ireland.osm.pbf, then started dispatcher
processes for osm-base and areas, followed by continuous updates:

./fetch_osc_and_apply.sh
http://planet.openstreetmap.ie/replication/15minute/ --meta=yes

Verified that DB update worked ok by cross-checking osc.gz contents with
osm3s_query results.

In parallel to this, areas were updated:

while true ; do ./osm3s_query --rules --progress <
../../src/rules/areas.osm3s ; sleep 60 ; done ;

This is how the area processing looks like here:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API">
<note>The data included in this document is from www.openstreetmap.org.
The data is made available under ODbL.</note>
<meta osm_base="2016-09-05T10:15:02Z" areas="2016-09-05T10:15:02Z"/>

After 0h0m15s: in "recurse", part 0, on line 26. Stack: 7416 of 67359 0 of 0
After 0h0m30s: in "recurse", part 0, on line 27. Stack: 15842 of 67359 0
of 8
After 0h0m45s: in "recurse", part 0, on line 27. Stack: 24524 of 67359 0
of 6
After 0h1m2s: in "recurse", part 0, on line 27. Stack: 31509 of 67359 0 of 7
After 0h1m17s: in "make-area", part 0, on line 29. Stack: 39292 of 67359
After 0h1m32s: in "make-area", part 0, on line 29. Stack: 47566 of 67359
After 0h1m47s: in "recurse", part 0, on line 27. Stack: 55967 of 67359 0
of 9
After 0h2m4s: in "recurse", part 0, on line 26. Stack: 61823 of 67359 0 of 0
After 0h2m19s: in "make-area", part 0, on line 257. Stack: 1453 of 41706
After 0h2m34s: in "make-area", part 0, on line 257. Stack: 12618 of 41706
After 0h2m49s: in "make-area", part 0, on line 257. Stack: 26676 of 41706
After 0h3m4s: in "make-area", part 0, on line 257. Stack: 31618 of 41706
After 0h3m19s: in "make-area", part 0, on line 257. Stack: 33267 of 41706
After 0h3m34s: in "make-area", part 0, on line 257. Stack: 34911 of 41706
After 0h3m49s: in "make-area", part 0, on line 257. Stack: 41700 of 41706

</osm>

After a few iterations, area* file sizes are still very modest.

 172M Sep  5 12:21 area_blocks.bin
 3,7K Sep  5 12:21 area_blocks.bin.idx
 3,8M Sep  5 12:21 areas.bin
   24 Sep  5 12:21 areas.bin.idx
  28M Sep  5 12:21 area_tags_global.bin
  901 Sep  5 12:21 area_tags_global.bin.idx
  18M Sep  5 12:21 area_tags_local.bin
 2,8K Sep  5 12:21 area_tags_local.bin.idx
   21 Sep  5 12:17 area_version


--