Overpass API developpement

[Donal Diamond <>]

Seems to be increasing - mmd you seeing the same?

overpass/db$ du -shc area*.bin
628M    area_blocks.bin
11M     areas.bin
71M     area_tags_global.bin
78M     area_tags_local.bin
786M    total

overpass/db$ ls -al area*
-rw-rw-r-- 1 overpass overpass 657981440 Sep  5 15:39 area_blocks.bin
-rw-rw-r-- 1 overpass overpass      3720 Sep  5 15:37 area_blocks.bin.idx
-rw-rw-r-- 1 overpass overpass      3720 Sep  5 15:37 area_blocks.bin.idx.shadow
-rw-rw-r-- 1 overpass overpass         8 Sep  5 15:37 area_blocks.bin.shadow
-rw-rw-r-- 1 overpass overpass  11010048 Sep  5 15:39 areas.bin
-rw-rw-r-- 1 overpass overpass        24 Sep  5 15:37 areas.bin.idx
-rw-rw-r-- 1 overpass overpass        24 Sep  5 15:37 areas.bin.idx.shadow
-rw-rw-r-- 1 overpass overpass         8 Sep  5 15:37 areas.bin.shadow
-rw-rw-r-- 1 overpass overpass         5 Sep  5 15:37 areas_shadow.lock
-rw-rw-r-- 1 overpass overpass  73465856 Sep  5 15:39 area_tags_global.bin
-rw-rw-r-- 1 overpass overpass       901 Sep  5 15:37 area_tags_global.bin.idx
-rw-rw-r-- 1 overpass overpass       901 Sep  5 15:37 area_tags_global.bin.idx.shadow
-rw-rw-r-- 1 overpass overpass         8 Sep  5 15:37 area_tags_global.bin.shadow
-rw-rw-r-- 1 overpass overpass  80805888 Sep  5 15:39 area_tags_local.bin
-rw-rw-r-- 1 overpass overpass      2777 Sep  5 15:37 area_tags_local.bin.idx
-rw-rw-r-- 1 overpass overpass      2777 Sep  5 15:37 area_tags_local.bin.idx.shadow
-rw-rw-r-- 1 overpass overpass         8 Sep  5 15:37 area_tags_local.bin.shadow
-rw-rw-r-- 1 overpass overpass        21 Sep  5 15:30 area_version
-rw-rw-r-- 1 overpass overpass        21 Sep  5 15:37 area_version.shadow

overpass/db$ cat replicate_id
50440

D

On 5 September 2016 at 13:45, Donal Diamond <> wrote:

I've reinstalled the db using 49728-ireland-and-northern-ireland.osm.pbf so we both should be in sync.

I'll keep an eye and see when it jumps again

The API has very little queries - only rules_loop.sh is currently causing any transactions.

irljidel@dev3:/home/roles/overpass/db$ du -shc area*bin*
261M    area_blocks.bin
4.0K    area_blocks.bin.idx
4.0K    area_blocks.bin.idx.shadow
4.0K    area_blocks.bin.shadow
4.6M    areas.bin
4.0K    areas.bin.idx
4.0K    areas.bin.idx.shadow
4.0K    areas.bin.shadow
33M     area_tags_global.bin
4.0K    area_tags_global.bin.idx
4.0K    area_tags_global.bin.idx.shadow
4.0K    area_tags_global.bin.shadow
30M     area_tags_local.bin
4.0K    area_tags_local.bin.idx
4.0K    area_tags_local.bin.idx.shadow
4.0K    area_tags_local.bin.shadow
327M    total

overpass/db$ cat replicate_id
50432

D

On 5 September 2016 at 13:29, mmd <> wrote:

Am 05.09.2016 um 13:19 schrieb Donal Diamond:
> Thanks for looking at this. My setup was the following:
>

Thanks a lot for the detailed description.

I have a quick question on the number of area creation runs needed to
reproduce this. After about 25 runs on my local machine, I see some very
slow growth (see below).

In addition to updating the db and updating areas, I also run an the
following osm3s_query in an endless loop:

  (area[~"."~"."];rel(pivot);>;);out geom meta;

I think I mentioned in some other post a few months ago [1], that file
keep growing over time and creating a clone db would bring the db size
down considerably: "Database size grows quite a lot during the update
process. A subsequent clone-db sometimes reduced that size by 50-60GB.
Again I didn't investigate where those large differences come from."

Maybe we're seeing similar effects here?

Do you happen to have any special queries running in parallel on your
machine, especially some long running ones?


208M Sep  5 14:16 area_blocks.bin
3,7K Sep  5 14:16 area_blocks.bin.idx
 12M Sep  5 14:16 areas.bin
  24 Sep  5 14:16 areas.bin.idx
 87M Sep  5 14:16 area_tags_global.bin
 901 Sep  5 14:16 area_tags_global.bin.idx
 28M Sep  5 14:16 area_tags_local.bin
2,8K Sep  5 14:16 area_tags_local.bin.idx
  21 Sep  5 14:11 area_version

--

[1] [PerfProject2016] Full attic database creation in 4 days