Overpass API developpement

Text archives Help


Re: [overpass] compressed database info


Chronological Thread 
  • From: mmd <mmd.osm@gmail.com>
  • To: overpass@listes.openstreetmap.fr
  • Subject: Re: [overpass] compressed database info
  • Date: Sun, 08 Mar 2015 10:51:50 +0100

A few additional remarks:

Am 03.03.2015 um 18:25 schrieb mmd:
>
> Other queries [2] involving the 'foreach' statement seemed to be even
> slower with backend compression.

<speculation>
I have the impression that OS buffer cache keeps uncompressed blocks in
case of file system compression. Compression backend on the other hand
seems decompress the same blocks again, if they're needed (which is not
unlikely in a 'foreach' scenario).

This could explain at least some of the differences. Further
measurements are needed in any case...
</speculation>

> Based on the good experience I have with LZO compression on btrfs, I
> would also highly welcome seeing LZO or even LZ4 as additional
> compression algorithms in the compression backend (optional: as
> configurable option on a per-file type basis). As you pointed out
> before, the design is already open for different algorithms.


I've created a new branch with LZ4 compression enabled by default [2],
to facilitate further performance testing. Compression will not be as
good as with ZLIB, but CPU consumption should be quite a bit lower.

To compile this branch you need to manually install LZ4 r127 [3], as
r127 is not included in Ubuntu 14.04.

The lz4 branch should be benchmarked against branch [1]. Reason being
that both branches include a few unrelated performance tweaks, which
Roland hasn't merged into the official version yet.

(Both branches are not meant to be used in production.)

[1] https://github.com/mmd-osm/Overpass-API/tree/test753_compression
[2] https://github.com/mmd-osm/Overpass-API/tree/test753_lz4
[3] https://github.com/Cyan4973/lz4/releases/tag/r127




Archive powered by MHonArc 2.6.18.

Top of page