Overpass API developpement

[mmd <>]

Hi,


> 
> BTW: Regarding the 2 GB limit: There seems to be a tiny difference of
> 4096 bytes (or 1 page), which causes the query to get stuck with just 1
> GB (+ 1 page).
> 
> * setrlimit will set the limit to   2147483648 bytes
> * mmap will however try to allocate 2147487744 bytes -> fail!
> 
> I've never really seen a query move up to 2 GB. Could you investigate
> this a bit further? Probably memory is requested this way by the
> underlying C++ memory management functions.
> 
> I could reproduce this via:
> 
> strace -e signal,setrlimit,mmap ./osm3s_query
> 
> ...
> 


I should have included munmap as well in the strace command above,
otherwise it's a bit difficult to see what's going on. In chronological
order, strace traces the following mmap/munmap calls:

- Allocate 512 MB     -> ok
- Deallocate 256 MB
- Allocate 1 GB       -> ok
- Deallocate 512 MB
- Allocate 2 GB       -> Cannot allocate memory due to RLIMIT_AS


So, for a short amount of time, we would have to allow ~3 GB to enable
queries with about 2 GB of memory consumption.

Due to the 2GB RLIMIT_AS, the memory expansion will be aborted, and
according to `/usr/bin/time -v ` the osm3s_query can only allocate 1 GB
at most.

Maximum resident set size (kbytes): 1067416

That's what I've noticed in top before and which I described in my
previous email as osm3s_query getting stuck at 1 GB.



Test query:
-----------

[out:json][timeout:25]
;
(
  node[~"[^(alt_name)][(:de)$]"~"."];
  way[~"[^(alt_name)][(:de)$]"~"."];
  relation[~"[^(alt_name)][(:de)$]"~"."];
);
out body;
>;
out skel qt;




strace:
-------

setrlimit(RLIMIT_CPU, {rlim_cur=110, rlim_max=110}) = 0
setrlimit(RLIMIT_AS, {rlim_cur=2097152*1024, rlim_max=2097152*1024}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fc6a3d26000
{
  "version": 0.6,
  "generator": "Overpass API",
  "osm3s": {
    "timestamp_osm_base": "2016-04-06T00:00:00Z",
    "copyright": "The data included in this document is from
www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

mmap(NULL, 638976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7fc6a22f9000
munmap(0x7fc6a22f9000, 638976)          = 0
[...]
mmap(NULL, 134221824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fc69248d000
munmap(0x7fc69a48e000, 67112960)        = 0
mmap(NULL, 268439552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fc68248c000
munmap(0x7fc69248d000, 134221824)       = 0
mmap(NULL, 536875008, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fc66248b000
munmap(0x7fc68248c000, 268439552)       = 0
mmap(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fc62248a000
munmap(0x7fc66248b000, 536875008)       = 0
mmap(NULL, 2147487744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 2147618816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 134217728, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fc69a294000
munmap(0x7fc69a294000, 30851072)        = 0
munmap(0x7fc6a0000000, 36257792)        = 0
mmap(NULL, 2147487744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
munmap(0x7fc62248a000, 1073745920)      = 0
munmap(0x7fc6a2294000, 1052672)         = 0


  ]
}
runtime error: Query run out of memory using about 2048 MB of RAM.
+++ exited with 0 +++