Overpass API developpement

Text archives Help


Re: [overpass] std::bad_alloc runtime error


Chronological Thread 
  • From: mmd <mmd.osm@gmail.com>
  • To: overpass@osm101.openstreetmap.fr
  • Subject: Re: [overpass] std::bad_alloc runtime error
  • Date: Wed, 27 Apr 2016 21:41:05 +0200

Hi,


>
> BTW: Regarding the 2 GB limit: There seems to be a tiny difference of
> 4096 bytes (or 1 page), which causes the query to get stuck with just 1
> GB (+ 1 page).
>
> * setrlimit will set the limit to 2147483648 bytes
> * mmap will however try to allocate 2147487744 bytes -> fail!
>
> I've never really seen a query move up to 2 GB. Could you investigate
> this a bit further? Probably memory is requested this way by the
> underlying C++ memory management functions.
>
> I could reproduce this via:
>
> strace -e signal,setrlimit,mmap ./osm3s_query
>
> ...
>


I should have included munmap as well in the strace command above,
otherwise it's a bit difficult to see what's going on. In chronological
order, strace traces the following mmap/munmap calls:

- Allocate 512 MB -> ok
- Deallocate 256 MB
- Allocate 1 GB -> ok
- Deallocate 512 MB
- Allocate 2 GB -> Cannot allocate memory due to RLIMIT_AS


So, for a short amount of time, we would have to allow ~3 GB to enable
queries with about 2 GB of memory consumption.

Due to the 2GB RLIMIT_AS, the memory expansion will be aborted, and
according to `/usr/bin/time -v ` the osm3s_query can only allocate 1 GB
at most.

Maximum resident set size (kbytes): 1067416

That's what I've noticed in top before and which I described in my
previous email as osm3s_query getting stuck at 1 GB.



Test query:
-----------

[out:json][timeout:25]
;
(
node[~"[^(alt_name)][(:de)$]"~"."];
way[~"[^(alt_name)][(:de)$]"~"."];
relation[~"[^(alt_name)][(:de)$]"~"."];
);
out body;
>;
out skel qt;




strace:
-------

setrlimit(RLIMIT_CPU, {rlim_cur=110, rlim_max=110}) = 0
setrlimit(RLIMIT_AS, {rlim_cur=2097152*1024, rlim_max=2097152*1024}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fc6a3d26000
{
"version": 0.6,
"generator": "Overpass API",
"osm3s": {
"timestamp_osm_base": "2016-04-06T00:00:00Z",
"copyright": "The data included in this document is from
www.openstreetmap.org. The data is made available under ODbL."
},
"elements": [

mmap(NULL, 638976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7fc6a22f9000
munmap(0x7fc6a22f9000, 638976) = 0
[...]
mmap(NULL, 134221824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fc69248d000
munmap(0x7fc69a48e000, 67112960) = 0
mmap(NULL, 268439552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fc68248c000
munmap(0x7fc69248d000, 134221824) = 0
mmap(NULL, 536875008, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fc66248b000
munmap(0x7fc68248c000, 268439552) = 0
mmap(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fc62248a000
munmap(0x7fc66248b000, 536875008) = 0
mmap(NULL, 2147487744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 2147618816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 134217728, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fc69a294000
munmap(0x7fc69a294000, 30851072) = 0
munmap(0x7fc6a0000000, 36257792) = 0
mmap(NULL, 2147487744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
munmap(0x7fc62248a000, 1073745920) = 0
munmap(0x7fc6a2294000, 1052672) = 0


]
}
runtime error: Query run out of memory using about 2048 MB of RAM.
+++ exited with 0 +++




Archive powered by MHonArc 2.6.18.

Top of page