Overpass API developpement

[mmd <>]

Hi Roland,

Am 24.04.2016 um 23:04 schrieb Roland Olbricht:
> Hi mmd,
> 
> 
> Thank you for forwarding the report.
> 
> Investigating this just yielded the fix
> 4abe454a5460e77a4cb32b97ec9ab165f554a6bf resp.
> e1efbf1112c83fbaba1aea83471c000400891903.
> 

I have another attic testcase, which currently fails with 2 GB limit.

Fortunately, I have already created a fix for it, which reduces memory
consumption down to 77 MB.

Here's the link to the pull request. It includes some background, which
hopefully makes some sense:

https://github.com/drolbr/Overpass-API/pull/174

And the respective attic query:

[date:"2015-09-25T00:00:00Z"]
[timeout:600];
area["name:de"="Deutschland"]
    [boundary=administrative];
node(area)[railway=switch];
out count;


BTW: Regarding the 2 GB limit: There seems to be a tiny difference of
4096 bytes (or 1 page), which causes the query to get stuck with just 1
GB (+ 1 page).

* setrlimit will set the limit to   2147483648 bytes
* mmap will however try to allocate 2147487744 bytes -> fail!

I've never really seen a query move up to 2 GB. Could you investigate
this a bit further? Probably memory is requested this way by the
underlying C++ memory management functions.

I could reproduce this via:

strace -e signal,setrlimit,mmap ./osm3s_query

...


setrlimit(RLIMIT_CPU, {rlim_cur=62, rlim_max=62}) = 0
setrlimit(RLIMIT_AS, {rlim_cur=2097152*1024, rlim_max=2097152*1024}) = 0


mmap(NULL, 2147487744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 2147618816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 134217728, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fb692b17000
mmap(NULL, 2147487744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)


  ]
}
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fb69c7e0000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fb69c7e0000
runtime error: Query run out of memory using about 2048 MB of RAM.
+++ exited with 0 +++





Best,
mmd