Overpass API developpement

Text archives Help


Re: [overpass] import of Belgium extract failed flush-size <> ram


Chronological Thread 
  • From: marc marc < >
  • To: " " < >
  • Subject: Re: [overpass] import of Belgium extract failed flush-size <> ram
  • Date: Wed, 20 Sep 2017 21:45:17 +0000
  • Accept-language: fr-BE, en-US
  • Authentication-results: listes.openstreetmap.fr; dkim=none (message not signed) header.d=none;listes.openstreetmap.fr; dmarc=none action=none header.from=hotmail.com;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99

Le 20. 09. 17 à 22:28, mmd a écrit :
>> I tried with --flush-size=6 and the import succeeded
>>
>> what is the exact meaning of --flush-size = 1 ? a flush after each
>> modified node/way/relation ? after each tag ? each changeset ?
>>
>> Is it a "formula" between flush-size and maximum consumed ram ?
>> I imagine something like x Mo + x Mo x flush-size = max malloc ram
>
> update_database.cc has the following definitions:
>
> Default value: flush_limit = 16*1024*1024
>
> When setting --flush-size=6 as a parameter, flush_limit will be set to
> 6*1024*1024.
>
> Very roughly speaking, when importing nodes/ways/relations, the number
> of objects will be counted, and once the flush-size threshold is
> reached, objects will be flushed to disk and the counter reset to zero.
> As objects have varying sizes, this is a rough approximation of the
> actual memory requirements.
>
> Setting it close to the maximum value your memory permits reduces
> the overall runtime of the initial load.

yes of course, it is what I 'm trying todo : found the hightest value to
speedup the initiale load AND having a value that avoid a out of memory.

But due to that flush-size is based on the number of objects and not
their size, it is difficult to prevent an out of memory when
a change occurs on a lot of "bigger than average" objects.
is it possible that flush-size uses the actual size of the malloc
instead of the number of objects ?
or is this option too little used to make it important?

Regards,
Marc



Archive powered by MHonArc 2.6.19+.

Top of page