Overpass API developpement

[Sameera Bandara <>]

Hey Yann,

I have a dedicated server only for hosting OSM with 2 GHz 8 cores and 16 GB RAM. But I didn't prioritize down the processes as still I am not using the server and thought the processes would finish fast. Below are the consumption stats. It consumes 100% CPU :O

  1  [                                       0.0%]   5  [                                       0.0%]

  2  [                                       0.0%]   6  [                                       0.0%]

  3  [||                                     1.3%]   7  [                                       0.0%]

  4  [||||||||||||||||||||||||||||||||||||||99.3%]   8  [                                       0.0%]

  Mem[||||||||||||||||||||||||||||||||4.99G/15.7G]   Tasks: 41, 103 thr; 2 running

  Swp[||                               33.2M/861M]   Load average: 1.00 1.00 1.00 

                                                     Uptime: 9 days, 22:19:58

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command

30013 sameera    20   0 4929M 4903M  2620 R 100. 30.6 10h24:01 ./osm3s_query --progress --rules

31945 sameera    20   0 29544  4012  3320 R  1.3  0.0  0:03.52 htop

 8737 sameera    20   0 19240  1680   680 S  0.0  0.0  2:19.47 /data/osm/osm-3s_v0.7.55/bin/dispatcher --ar

 8692 sameera    20   0 33476 14168     4 S  0.0  0.1  2:22.61 /data/osm/osm-3s_v0.7.55/bin/dispatcher --os

 1757 root       20   0  283M   804     0 S  0.0  0.0  0:12.68 /usr/lib/accountsservice/accounts-daemon

 8546 root       20   0 69960  4240  3036 S  0.0  0.0  0:11.26 /usr/sbin/apache2 -k start

 1779 root       20   0  283M   804     0 S  0.0  0.0  0:12.58 /usr/lib/accountsservice/accounts-daemon

On Thu, 25 Jul 2019 at 09:01, yann Guillerm <> wrote:

whhooo 24H ! 

i use rules_loop on 2 server where i have priorize the process down and it always make it (for world db) within a few hours.

Actually for France only db, but the server is use for a lot more processes using sometime 100% of the processor, and it take 1h20 to finish the loop. The server specification : intel Xeon E3 1230 v6, 16go Ram.

in the pre-requirement, the documentation said to have at least 4Go ram, a lot of hard disk and that processor is not revelant.

are you sure that processor and ram are free to use ? 

sincerly, 

Yann.

Le jeu. 25 juil. 2019 à 08:12, Sameera Bandara <> a écrit :

Hi Roland,

Thank you very much for the reply. I actually already sorted out the issue after checkin nohup.out. As you have mentioned the issue was something with absolute paths. I specified the relative path to the database_dir and it worked. Below is the error I got initially.

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 40: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 41: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules/areas.osm3s: No such file or directory

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 42: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory

Now the areas are populated and I can issue queries for areas and get results, however the log says the query was timed-out. And it ran exactly 24 hours.

After 23h59m3s: in "make-area", part 0, on line 257. Stack: 0 of 0 8442756 of 0

After 23h59m18s: in "recurse", part 0, on line 255. Stack: 0 of 0 8445531 of 0 0 of 0

After 23h59m33s: in "recurse", part 0, on line 255. Stack: 0 of 0 8447843 of 0 0 of 0

After 23h59m48s: in "make-area", part 0, on line 257. Stack: 0 of 0 8450335 of 0

After 24h0m1s: in "make-area", part 0, on line 257. Stack: 0 of 0 8452832 of 0

</osm>

runtime error: Query timed out in "make-area" at line 257 after 86401 seconds.

<?xml version="1.0" encoding="UTF-8"?>

<osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77">

<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>

<meta osm_base=""/>

kind regards,

Sameera

On Wed, 24 Jul 2019 at 21:55, Roland Olbricht <> wrote:

Hi Sameera,

could you please check what the file nohup.out contains. There should be error messages there.

As a blind guess, please try the following:

- Stop the rules_loop.sh process, e.g. by "kill 9990".

- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
   replace there the line
     DB_DIR="`pwd`/$1"
   with
     DB_DIR="$1"

- Restart the process, i.e. run the command again please.

I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.

Kind regards,
Roland