Overpass API developpement

[mmd <>]

Hi there, again,


I used JMeter to fire about 40-50 requests/second using 8 parallel
threads against my server, using the following rather trivial QL query:

[out:xml];
node(${NODEID});
out;

Fortunately, I was able to reproduce the error message "End of script
output before headers: interpreter", but only if I used the HTTP/1.1
keep alive option in JMeter:

Use keep-alive -> Error rate about 0,25%  (after a few hundred requests)
No keep-alive  -> Error rate:      0,00%  (after 25'000 requests)


strace releaved that the interpreter binary was in fact killed by a
SIGTERM, which it received from another thread of the Apache Server itself.

At that time, the interpreter binary was still reading data from disk
and didn't output anything at all yet (not even an HTTP header).

fstat(5, {st_mode=S_IFREG|0666, st_size=16290895, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fd2e6025000
fstat(5, {st_mode=S_IFREG|0666, st_size=16290895, ...}) = 0
lseek(5, 16290895, SEEK_SET)            = 16290895
write(5, "2016-04-24 11:56:32 [5110] reque"..., 56) = 56
close(5)                                = 0
munmap(0x7fd2e6025000, 4096)            = 0
sendto(4, "\311\0\0\0", 4, 0, NULL, 0)  = 4
sendto(4, "\264\0\0\0", 4, 0, NULL, 0)  = 4
sendto(4, "\0\0\0 \0\0\0\0", 8, 0, NULL, 0) = 8
sendto(4, "\304\255\202T", 4, 0, NULL, 0) = 4
recvfrom(4, "\0\0\0\0", 4, 0, NULL, NULL) = 4
select(1024, NULL, NULL, NULL, {0, 300000}) = ? ERESTARTNOHAND (To be
restarted if no handler)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=28044, si_uid=33} ---
+++ killed by SIGTERM +++


On the Apache side of things, there's usually quite some lengthy
epilogue to transfer HTTP Request and header to the CGI socket:

epoll_ctl(15, EPOLL_CTL_ADD, 16, {EPOLLIN, {u32=1074381544,
u64=140596828816104}}) = 0
futex(0x7fdf46abe5fc, FUTEX_WAIT_PRIVATE, 12882, NULL) = 0
futex(0x7fdf46abe5a8, FUTEX_WAKE_PRIVATE, 1) = 0
read(18, "POST /api/interpreter HTTP/1"..., 8000) = 275
stat("/home/mmd/osm3s_754/cgi-bin/interpreter", {st_mode=S_IFREG|0755,
st_size=77175266, ...}) = 0
open("/home/mmd/osm3s_754/cgi-bin/.htaccess", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
lstat("/home/mmd/osm3s_754/cgi-bin/interpreter", {st_mode=S_IFREG|0755,
st_size=77175266, ...}) = 0
socket(PF_LOCAL, SOCK_STREAM, 0)        = 17
connect(17, {sa_family=AF_LOCAL,
sun_path="/var/run/apache2/cgisock.28038"}, 32) = 0
writev(17,
[{"\1\0\0\0\0\0\0\0S\0\0\0\0\0\0\0\206m\0\0\32\0\0\0\377\377\377\377\377\377\377\377"...,
152}, {"/home/mmd/osm3s_754/cgi-bin/inte"..., 39}, {"/home/mmd/osm3s_7
54/cgi-bin/inte"..., 39}, {"/api/interpreter", 20}], 4) = 250
writev(17, [{"\32\0\0\0\0\0\0\0", 8}, {"HTTP_CONNECTION=keep-alive",
26}], 2) = 34
writev(17, [{".\0\0\0\0\0\0\0", 8},
{"CONTENT_TYPE=application/x-www-f"..., 46}], 2) = 54
writev(17, [{"\21\0\0\0\0\0\0\0", 8}, {"CONTENT_LENGTH=35", 17}], 2) = 25
writev(17, [{"\35\0\0\0\0\0\0\0", 8}, {"HTTP_USER_AGENT=Java/1.8.0_77",
29}], 2) = 37

In case of the error, this looks very stripped down, lacking all
relevant details.
Here's where we also see the kill SIGTERM to kill the interpreter binary.


times({tms_utime=159, tms_stime=169, tms_cutime=0, tms_cstime=0}) =
1991522777
socket(PF_LOCAL, SOCK_STREAM, 0)        = 19
connect(19, {sa_family=AF_LOCAL,
sun_path="/var/run/apache2/cgisock.28038"}, 32) = 0
write(19,
"\3\0\0\0\0\0\0\0X\0\0\0\0\0\0\0\206m\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
152) = 152
read(19, "\202\36\0\0", 4)              = 4
kill(7810, SIGTERM)                     = 0
kill(7810, SIG_0)                       = 0
select(0, NULL, NULL, NULL, {0, 10000}) = 0 (Timeout)
kill(7810, SIG_0)                       = -1 ESRCH (No such process)
close(19)                               = 0


Not sure how to move on from here. Might be some CGI issue in Apache in
conjunction with the HTTP/1.1 keep alive option, maybe?


Cheers