spoa-mirror failing

[mark-k-english]

Version 1.2.6 build 2388, centos 6

POSTing a message causes the process to core dump.

gdb shows:

Program terminated with signal 11, Segmentation fault.
#0 0x00000000004051cf in mir_ptr_free (data=0x7f2a7119e121) at spoa-message.c:476
476 PTR_FREE((*data)->path);

mirror.log

[zaga00]

Hello @mark-k-english,

I apologize for not answering so far. I should have some free time soon so I'll pay attention to this.

[mark-k-english]

I understand. I pulled down the latest version spoa-mirror v1.2.9 [build 2407] by Miroslav Zagorac <mzagorac@haproxy.com>, Apr 25 2020 In spoa-message.c I added the following at line 499 to address a seg fault. if (! _NULL((*data)->hdrs)) list_for_each_entry_safe(hdr, hdr_back, (*data)->hdrs, list) buffer_ptr_free(&hdr); Also, I've come across issues with header parsing/processing in util.c. I have not found the true issue but I worked around it with the following which I hope you can address properly; i realize this is a major hack. I started getting malloc errors because n would equal a very very large number and then pass that to buffer_grow which failed due to OOM. This is within buffer_grow_va /* if (_nNULL(src = va_arg(ap, typeof(src)))) n = va_arg(ap, typeof(n)); */ if (_nNULL(src = va_arg(ap, typeof(src)))) { int l = strlen((const char *) src); n = va_arg(ap, typeof(n)); if(l == 0) { if (_nNULL(src = va_arg(ap, typeof(src)))) { l = strlen((const char *) src); n = va_arg(ap, typeof(n)); } } It's been years since I've done C so don't laugh to loud. Currently with these changes I've not run into anything else. Thanks - ME

…

On Mon, Apr 27, 2020 at 11:49 AM Miroslav Zagorac ***@***.***> wrote: Hello @mark-k-english <https://github.com/mark-k-english>, I apologize for not answering so far. I should have some free time soon so I'll pay attention to this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZ5AC6XT67DP5CBQWSVJD3ROXHSTANCNFSM4LFI76FQ> .

[mark-k-english]

Oh, one more thing for you to review. In spoa-message.c line 282 you're breaking when a header has no value, shouldn't this be a continue to finish processing the other headers? TX ME On Tue, Apr 28, 2020 at 10:14 AM Mark English <mark.k.english@gmail.com> wrote:

…

I understand. I pulled down the latest version spoa-mirror v1.2.9 [build 2407] by Miroslav Zagorac ***@***.***>, Apr 25 2020 In spoa-message.c I added the following at line 499 to address a seg fault. if (! _NULL((*data)->hdrs)) list_for_each_entry_safe(hdr, hdr_back, (*data)->hdrs, list) buffer_ptr_free(&hdr); Also, I've come across issues with header parsing/processing in util.c. I have not found the true issue but I worked around it with the following which I hope you can address properly; i realize this is a major hack. I started getting malloc errors because n would equal a very very large number and then pass that to buffer_grow which failed due to OOM. This is within buffer_grow_va /* if (_nNULL(src = va_arg(ap, typeof(src)))) n = va_arg(ap, typeof(n)); */ if (_nNULL(src = va_arg(ap, typeof(src)))) { int l = strlen((const char *) src); n = va_arg(ap, typeof(n)); if(l == 0) { if (_nNULL(src = va_arg(ap, typeof(src)))) { l = strlen((const char *) src); n = va_arg(ap, typeof(n)); } } It's been years since I've done C so don't laugh to loud. Currently with these changes I've not run into anything else. Thanks - ME On Mon, Apr 27, 2020 at 11:49 AM Miroslav Zagorac < ***@***.***> wrote: > Hello @mark-k-english <https://github.com/mark-k-english>, > > I apologize for not answering so far. I should have some free time soon > so I'll pay attention to this. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#11 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHZ5AC6XT67DP5CBQWSVJD3ROXHSTANCNFSM4LFI76FQ> > . >

[zaga00]

Hello @mark-k-english,

regarding spoa-message.c:282 line, in case the buffer_grow() function cannot allocate memory, the processing of the other headers is interrupted because it no longer makes sense.

As for the rest, I'll take a look when I get more free time. Thanks for your understanding and for reporting the bug. :)

[zaga00]

Hello, @mark-k-english,

please check if the latest version of the program solves your problem.

Best regards,
Miroslav Zagorac

[poncheg]

@zaga00 no it doesn help
local env get the same error
`
#сборка haproxy бинарника
FROM centos:7 as build
RUN yum -y install autoconf automake git libev-devel gcc make openssl-devel git pkgconfig libcurl-devel
RUN git clone https://github.com/haproxytech/spoa-mirror &&
cd spoa-mirror &&
ls -la &&
./scripts/bootstrap &&
./configure --enable-debug &&
make all &&
ls ./src/spoa-mirror
RUN yum -y install dnf-plugins-core
#RUN yum config-manager --set-enabled powertools
RUN yum -y install wget gcc pcre-static pcre-devel openssl-devel make haproxy:2.3.6
RUN wget http://www.haproxy.org/download/2.3/src/haproxy-2.3.6.tar.gz -O ~/haproxy.tar.gz
RUN tar xzvf ~/haproxy.tar.gz -C ~/
WORKDIR /root/haproxy-2.3.6
RUN make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_CRYPT_H=1 USE_LIBCRYPT=1
RUN make install
RUN ls

FROM centos:7
RUN yum -y install ca-certificates libev pkgconfig libcurl autoconf automake git libev-devel gcc make openssl-devel git pkgconfig libcurl-devel
WORKDIR /root/
COPY --from=build /spoa-mirror/src/spoa-mirror /usr/local/bin/spoa-mirror
COPY --from=build /root/haproxy-2.3.6/haproxy /usr/local/bin/haproxy
RUN chmod +x /usr/local/bin/haproxy
RUN chmod +x /usr/local/bin/spoa-mirror
RUN ls /usr/local/sbin/
RUN groupadd --system haproxy && useradd --gid haproxy --home-dir /var/lib/haproxy --no-create-home --system haproxy
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
COPY mirror.conf /usr/local/etc/haproxy/mirror.conf
COPY ssl/ /usr/local/etc/ssl/
COPY docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod +x /docker-entrypoint.sh
ENTRYPOINT [ "/docker-entrypoint.sh" ]
CMD "haproxy" "-f" "/usr/local/etc/haproxy/haproxy.cfg"
`

log debug output:
<134>Mar 14 19:44:41 haproxy[9]: Connect from 172.17.0.1:61558 to 172.17.0.3:80 (prod_test_api/HTTP) [ 1][ 19.613873] worker_accept_cb(0x7f946166ea40, 0x7ffc18811180, 0x00000001) [ 0][ 19.613922] socket_set_nonblocking(28) [ 0][ 19.613931] socket_set_keepalive(28, 1, -1, -1, -1) [ 1][ 19.614031] worker_async_cb(0x7f94540008c0, 0xa5c3f0, 0x00080000) [ 1][ 19.614809] read_frame_cb(0x7f94540008c0, 0xa5e698, 0x00000001) [ 1][ 19.614895] acquire_incoming_frame(0xa5e680) [ 1][ 19.614922] reset_frame(0x7f945400f430) [ 1][ 19.614992] buffer_free(0x7f945400f4c0) [ 1][ 19.614998] frame_recv(0x7f945400f430) [ 1][ 19.615005] tcp_recv(0x7f945400f430, 4, " length") [ 1][ 19.615012] tcp_recv(0x7f945400f430, 129, " data") [ 1][ 19.615032] handle_hahello(0x7f945400f430) [ 1][ 19.615062] spoe_decode_frame("HAPROXY-HELLO", 0x7f945400f430, 1, -1, 7, ...) [ 1][ 19.615134] spoe_vdecode(0x7f945400f430, 0x7f945cb07af8:0x7f945400f4ec, 0x7f945400f56d, 0, 0x7f945cb07990) [ 1][ 19.615206] spoe_decode_kv(0x7f945400f430, 0x7f945cb07c78:0x7f945400f4f3, 0x7f945400f56d, ...) [ 1][ 19.615215] spoe_decode_kv_item(0x7f945400f430, 0x7f945cb079f0:0x7f945400f506, 0x7f945400f56d, 5, 0x40ce48) [ 1][ 19.615230] check_proto_version_cb(0x7f945400f430, 0x7f945400f508, 0x7f945cb077b8) [ 1][ 19.615238] spoe_decode_kv_item(0x7f945400f430, 0x7f945cb079f0:0x7f945400f51a, 0x7f945400f56d, 3, 0x40cf0c) [ 1][ 19.615242] check_max_frame_size_cb(0x7f945400f430, 0x7f945cb077c8, (nil)) [ 1][ 19.615245] spoe_decode_kv_item(0x7f945400f430, 0x7f945cb079f0:0x7f945400f52b, 0x7f945400f56d, 5, 0x40d09b) [ 1][ 19.615266] check_capabilities_cb(0x7f945400f430, 0x7f945400f52d, 0x7f945cb077b8) [ 1][ 19.615423] spoe_decode_kv_item(0x7f945400f430, 0x7f945cb079f0:0x7f945400f547, 0x7f945400f56d, 5, 0x40d3bf) [ 1][ 19.615479] check_engine_id_cb(0x7f945400f430, 0x7f945400f549, 0x7f945cb077b8) [ 1][ 19.615483] mem_dup(0x7f945400f549, 36) [ 1][ 19.615493] prepare_agenthello(0x7f945400f430) [ 1][ 19.615581] spoe_encode_frame("AGENT-HELLO", 0x7f945400f430, 2, 101, 0x00000001, 0, ...) [ 1][ 19.615626] spoe_vencode(0x7f945400f430, 0x7f945cb05ad8:0x7f945400f4f1, 0, 0x7f945cb05ae0) [ 1][ 19.615678] write_frame(0xa5e680, 0x7f945400f430) [ 1][ 19.615721] worker_async_cb(0x7f94540008c0, 0xa5c3f0, 0x00080000) [ 1][ 19.615745] write_frame_cb(0x7f94540008c0, 0xa5e6c8, 0x00000002) [ 1][ 19.615763] acquire_outgoing_frame(0xa5e680) [ 1][ 19.615768] frame_send(0xa5e680, 0x7f945400f430) [ 1][ 19.615772] tcp_send(0xa5e680, 0x7f945400f430, 4, " length") [ 1][ 19.615810] tcp_send(0xa5e680, 0x7f945400f430, 54, " data") [ 1][ 19.615878] release_frame(0x7f945400f430) [ 1][ 19.615883] buffer_free(0x7f945400f4c0) [ 1][ 19.615902] worker_async_cb(0x7f94540008c0, 0xa5c3f0, 0x00080000) [ 1][ 19.615973] read_frame_cb(0x7f94540008c0, 0xa5e698, 0x00000001) [ 1][ 19.616010] acquire_incoming_frame(0xa5e680) [ 1][ 19.616031] reset_frame(0x7f945400f430) [ 1][ 19.616069] buffer_free(0x7f945400f4c0) [ 1][ 19.616119] frame_recv(0x7f945400f430) [ 1][ 19.616128] tcp_recv(0x7f945400f430, 4, " length") [ 1][ 19.616133] tcp_recv(0x7f945400f430, 483, " data") [ 1][ 19.616137] handle_hanotify(0x7f945400f430) [ 1][ 19.616141] spoe_decode_frame("HAPROXY-NOTIFY", 0x7f945400f430, 3, 0, 7, ...) [ 1][ 19.616157] spoe_vdecode(0x7f945400f430, 0x7f945cb07b98:0x7f945400f4ec, 0x7f945400f6cf, 0, 0x7f945cb07a30) [ 1][ 19.616162] acc_payload(0x7f945400f430) [ 1][ 19.616166] process_incoming_frame(0x7f945400f430) [ 1][ 19.616197] worker_async_cb(0x7f94540008c0, 0xa5c3f0, 0x00080000) [ 1][ 19.616259] process_frame_cb(0x7f94540008c0, 0x7f945400f468, 0x00000100) [ 1][ 19.616284] spoe_vdecode(0x7f945400f430, 0x7f945cb07cc0:0x7f945400f4f3, 0x7f945400f6cf, 4, 0x7f945cb07ba0) [ 1][ 19.616360] spoa_msg_mirror(0x7f945400f430, 0x7f945cb07cc0:0x7f945400f4fa, 0x7f945400f6cf) [ 1][ 19.616369] spoe_vdecode(0x7f945400f430, 0x7f945cb07c18:0x7f945400f4fa, 0x7f945400f6cf, 0, 0x7f945cb07ae0) [ 1][ 19.616373] spoe_vdecode(0x7f945400f430, 0x7f945cb07c18:0x7f945400f4fb, 0x7f945400f6cf, 4, 0x7f945cb07ae0) [ 1][ 19.616377] spoa_msg_arg_dup(0x7f945400f430, 0, "arg_method", 10, 0x7f945cb07c30, 0x7f9454013520:(nil), (nil), "allocate memory for headers") [ 1][ 19.616402] mem_dup(0x7f945400f508, 4) [ 1][ 19.616406] spoe_vdecode(0x7f945400f430, 0x7f945cb07c18:0x7f945400f50c, 0x7f945400f6cf, 4, 0x7f945cb07ae0) [ 1][ 19.616451] spoa_msg_arg_dup(0x7f945400f430, 1, "arg_path", 8, 0x7f945cb07c30, 0x7f9454013518:(nil), (nil), "allocate memory for headers") [ 1][ 19.616475] mem_dup(0x7f945400f517, 31) [ 1][ 19.616480] spoe_vdecode(0x7f945400f430, 0x7f945cb07c18:0x7f945400f536, 0x7f945400f6cf, 4, 0x7f945cb07ae0) [ 1][ 19.616484] spoa_msg_arg_dup(0x7f945400f430, 2, "arg_ver", 7, 0x7f945cb07c30, 0x7f9454013530:(nil), (nil), "allocate memory for headers") [ 1][ 19.616487] mem_dup(0x7f945400f540, 3) [ 1][ 19.616491] spoe_vdecode(0x7f945400f430, 0x7f945cb07c18:0x7f945400f543, 0x7f945400f6cf, 4, 0x7f945cb07ae0) [ 1][ 19.616526] spoa_msg_arg_hdrs(0x7f945400f430, 0x7f945400f54f, 0x7f945400f646, 0x7f9454013538) [ 1][ 19.616530] spoe_vdecode(0x7f945400f430, 0x7f945cb07b70:0x7f945400f54f, 0x7f945400f646, 4, 0x7f945cb07a70) [ 1][ 19.616535] buffer_alloc(16384, 0x7f945400f550, ...) [ 1][ 19.616545] buffer_grow(0x7f94540135e0, 0x7f945400f550, 12) [ 1][ 19.616549] spoe_vdecode(0x7f945400f430, 0x7f945cb07b70:0x7f945400f55c, 0x7f945400f646, 4, 0x7f945cb07a70) [ 1][ 19.616571] buffer_grow_va(0x7f94540135e0, 0x416d33, 2, ...) [ 1][ 19.616577] buffer_grow(0x7f94540135e0, 0x416d33, 2) [ 1][ 19.616610] buffer_grow(0x7f94540135e0, 0x7f945400f55d, 5) [ 1][ 19.616615] buffer_grow(0x7f94540135e0, 0x416d31, 140273631887361) <132>Mar 14 19:44:41 haproxy[9]: SPOE: [mirror] <EVENT:on-backend-http-request> sid=1 st=257 0/8/8/-1/16 0/0 0/0 1/1 [NOTICE] 072/194441 (1) : haproxy version is 2.3.6-7851701 [NOTICE] 072/194441 (1) : path to executable is /usr/local/bin/haproxy [ALERT] 072/194441 (1) : Current program 'mirror' (8) exited with code 139 (Segmentation fault) [ALERT] 072/194441 (1) : exit-on-failure: killing every processes with SIGTERM [ALERT] 072/194441 (1) : Current worker #1 (9) exited with code 143 (Terminated) [WARNING] 072/194441 (1) : All workers exited. Exiting... (139)