`Test::test_buffer_growth_threaded_comm` is throwing errors

[bdangit]

When I compile test_cat and run it, I get the following error:

$ cd test
$ g++ -Wall -lpthread -o test_env test_env.cc
...
$ ./test_env
Test::test_cat_pipe_redirection
END_TEST
Test::test_cat_file_redirection
END_TEST
20480
free(): invalid next size (normal)

[arun11299]

Hello @bdangit
Thanks for reporting the error.
Can you give me the output from valgrind as well ? That would be really helpful.

[bdangit]

on second look, it appears to be related to check_output where one of the buffers is not getting deallocating.

[vagrant@localhost test]$ ./test_cat
Test::test_cat_pipe_redirection
END_TEST
Test::test_cat_file_redirection
END_TEST
20480
*** Error in `./test_cat': free(): invalid next size (normal): 0x0000000001f1b940 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81499)[0x7f75d402d499]
./test_cat[0x40de48]
./test_cat[0x40c460]
./test_cat[0x409ed7]
./test_cat[0x408331]
./test_cat[0x404bb4]
./test_cat[0x4039b0]
./test_cat[0x403b25]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f75d3fce445]
./test_cat[0x402ec9]
======= Memory map: ========
00400000-00427000 r-xp 00000000 fd:02 98                                 /home/vagrant/cpp-subprocess/test/test_cat
00626000-00627000 r--p 00026000 fd:02 98                                 /home/vagrant/cpp-subprocess/test/test_cat
00627000-00628000 rw-p 00027000 fd:02 98                                 /home/vagrant/cpp-subprocess/test/test_cat
01f19000-01f3a000 rw-p 00000000 00:00 0                                  [heap]
7f75cc000000-7f75cc021000 rw-p 00000000 00:00 0
7f75cc021000-7f75d0000000 ---p 00000000 00:00 0
7f75d37ab000-7f75d37ac000 ---p 00000000 00:00 0
7f75d37ac000-7f75d3fac000 rw-p 00000000 00:00 0
7f75d3fac000-7f75d416f000 r-xp 00000000 fd:00 33603975                   /usr/lib64/libc-2.17.so
7f75d416f000-7f75d436e000 ---p 001c3000 fd:00 33603975                   /usr/lib64/libc-2.17.so
7f75d436e000-7f75d4372000 r--p 001c2000 fd:00 33603975                   /usr/lib64/libc-2.17.so
7f75d4372000-7f75d4374000 rw-p 001c6000 fd:00 33603975                   /usr/lib64/libc-2.17.so
7f75d4374000-7f75d4379000 rw-p 00000000 00:00 0
7f75d4379000-7f75d438e000 r-xp 00000000 fd:00 34115950                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f75d438e000-7f75d458d000 ---p 00015000 fd:00 34115950                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f75d458d000-7f75d458e000 r--p 00014000 fd:00 34115950                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f75d458e000-7f75d458f000 rw-p 00015000 fd:00 34115950                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f75d458f000-7f75d4690000 r-xp 00000000 fd:00 33603983                   /usr/lib64/libm-2.17.so
7f75d4690000-7f75d488f000 ---p 00101000 fd:00 33603983                   /usr/lib64/libm-2.17.so
7f75d488f000-7f75d4890000 r--p 00100000 fd:00 33603983                   /usr/lib64/libm-2.17.so
7f75d4890000-7f75d4891000 rw-p 00101000 fd:00 33603983                   /usr/lib64/libm-2.17.so
7f75d4891000-7f75d497a000 r-xp 00000000 fd:00 33604029                   /usr/lib64/libstdc++.so.6.0.19
7f75d497a000-7f75d4b79000 ---p 000e9000 fd:00 33604029                   /usr/lib64/libstdc++.so.6.0.19
7f75d4b79000-7f75d4b81000 r--p 000e8000 fd:00 33604029                   /usr/lib64/libstdc++.so.6.0.19
7f75d4b81000-7f75d4b83000 rw-p 000f0000 fd:00 33604029                   /usr/lib64/libstdc++.so.6.0.19
7f75d4b83000-7f75d4b98000 rw-p 00000000 00:00 0
7f75d4b98000-7f75d4baf000 r-xp 00000000 fd:00 33604001                   /usr/lib64/libpthread-2.17.so
7f75d4baf000-7f75d4dae000 ---p 00017000 fd:00 33604001                   /usr/lib64/libpthread-2.17.so
7f75d4dae000-7f75d4daf000 r--p 00016000 fd:00 33604001                   /usr/lib64/libpthread-2.17.so
7f75d4daf000-7f75d4db0000 rw-p 00017000 fd:00 33604001                   /usr/lib64/libpthread-2.17.so
7f75d4db0000-7f75d4db4000 rw-p 00000000 00:00 0
7f75d4db4000-7f75d4dd6000 r-xp 00000000 fd:00 33603968                   /usr/lib64/ld-2.17.so
7f75d4fc8000-7f75d4fce000 rw-p 00000000 00:00 0
7f75d4fd2000-7f75d4fd5000 rw-p 00000000 00:00 0
7f75d4fd5000-7f75d4fd6000 r--p 00021000 fd:00 33603968                   /usr/lib64/ld-2.17.so
7f75d4fd6000-7f75d4fd7000 rw-p 00022000 fd:00 33603968                   /usr/lib64/ld-2.17.so
7f75d4fd7000-7f75d4fd8000 rw-p 00000000 00:00 0
7ffc4c72c000-7ffc4c74d000 rw-p 00000000 00:00 0                          [stack]
7ffc4c7f1000-7ffc4c7f3000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted
[vagrant@localhost test]$
[vagrant@localhost test]$ valgrind --leak-check=yes ./test_cat
==23799== Memcheck, a memory error detector
==23799== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==23799== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==23799== Command: ./test_cat
==23799==
Test::test_cat_pipe_redirection
END_TEST
Test::test_cat_file_redirection
END_TEST
==23799== Syscall param read(buf) points to unaddressable byte(s)
==23799==    at 0x4E447FD: ??? (in /usr/lib64/libpthread-2.17.so)
==23799==    by 0x4035C7: subprocess::util::read_atmost_n(int, char*, unsigned long) (subprocess.hpp:280)
==23799==    by 0x403B8E: int subprocess::util::read_all<std::vector<char, std::allocator<char> > >(int, std::vector<char, std::allocator<char> >&) (subprocess.hpp:319)
==23799==    by 0x406B80: subprocess::detail::Communication::communicate(char const*, unsigned long) (subprocess.hpp:1435)
==23799==    by 0x404FC7: subprocess::detail::Streams::communicate(char const*, unsigned long) (subprocess.hpp:870)
==23799==    by 0x405040: subprocess::Popen::communicate(char const*, unsigned long) (subprocess.hpp:992)
==23799==    by 0x4050A7: subprocess::Popen::communicate() (subprocess.hpp:1006)
==23799==    by 0x40B7A3: subprocess::Buffer subprocess::detail::check_output_impl<std::initializer_list<char const*>>(std::initializer_list<char const*>&) (subprocess.hpp:1533)
==23799==    by 0x409246: subprocess::Buffer subprocess::check_output<>(std::initializer_list<char const*>)
(subprocess.hpp:1600)
==23799==    by 0x403960: test_buffer_growth() (test_cat.cc:28)
==23799==    by 0x403B24: main (test_cat.cc:45)
==23799==  Address 0x5c47680 is 0 bytes after a block of size 16,384 alloc'd
==23799==    at 0x4C2A243: operator new(unsigned long) (vg_replace_malloc.c:334)
==23799==    by 0x40DEDB: __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) (new_allocator.h:104)
==23799==    by 0x40C5A8: std::_Vector_base<char, std::allocator<char> >::_M_allocate(unsigned long) (stl_vector.h:168)
==23799==    by 0x409FD9: std::vector<char, std::allocator<char> >::_M_default_append(unsigned long) (vector.tcc:549)
==23799==    by 0x40837C: std::vector<char, std::allocator<char> >::resize(unsigned long) (stl_vector.h:667)
==23799==    by 0x403C3A: int subprocess::util::read_all<std::vector<char, std::allocator<char> > >(int, std::vector<char, std::allocator<char> >&) (subprocess.hpp:324)
==23799==    by 0x406B80: subprocess::detail::Communication::communicate(char const*, unsigned long) (subprocess.hpp:1435)
==23799==    by 0x404FC7: subprocess::detail::Streams::communicate(char const*, unsigned long) (subprocess.hpp:870)
==23799==    by 0x405040: subprocess::Popen::communicate(char const*, unsigned long) (subprocess.hpp:992)
==23799==    by 0x4050A7: subprocess::Popen::communicate() (subprocess.hpp:1006)
==23799==    by 0x40B7A3: subprocess::Buffer subprocess::detail::check_output_impl<std::initializer_list<char const*>>(std::initializer_list<char const*>&) (subprocess.hpp:1533)
==23799==    by 0x409246: subprocess::Buffer subprocess::check_output<>(std::initializer_list<char const*>)
(subprocess.hpp:1600)
==23799==
20480
Test::test_buffer_growth_threaded_comm
--23799-- VALGRIND INTERNAL ERROR: Valgrind received a signal 11 (SIGSEGV) - exiting
--23799-- si_code=128;  Faulting address: 0x0;  sp: 0x1002ca9e10

valgrind: the 'impossible' happened:
   Killed by fatal signal

host stacktrace:
==23799==    at 0x5804DB36: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux)
==23799==    by 0x5800C524: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux)
==23799==    by 0x5800C74A: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux)
==23799==    by 0x580961A7: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux)
==23799==    by 0x580A4A4A: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux)

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable (lwpid 23799)
==23799==    at 0x4C2A243: operator new(unsigned long) (vg_replace_malloc.c:334)
==23799==    by 0x510FAB8: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in /usr/lib64/libstdc++.so.6.0.19)
==23799==    by 0x5111340: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (in /usr/lib64/libstdc++.so.6.0.19)
==23799==    by 0x5111777: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (in /usr/lib64/libstdc++.so.6.0.19)
==23799==    by 0x403A29: test_buffer_growth_threaded_comm() (test_cat.cc:36)
==23799==    by 0x403B29: main (test_cat.cc:46)


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

[arun11299]

Hi @bdangit
Since I am not able to reproduce it, I have committed a possible fix.
Can you try it again ? Also, which compiler and OS are you running it on ?

Thanks.

[bdangit]

hey @arun11299,

just did a pull, re-compiled and tested ./test_cat. It works fine.

[vagrant@localhost test]$ g++ -g -Wall -std=c++11 -lpthread -o test_cat test_cat.cc
[vagrant@localhost test]$ ./test_cat
Test::test_cat_pipe_redirection
END_TEST
Test::test_cat_file_redirection
END_TEST
44190
Test::test_buffer_growth_threaded_comm
44190
END_TEST

Environment

[vagrant@localhost test]$ g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[vagrant@localhost test]$ uname -a
Linux localhost.localdomain 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[vagrant@localhost test]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[bdangit]

Since this works, would it be possible for you to tag a release as 1.0.2?

[arun11299]

Thanks for testing it out @bdangit .
Yes, I will release it as 1.0.2.

[bdangit]

you rock! thanks @arun11299