New installation crashes appear to be ssh-related

Zeek
1

We have a new Bro installation, built from source on Debian wheezy, that keeps core dumping. It looks like it’s choking on some code related to ssh. Here is the diag for the latest crash. It is identical to the other one I have:

[BroControl] > diag

[bro]

Bro 2.3-633

Linux 3.2.0-4-686-pae

No gdb installed.

==== No reporter.log

==== stderr.log

listening on eth1, capture length 8192 bytes

bro: /root/bro/build/src/analyzer/protocol/ssh/ssh_pac.cc:1382: int binpac::SSH::SSH2_KEXINIT::Parse(binpac::const_byteptr, binpac::const_byteptr, binpac::SSH::ContextSSH*, int): Assertion `t_dataptr_after_cookie <= t_end_of_data’ failed.

/usr/local/bro/share/broctl/scripts/run-bro: line 100: 10307 Aborted (core dumped) nohup “$mybro” “$@”

==== stdout.log

max memory size (kbytes, -m) unlimited

data seg size (kbytes, -d) unlimited

virtual memory (kbytes, -v) unlimited

core file size (blocks, -c) unlimited

==== .cmdline

-i eth1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto

==== .env_vars

PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site

CLUSTER_NODE=

==== .status

RUNNING [net_run]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[BroControl] >

This is just running the default setup, with the local subnets configured, as we are just starting with Bro. This is a really low end server, but the capture interface is only running at 100 meg so there are really no resource issues. (Yes, this is a 32-bit box. It’s pretty old. That’s why I built from source.)

The first crash occurred after a few minutes. Then it ran for nearly 24 hours before this crash. Is there something I can tweak to prevent this?

Thanks,

Ted Llewellyn

2

Hi Ted,

Thanks for reporting this. I’ll look into it.

–Vlad

3

Vlad,

I happened to run across HILTI while I was looking at something not related to Bro. My output seems to come from binpac, and according to the HILTI folks they develop on 64-bit platforms and promise nothing if it’s run on 32-bit hardware. I thought binpac was just supposed to be a plugin, which says to me it can be turned off or I could rebuild without it, if I could find out how. “./configure –help” wasn’t very helpful about this. Does this sound plausible?

Thanks,

Ted

4

Ted, mind filing a ticket so that we track this one?

Robin

5

Also, do you happen to have a core dump of this? It would help with debugging.

To answer your question about BinPAC - BinPAC is a Binary Protocol Analyzer Compiler. Some analyzers in Bro are written in a language that BinPAC will compile to C++. When you compile Bro, this compilation happens, and then that C++ code gets compiled with the rest of Bro. So, it’s not really a plugin - you could technically build Bro without BinPAC, but in practice, you wouldn’t want to do that.

Hope that makes sense,

–Vlad

6

Yes, I have core dumps. Is there an upload site? I’m concerned about information leakage, also. This is a lab environment, but I still don’t want too much information about it being distributed in public forums.

Ted Llewellyn

Sr. Network Planning Engineer

VoIP Engineering

Frontier Communications

120 Plymouth Ave. N.

Rochester, NY 14608

585-413-9743

7

Robin,

I submitted a ticket, 1361. It won't let me attach the core dump as it's too big. How do I upload that?

Thanks,
Ted

8

Vlad,

It crashed again this morning. The crash on 3/29 was at 6:29 local time, and the crash this morning was at 6:27 local time. I’m not aware of anything that happens here around that time on a regular basis.

The diag looks pretty much the same:

[BroControl] > diag

[bro]

Bro 2.3-633

Linux 3.2.0-4-686-pae

No gdb installed.

==== No reporter.log

==== stderr.log

listening on eth1, capture length 8192 bytes

bro: /root/bro/build/src/analyzer/protocol/ssh/ssh_pac.cc:1382: int binpac::SSH::SSH2_KEXINIT::Parse(binpac::const_byteptr, binpac::const_byteptr, binpac::SSH::ContextSSH*, int): Assertion `t_dataptr_after_cookie <= t_end_of_data’ failed.

/usr/local/bro/share/broctl/scripts/run-bro: line 100: 24675 Aborted (core dumped) nohup “$mybro” “$@”

==== stdout.log

max memory size (kbytes, -m) unlimited

data seg size (kbytes, -d) unlimited

virtual memory (kbytes, -v) unlimited

core file size (blocks, -c) unlimited

==== .cmdline

-i eth1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto

==== .env_vars

PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site

CLUSTER_NODE=

==== .status

RUNNING [net_run]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[BroControl] >

Here is the ssh.log, with the local addresses obfuscated:

#separator \x09

#set_separator ,

#empty_field (empty)

#unset_field -

#path ssh

#open 2015-03-31-06-12-54

#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p version auth_success direction client server cipher_alg mac_alg compression_alg kex_alg host_key_alg host_key remote_location.country_code remote_location.region remote_location.city remote_location.latitude remote_location.longitude

#types time string addr port addr port count bool enum string string string string string string string string string string string double double

1427796767.723015 CekWob4QEqOlP0oqp8 115.239.230.133 57922 10.10.20.217 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427796768.761095 Ctm96W1UH7UUMJkEhk 115.239.230.133 42380 10.10.24.233 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427796773.022767 CBJCTy0vfPn8efye4 115.239.230.133 45326 10.10.20.194 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427796998.016420 CPC3hO10j08ML06CRj 115.231.218.130 56223 10.10.20.217 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss - - - - - -

1427796998.641613 CMUo9V3XqIY3J45Arl 115.231.218.130 51297 10.10.20.194 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427797000.236567 C4F5Ca2TZOVL55re0i 115.231.218.130 60792 10.10.24.233 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427797056.937244 CeElA5RdppTwHbR6b 183.136.216.4 34758 10.10.24.233 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427797056.134247 CDKUcz2vwqwCQ6FMP 183.136.216.4 57005 10.10.20.217 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427797314.991566 CPkA7E3jOaA4O3n6Zj 115.239.248.238 46652 10.10.20.217 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427797315.312565 CF4kqy4fSKVNiRwHKa 115.239.248.238 34778 10.10.20.194 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427797316.044014 CfKqmt3d5HTfWS7xyc 115.239.248.238 50058 10.10.24.233 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

1427797665.315966 CUdfQY3IPL1xx4UtY7 115.231.218.131 57464 10.10.20.194 22 2 - - SSH-2.0-PUTTY SSH-2.0-2.0.12 3des-cbc hmac-sha1 none diffie-hellman-group1-sha1 ssh-dss 92:fe:da:65:a4:2e:ae:30:a4:26:a9:62:56:35:30:37 - - - - -

I can only get the core files down to about 15 meg, so they won’t attach to the ticket. Should I try sending it directly to your gmail account, so the whole list doesn’t get it?

Ted

9

Thanks for filing the ticket. For the core, actually what would be
most helpful right now I believe is a stack backtrace. Your crash
report didn't have that, it looks like there's no gdb installed. Can
you install gdb and then run "gdb bro core" + "bt" as described here:
https://www.bro.org/support/reporting-problems.html#getting-more-information-after-acrash

For the core itself, I think the best thing might be to hold on to it
for now, just the core won't be useful for others much anyways, as one
also needs to the binary and potentially a similar system to use it.
So if you could keep binary and core somewhere until this is resolved,
that would be best for now.

Robin

10

Robin,

I have attached the backtrace to the ticket, but here it is also:

(gdb) bt
#0 0xb76e6424 in __kernel_vsyscall ()
#1 0xb71b4661 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0xb71b7a92 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#3 0xb71ad878 in __assert_fail () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#4 0x083eaabe in binpac::SSH::SSH2_KEXINIT::Parse (this=0xac7ac978,
    t_begin_of_data=t_begin_of_data@entry=0xac533ff6 "",
    t_end_of_data=t_end_of_data@entry=0xac534008 "\210>%\255\035",
    t_context=t_context@entry=0xad9419e8, t_byteorder=t_byteorder@entry=0)
    at /root/bro/build/src/analyzer/protocol/ssh/ssh_pac.cc:1382
#5 0x083eac60 in binpac::SSH::SSH2_Message::Parse (this=0xad22d938,
    t_begin_of_data=t_begin_of_data@entry=0xac533ff6 "",
    t_end_of_data=t_end_of_data@entry=0xac534008 "\210>%\255\035",
    t_context=t_context@entry=0xad9419e8, t_byteorder=t_byteorder@entry=0)
    at /root/bro/build/src/analyzer/protocol/ssh/ssh_pac.cc:1216
#6 0x083eb160 in binpac::SSH::SSH2_Key_Exchange::ParseBuffer (
    this=0xab743610, t_flow_buffer=0xafd04dc0, t_context=0xad9419e8,
    t_byteorder=0) at /root/bro/build/src/analyzer/protocol/ssh/ssh_pac.cc:1090
#7 0x083eb4d6 in binpac::SSH::SSH_Key_Exchange::ParseBuffer (this=0xaeb2e878,
    t_flow_buffer=0xafd04dc0, t_context=0xad9419e8, t_byteorder=0)
    at /root/bro/build/src/analyzer/protocol/ssh/ssh_pac.cc:520
#8 0x083eb6ff in binpac::SSH::SSH_PDU::ParseBuffer (this=0xaeb323f8,
    t_flow_buffer=0xafd04dc0, t_context=0xad9419e8)
    at /root/bro/build/src/analyzer/protocol/ssh/ssh_pac.cc:360
---Type <return> to continue, or q <return> to quit---
#9 0x083eb982 in binpac::SSH::SSH_Flow::NewData (this=0xafd635b8,
    t_begin_of_data=0xac533ff0 "", t_end_of_data=0xac534008 "\210>%\255\035")
    at /root/bro/build/src/analyzer/protocol/ssh/ssh_pac.cc:2913
#10 0x083e2855 in analyzer::SSH::SSH_Analyzer::DeliverStream (this=0xadc3e1f0,
    len=24, data=0xac533ff0 "", orig=true)
    at /root/bro/src/analyzer/protocol/ssh/SSH.cc:71
#11 0x08479f34 in analyzer::Analyzer::NextStream (this=0xadc3e1f0, len=24,
    data=0xac533ff0 "", is_orig=true) at /root/bro/src/analyzer/Analyzer.cc:245
#12 0x0847a72c in analyzer::Analyzer::ForwardStream (this=0xae014040, len=24,
    data=0xac533ff0 "", is_orig=true) at /root/bro/src/analyzer/Analyzer.cc:331
#13 0x0840ddec in analyzer::tcp::TCP_Reassembler::DeliverBlock (
    this=this@entry=0xadbc1cb0, seq=16, len=len@entry=24, data=0xac533ff0 "")
    at /root/bro/src/analyzer/protocol/tcp/TCP_Reassembler.cc:647
#14 0x0840e2cc in BlockInserted (start_block=<optimized out>,
    this=<optimized out>)
    at /root/bro/src/analyzer/protocol/tcp/TCP_Reassembler.cc:393
#15 analyzer::tcp::TCP_Reassembler::BlockInserted (this=0xadbc1cb0,
    start_block=0xac648218)
    at /root/bro/src/analyzer/protocol/tcp/TCP_Reassembler.cc:368
#16 0x0840db2e in analyzer::tcp::TCP_Reassembler::DataSent (this=0xadbc1cb0,
    t=1427797676.2736609, seq=16, len=<optimized out>, data=0xa8d1a4a "",
    replaying=true)
    at /root/bro/src/analyzer/protocol/tcp/TCP_Reassembler.cc:492
---Type <return> to continue, or q <return> to quit---
#17 0x0840beeb in analyzer::tcp::TCP_Endpoint::DataSent (this=0xadc74340,
    t=1427797676.2736609, seq=16, len=24, caplen=24, data=0xa8d1a4a "",
    ip=0xbfbeacac, tp=0xa8d1a2a)
    at /root/bro/src/analyzer/protocol/tcp/TCP_Endpoint.cc:205
#18 0x08408c76 in DeliverData (flags=..., is_orig=<optimized out>,
    rel_data_seq=16, endpoint=0xadc74340, tp=0xa8d1a2a, ip=0xbfbeacac,
    caplen=<optimized out>, len=<optimized out>, data=<optimized out>,
    t=<optimized out>, this=0xae014040)
    at /root/bro/src/analyzer/protocol/tcp/TCP.cc:947
#19 analyzer::tcp::TCP_Analyzer::DeliverPacket (this=0xae014040, len=24,
    data=0xa8d1a4a "", is_orig=true, seq=18446744073709551615, ip=0xbfbeacac,
    caplen=24) at /root/bro/src/analyzer/protocol/tcp/TCP.cc:1347
#20 0x0847a118 in analyzer::Analyzer::NextPacket (this=0xae014040, len=56,
    data=0xa8d1a2a "\230", <incomplete sequence \335>, is_orig=true,
    seq=18446744073709551615, ip=0xbfbeacac, caplen=56)
    at /root/bro/src/analyzer/Analyzer.cc:222
#21 0x081951c4 in Connection::NextPacket (this=0xafd52858,
    t=1427797676.2736609, is_orig=1, ip=0xbfbeacac, len=56, caplen=56,
    data=@0xbfbeaa68: 0xa8d1a2a "\230", <incomplete sequence \335>,
    record_packet=@0xbfbeaa70: 1, record_content=@0xbfbeaa74: 1,
    hdr=0xa097074, pkt=0xa8d1a08 "", hdr_size=14) at /root/bro/src/Conn.cc:260
#22 0x08238ca0 in NetSessions::DoNextPacket (this=this@entry=0xa8d3a10,
    t=1427797676.2736609,
---Type <return> to continue, or q <return> to quit---
    t@entry=<error reading variable: Could not find type for DW_OP_GNU_const_type>, hdr=hdr@entry=0xa097074, ip_hdr=ip_hdr@entry=0xbfbeacac,
    pkt=pkt@entry=0xa8d1a08 "", hdr_size=hdr_size@entry=14,
    encapsulation=encapsulation@entry=0x0) at /root/bro/src/Sessions.cc:760
#23 0x0823a3bc in NetSessions::NextPacket (this=0xa8d3a10,
    t=1427797676.2736609, hdr=0xa097074, pkt=0xa8d1a08 "", hdr_size=14)
    at /root/bro/src/Sessions.cc:231
#24 0x08205de6 in net_packet_dispatch (t=1427797676.2736609, hdr=0xa097074,
    pkt=0xa8d1a08 "", hdr_size=14, src_ps=0xa096f88)
    at /root/bro/src/Net.cc:281
#25 0x0844d5ce in iosource::PktSrc::Process (this=0xa096f88)
    at /root/bro/src/iosource/PktSrc.cc:411
#26 0x0820631a in net_run () at /root/bro/src/Net.cc:329
#27 0x0815e588 in main (argc=19, argv=0xbfbeb214) at /root/bro/src/main.cc:1212
(gdb)

Ted