Segmentation fault while using own signature.

Zeek
1

Hi all,

So I have a case where if I use following regex in sig file, it works, but when I edit it and make it more strict I get segmentation fault in like 5 minutes after bro gets normally started:

The working version:

signature rootkit-potential {
payload /.[0-9.]{7,15}|[0-9]{1,5}./
event “Potential rootkit”

tcp-state originator
}

signature rootkit-malware {
payload /.SSH-2.5-OpenSSH_6.1.9.[0-9.]{7,15}|\d{1,5}./
event “rootkit malware”
tcp-state originator
}

When I change regex to be more restrictive, Seg fault occurs:

signature rootkit-potential {
payload /.(?:\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}|\d{1,5})./
event “Potential rootkit”
tcp-state originator
}

signature rootkit-malware {
payload /.SSH-2.5-OpenSSH_6.1.9.(?:\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}|\d{1,5})./

event “rootkit malware”
tcp-state originator
}

Any idea what might be going wrong?

Thanks,
Fatema.

2

Did anyone follow up on this?

    Vern

3

I could be mistaken, but some of these don't look like correct escape sequences for Bro regular expressions.

Check out the PATTERNS section of the flex documentation:

http://dinosaur.compilertools.net/flex/manpage.html

  --Vlad

fatema bannatwala <fatema.bannatwala@gmail.com> writes:

4

That’s a good catch, I think \d isn’t supported, so you’ll want to use [0-9].

I’ve chatted with Fatema off-list and I don’t think this is the problem though. The \d should just cause the signature to not match correctly.

-Dop

5

Hi Fatema,

Have you been able to get a stack trace? That would be the most helpful. I suspect that Dop is right though, the problem you're encountering with Bro crashing much be somewhere else. I have a hard time believing that this is the cause of the crash.

Another small note about the regular expressions you are writing is that Bro doesn't support the (?:abc) mechanism to prevent captures from occurring. You can leave out the "?:" when writing regular expressions. Bro has "flex-ish" regular expressions but it doesn't support all of the features that flex has.

  .Seth

6

You need to collect a core dump when the crash happens and get a stack trace from that. If this is on Linux, you will need to set your kernel.core_pattern sysctl value to something like the following....

sudo sysctl -w kernel.core_pattern=core.%e-%t-%p

If you have things set this way and you have gdb installed, broctl should automatically generate a stack trace when it restarts the dead process.

  .Seth

7

Hi Seth,

On one of our sensors, I did:
$ sudo sysctl -w kernel.core_pattern=core.%e-%t-%p

$ sudo sysctl -a | grep “kernel.core”
kernel.core_pattern = core.%e-%t-%p

Also, verified that I have gdb installed:

$ which gdb
/usr/bin/gdb

Also, I m starting bro with following commands on manager:

sudo -u bro /usr/local/bro/2.5/bin/broctl install
sudo -u bro /usr/local/bro/2.5/bin/broctl restart

However, when seeing the crash report on the sensor, it says No core file was found:
(Any idea, why broctl isn’t generating the core dump, or do I have to include any file in local.bro for the same?)

$ cd /mnt/brolog/spool/tmp/post-terminate-worker-2017-01-17-15-50-21-90688-crash
$ less .crash-diag.out

No core file found.

Bro 2.5
Linux 3.10.0-327.36.3.el7.x86_64

Bro plugins: (none found)

==== No reporter.log

==== stderr.log
internal warning in /usr/local/bro/2.5/share/bro/site/connStats.bro, line 3: Discarded extraneous Broxygen comment: aashish: need to port to file analysis framework
warning in /usr/local/bro/2.5/share/bro/site/connStats.bro, line 39: dangerous assignment of double to integral (ConnStats::out$EstinboundConns = ConnStats::result[EstinboundConns]$sum)
warning in /usr/local/bro/2.5/share/bro/site/connStats.bro, line 40: dangerous assignment of double to integral (ConnStats::out$EstoutboundConns = ConnStats::result[EstoutboundConns]$sum)
Warning: Kernel filter failed: Bad address
listening on em1

Warning: Kernel filter failed: Bad address
1484685887.668496 processing suspended
1484685887.668496 processing continued
/usr/local/bro/2.5/share/broctl/scripts/run-bro: line 107: 121052 Segmentation fault nohup ${pin_command} $pin_cpu “$mybro” “$@”

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-i em1 -U .status -p broctl -p broctl-live -p local -p worker-1-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

==== .env_vars
PATH=/usr/local/bro/2.5/bin:/usr/local/bro/2.5/share/broctl/scripts:/usr/local/bin:/usr/bin
BROPATH=/mnt/brolog/spool/installed-scripts-do-not-touch/site::/mnt/brolog/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/2.5/share/bro:/usr/local/bro/2.5/share/bro/policy:/usr/local/bro/2.5/share/bro/site
CLUSTER_NODE=worker-1-9

==== .status
RUNNING [net_run]

==== prof.log
1484686157.516259 TCP-States: Inact. Syn. SA Part. Est. Fin. Rst.
1484686157.516259 TCP-States:Inact. 24 4 3 2
1484686157.516259 TCP-States:Syn. 118 1
1484686157.516259 TCP-States:SA 6
1484686157.516259 TCP-States:Part. 38 335 9 2
1484686157.516259 TCP-States:Est. 602 81 2
1484686157.516259 TCP-States:Fin. 3 5 3 107 1
1484686157.516259 TCP-States:Rst. 2
1484686157.516259 Connections expired due to inactivity: 1525
1484686157.516259 Total reassembler data: 1178K

==== No packet_filter.log

==== No loaded_scripts.log

8

Ah! I suspect the problem is that you're starting Bro as the Bro user which probably doesn't have permission to increase it's maximum core file size to unlimited.

You can edit /etc/security/limits.conf and add the following line to it...

* soft core unlimited

That should make it possible for Bro to have arbitrarily large core dumps.

  .Seth

9

Hi Seth,

Thanks for the suggestions, still getting No core dump:

$ less /etc/security/limits.conf

#Editing the core dump limit to unlimited for Bro debugging
#* soft core 0

  • soft core unlimited

$ less .crash-diag.out

No core file found.

Bro 2.5
Linux 3.10.0-327.36.3.el7.x86_64

Bro plugins: (none found)

==== No reporter.log

I will check to see what am I missing.

Thanks,
Fatema.

10

I’ve run into issues with getting core dumps in the past. I documented some of them as comments against broala KBs, but I’m not sure where those exist now that it has been renamed. What OS are you running? Recalling from memory, there are different things that can stop successful cores using the afore-mentioned config depending on the platform (I think it was ABRT?). Happy to pull that back up again if you continue to have an issue.

Jon

11

Hi Jon,

Thanks for lending some help. Appreciate it.
We are running CentOS on our bro sensors as well as on manager.

Here’s the full info:
Linux sensor1.xx.xx 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

CentOS Linux release 7.2.1511 (Core)

Thanks,
Fatema.

12

I'd just run bro from a shell.. you said it crashes pretty quickly right?

sudo su -
mkdir /tmp/brotest
cd /tmp/brotest
ulimit -c unlimited
/usr/local/bro/2.5/bin/bro -i eth0 local

then it should crash and dump the core file right there.

(replace eth0 with whatever)

13

Here are some reading materials that may help.

Jon

14

Thanks Jon for the links!

Thanks Justin for alternative.

We have our cluster in production, hence currently that sig file is disabled so that the cluster runs properly.
Hence, to recreate the seg fault issue this time, rather than enabling it for the whole cluster, I just enabled it (in local.bro) for the previous version
of bro that we still have around, and ran a single bro process for that old version, as you suggested.
This time I was able to generate core dump for that single process.

I ran the core dump through the crash-diag script:

15

I have come across the same behavior while testing a script against some odd PCAPs and found that most of the time tweaking the filter to be more restrictive solved the issue. so, what you can do is:

1-Test against one signature only to determine which specific signature causes the issue.
2-Count # of “Potential rootkit” string existence for before and after and see which one got more hits (supposedly this should be the one causing the issue -less restrictive). This might validate that regex is working as expected…usage of debugging PRINT also might come handy.

Thanks

16

Thank you all for helping to troubleshoot the problem. :slight_smile: :slight_smile:
Finally, was able to get the issue resolved.

So the problem boiled down to having very loose regex.

The regex expression itself wasn’t the problem, but the bro script that was using that signature was expecting a particular type of data type (port format),
but since the regex was so loose that anything matching was passed onto the script, and then the script start complaining that the port format extracted is wrong.
Following would help to clarify it more:

my sig file had a signature with following payload match:
(It was expected to match “|” type of data in payload)

payload /.[0-9.]{7,15}|[0-9]{1,5}./

And the bro script was splitting that particular chunk of data from the payload and assigning it to an IP n port type variable.
v_port = to_port(fmt("%s/tcp",v_strs[i]));

When I ran the script and sig file with a bro instance, got following error messages:

/rootkit.bro, line 57: wrong port format, must be /[0-9]{1,5}/(tcp|udp|icmp)/ (to_port(fmt(%s/tcp, Site::v_strs[Site::i])))

Little troubleshooting revealed that the v_strs[i] was getting data like:
1;29645166|29663045|1;;cs=v%3fhttp://ad.doubleclick.net/dot.gif?1258562851900657 HTTP/1.1\x0d\x0aHost: ad…

which is definitely not a port, and hence I changed the loose regex to something like:
payload /.([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}|[0-9]{1,5})./

to match the “|” more accurately, and only get triggered when payload actually has a legit IP and port.

Tested it and working so far without any seg fault/issues.
Haven’t tried on prod though.

Mike - Don’t know whether you came across this type of errors, but might also want to restrict the first payload to match |.

Thanks,
Fatema.

17

Fatema, thanks for continuing to dig into this.

"which is definitely not a port, and hence I changed the loose regex to something like:
payload /.([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}|[0-9]{1,5})./

to match the “|” more accurately, and only get triggered when payload actually has a legit IP and port."

I had been running a regex similar to this already, but without the ( )'s. I’d like clarification on if that’s what ends up getting passed to the signature match event as ‘data’. I’ve added the ( ) 's today just in case. Regardless, looks like I could do a little cleanup further into the script logic as well. I really appreciate the feedback.

Interestingly, we just had a hit today that was an oddly formatted HTTP Cookie, but it was just a “potential” and obviously didn’t result in a callback.

-Dop