Looking through the archives, it looks like this has come up, but I’ll ask it again since doesn’t look like it’s been asked recently. Does bro support bitwise operations such as ‘and’, ‘or’, and ‘xor’ ?
Shifts would be nice too.
Looks like it wouldn’t be too difficult to add bitwise operators that work on integral types (int and count) – I modified the bro source to add ‘&’, ‘^’, ‘<<’, and ‘>>’. I had some clash with ‘|’ due to it being used in other places (and there’s probably numerous other bugs since I’m not much of a lex/yacc expert) but it seems like something that’d be do-able.
Looks like it wouldn't be too difficult to add bitwise operators that work
on integral types (int and count)
Sure. But can you please sketch a compelling use case for which it's
important to add this functionality? That's the general bar for deciding
what sort of features to add.
Vern
Well, one reason would be to aid in detecting malware c2 traffic that can’t be detected with simple signatures or regular math operations.
As a grossly simplified example, imagine you’ve reverse engineered a piece of c2 malware and have figured out what their handshake protocol looks like. This malware always puts a key somewhere in the packet and then uses that key to xor data in other parts of the packet. This method would be used as a simple traffic obfuscation technique to prevent traditional signature detection.
As it stands there’s very little way (frankly, no way) for Bro to detect this sort of stuff (and that was my response when someone asked if we could implement something in Bro to detect some c2 traffic we’re trying to track).
Assuming you have the full range of the bro language to leverage in the signature framework’s eval function, this is pretty much a requirement for writing more advanced signatures and one of the reasons Snort introduced Shared Object Rules into their system (http://blog.snort.org/2011/02/snort-shared-object-rules.html).
Bitwise operations on user defined stream fields for custom protocol (or specific “data chunks”) analyzers within scriptland is one reason.
-AK
Bitwise operations on user defined stream fields for custom protocol
Okay, these examples make sense to me. Let me ask then about what such
operators should look like. M K originally sketched them as operating on
integral types. However, I'd think that if it's for manipulating blobs
of C&C, then instead working on strings would be the right target ... ?
Vern
My method was to take a string of bytes and convert them to integral types I wanted.
So if I received a ‘string’ type in a function I could do:
local foo1 = bytestring_to_count(sub_bytes(string, 0, 4));
local foo2 = bytestring_to_count(sub_bytes(string, 4, 2));
local foo3 = bytestring_to_count(sub_bytes(string, 6, 2));
bar = foo1 ^ 0x12345678;
bah = (foo2 + foo3) & 0xFFFF;
if ( bar == 0xDEADBEEF && bah > 0x1234 ) {
#do a barrel roll
}
I’d love to see these operations built into Bro. However, considering Bro’s focus on scale, individual c2 operations analysis (especially when applied to specific connections within a large pipe) may be better suited for something like chopshop or another framework focussed on individual connections. Any other opinions?
-AK
ChopShop is too slow to handle detection of c2 traffic in real-time on a large pipe. Frankly, it was never meant for it, even though a lot of folks would like to run it like that. For detection of c2 in real-time on a large pipe you need some capability that has the ability to scale (and just be faster overall). If the performance hit added by the translation of bro script is too huge, then it would be slow, but given Bro Cluster’s ability to scale I would hope that it could be somehow managed.
The alternative would be to add another bro framework (or augment the signature framework) to allow for C or C++ like code to be directly executed on packets akin to Shared Object Rules or to add some abstracted math language to it that gets parsed and directly executed. The other alternative, of course, is to just use another detection system.
P.S – full disclosure – I’m the primary author of ChopShop.
In my mind malware C2 communications comes in three flavors.
- repurposed HTTP (RFC compliant)
- modified HTTP (just enough to make it not work with Bro's HTTP analyzer)
- custom binary
A bitwise operations within scriptland would be very useful for the
first type, where xor'ing an HTTP body or header value can reveal an
underlying communications mechanism. This assumes you're looking to
take advantage of Bro's HTTP analyzer. The second and third types of
C2 would require completely new protocol analyzers. If the analyzer is
written in C/C++, the scriptland bitwise operator is pointless. If the
analyzer was written in scriptland, e.g. hooking events such as
tcp_contents, the bitwise operator would be very handy but Bro's speed
would likely be compromised.
-AK
P.S -- full disclosure -- I'm a fan of ChopShop 
In my mind malware C2 communications comes in three flavors.
- repurposed HTTP (RFC compliant)
- modified HTTP (just enough to make it not work with Bro's HTTP analyzer)
- custom binary
Nice list. I think you've nailed it with these. Fortunately there has been work in progress for several years that should help address points 2 and 3. 
For the first case, it's possible to implement xor in scriptland (I attached an implementation to this email). I can't promise how well it will perform, but it's unlikely you'd be doing it constantly either.
hooking events such as tcp_contents, the bitwise operator would be very handy but Bro's speed
would likely be compromised.
Yeah, I *definitely* don't recommend that.
.Seth
xor.bro (8.57 KB)
“Fortunately there has been work in progress for several years that should help address points 2 and 3. ;)”
I assume you’re talking about the dynamic plugin capability Robin is working on – or is it something else? Are there any details you can share?
“For the first case, it’s possible to implement xor in scriptland (I attached an implementation to this email). I can’t promise how well it will perform, but it’s unlikely you’d be doing it constantly either.”
Although you can implement all of the bitwise operators in bro language using arithmetic operators, it seems overly cumbersome to use them for operators that are implemented literally using one instruction in almost every hardware platform (that’s not to say writing bro script will turn it into one instruction, it’s just to say individual math operations are fast). And for my use case, which is specifically using it as part of the signature framework it’s probably highly likely that it would be doing it constantly.
As an example (I wanted to make sure it would actually work), I created a test signature that used eval to call a bro function I wrote that used the ^ and & operators to detect a specific sequence that’s seen during the handshake/setup of a piece of C2 and ran this on a canned sample pcap. It worked and fired off a signature match event. Obviously this is more anecdotal than anything else and would require further testing to ensure that my code isn’t a massive false positive generator and would actually perform well on live traffic, but it shows that such operators have some potential. At the least, if not as individual operators, it’d be beneficial to create some built-in’s that can take care of this at a lower level.
The caveat with what I’ve mentioned though is that all of this is just an idea right now that I’m still formulating the extent of and there’s no actual plans for implementation anytime soon, so it’s possible you guys might add some capability in the meantime to Bro that obviates the need for this.
I assume you're talking about the dynamic plugin capability Robin is working on -- or is it something else? Are there any details you can share?
Partly that, but generally I don't think that it's a good idea for most people to be writing c/c++ code that is parsing network traffic. I was primarily referring to Binpac++, but that's still research (nod to Robin) so it's probably better to not discuss it concretely in public quite yet.
it seems overly cumbersome to use them for operators that are implemented literally using one instruction in almost every hardware platform
Sounds like a pre-optimization to me. (although I do agree, my script is an enormous hack). 
so it's possible you guys might add some capability in the meantime to Bro that obviates the need for this.
We've been talking about having a release that focuses on language issues and additions and this could definitely fit in as a part of that.
.Seth