Can type notation like "vector of string" and "set[addr]" be parsed in BiF declarations? Doesn't work for me and I don't see any examples of this in current code. Looks like the current practice is to declare a type alias in bro.init and then use that alias in the BiF.
- Jon
You got it right.
.Seth
D'oh! I never realized that that's reason for the type aliases. Seems
something we should teach bifcl at some point.
Robin
having looked at bifcl recently, I might actually make sense to consider
extending the Bro language parser to be able to also parse bifs (i.e.,
having a "bif-mode" but still compiling a bifcl binary).
Type definitions in bro can be quite complex and one would have to
completely duplicate that in bif.
Similarly, when I was working on the bif syntax to make record types,
enums, etc. available to C++ I thought that it would have been great if
I could just specify the record in the BiF. Instead I have to specify
the record in a bro file and put a simple "type foo: record;" in the
BiF. Otherwise I would have had to copy all record-type-definition syntax.
I think the more we use BiFs the more we will encounter such situations
were the BiF-syntax is not powerful enough and were we would like to
have elements of the Bro syntax available.
Just my 2ct.
gregor
This isn't likely in the same vein, but is the syntax below correct?
It doesn't appear to be working properly.
# ... unless the lookup comes from one of these hosts.
const okay_to_lookup_sensitive_hosts: set[addr] = {
[172.0.0.1],
[172.0.0.2],
} &redef;
Thanks,
-Will
Type definitions in bro can be quite complex and one would have to
completely duplicate that in bif.
Does bifcf really need to parse the type fully? Could it just treat
more complex types as opaque strings and carry them through into the
output?
I could just specify the record in the BiF. Instead I have to specify
the record in a bro file and put a simple "type foo: record;" in the
BiF. Otherwise I would have had to copy all record-type-definition syntax.
Likewise, would some simpler parsing rule work here that doesn't
understand all the pieces going into the record? Haven't really
thought this through admittedly.
Robin
Normally you'd write:
const okay_to_lookup_sensitive_hosts: set[addr] = {
172.0.0.1,
172.0.0.2,
} &redef;
But your version seems to work as well. With both versions, I get:
# bro -e "print okay_to_lookup_sensitive_hosts" foo.bro
{
172.0.0.1,
172.0.0.2
}
What's not working?
Robin
Type definitions in bro can be quite complex and one would have to
completely duplicate that in bif.Does bifcf really need to parse the type fully? Could it just treat
more complex types as opaque strings and carry them through into the
output?
I thought about that too. However the problem with treating the types as
opaque is that the parser still has to parse over them as they contain
keyword, tokens, braces, etc.
I could just specify the record in the BiF. Instead I have to specify
the record in a bro file and put a simple "type foo: record;" in the
BiF. Otherwise I would have had to copy all record-type-definition syntax.Likewise, would some simpler parsing rule work here that doesn't
understand all the pieces going into the record? Haven't really
thought this through admittedly.
This has the same problem as the types in the header, however, it might
be easier to solve: One solution might be to introduce a new kind of
brackets/paranthese so that the scanner can skip over the record
definition, e.g.,
type asdf: record @{
# opaque string comes here
@}
However, that's yet another brace-style....
(maybe one could also re-use another bracket or brace, have to look at
the bif parser for that though).
Still, one "problem" is that if there is a syntax error in the record
definition, you will only see it during Bro-runtime in the generated
.bro file. That implies that we probably have to add markers to the
generated bro-file so that a user knows where the error actually came
from. But that's more a cosmetic issue.
cu
Gregor
I thought about that too. However the problem with treating the types as
opaque is that the parser still has to parse over them as they contain
keyword, tokens, braces, etc.
Wouldn't it only need to follow brackets and parentheses? That's
pretty straight-forward, and everything read while doing so would just
be concatenated into a single string.
However, that's yet another brace-style....
Yeah, I'm already confused by those we have. 
But can't we get away
here with a similar heuristic as above?
Still, one "problem" is that if there is a syntax error in the record
definition, you will only see it during Bro-runtime in the generated
.bro file.
I could live with that as it's not something users are supposed to
mess with anyway. And our testing should already make sure that
this doesn't happen in released versions.
Robin
I thought about that too. However the problem with treating the types as
opaque is that the parser still has to parse over them as they contain
keyword, tokens, braces, etc.Wouldn't it only need to follow brackets and parentheses? That's
pretty straight-forward, and everything read while doing so would just
be concatenated into a single string.
I don't think that you can specify a yacc grammar that will work. E.g.,
consider this grammar:
type_def: TOK_TYPE opt_ws TOK_ID opt_ws ':' opt_ws TOK_RECORD opt_ws ';'
the trailing ';' would have to be replaced with something like
.... TOK_RECORD opt_ws { THE_OPAQUE_STRING }
but I don't see how you can specify THE_OPAQUE_STRING in a short way
without reproducing a lot of Bro's grammar.
But maybe I'm missing something. I'm not a yacc expert.
cu
Gregor
Thanks for the quick reply and sorry for the delayed response. I have
been trying to troubleshoot a few other issues.
It doesn't appear to be working properly.
Normally you'd write:
const okay_to_lookup_sensitive_hosts: set[addr] = {
172.0.0.1,
172.0.0.2,
} &redef;But your version seems to work as well. With both versions, I get:
# bro -e "print okay_to_lookup_sensitive_hosts" foo.bro
{
172.0.0.1,
172.0.0.2
}What's not working?
So, this variable defines hosts in dns.bro that we intend to ignore
sensitive DNS queries from (recursive DNS servers). A couple of weeks
ago I installed and started using broctl on this box and with a few
minor changes everything continued working fine except we are now
getting alerts from the two hosts defined here.
I wasn't aware of being able to specify and print a single variable
from bro, as you did above, but ecstatic about how much easier that
will make things when troubleshooting.
On my first attempt, I think I broke something else. I actually have
scan.bro commented out, so I'm assuming I this indicates a bigger
issue?
/usr/local/bro/share/bro/site]# bro -e "print
okay_to_lookup_sensitive_hosts" local.bro
/usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS
error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long)
Abort trap: 6 (core dumped)
After rebuilding bro and broctl, I created a skeleton script that just
had the statements above and it worked just as you showed.
Unfortunately, we are still getting notices from the hosts defined in
that variable.
Like I said, the only change (that I am aware I made) is upgrading bro
from 1.5.1 to 1.5.3 and installing broctl.
This may be something that should be on the user list than dev?
Thanks again!
Will
.... TOK_RECORD opt_ws { THE_OPAQUE_STRING }
How about something like this:
.... TOK_RECORD opt_ws { opaque_string }
opaque_string : TOK_NO_BRACKET_STRING opaque_string
> '[' opaque_string ']'
> '{' opaque_string '}'
> '(' opaque_string ')'
>
TOK_NO_BRACKET_STRING is something like '[^{}()]+'.
And for each rule, we append the $1,...,$n to a global
current_opaque_string.
This is probably too simplistic, I'm sure I'm missing something (and
it doesn't preserve white-space; we'd need switch the lexer into a
separate mode). But some hack like that.
Robin
I wasn't aware of being able to specify and print a single variable
from bro, as you did above, but ecstatic about how much easier that
will make things when troubleshooting.
Are you aware of broctl's "print" command? That shows you the value of
variable at runtime. Try running that with
"okay_to_lookup_sensitive_hosts" to see if the broctl configuration
gets it right.
/usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS
error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long)
Abort trap: 6 (core dumped)
These kind of errors usually indicate trouble with the system's DNS
setup. However, I don't think I've ever seen the "message too long"
message.
Robin
.... TOK_RECORD opt_ws { THE_OPAQUE_STRING }
How about something like this:
.... TOK_RECORD opt_ws { opaque_string }
opaque_string : TOK_NO_BRACKET_STRING opaque_string
> '[' opaque_string ']'
> '{' opaque_string '}'
> '(' opaque_string ')'
>
you probably only need to take care of '{' and '}'. Should be able to
handle brackets and parentheses as opaque as well.
TOK_NO_BRACKET_STRING is something like '[^{}()]+'.
For this to work yacc has to be able to switch the lexer mode. I don't
know whether yacc can to do this (I think only the lexer can switch its
mode/state).
Finally, (although not a show-stopper because unlikely), the above has
problems with quoted strings that have an unmatched number of braces in
them.
cu
Gregor
I wasn't aware of being able to specify and print a single variable
from bro, as you did above, but ecstatic about how much easier that
will make things when troubleshooting.Are you aware of broctl's "print" command? That shows you the value of
variable at runtime. Try running that with
"okay_to_lookup_sensitive_hosts" to see if the broctl configuration
gets it right.
No, I wasn't til now. Forgive my nubness, but what specifically are
<id> and <node>? i.e. process id of parent ps or variable name? If
standalone, would node be just bro, localhost or something completely
different?
# broctl print "okay_to_lookup_sensitive_hosts" bro
/usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS
error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long)
Abort trap: 6 (core dumped)These kind of errors usually indicate trouble with the system's DNS
setup. However, I don't think I've ever seen the "message too long"
message.
So everything has been running smoothly for the last 24 hours or so,
then another crash. More details:
Cannot access memory at address 0x5
==== stderr.log
pcap bufsize = 32768
listening on bge1
1305661424.619015 run-time error: string without NUL terminator:
"\xff\xff*^Hbc0975.0\xc0^L\0^A\0^A\0\0^D\xb0\0^D^J^D^D.^Hbc097531\xc0^L\0^A\0^A\0\0^D\xb0\0^D\xac^Q(\xa7^Hbc097532.\xff\xff*^Hbc0975.0\xc0^L\0^A\0^A\0\0^D\xb0\0^D^J^D^D.^Hbc097531\xc0^L\0^A\0^A\0\0^D\xb0\0^D\xac^Q(\xa7^Hbc097532"
1305707636.214027 run-time error: string without NUL terminator:
"hosta^Ecompany^Corg\0\xc0^L\0!\0^A\0\0^Bx\0\x1c\0\0\0d^A\x85^J04c2nvrs-a^Ecompa"
1305707636.259675 run-time error: string without NUL terminator:
"hosta^Ecompany^Corg\0\xc0^L\0!\0^A\0\0^Bx\0\x1c\0\0\0d^A\x85^J04c2nvrs-a^Ecompa"
1305734703.016096 run-time error: string with embedded NUL:
"oo^M\xc3\xca\0^A\0^A\0"
1305735623.373659 internal error: NB-DNS error in DNS_Mgr::Process
(ns_initparse(): Message too long)
/usr/local/bro/share/broctl/scripts/run-bro: line 73: 31891 Abort
trap: 6 (core dumped) nohup tmpbro @
==== stdout.log
==== .status
TERMINATED [internal_error]
==== No prof.log.
bro.core
Core was generated by `bro'.
Program terminated with signal 6, Aborted.
#0 0x286e8a27 in kill () from /lib/libc.so.7
#0 0x286e8a27 in kill () from /lib/libc.so.7
#1 0x286e8986 in raise () from /lib/libc.so.7
#2 0x286e756a in abort () from /lib/libc.so.7
#3 0x080517a4 in internal_error () at SSLInterpreter.cc:30
#4 0x080a1691 in DNS_Mgr::Process (this=0xbfbfe554) at DNS_Mgr.cc:1069
#5 0x08147285 in net_run () at Net.cc:528
#6 0x0804fbff in main (argc=) at main.cc:999
Followed by this crash when broctl tried to restart 5 minutes later.
So, do you still think this looks like a host configuration issue?
This is on a freebsd 7.3 host, fyi.
[bro]
Variable "this" is not available.
==== stderr.log
/usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS
error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long)
/usr/local/bro/share/broctl/scripts/run-bro: line 73: 55221 Abort
trap: 6 (core dumped) nohup tmpbro @
==== stdout.log
==== .status
TERMINATED [internal_error]
==== No prof.log.
bro.core
Core was generated by `bro'.
Program terminated with signal 6, Aborted.
#0 0x286e8a27 in kill () from /lib/libc.so.7
#0 0x286e8a27 in kill () from /lib/libc.so.7
#1 0x286e8986 in raise () from /lib/libc.so.7
#2 0x286e756a in abort () from /lib/libc.so.7
#3 0x080517a4 in internal_error () at SSLInterpreter.cc:30
#4 0x080a199c in DNS_Mgr::Resolve (this=) at DNS_Mgr.cc:580
#5 0x080a1dbd in DNS_Mgr::LookupHost (this=0x82dc800, name=0x85a3939
"test-scooter.av.pa-x.dec.com") at DNS_Mgr.cc:468
#6 0x080682b7 in brolex () at scan.l:324
#7 0x08053bbc in yyparse () at p.c:2260
#8 0x0804ee16 in main (argc=17, argv=0xbfbfeb5c) at main.cc:749
Thanks,
Will
For this to work yacc has to be able to switch the lexer mode. I don't
know whether yacc can to do this (I think only the lexer can switch its
mode/state).
It can if you do it carefully. Basically, there's a "preproduction"
(something reduced early on) that has an action to call into the scanner
and flag for it to switch.
Vern
I wasn't aware of being able to specify and print a single variable
from bro, as you did above, but ecstatic about how much easier that
will make things when troubleshooting.Are you aware of broctl's "print" command? That shows you the value of
variable at runtime. Try running that with
"okay_to_lookup_sensitive_hosts" to see if the broctl configuration
gets it right.No, I wasn't til now. Forgive my nubness, but what specifically are
<id> and <node>? i.e. process id of parent ps or variable name? If
standalone, would node be just bro, localhost or something completely
different?# broctl print "okay_to_lookup_sensitive_hosts" bro
/usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS
error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long)
Abort trap: 6 (core dumped)These kind of errors usually indicate trouble with the system's DNS
setup. However, I don't think I've ever seen the "message too long"
message.
Looks like a ticket is open for a similar issue:
http://tracker.bro-ids.org/bro/ticket/255
The odd thing is, I don't know what changed that would cause these to
start occurring recently.
No, I wasn't til now. Forgive my nubness, but what specifically are
<id> and <node>? i.e. process id of parent ps or variable name? If
standalone, would node be just bro, localhost or something completely
different?
ID is the name of the variable that you would like printed and node is the name of the Bro instance that you'd like to inspect the value on. If you don't give a node it will print the value from all nodes.
/usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS
error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long)
Abort trap: 6 (core dumped)These kind of errors usually indicate trouble with the system's DNS
setup. However, I don't think I've ever seen the "message too long"
message.1305735623.373659 internal error: NB-DNS error in DNS_Mgr::Process
(ns_initparse(): Message too long)
/usr/local/bro/share/broctl/scripts/run-bro: line 73: 31891 Abort
trap: 6 (core dumped) nohup tmpbro @
==== stdout.log
==== .status
TERMINATED [internal_error]
==== No prof.log.
bro.core
Core was generated by `bro'.
Program terminated with signal 6, Aborted.
#0 0x286e8a27 in kill () from /lib/libc.so.7
#0 0x286e8a27 in kill () from /lib/libc.so.7
#1 0x286e8986 in raise () from /lib/libc.so.7
#2 0x286e756a in abort () from /lib/libc.so.7
#3 0x080517a4 in internal_error () at SSLInterpreter.cc:30
Question for Robin/Vern/whomever, should this really be calling internal_error? It's definitely worthwhile to log that some sort of error was received from libbind, but I don't think Bro should be shutting down when it has a problem like this.
.Seth
It's usually that something has changed external to Bro. Perhaps a DNS server that hosts on your network tend to do traffic with occasionally started returning odd responses. This seems to be how most of the edge cases are found. 
.Seth
The odd thing is, I don't know what changed that would cause these to
start occurring recently.It's usually that something has changed external to Bro. Perhaps a DNS server that hosts on your network tend to do traffic with occasionally started returning odd responses. This seems to be how most of the edge cases are found.
Yes, found this to be exactly the case. We built a new box and just
transferred over the same bro scripts and haven't had any problems.
Thanks!