I noticed that the DNS.log is no longer being created when running the cluster. I don't think I changed anything to disable it. It is still being loaded by local.bro. I am running the latest trunk.
$ broctl print DNS::logging
manager DNS::logging = F
proxy-1 DNS::logging = F
worker-1 DNS::logging = F
The dns.bro script shows logging defaulted to true. Where is it getting set to false when the cluster runs?
Tyler
Mine is set to false here:
$BROHOME/share/bro/broctl/cluster.dns.bro
-Lou
(who also spent some time trying to figure this out)
Thanks, that did the trick. -Tyler
$BROHOME/share/bro/broctl/cluster.dns.bro
Yes, indeed. The cluster config is changing some defaults to values
which seem to be more reasonable in a large setting. It's of course
debatable what the definition of "reasonable" here is 
With DNS
one gets these huge logs which often aren't very helpful.
So, the general guideline is when you're looking for a specific
setting, also grep through the cluster's *.bro scripts.
(who also spent some time trying to figure this out)
Sorry.
Robin
On a possibly related issue, I noticed that I am no longer seeing many ServerFound notices. I used to see a lot of these notices with the older 1.4.19, Robin's branch. The log format also changed around the time I stopped seeing the notices. This was between Sept. 2nd and Sept. 5th, 2009.
The alarm.log messages through Sept. 2nd looked like this.
Sep 2 12:26:14 ServerFound 128.x.x.x: SSH server on port 2222/tcp
On/after Sept. 5th changed to this format:
Sep 5 05:12:25 no=ServerFound na=NOTICE_ALARM_ALWAYS es=worker-1 sa=128.x.x.x da=128.y.y.y dp=3919/tcp p=3919/tcp num=32 msg=128.x.x.x:\ SSH\ server\ on\ port\ 3919/tcp sub=SSH tag=@c5-2f10-bf17
I think the log format change happened when I switched from a stand-alone config to the cluster config with a single worker. I don't understand why the ServerFound detections dropped so dramatically. I went from detecting 261 servers when running stand-alone to only 5 when running as a cluster.
In my new cluster config, with the latest trunk, in local-manager.bro, [ProtocolDetector::ServerFound] = file_if_remote. I changed file_if_remote to file_notice, but that didn't seem to make a difference.
I also commented ServerFound out of cluster-manager.detect-protocols.bro, but that didn't help either.
Any ideas what changed?
Tyler
I think the log format change happened when I switched from a
stand-alone config to the cluster config with a single worker.
Yes, the cluster switches to the better parseable log format; the
option for that is "use_tagging" in notice.bro.
(To extend my earlier note about the cluster configuration setting a
few defaults differently: that's the case for a number features we
have added to Bro in the past that are in some way incompatible with
older Bro installations, like changes in log format. We have rarely
turned these on per default to not break anything. The cluster now
flips over some of these switches to get the new behaviour for new
installations. Another example for that are DPD-based conn.logs: the
service field in conn.log is now determined via DPD so you may for
example now see "ssh" there for an SSH session on port 80, while the
standard Bro default would still say "http".)
There shouldn't be a difference though between broctl's cluster and
standlone modes in this regard. I've just checked this for
use_tagging setting, and that's enabled by default in the standlone
setting as well now; it might not have in earlier versions.
The alarm.log messages through Sept. 2nd looked like this.
Sep 2 12:26:14 ServerFound 128.x.x.x: SSH server on port 2222/tcpOn/after Sept. 5th changed to this format:
Sep 5 05:12:25 no=ServerFound na=NOTICE_ALARM_ALWAYS es=worker-1
sa=128.x.x.x da=128.y.y.y dp=3919/tcp p=3919/tcp num=32 msg=128.x.x.x:\
SSH\ server\ on\ port\ 3919/tcp sub=SSH tag=@c5-2f10-bf17understand why the ServerFound detections dropped so dramatically. I
went from detecting 261 servers when running stand-alone to only 5 when
running as a cluster.
I don't think the differences in the output format is (directly)
linked to the missing ServerFounds. There must be another reason why
you're seeing less. Have you looked at notice.log whether there are
more ServerFounds in there? If yes, then they are filtered out
somewhere before they reach alarm.log; if not, then they are not
generated in the first place.
Robin
(To extend my earlier note about the cluster configuration setting a
few defaults differently: that's the case for a number features we
have added to Bro in the past that are in some way incompatible with
older Bro installations, like changes in log format. We have rarely
turned these on per default to not break anything. The cluster now
flips over some of these switches to get the new behaviour for new
installations. Another example for that are DPD-based conn.logs: the
service field in conn.log is now determined via DPD so you may for
example now see "ssh" there for an SSH session on port 80, while the
standard Bro default would still say "http".)There shouldn't be a difference though between broctl's cluster and
standlone modes in this regard. I've just checked this for
use_tagging setting, and that's enabled by default in the standlone
setting as well now; it might not have in earlier versions.
Thanks for explaining how that works.
I don't think the differences in the output format is (directly)
linked to the missing ServerFounds. There must be another reason why
you're seeing less. Have you looked at notice.log whether there are
more ServerFounds in there? If yes, then they are filtered out
somewhere before they reach alarm.log; if not, then they are not
generated in the first place.
Yes, there are fewer showing up in the notice.log as well. When I get a chance, I'll try rolling back to an older version, and see if there is a difference. We also moved our SPAN port from a core-to-core link to a Internet-to-core link. That may have caused a difference, but I had expected to see more ServerFounds.
Tyler
Are there any internal systems for which you can confirm that they
should be reported? If so, capturing a trace and running it through
Bro offline could show whether it's problem of the cluster config or
something else in Bro.
Robin