Duplicate Notice Actions Regression?

Development
1

Recently, I've been seeing Bro perform duplicate notice actions. I think this commit might have introduced a regression: <http://git.bro-ids.org/bro.git/commitdiff/290c2a0b4df2db38ade684cf386a5c9b6b271d9e&gt;

# The notice policy is completely handled by the manager and shouldn't be
# done by workers or proxies to save time for packet processing.
-event bro_init() &priority=11
- {
- Notice::policy = table();
- }
+redef Notice::policy = table();

Specifically, reading from the commit that fixed the duplicate notice action issue (<http://git.bro-ids.org/bro.git/commitdiff/53d9832d5adb526bf80f1d225a13941c7d05bdb2&gt;\):

The problem was that Notice::policy is used to populate the internal Notice::ordered_policy vector in a priority 10 bro_init handler (in scripts/base/frameworks/notice/main.bro) and then that is what is used when applying policy to notices. In order for scripts/base/frameworks/notice/cluster.bro to prevent Notice::policy from being used on non-manager nodes, it needs to clear it in a bro_init hander of higher priority than 10.

Am I on the right track here? If not, does anyone have any other ideas of what might be causing this?

Thanks,

  --Vlad

2

Recently, I've been seeing Bro perform duplicate notice actions. I think this commit might have introduced a regression: <http://git.bro-ids.org/bro.git/commitdiff/290c2a0b4df2db38ade684cf386a5c9b6b271d9e&gt;

# The notice policy is completely handled by the manager and shouldn't be
# done by workers or proxies to save time for packet processing.
-event bro_init() &priority=11
- {
- Notice::policy = table();
- }
+redef Notice::policy = table();

I also thought that could have broken the notice de-duplication/suppression, but it seemed to work in my testing. A simple check is to do `broctl print Notice::ordered_policy`. If it's empty on all the worker nodes, but populated for the manager node, then it's still working like I expected and probably something else is wrong.

Am I on the right track here? If not, does anyone have any other ideas of what might be causing this?

Are you getting 2 of the same exact email as if from both the worker and manager, or is it just that you get many emails within the suppression interval for the same "logical" notice $identifier?

And is it for all notice types or just certain ones? If it's certain custom ones you're creating, can you post examples of how you call NOTICE() to generate them?

Have you changed any of the "suppression_interval" settings?

    Jon

3

I also thought that could have broken the notice de-duplication/suppression, but it seemed to work in my testing. A simple check is to do `broctl print Notice::ordered_policy`. If it's empty on all the worker nodes, but populated for the manager node, then it's still working like I expected and probably something else is wrong.

It's populated on all the nodes. I'm not redefing Notice::emailed_types, which is what the original commit says causes this, but I am redefing Notice::mail_dest.

Are you getting 2 of the same exact email as if from both the worker and manager, or is it just that you get many emails within the suppression interval for the same "logical" notice $identifier?

Same exact e-mail.

And is it for all notice types or just certain ones? If it's certain custom ones you're creating, can you post examples of how you call NOTICE() to generate them?

Hmm. I believe only custom ones. I don't think I'm doing anything with the default ones, except for ACTION_LOG, which isn't duplicated. I used to use sync_functions to generate them (example here: https://gist.github.com/grigorescu/2925e938f1bcc13a1964), but I've changed to just using the notice event to see if that fixes this, e.g.:

event notice(n: Notice: Info) &priority=-5
      {
            if ( ACTION_EMAIL_ISO_IR in n$actions )
                  email_notice_to(n, "iso-ir@cmu.edu", T);
      }

Have you changed any of the "suppression_interval" settings?

Some of my notices have a non-default suppress_for interval, but I haven't change the interval globally.

Thanks,

  --Vlad

4

I also thought that could have broken the notice de-duplication/suppression, but it seemed to work in my testing. A simple check is to do `broctl print Notice::ordered_policy`

It's populated on all the nodes.

Then I think that change to using redef to clear Notice::policy isn't working, probably because it depends on @load ordering -- it would always have to be the last script to be loaded that redefs Notice::policy.

Seth said he'll make some changes so that the application of notice policy is better delegated to the manager in a cluster environment. Can you create a bug ticket for him?

    Jon