patterns and &&/|| vs. &/| operators

In working on adding bitwise &/| operators for counts, I've come across
apparently undocumented && and || operators for patterns:

  p1 && p2 yields a pattern that matches a p1 followed by a p2
  p1 || p2 yields a pattern that matches either p1 or p2

Confusingly, Bro also supports "p1 | p2", which means the same as "p1 || p2"
above, but *only* if p1 and p2 are literal patterns, not if they are
variables of type "pattern". (This functionality is in common use.)
It doesn't support "p1 & p2" in any form.

I searched a large corpus of scripts and didn't find any instances of
"p1 && p2" or "p1 || p2" for literal patterns, so I suspect the current
feature is basically unused.

Proposal: as part of adding bitwise &/| operators for counts, I'll
also implement &/| operators for patterns, and remove the current
&&/|| functionality.

This seems pretty straightforward to me - but I've mistakenly thought
that about other things before! :stuck_out_tongue: So if anyone has comments, plz
speak up ...

    Vern

% bro -e 'print (/foo/ && /bar/) in "xfoobary"'
T

Sounds good to me...

Johanna

The meaning of "p1 & p2" would be "yields a pattern that matches both
p1 and p2" versus the meaning of "p1 && p2" currently being "yields a
pattern that matches a p1 followed by a p2" ?

I'd generally say that deprecating (emit a warning message pointing to
each usage) for a time period is a more cautious approach.

- Jon

The meaning of "p1 & p2" would be "yields a pattern that matches both
p1 and p2" versus the meaning of "p1 && p2" currently being "yields a
pattern that matches a p1 followed by a p2" ?

No, p1 & p2 would be the new way to express p1 && p2.

I'd generally say that deprecating (emit a warning message pointing to
each usage) for a time period is a more cautious approach.

Easy 'nuf, though I'd be amazed if anyone is using p1 && p2 given it's
not documented and not intuitive!

    Vern

Ok, I see now, yeah & would be better than && for that semantic,
though maybe p1 + p2 would be even better at expressing that
concatenation is happening?

I also notice from [1]:

    ‘r/s’: an ‘r’ but only if it is followed by an ‘s’ ...

Maybe another option?

Just making suggestions since I didn't quite get what p1 & p2 would do at first.

- Jon

[1] http://westes.github.io/flex/manual/Patterns.html

though maybe p1 + p2 would be even better at expressing that
concatenation is happening?

I think this is somewhat problematic, since '+' already has a
regular-expression meaning which is different. In addition, '&' is
a more natural dual to '|' than '+' is. Indeed, in some contexts
'|' and '+' are synonyms (e.g., I originally wanted them both for
set union).

I also notice from [1]:

    `r/s': an `r' but only if it is followed by an `s' ...

Maybe another option?

Note that Bro's REs don't support that ... and in general that operator
is a PITA to support correctly+efficiently. It would also step on the
current syntax of '/'s being used to express /re/ 's.

Just making suggestions since I didn't quite get what p1 & p2 would do at first.

Interestingly, I discovered that we have a BIF merge_pattern(p1, p2) which
does the same thing as "p1 & p2" (in the new syntax). As best as I can
tell it's not used anywhere - plus it's funky (only allows itself to be
called if Bro isn't processing traffic yet). Perhaps we can deprecate it, too?

    Vern

> though maybe p1 + p2 would be even better at expressing that
> concatenation is happening?

I think this is somewhat problematic, since '+' already has a
regular-expression meaning which is different. In addition, '&' is
a more natural dual to '|' than '+' is.

Yeah, agree w/ that.

Interestingly, I discovered that we have a BIF merge_pattern(p1, p2) which
does the same thing as "p1 & p2" (in the new syntax). As best as I can
tell it's not used anywhere - plus it's funky (only allows itself to be
called if Bro isn't processing traffic yet). Perhaps we can deprecate it, too?

If there actually is no (longer) problems with concatenating patterns
at run-time, I'd agree to deprecate.

I'm imagine it existed because there was such a problem with
dynamically creating patterns at run-time, but don't know/remember
what it was.

- Jon

If there actually is no (longer) problems with concatenating patterns
at run-time, I'd agree to deprecate.

I'm imagine it existed because there was such a problem with
dynamically creating patterns at run-time, but don't know/remember
what it was.

Now that you mention it - yes, there still is a problem as far as I know.
[BIT-328] - Bro Tracker seems to be the
relevant ticket.

This probably means that we will either have to limit p1 & p2 to only be
allowed when Bro is not processing traffic yet too - or fix the cleanup of
the DFA data structures.

Johanna