# Broker's remote logging (BIT-1784)
**URL:** https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635
**Category:** Development
**Tags:** development
**Created:** [January 31, 2017, 10:41pm UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635 "2017-01-31T22:41:45Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@robin](https://community.zeek.org/u/robin)
#### Post date: [January 31, 2017, 10:41pm UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/1 "2017-01-31T22:41:45Z")
Taking from ticket to the mailing list as I'm looking for some input.
[[BIT-1784] - Bro Tracker](https://bro-tracker.atlassian.net/browse/BIT-1784) says:
> The change from the older communication code is that
> RemoteSerializer::ProcessLogWrite used to do
>
> success = log\_mgr-\>Write(id\_val, writer\_val, path, num\_fields, val);
>
> Where bro\_broker::Manager::Process uses
>
> log\_mgr-\>Write(stream\_id-\>AsEnumVal(), columns-\>AsRecordVal());
The fact that RemoteSerializer and broker::Manager are calling
different Write() functions seems to be a broader issue: we get
different semantics that way. For RemoteSerializer, log events and log
filters run only on the originating nodes; those guys make all
decisions about what's getting logged exactly and they then send that
on to the manager, which just writes out the data it receives. With
Broker, however, both events and filters run (also) on the manager, so
that it's making its own decisions on what to record. The filters can
be different on the manager, and they will have access to different
state.
I'm not sure what approach is better actually, I think the Broker
semantics can be both helpful and harmful, depending on use case. In
any case, it's a change in semantics compacted to the old
communication system, and I'm not sure we want that.
I'm wondering if there's a reason that in the Broker case things
\*have\* to be this way. Is there something that prevents the Broker
manager from doing the same as the RemoteSerializer?
Robin
---
### Author:  [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)
#### Post date: [January 31, 2017, 11:41pm UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/2 "2017-01-31T23:41:19Z")
>
...
> The fact that RemoteSerializer and broker::Manager are calling
> different Write() functions seems to be a broader issue: we get
> different semantics that way. For RemoteSerializer, log events and log
> filters run only on the originating nodes; those guys make all
> decisions about what's getting logged exactly and they then send that
> on to the manager, which just writes out the data it receives. With
> Broker, however, both events and filters run (also) on the manager, so
> that it's making its own decisions on what to record. The filters can
> be different on the manager, and they will have access to different
> state.
>
> I'm not sure what approach is better actually, I think the Broker
> semantics can be both helpful and harmful, depending on use case. In
> any case, it's a change in semantics compacted to the old
> communication system, and I'm not sure we want that.
I think we want the old behavior for 2 reasons:
1. The workers only send the &log fields to the managers, so the events are raised with half the fields missing.
2. Having the logger node be as much of a dumb byte mover as possible is best for performance reasons. Having the log events and log filters run on the workers lets that functionality scale out across the nodes. Especially if a filter is used that would remove a large percent of the entries.
If someone really wanted the log\_\* events to run on the manager, they could redef Cluster::worker2manager\_events right?
> I'm wondering if there's a reason that in the Broker case things
> \*have\* to be this way. Is there something that prevents the Broker
> manager from doing the same as the RemoteSerializer?
Jon would know best, but I'd guess one form was more convenient to use than the other and it may have been assumed that they both did the same thing.
---
### Author:  [@Alan\_Commike](https://community.zeek.org/u/Alan_Commike)
#### Post date: [February 1, 2017, 12:06am UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/3 "2017-02-01T00:06:30Z")
> 1. Having the logger node be as much of a dumb byte mover as possible is best for performance reasons. Having the log events and log filters run on the workers lets that functionality scale out across the nodes. Especially if a filter is used that would remove a large percent of the entries.
This.
Especially over time as we see more and more cores per processor, it’s best to distribute the processing load. By putting the filter in the logger, the logger will then need to enter the interpreter for each log message to determine if it needs to throw away data it just received. That’s expensive and limits scalability on multiple fronts.
…alan
---
### Author:  [@Matthias\_Vallentin1](https://community.zeek.org/u/Matthias_Vallentin1)
#### Post date: [February 1, 2017, 2:44am UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/4 "2017-02-01T02:44:50Z")
> I'm wondering if there's a reason that in the Broker case things
> \*have\* to be this way. Is there something that prevents the Broker
> manager from doing the same as the RemoteSerializer?
Some background: when Broker sends to a log topic, the message has the
structure of a pair (id, (x, y, z, ..)) where id is an enum with the log
stream name and (x, y, z, ...) a record of log columns. Therefore, in
broker::Manager::Process() where messages are parsed and dispatched, the
log messages go into logging::Manager::Write(EnumVal\*, RecordVal\*).
Such messages get created via broker::Manager::Log(EnumVal\*, RecordVal\*,
RecordType\*). The only caller of this function is logging::Manager.
Purely from an API perspective, could we just move the call from one
Write() function to the other?
Matthias
---
### Author:  [@robin](https://community.zeek.org/u/robin)
#### Post date: [February 1, 2017, 4:05am UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/5 "2017-02-01T04:05:02Z")
I think the answer is yes, but I've looked at bit more at the code and
I think I see where the challenge is: that 2nd Write() method (the one
the RemoteSerializer is using to output already filtered logs) takes
threading::Values, not Vals. That means switching over from one
Write() to the other isn't straight-forward because we don't have code
that sends threading::Values over the Broker connection. We could
convert the Val's into threading::Values once received, but that'd be
kind of odd:: I'm pretty sure the distinction was due to
threading::Values being quite a bit more efficient to send.
It should be pretty straight-forward to add the necessary
threading::Value-to-Broker conversion code (just a bit tedious :).
I'll look into that.
Robin
---
### Author:  [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)
#### Post date: [February 1, 2017, 3:02pm UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/6 "2017-02-01T15:02:45Z")
Just to pile onto this, I think it should be this way too. I'd really like to avoid script code executing on the logger.
.Seth
---
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [February 5, 2017, 10:04pm UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/7 "2017-02-05T22:04:04Z")
I think I was aware of the differences and went ahead with that approach because there's the extra technical work of writing code to convert value types as Robin mentions and also it's conceptually more flexible than the old way.
I understand the argument that the old semantics (manager not running log events/filters) may be more performant, though, I’d consider whether the internal comm. framework or the base/user scripts should be the one to decide.
I think the later is better, so the problem breaks down into (1) does the user have the ability to fully control whether log events/filters run on any given node via scripts? and (2) are the default settings/scripts sane for the common use-case?
(1) is likely true, so (2) sounds like it needs to be fixed.
Just a different idea on how to approach solving the issue without having to touch the framework's internals. (it’s been a while, hope it’s not way off base)
- Jon
---
### Author:  [@robin](https://community.zeek.org/u/robin)
#### Post date: [February 6, 2017, 4:43pm UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/8 "2017-02-06T16:43:39Z")
I agree that generally it'd be nice to be able to do it either way.
However, I'm pretty sure at this point that we need the separate
high-performance path that the old communication introduced, for the
reasons discussed in this thread and also for consistency. I'm working
on adding that code, and I think it should be the standard model, just
as it is currently. In addition to that, one can always send log\_\*
events around and then do custom processing on the receiving side.
That's not quite as transparent as "normal" log messages would be with
their configuration and filters, but that might actually be a good
thing: if we actually had both mechanisms (sender- and receiver-side
filtering) built transparently into the logging framework, it could
end up being quite confusing what's used when.
I propose that for now we make Broker work like the current model, and
then we go from there. If we need more, we can add that later. The
less semantic differences we have between old and new communication,
the easier the switch will be.
Robin
---
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [February 7, 2017, 2:46am UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/9 "2017-02-07T02:46:42Z")
> I propose that for now we make Broker work like the current model, and
> then we go from there. If we need more, we can add that later. The
> less semantic differences we have between old and new communication,
> the easier the switch will be.
Yeah, I agree that it should behave the same. Was just also trying to reframe the problem to give ideas for solutions that give back old semantics (i.e. via script-level mechanisms). No big deal, since there’s no demand for it.
> In addition to that, one can always send log\_\*
> events around and then do custom processing on the receiving side.
> That's not quite as transparent as "normal" log messages would be with
> their configuration and filters, but that might actually be a good
> thing: if we actually had both mechanisms (sender- and receiver-side
> filtering) built transparently into the logging framework, it could
> end up being quite confusing what's used when.
It was actually always confusing to me that a remote log entry versus a local log entry would be processed differently regarding the log\_\* events. The event is a property of the Log::Stream definition and the logging API or docs don’t distinguish between outgoing versus incoming log entries there, or do they? Or is a Stream meant to be thought of from only the perspective of an outgoing sequence of log entries?
Could be I misunderstood the log framework the whole time, and that’s why broker behaved the way it did 🙂
- Jon
---
### Author:  [@robin](https://community.zeek.org/u/robin)
#### Post date: [February 7, 2017, 4:19pm UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/10 "2017-02-07T16:19:26Z")
I know, it's a bit confusing. Some of that is historic and part of
trying to maintain semantics as things were involving (both logging
framework and communication; quite similar actually to what we've been
discussing here: what should be done where). It all came out of
"remote printing" where a print-statement would just send what it
would normally print into a file, over to another node---that means
everything was fully procecessed already as it was received. The other
part is the performance optimization: special-casing log transmission
for batching and volume, so that it doesn't become a bottleneck.
Thinking about it as just outgoing entries fits its best I think. Onw
the receiving side, the entries don't "really" enter the flu logging
framework, they just take a fast path directly into the writers. One
thing I'm doing is renaming methods to make that bit clearer; the two
Write() methods are clearly misleading.
Robin
---
### Author:  [@system](https://community.zeek.org/u/system)
#### Post date: [May 6, 2022, 3:44pm UTC](https://community.zeek.org/t/brokers-remote-logging-bit-1784/4635/11 "2022-05-06T15:44:34Z")