Hi Jon,
I'm curious what's the use case for this? Generally I think it's best
to use a combination of Broker-internal forwarding and topics to
control where messages get propagated too, as that will better
generalize to larger topologies later. The less of the routing we
control manually at the Bro level, the better I'd say. But that's not
an absolute rule of course, and I may just be missing what this is
aiming at.
Robin
I'm curious what's the use case for this?
From last month's thread regarding cluster API/topology changes, the
message pattern of sending an event from a worker to all other workers
or proxy/data to all other proxy/data nodes (within these classes,
nodes aren't connected with each other) was frequent enough to warrant
putting in a single, simple function call like this.
e.g. the next thing to come will be adding a function call to do this:
Cluster::relay_round_robin(Cluster::proxy_pool, key,
Cluster::worker_topic, event, event_args);
Which you could use to send an event from a worker to all other
workers via a rotating proxy/data node. I think at least Justin has
been pushing for something like this.
Generally I think it's best
to use a combination of Broker-internal forwarding and topics to
control where messages get propagated too, as that will better
generalize to larger topologies later.
At least currently, Bro has the forwarding capability of Broker turned
off. IIRC, it was very easy to unintentionally create routing loops
when it was turned on and when I asked about it, no one gave any
justification or use-case for it, thus it got turned off.
The less of the routing we
control manually at the Bro level, the better I'd say. But that's not
an absolute rule of course, and I may just be missing what this is
aiming at.
I'm probably definitely missing some info on long-term plans for
larger topologies and deep-clustering, so let me know if it's
something important to be considering, though it seems like in the
near-term goals and use-cases for the cluster topology, the implicit
routing actually may be more confusing/difficult/error-prone to use
than a simple function call like this which clearly convey's the
sender's intent and expectations.
- Jon
or proxy/data to all other proxy/data nodes (within these classes,
nodes aren't connected with each other) was frequent enough to warrant
putting in a single, simple function call like this.
Yeah, though then I'd actually say it's worth keeping multi-hop
topologies in mind at least. We don't need to fully solve this right
now; nobody really knows yet how topologies may end up looking in the
future. But a good rule of thumb I think is considering that generally
nodes may not be reachable directly, but only through other Broker
hops, and that Broker takes care of the routing to get messages there
as long as topic subscriptions are set up appropiately. Seems that's
actually what we have here already for some nodes. Could we use
Broker-level routing here to get connectivity between the nodes that
aren't directly connected?
At least currently, Bro has the forwarding capability of Broker turned
off. IIRC, it was very easy to unintentionally create routing loops
when it was turned on and when I asked about it, no one gave any
justification or use-case for it, thus it got turned off.
It's off because we indeed don't really have understood yet how to use
it. 
But Broker has message TTLs already for detecting loops, and
generally I don't think there's anything preventing us from turning it
on; it should work.
That all said, the priority right now is on getting a basic cluster to
work, so no problem doing the relaying as you have it now if that gets
us there the quickest. Let's just keep thinking about how such
mechanisms will look in the future in other topologies, so that we
don't back us into a corner (and I hear your of course: to really do
that, we need to understand better where we want to get to).
Robin
Yeah, though then I'd actually say it's worth keeping multi-hop
topologies in mind at least. We don't need to fully solve this right
now; nobody really knows yet how topologies may end up looking in the
future. But a good rule of thumb I think is considering that generally
nodes may not be reachable directly, but only through other Broker
hops, and that Broker takes care of the routing to get messages there
as long as topic subscriptions are set up appropiately. Seems that's
actually what we have here already for some nodes. Could we use
Broker-level routing here to get connectivity between the nodes that
aren't directly connected?
For the case of worker to all other workers via a single proxy, I'm
not sure it's simple to do with implicit routing. You would need a
topic subscription in common between the proxy and workers and you
also want that topic to be unique among all proxies, otherwise you'd
generate many duplicate messages as you send to all proxies and then
all proxies send to all workers. I think you'd end up with a long
subscription lists that look like:
proxy-1: relay_to_workers_via_proxy_1
proxy-2: relay_to_workers_via_proxy_2
...
worker-1: relay_to_workers_via_proxy_1, relay_to_workers_via_proxy_2, ...
worker-2: same as worker-1
And that works and could be automatically generated, though I think
it's conflicting w/ how I intuitively want to try and categorize
topics/subscriptions: what I end up looking for is to have topics
identify either a single node or a whole class of nodes, so instead of
encoding the intent of the message pattern in the topic names, I want
function calls that define the semantics of the messaging pattern and
supply to it the identities/classes of which nodes should be involved
in that pattern.
Another thing I like more about an explicit relay function is that
it's clear that I don't want the relaying node(s) to actually do any
any event handling, I just want them to forward it on, but if I did
want them to handle it, then I still have the option of sending to
them directly using the publish() function. With the internal routing
mechanism, the sender doesn't have as good control/flexibility of
which nodes on the route will end up handling events, at least not
without doing more planning/configuration in advance to set up those
separate avenues.
At least currently, Bro has the forwarding capability of Broker turned
off. IIRC, it was very easy to unintentionally create routing loops
when it was turned on and when I asked about it, no one gave any
justification or use-case for it, thus it got turned off.It's off because we indeed don't really have understood yet how to use
it.
generally I don't think there's anything preventing us from turning it
on; it should work.
I meant that I was the one who turned it off when I started porting
scripts because the first thing I tried to do ended up creating a
loop. Yeah, the 20-hop TTL eventually killed things eventually,
though it wasn't a good first impression
And I think that particular loop would still be there if it got turned
on right now... also not certain, but that particular one might not
be so harmless to leave routing around, it might actually have side
effects with how cluster nodes track each other's identities when they
join/leave the cluster and completely break it.
That all said, the priority right now is on getting a basic cluster to
work, so no problem doing the relaying as you have it now if that gets
us there the quickest. Let's just keep thinking about how such
mechanisms will look in the future in other topologies, so that we
don't back us into a corner (and I hear your of course: to really do
that, we need to understand better where we want to get to).
Yeah, agree that it should become easier to think about how the
extended cluster/routing stuff should come into play after the first
step of porting to broker is done and is available to iterate upon
instead of trying to do both steps in one go.
- Jon