I’ve been playing with sending event data to a peer using the broker interface. I’m able to send records over just fine and my python script can receive and interpret them using the swig-generated wrapper as long as all the fields are present.
If I try to send one with optional fields such as pkt_hdr where not all of the fields are present (as is always the case with pkt_hdr), I get various segmentation violations either direct in the swig-generated code or assertion failures in the ‘optional’ class.
Seems like there should be a more intelligent iterator for the record fields in the swig source. I was thinking it would make sense to return a None value in the slot where a non-present optional value goes and then you could just test for that, but I don’t know enough about swig to create the iterator. I’ve tried several combinations of %extend, %pythoncode and so forth, but can’t figure out the right magic words.
Anybody know the right way to do this?
There’s a brief example of sending/receiving a record with an empty field in tests/test_messages.py. You can call the valid() method on a field to test if there’s data there that you’re allowed to access. If that doesn’t help clarify the issue, can you post some example code?
- Jon
Yeah. Using valid() works but it makes the code clumsy and unpythonic. Below is my code. Uncomment the second “sub_recs = …” line in ppkt() (in place of the first one) to see the issue.
With this .bro loaded:
const broker_port: port = 9999/tcp &redef;
redef BrokerComm::endpoint_name = “events”;
export {
global jb_packet: event(p: pkt_hdr);
}
event bro_init()
{
BrokerComm::enable();
BrokerComm::listen(broker_port, “127.0.0.1”);
BrokerComm::auto_event(“bro/event/jb_packet”, jb_packet);
}
event new_packet(c:connection, p: pkt_hdr) { event jb_packet(p); }
I have this script:
#!/usr/bin/env python
from select import select
import pybroker
def get_fields(fields, n_fields):
new_fields = []
for n in range(n_fields):
f = fields[n]
if f.valid():
new_fields.append(f.get())
else:
new_fields.append(None)
return new_fields
def ppkt(p):
rec = p.as_record()
sub_recs = get_fields(rec.fields(), rec.size())
#===>>> sub_recs = [f.get() for f in fields]
print sub_recs
def pmsg(msg_type, obj):
msg_type = msg_type.as_string()
pobj = {
“jb_packet”: ppkt,
}[msg_type]
print "%s: " % msg_type,
pobj(obj)
def main():
epc = pybroker.endpoint(“connector”)
epc.peer(“127.0.0.1”, 9999, 1)
ocsq = epc.outgoing_connection_status()
select([ocsq.fd()], [], [])
conns = ocsq.want_pop()
for m in conns:
print(“outgoing connection”, m.peer_name, m.status)
mql = pybroker.message_queue(“bro/event”, epc)
while True:
select([mql.fd()], [], [])
msgs = mql.want_pop()
for m in msgs:
pmsg(*m)
main()
Does something like the following work to transform it into the format you want?
sub_recs = [f.get() if f.valid() else None for f in fields]
- Jon
Jon,
That does seem to work at the top-most level (i.e. pkt_hdr). But then in trying to break apart the next layer (ip_hdr), it doesn’t. I get a seg fault in the “vector_of_field” iterator.
With:
rec = p.as_record()
fields = rec.fields()
ip, ip6, tcp, udp, icmp = [f.get() if f.valid() else None for f in fields]
if ip is not None:
fields = ip.as_record().fields()
fields = [f for f in fields]
This happens in the last line:
Program received signal SIGSEGV, Segmentation fault.
0xb76a1386 in broker::util::optionalbroker::data::optional (this=0x846f688, other=…) at /home/jbarber/src/bro/aux/broker/broker/util/optional.hh:84
84 { if ( other.is_valid ) create(other.value); }
(gdb) bt
#0 0xb76a1386 in broker::util::optionalbroker::data::optional (this=0x846f688, other=…) at /home/jbarber/src/bro/aux/broker/broker/util/optional.hh:84
#1 0xb7704471 in swig::traits_from<broker::util::optionalbroker::data >::from (val=…)
at /home/jbarber/src/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:3909
#2 0xb770419a in swig::from<broker::util::optionalbroker::data > (val=…)
at /home/jbarber/src/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:3928
#3 0xb7703cc1 in swig::from_oper<broker::util::optionalbroker::data >::operator() (this=0x84345a4, v=…)
at /home/jbarber/src/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:4481
#4 0xb77030ec in swig::SwigPyIteratorClosed_T<__gnu_cxx::__normal_iterator<broker::util::optionalbroker::data*, std::vector<broker::util::optionalbroker::data, std::allocator<broker::util::optionalbroker::data > > >, broker::util::optionalbroker::data, swig::from_oper<broker::util::optionalbroker::data > >::value (
this=0x8434598) at /home/jbarber/src/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:4549
#5 0xb769f12c in swig::SwigPyIterator::next (this=0x8434598) at /home/jbarber/src/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:3243
#6 0xb76358b2 in _wrap_SwigPyIterator_next (args=0xb77af38c) at /home/jbarber/src/bro/build/aux/broker/bindings/python/pybrokerPYTHON_wrap.cxx:7044
#7 0x08156a91 in PyEval_EvalFrameEx ()
…
(gdb) print other
$1 = (const broker::util::optionalbroker::data &) @0x0:
The problem here was ip.as_record() returns a new object, but then .fields() on it returns a reference to something owned by that object whose reference count is going to drop to zero immediately after the line. So it ends up accessing invalid memory of the object which went out of scope. If you want to workaround the bug, assign the record to a temporary variable for as long as you need to access fields coming from it. E.g.:
rec = ip.as_record() # assign to rec in order to keep a reference alive
fields = rec.fields()
# operate on fields…
Or if you want to patch/update the broker source code, the real fix is here:
https://github.com/bro/broker/commit/8fc6938017dc15acfb26fa29e6ad0933019781c5
That’s also in the master branch of the bro and broker repositories, but should eventually make it into the final 2.4 Bro release and a 0.3.1 release of Broker as well.
- Jon