I was curious if anyone has any experience running bro between
load-balancers (such as Netscaler) and web applications. We are
currently trying to get HTTP logs generated for a web application. We
couldn't figure out why bro was not triggering the HTTP analyzer, but
I now believe that this is because it is never seeing the original SYN
+ SYN/ACK for the conversation. When viewing the conversations in
Wireshark, I can see that all the TCP streams for this particular
application begin with the GET and do not include the initial 3-way
handshake.
Here is an entry in the conn.log for this stream which shows the states:
Other web applications on the wire, which do have the 3-way handshake
visible for all connections, seem to work just fine and I get http
logs.
My questions are:
Am I correct in assuming that the lack of initial connection
establishment is why the HTTP analysis is never occurring (and
therefore I'm not getting entries in http.log)?
Is there a way to force bro to analyze the traffic even though there
is no proper 3-way handshake visible?
Not sure what Netscalar does, but it all should act the same. The host
TCP stack would drop any attempted connection for which a session was
not established regardless of what was upstream from it. Quick and
dirty, you sould be able to fire up tcpdump and see the session
initialization.
That's what I'm finding strange. After running a tcpdump capture on
the interface and analyzing it with Wireshark, I do not see any 3-way
handshakes for this particular web application. For any HTTP GET that
I see in Wireshark that pertains to this application, when I "Follow
TCP Stream", the first entry in Wireshark is always the GET message
itself. For all other applications on the network, doing the above
results in the first entry being the SYN.
I've generated a few dumps with the same results. I wonder if the
load balancer is somehow keeping a session active for very long
periods (if this even makes sense).
If you have any suggestions or thoughts, I'd be very interested.
Just to make sure this isn't it, what bpf filter if any are you using with
tcpdump and bro?
If it's not the filter, the only thing I can think of is that the load balancer
is opening a persistent (http/1.1 keep-alive) connection to the backend servers.
I don't know how common that sort of thing is, but it would be easy to check
for, you would see the http/1.1 connection: header in the GET request..
you could also see if tcpdump sees a 3-way handshake if you restart one of the
webservers.
Thanks for the response. I do actually see a "Connection:
Keep-Alive\r\n" in the GET packet. From this, can I assume that a
persistent connection is being held, thus the confusion by bro?
If so, do you have any ideas or suggestions on how I can get the HTTP
analyzer to still process these as if the connection had been
established normally?
Sorry, I meant to add that no bpf filters are being added to tcpdump
or bro at this time. Also, I don't have the ability to restart these
servers at this time.
Thanks for the response. I do actually see a "Connection:
Keep-Alive\r\n" in the GET packet. From this, can I assume that a
persistent connection is being held, thus the confusion by bro?
Right... if the connection is long lived, there was a handshake, it just
happened before Bro started.
If so, do you have any ideas or suggestions on how I can get the HTTP
analyzer to still process these as if the connection had been
established normally?
Regards,
Bill
I'm not sure about the Bro side, but you could probably turn off keepalives on
the load balancer. It also might have an option somewhere called 'maximum
number of requests' that might be a high number like 5000, you could try
lowering that a bunch, which would cause a handshake to happen frequently
enough to keep Bro happy.
Thanks, that makes sense. I'm not sure if I'll be able to tweak the
load-balancer settings at this time, but if I am, that may indeed
solve this issue. Hopefully, come Monday morning when many more
sessions will be generated, I may actually see some of these
connections as established and bro will operate as expected.
Am I correct in assuming that the lack of initial connection
establishment is why the HTTP analysis is never occurring (and
therefore I'm not getting entries in http.log)?
Yes. The appended patch should cause the analyzer to function anyway.
Thanks for the response. I do actually see a "Connection:
Keep-Alive\r\n" in the GET packet. From this, can I assume that a
persistent connection is being held, thus the confusion by bro?
That's doubtless what's happening. (FYI, seeing that from the client side
it's just a request, than necessarily imposed on the connection. However,
in the setup you describe, surely the server is accepting this, and hence
the seemingly missing SYNs for later requests.)
Wow, thanks for the quick response! I am going to look deeper into
the causes, but this is exactly the type of short-term solution I was
looking for. Much obliged. I will be sure to report back if I have
any interesting findings.
I have a few machines that can produce large flows in access 100Mbytes over NLR. This cause bro to drop packets. Right now I am using pcap filters and extra workers on the sensor machine to insures that only packets dropped are from the file transfers machines. This is not that big down side but it make we wonder if just what is the biggest flow that bro can handle on a single cpu.