Hey all, I’m trying to follow and correlate web sessions that go through a proxy server pool and then through a NATing firewall - and I’m curious if Bro would be the tool to use?
My immediate need is to troubleshoot a nagging performance issue with our web browsing. I’d like to be able to analyze all our web sessions during a period and discover those that show slow responses and trace back to where the delay is located.
What makes this tricky is that the sample would be 1000’s of clients, 100’s of GB per hour in traffic, a pool of proxy servers, and a FW that is doing NATing/PATing and protocol inspection (it doesn’t blindly forward every packet, it is in the middle of the TCP sessions randomizing SEQ numbers and such).
Specifically, we have a few thousand web clients that are transparently proxied (WCCP) for HTTP/S. After their traffic leaves the proxy, it goes through a FW and out to the Internet.
I’ve analyzed specific flows, manually tracing through the hops by inserting unique headers, but that is fairly laborious and I don’t know for sure if I’m truly capturing what the clients are seeing.
I can capture traffic at each stage simultaneously, so I should be able to get a good view.
I believe I would need to:
-Create a list of all client->proxy TCP sessions
–Match them to any DNS queries
–understand what a complete page/request is (i.e. a complete yahoo.com request goes to dozens of sites to build the page)
-Create a list of proxy->FW sessions
–Correlate to client->proxy session list via destination, client headers, time, etc.
-Create a list of ASA->Internet sessions via destination, client headers, time, etc.
–Correlate to client->proxy->ASA list
Once I have a complete flow from client to Internet, then I need to analyze and look for patterns of delays. Are delays caused by one or more proxy servers all the time, periodically, based on certain destinations, etc? I noticed the FW normally forwards packets one for one unless a packet is dropped from the Internet - then it buffers the packets until it can deliver a complete response back to the proxy. I assume the FW wants a complete response so that it can do its protocol inspection.
(After I typed this out, I’m thinking it’ll probably be quicker to use PhantomJS/other and drive some test workstations, but I am curious how/if Bro can do this because analysis of actual traffic would be most accurate imho).
Thanks for any input