Cannot catch http.log on internal network machines, but it can be found on external network machines

Zeek
1

Hello, I would like to obtain the HTTP request header, request body, response header, and response body through Zeek. Therefore, I built vmware17pro, zeek3.0.1, and metronpro plugin kafka on an external laptop. Everything is progressing smoothly, including pushing data into Kafka. However, during the process of exporting ovf from vmware to my intranet machine, zeek will not generate http.log, but can generate conn. log, dns. log, and so on. Do you need any special configurations to make the internal network effective?

I have also built a local zeek on the external network, but cannot capture http.log on the internal machine

The link I referenced https://mp.weixin.qq.com/s?__biz=MzAwMDQwNTE5MA==&mid=2650246104&idx=1&sn=fcc7446358e44e2610eb074ecce42e3e&chksm=82ea5071b59dd9674451f067b92896805e6932cfac3d7ee8e51810f45f18f85838bd6af6e657&scene=126&sessionid=1654532076&key=acb1db43c4248adcb8aba20712f8f2b82f2861be05a9ca396ad1117a85231f97a17181f46aba51e464c4eae6cf5e7d7dbecb2dc8f3321abaeac6c6ec8774bf24c1294bb239f4f75bb5252d535e1b620b15406eb3c91b6d62b13b8c09795fbc92c9ccb164fa6259ebb53283cc531a83a52cbc8085745f07bf72cf536a9c421c35&ascene=15&uin=NTY2NTA4NjQ%3D&devicetype=Windows +Server+2016+x64&Version=63060012&lang=zh CN&Session_ Us=gh_ B902de9201a8&exportkey=A8RDeGV7z% 2FZHnMQmLHfHrn8% 3D&acctmode=0&pass_ Ticket=ug9ebHcf9fKbS3urjDlVAKXSYK6y4cQY9T5Al4wODa4jVvP1ccrZYm4gcaJ12&wx_ Header=0&fontgear=2

2

Hey @lideliang,

Could you instead use Zeek 6.0.0 if you’re starting in a new environment. Zeek 3.0.1 was released in 2019.

For the rest of your observations, is there a way for you could confirm with another network monitoring tool like tcpdump or wireshark that the traffic you’re expecting is received by the monitoring interface? Much HTTP communication is encrypted these days, so http.log may be empty. You could cross check the ssl.log.

Hope this helps,
Arne

3

Later on, I also used the latest zeek 6.0.0, which was unable to generate http.log. However, I found through conn.log that there was indeed related traffic. I installed Wireshark on the virtual machine and observed HTTP traffic.

I installed the OVF to the internal network through vmware export, modified the network card to ensure that the virtual machine can connect to the host, and modified network.cfg and node.cfg

4

However, I found through conn.log that there was indeed related traffic. I installed Wireshark on the virtual machine and observed HTTP traffic.

Is it possible to capture a subset (preferably just one flow) of traffic as .pcap file using tcpdump or wireshark, the compare the result between zeek -r and wireshark. In case of unexpected differences we’d be happy if you could share the pcap to take a look.

Thanks,
Arne

5

Why can I monitor HTTP traffic using domain name access, but not HTTP traffic using IP access.

I built a centros (with GUI) using a virtual machine, and accessing the website using the browser on the virtual machine did not generate an http.log by entering an IP address. However, accessing the website using a domain name generated an http.log. Why is this happening? How to restore the generation of http.log through IP access to websites.

Will there be such a problem when mirroring traffic through a switch. For example, if I access a website through IP and the traffic flows into Zeek, will it not generate an HTTP. log?

6

I know why http.log cannot be monitored because it is defined in/usr/local/zeek/share/zeek/base/protocols/http/main.zeek

Const ports={

80/tcp, 81/tcp, 631/tcp, 1080/tcp, 3128/tcp,

8000/TCP, 8080/TCP, 8888/TCP,

};

Redef Likely_ Server_ Ports+={ports};

Is this a way of writing that only monitors these few port numbers? If I want to monitor all ports, should I enumerate all ports?

7

For non-standard ports, Zeek uses Dynamic Protocol Detection (DPD). Basically, analyzers are enabled based on signatures. The following is used for HTTP.

If this doesn’t work reliable for you, please share a pcap capture of your traffic.

Another thought: Given you setup involves virtual machines: Have you looked at the ignore_checksums option? If the history field in your conn.log contains c or C the packets might have incorrect checksums. Zeek ignores these by default.

Hope this helps,
Arne