monitoring proxied web traffic

Zeek
1

Hello,

I need to monitor web traffic with Zeek that is going through an implicit web proxy. I would like to be able to see real client IPs in which case tap would be placed before the proxy, however I will not be able to see the real destination IP as it will always be proxy IP. I can also tap on the other side of the proxy where source will always be proxy IP at the same time.

Is there a way with Zeek to correlate the two sessions (before and after proxy) somehow?

I realize that UID or Community ID are probably not going do it since different source/destinations pairs for each for each session, but maybe based on the timestamp/URI/SNI/byte or packet counts/etc?

I was also thinking “X-Forwarded-For” header but I guess that would only work for HTTP and not HTTPS connections.

Thank You,

Konrad

2

There is no way for Zeek to really correlate if the proxy does everything at the application level, like Squid. The proxy

terminates the connection and then it’s free to start the second leg, to the destination host, as it wishes to.

Is your traffic from clients to proxy encrypted as well?

If the traffic between clients are the proxy is not encrypted (as it’s usually the case) then even if the client will establish a TLS session through the proxy you will see a destination with the “service” field http,ssl and all the conn + ssl + http logs.

We have that + proxy logs and that’s pretty much a full visibility.

3

Thank You Michał

Client to Proxy would not be encrypted. So based on that I will have all the logs, however my destination IP for proxied connections would be proxy IP (that was my understanding). I guess ingesting proxy logs would then provide full visibility as you mentioned. I am thinking maybe if for some reason I cannot get the proxy logs ingested and also tap after the proxy, I could possibly correlate both ends of the connections based on URI/SNI/etc and same timestamps in my analytics platform. Anyways I will give that a try as well. Thanks for your help again.

Konrad

4

You’re welcome!!

If the traffic from the client to the proxy is not encrypted and the proxy is not “transparent” (meaning you configured it on the client) then you have the same case as I do here.

So there’s this host
source IP 10.48.75.81

trying to talk to clr.telemetry.intel.com

via our proxy at 10.48.122.10 / 10.48.74.16 / 10.48.74.17

(for anyone working at a place where disclosing an internal IP or an FQDN equal incident - you know a lot of great places are hiring, right?)

I used the “uid” to correlate this connection across three types of logs - conn.log, http.log and ssl.log. You’re gonna get them due to the way proxies operate.

Please ignore tons of metadata from our SIEM (MozDef = custom Python + pure ES), I’m going to C&P three logs as they are without any redactions. This post is HUGE!!

conn.log - host → proxy

{
“_index”: “events-20191108”,
“_type”: “_doc”,
“_id”: “JmLRSW4BiWsbe3JQtC8S”,
“_version”: 1,
“_score”: null,
“_source”: {
“type”: “nsm”,
“details”: {
“ts”: 1573196540.178897,
"uid": “C9PFt711Odfwpbr0Ti”,
"proto": “tcp”,
"service": “http,ssl”,
“duration”: 0.183212,
“orig_bytes”: 1604,
“resp_bytes”: 7180,
“conn_state”: “SF”,
“local_orig”: true,
“local_resp”: true,
“missed_bytes”: 0,
"history": “ShADadFf”,
"orig_pkts": 14,
"resp_pkts": 13,
“peer”: “nsm2-af_packet-enp18s0f0-8”,
"orig_l2_addr": “b4:0c:25:e0:40:10”,
"resp_l2_addr": “00:50:56:a1:48:64”,
"sourceipaddress": “10.48.75.81”,
"sourceport": 55330,
"destinationipaddress": “10.48.122.10”,
"destinationport": 3128,
"originipbytes": 2340,
"responseipbytes": 7864,
“sourceipv4address”: “10.48.75.81”,
“destinationipv4address”: “10.48.122.10”
},
“customendpoint”: “bro”,
“hostname”: “nsm2.private.mdc1.xxxxxxxxxx.com”,
“tags”: “bro”,
“category”: “bro”,
"source": “conn”,
"utctimestamp": “2019-11-08T07:02:20.178897+00:00”,
“timestamp”: “2019-11-08T07:02:20.178897+00:00”,
“receivedtimestamp”: “2019-11-08T07:02:27.686061+00:00”,
“eventsource”: “nsm”,
“severity”: “INFO”,
“mozdefhostname”: “mozdef5.private.mdc1.xxxxxxxxxx.com”,
“summary”: “10.48.75.81:55330 → 10.48.122.10:3128 ShADadFf 2340 bytes / 7864 bytes”,
“plugins”: [
“customDocType”,
“broFixup”,
“ipFixup”,
“geoip”
],
“processid”: “UNKNOWN”,
“processname”: “UNKNOWN”
},
“fields”: {
“receivedtimestamp”: [
“2019-11-08T07:02:27.686Z”
],
“timestamp”: [
“2019-11-08T07:02:20.178Z”
],
“utctimestamp”: [
“2019-11-08T07:02:20.178Z”
]
},
“highlight”: {
“summary”: [
“@kibana-highlighted-field@10.48.75.81@/kibana-highlighted-field@:55330 → 10.48.122.10:3128 ShADadFf 2340 bytes / 7864 bytes”
],
“details.sourceipv4address”: [
“@kibana-highlighted-field@10.48.75.81@/kibana-highlighted-field@”
],
“details.uid”: [
“@kibana-highlighted-field@C9PFt711Odfwpbr0Ti@/kibana-highlighted-field@”
],
“category”: [
“@kibana-highlighted-field@bro@/kibana-highlighted-field@”
]
},
“sort”: [
1573196540178
]
}

http.log (because the connection from the client to the proxy was technically a http connection)

{
“_index”: “events-20191108”,
“_type”: “_doc”,
“_id”: “jPDRSW4BC2j90LU9UdgL”,
“_version”: 1,
“_score”: null,
“_source”: {
“type”: “nsm”,
“details”: {
“ts”: 1573196540.202888,
"uid": “C9PFt711Odfwpbr0Ti”,
“trans_depth”: 1,
“method”: “CONNECT”,
"host": “clr.telemetry.intel.com”,
"uri": “clr.telemetry.intel.com:443”,
“version”: “1.1”,
“request_body_len”: 0,
“response_body_len”: 0,
"status_code": 200,
"status_msg": “Connection established”,
“tags”: [],
“proxied”: [
“PROXY-CONNECTION → Keep-Alive”
],
"sourceipaddress": “10.48.75.81”,
"sourceport": 55330,
"destinationipaddress": “10.48.122.10”,
"destinationport": 3128,
“sourceipv4address”: “10.48.75.81”,
“destinationipv4address”: “10.48.122.10”
},
“customendpoint”: “bro”,
“hostname”: “nsm2.private.mdc1.xxxxxxxxxx.com”,
“tags”: “bro”,
“category”: “bro”,
"source": “http”,
"utctimestamp": “2019-11-08T07:02:20.202888+00:00”,
“timestamp”: “2019-11-08T07:02:20.202888+00:00”,
“receivedtimestamp”: “2019-11-08T07:02:21.427537+00:00”,
“eventsource”: “nsm”,
“severity”: “INFO”,
“mozdefhostname”: “mozdef7.private.mdc1.xxxxxxxxxx.com”,
“summary”: “HTTP CONNECT 10.48.75.81 → 10.48.122.10:3128”,
“plugins”: [
“customDocType”,
“broFixup”,
“ipFixup”,
“geoip”
],
“processid”: “UNKNOWN”,
“processname”: “UNKNOWN”
},
“fields”: {
“receivedtimestamp”: [
“2019-11-08T07:02:21.427Z”
],
“timestamp”: [
“2019-11-08T07:02:20.202Z”
],
“utctimestamp”: [
“2019-11-08T07:02:20.202Z”
]
},
“highlight”: {
“summary”: [
“HTTP CONNECT @kibana-highlighted-field@10.48.75.81@/kibana-highlighted-field@ → 10.48.122.10:3128”
],
“details.sourceipv4address”: [
“@kibana-highlighted-field@10.48.75.81@/kibana-highlighted-field@”
],
“details.uid”: [
“@kibana-highlighted-field@C9PFt711Odfwpbr0Ti@/kibana-highlighted-field@”
],
“category”: [
“@kibana-highlighted-field@bro@/kibana-highlighted-field@”
]
},
“sort”: [
1573196540202
]
}

And the ssl.log - over the previously established http connection the client told proxy to establish a tunnel to the destination host - and Zeek parsed that as TLS (find me another IDS/NSM that can do that)

{
“_index”: “events-20191108”,
“_type”: “_doc”,
“_id”: “sfDRSW4BC2j90LU9S8_Q”,
“_version”: 1,
“_score”: null,
“_source”: {
“type”: “nsm”,
“details”: {
“ts”: 1573196540.253057,
"uid": “C9PFt711Odfwpbr0Ti”,
"version": “TLSv12”,
"cipher": “TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”,
"curve": “secp256r1”,
"server_name": “clr.telemetry.intel.com”,
"resumed": false,
"established": true,
"cert_chain_fuids": [
"FknnGH2L2UIKGERW3b",
"FGtbhK1HoQMWdgyKW",
"F8JbGM30JvqFVi41A6",
"Fsc2fRRTyTDQrT1Gc"
],
"client_cert_chain_fuids": [],
"subject": “CN=c-all.telemetry.intel.com,OU=Multi-Domain SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200 Mission College Blvd,street=FM1-110,L=Santa Clara,ST=California,postalCode=95054-1537,C=US”,
"issuer": “CN=COMODO RSA Organization Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB”,
"validation_status": “ok”,
"server_cert_md5": “27ba7ede8ceab7c896ee4b6bcb509f63”,
"server_cert_sha1": “9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21”,
"pfs": true,
"ja3": “f436b9416f37d134cadd04886327d3e8”,
"ja3s": “303951d4c50efb2e991652225a6f02b1”,
"sourceipaddress": “10.48.75.81”,
"sourceport": 55330,
"destinationipaddress": “10.48.122.10”,
"destinationport": 3128,
“sourceipv4address”: “10.48.75.81”,
“destinationipv4address”: “10.48.122.10”
},
“customendpoint”: “bro”,
“hostname”: “nsm2.private.mdc1.xxxxxxxxxx.com”,
“tags”: “bro”,
“category”: “bro”,
“source”: “ssl”,
“utctimestamp”: “2019-11-08T07:02:20.253057+00:00”,
“timestamp”: “2019-11-08T07:02:20.253057+00:00”,
“receivedtimestamp”: “2019-11-08T07:02:21.464921+00:00”,
“eventsource”: “nsm”,
“severity”: “INFO”,
“mozdefhostname”: “mozdef6.private.mdc1.xxxxxxxxxx.com”,
“summary”: “SSL: 10.48.75.81 → 10.48.122.10:3128”,
“plugins”: [
“customDocType”,
“broFixup”,
“ipFixup”,
“geoip”
],
“processid”: “UNKNOWN”,
“processname”: “UNKNOWN”
},
“fields”: {
“receivedtimestamp”: [
“2019-11-08T07:02:21.464Z”
],
“timestamp”: [
“2019-11-08T07:02:20.253Z”
],
“utctimestamp”: [
“2019-11-08T07:02:20.253Z”
]
},
“highlight”: {
“summary”: [
“SSL: @kibana-highlighted-field@10.48.75.81@/kibana-highlighted-field@ → 10.48.122.10:3128”
],
“details.sourceipv4address”: [
“@kibana-highlighted-field@10.48.75.81@/kibana-highlighted-field@”
],
“details.uid”: [
“@kibana-highlighted-field@C9PFt711Odfwpbr0Ti@/kibana-highlighted-field@”
],
“category”: [
“@kibana-highlighted-field@bro@/kibana-highlighted-field@”
]
},
“sort”: [
1573196540253
]
}

Not enough? Let’s look for the certificate’s MD5

{
“_index”: “events-20191108”,
“_type”: “_doc”,
“_id”: “_CwBSm4BC2j90LU9vfNt”,
“_version”: 1,
“_score”: null,
“_source”: {
“type”: “nsm”,
“details”: {
“ts”: 1573199701.992707,
“id”: “FdE5qt1b93z9QOJnz1”,
“san.dns”: [
“c-all.telemetry.intel.com”,
“c.telemetry.intel.com”,
“c1.telemetry.intel.com”,
“c2.telemetry.intel.com”,
“c3.telemetry.intel.com”,
“c4.telemetry.intel.com”,
“c5.telemetry.intel.com”,
“c6.telemetry.intel.com”,
“c7.telemetry.intel.com”,
“c8.telemetry.intel.com”,
“c9.telemetry.intel.com”,
“clr.telemetry.intel.com”,
“sur.telemetry.intel.com”
],
“md5”: “27ba7ede8ceab7c896ee4b6bcb509f63”,
“sha1”: “9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21”,
“certificate”: {
“basic_constraints_ca”: false,
“exponent”: “65537”,
“issuer”: “CN=COMODO RSA Organization Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB”,
“key_alg”: “rsaEncryption”,
“key_length”: 2048,
“key_type”: “rsa”,
“not_valid_after”: “2020-08-02T23:59:59+00:00”,
“not_valid_before”: “2018-08-03T00:00:00+00:00”,
“sig_alg”: “sha256WithRSAEncryption”,
“subject”: “CN=c-all.telemetry.intel.com,OU=Multi-Domain SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200 Mission College Blvd,street=FM1-110,L=Santa Clara,ST=California,postalCode=95054-1537,C=US”,
“version”: 3,
“serial”: “1406C0A906762390B7252C9598F2DB3C”
}
},
“customendpoint”: “bro”,
“hostname”: “nsm1-ber3.private.ber3.xxxxxxxxxx.com”,
“tags”: “bro”,
“category”: “bro”,
“source”: “x509”,
“utctimestamp”: “2019-11-08T07:55:01.992707+00:00”,
“timestamp”: “2019-11-08T07:55:01.992707+00:00”,
“receivedtimestamp”: “2019-11-08T07:55:17.040196+00:00”,
“eventsource”: “nsm”,
“severity”: “INFO”,
“mozdefhostname”: “mozdef1.private.mdc1.xxxxxxxxxx.com”,
“summary”: “X509 certificate seen”,
“plugins”: [
“customDocType”,
“broFixup”
],
“processid”: “UNKNOWN”,
“processname”: “UNKNOWN”
},
“fields”: {
“details.certificate.not_valid_after”: [
“2020-08-02T23:59:59.000Z”
],
“receivedtimestamp”: [
“2019-11-08T07:55:17.040Z”
],
“details.certificate.not_valid_before”: [
“2018-08-03T00:00:00.000Z”
],
“timestamp”: [
“2019-11-08T07:55:01.992Z”
],
“utctimestamp”: [
“2019-11-08T07:55:01.992Z”
]
},
“highlight”: {
“details.md5”: [
“@kibana-highlighted-field@27ba7ede8ceab7c896ee4b6bcb509f63@/kibana-highlighted-field@”
],
“source”: [
“@kibana-highlighted-field@x509@/kibana-highlighted-field@”
],
“category”: [
“@kibana-highlighted-field@bro@/kibana-highlighted-field@”
]
},
“sort”: [
1573199701992
]
}

Let’s try to splice the first and the second half of the connection

category:bro AND source:dns AND details.query:clr.telemetry.intel.com

clr.ats.infra-host.com, 52.37.21.110, 34.208.200.27

Back to looking at conn.log with destination IP like above and source = proxies (from there you can again do the crazy dance with ssl.log but there’s no need to)

{
“_index”: “events-20191108”,
“_type”: “_doc”,
“_id”: “jvDRSW4BC2j90LU9UdgL”,
“_version”: 1,
“_score”: null,
“_source”: {
“type”: “nsm”,
“details”: {
“ts”: 1573196540.253057,
“uid”: “CrcIhj4lIsSLpPwPe5”,
“version”: “TLSv12”,
“cipher”: “TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”,
“curve”: “secp256r1”,
“server_name”: “clr.telemetry.intel.com”,
“resumed”: false,
“established”: true,
“cert_chain_fuids”: [
“FpfU1u4yMxOBNKQJk2”,
“FYiimM3T6A5AKcTeze”,
“FfAQxKi177b5fl95a”,
“FWHqmB2emMx082lxO1”
],
“client_cert_chain_fuids”: [],
“subject”: “CN=c-all.telemetry.intel.com,OU=Multi-Domain SSL,OU=Hosted by Intel Corporation,OU=SSG,O=Intel Corporation,street=2200 Mission College Blvd,street=FM1-110,L=Santa Clara,ST=California,postalCode=95054-1537,C=US”,
“issuer”: “CN=COMODO RSA Organization Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB”,
“validation_status”: “ok”,
“server_cert_md5”: “27ba7ede8ceab7c896ee4b6bcb509f63”,
“server_cert_sha1”: “9077d0ce724c0f7bf1866d7cf8c4ec1303dc6c21”,
“pfs”: true,
“ja3”: “f436b9416f37d134cadd04886327d3e8”,
“ja3s”: “303951d4c50efb2e991652225a6f02b1”,
“sourceipaddress”: “10.48.74.16”,
“sourceport”: 58860,
“destinationipaddress”: “52.37.21.110”,
“destinationport”: 443,
“sourceipv4address”: “10.48.74.16”,
“destinationipv4address”: “52.37.21.110”,
“destinationipgeolocation”: {
“city”: “Boardman”,
“continent”: “NA”,
“country_code”: “US”,
“country_name”: “United States”,
“dma_code”: 810,
“latitude”: 45.8491,
“longitude”: -119.7143,
“metro_code”: “Boardman, OR”,
“postal_code”: “97818”,
“region_code”: “OR”,
“time_zone”: “America/Los_Angeles”
},
“destinationipgeopoint”: “45.8491,-119.7143”
},
“customendpoint”: “bro”,
“hostname”: “nsm2.private.mdc1.xxxxxxxxxx.com”,
“tags”: “bro”,
“category”: “bro”,
“source”: “ssl”,
“utctimestamp”: “2019-11-08T07:02:20.253057+00:00”,
“timestamp”: “2019-11-08T07:02:20.253057+00:00”,
“receivedtimestamp”: “2019-11-08T07:02:21.465325+00:00”,
“eventsource”: “nsm”,
“severity”: “INFO”,
“mozdefhostname”: “mozdef7.private.mdc1.xxxxxxxxxxxxxxxxxxxx.com”,
“summary”: “SSL: 10.48.74.16 → 52.37.21.110:443”,
“plugins”: [
“customDocType”,
“broFixup”,
“ipFixup”,
“geoip”
],
“processid”: “UNKNOWN”,
“processname”: “UNKNOWN”
},
“fields”: {
“receivedtimestamp”: [
“2019-11-08T07:02:21.465Z”
],
“timestamp”: [
“2019-11-08T07:02:20.253Z”
],
“utctimestamp”: [
“2019-11-08T07:02:20.253Z”
]
},
“highlight”: {
“category”: [
“@kibana-highlighted-field@bro@/kibana-highlighted-field@”
]
},
“sort”: [
1573196540253
]
}

I could now paste the third (?!) leg of this connection - after SNAT, but you get the idea.

Unfortunately, this sort of visibility partially goes away soon, due to some pseudo-privacy mechanisms AKA a classic shoot in the foot, while trying to kill a mosquito. From a shotgun. ← my private opinion BTW. I’m well
after hours :wink:

5

Thanks again Michał for the breakdown. I have the full picture now :slight_smile:

Take care

Konrad