Zeek Telemetry /services.json issue

Zeek
1

Hi,

I am trying to set up metric exporting in Zeek following the documentation here. So far I’ve been reasonably successful and have all of my nodes exporting metrics to their respective ports and can see these on the /services.json endpoint of my manager node (see output of curl below).

$curl http://127.0.0.1:9990/services.json
Output:
[{"targets":["127.0.0.1:9995","127.0.0.1:9997","127.0.0.1:9996","127.0.0.1:9994","127.0.0.1:9992","127.0.0.1:9993","127.0.0.1:9991"],"labels":{}}]

The issue I am running into is when using Prometheus Service discovery to scrape these endpoints. Prometheus is running on a different machine and is not able to resolve the references to localhost returned by /services.json . I need to somehow change this address to the ip of my host which I believe will resolve the issue.

One thing I have tried is setting Telemetry::metrics_address in local.zeek. This changes the address of the metrics api but does not affect the /services.json endpoint. I did poke around in the source code a bit and it seems the ip returned by /services.json is fetched from the Cluster::node variable so this explains the discrepancy. If you are interested you can find the relevant code for the /services.json endpoint here.

I guess a temporary fix is to just not use http service discovery but I’d like to get this working if possible. Any ideas? As you can probably guess I’m pretty new to zeek so be gentle!

Kind regards,

Blair

2

Hello @blairmcc03 :waving_hand:

I ran into this as well, but never attempted to straighten it out or create a ticket/discussion as I had found a workaround. The zeek-demo repo has the following in the prometheus.yaml file which rewrites the 127.0.0.1’s at scrape time on the Prometheus server side:

  - job_name: zeek-local-http-sd
    scrape_interval: 5s
    http_sd_configs:
      - url: http://zeek:9991/services.json
        refresh_interval: 10s
    # As of Zeek 8.2-dev, the manager returns 127.0.0.1 because this is
    # what the cluster layout generates?
    relabel_configs:
      - source_labels: [__address__]
        regex: '127\.0\.0\.1(:\d+)'
        replacement: 'zeek${1}'
        target_label: __address__

Could you see if this relabeling provides a work-around for you? If you run a Zeek cluster over multiple systems, you’ll need to do something else, however.

A better fix would be to construct services.json not with the $ip in cluster-layout (used for cluster-internal-process communication), but instead take the Telemetry::metrics_address or of the processes. However, with the current static cluster-layout this is slightly cumbersome and (IMO) the proper solution would be to collect the chosen metrics address (and port?) from other processes to the manager, which then can dynamically construct /services.json. Discussion #5208 goes into that direction - this would also allow to remove metrics_port from the cluster-layout.zeek file.

Feel free to summarize in a new GitHub issue, you’ll likely aren’t the only one running into this.

Thanks,
Arne

3

Hi Arne,

Thanks for your help on this. The temporary workaround on the prometheus side works for now. Eventually our monitoring will need to support multiple (four) systems. Again I’m sure I will be able to find a fix on the prometheus side.

As you mentioned I will create a GitHub issue and suggest constructing services.json from Telemetry:metrics_address . I will also refer to the discussion that you linked.

Thanks again for your help.

Kind regards,
Blair

4

If you do this with ZeekControl, the $ip rendered in the cluster-layout.zeek should use the address from host= in node.cfg which will not be 127.0.0.1 and so that should work out.

Thanks for creating the issue :+1: