# Zeek Supervisor Command-Line Client **URL:** \ **Category:** Development\ **Tags:** development\ **Created:** [June 18, 2020, 2:07am UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158 "2020-06-18T02:07:53Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [June 18, 2020, 2:07am UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/1 "2020-06-18T02:07:53Z")
Don't recall any basic "project infrastructure" discussions happening yet for the upcoming replacement/alternative for ZeekControl that we want to introduce in Zeek 3.2 (roadmap/design links found at [1]), so here's starting questions. # What to Name It ? Suggestion: `zeekcl`, Zeek (Command-Line) CLlient. Open to ideas, but will use `zeekcl` below. # What Programming Language ? `zeekcl` has different/narrower scope than ZeekControl. It's more clearly a "client" with sole job of handling requests/responses via Broker without many (any?) system-level operations/integrations. Meaning there may be less of an approachability/convenience gap between C++ versus Python with `zeekcl` than there was with ZeekControl. Also nice if `zeekcl` doesn't require more dependencies beyond what `zeek` needs since they're expected to be used together. Is use of Python still desirable for other reasons? Otherwise, I lean towards `zeekcl` being C++. For reference/sanity-check in terms of what people expect `zeekcl` to be: in my testing of the SupervisorControl framework [2], I had a sloppy Zeek script implementing the full "client side" (essentially the majority of what `zeekcl` will do) in ~100 LOC. Most operations are that simple: send request and display response. That does mean the third option to consider besides either Python or C++ is Zeek's scripting language (e.g. `ctl.zeek`), but I don't suggest that since (1) using a full `zeek` process is way more than we need and (2) the command-line interface is awkward (`zeek ctl Supervisor::cmd="status"` versus `zeekcl status`) # Where's the Source Code Live ? Past experiences with ZeekControl being in a separate repo than Zeek are negative in terms of CI/testing: changes in Zeek have broken ZeekControl, but go uncaught for a while since it is tested independently. Since common use/maintenance will involve both `zeek` and `zeekcl`, and also don't expect the later to accrue large amounts of code deserving of a separate project, I plan to have `zeekcl` code/tests live inside the main Zeek repo. - Jon [1] [https://github.com/zeek/zeek/issues/582](https://github.com/zeek/zeek/issues/582) [2] [https://github.com/zeek/zeek/blob/689a242836092fba7818ba24724b74a7a7902e48/scripts/base/frameworks/supervisor/control.zeek](https://github.com/zeek/zeek/blob/689a242836092fba7818ba24724b74a7a7902e48/scripts/base/frameworks/supervisor/control.zeek) ---
**Author:** ![Vlad\_Grigorescu](https://avatars.discourse-cdn.com/v4/letter/v/a3d4f5/32.png) [@Vlad\_Grigorescu](https://community.zeek.org/u/Vlad_Grigorescu)\ **Post date:** [June 18, 2020, 3:32am UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/2 "2020-06-18T03:32:52Z")
I’m still fuzzy on the Supervisor framework, as we’re still in the process of upgrading systems to the point of supporting the new C++ requirements. As a concrete example, what does a cluster upgrade look like? Today, that means install the new version on the manager, and then do `zeekctl deploy`, which copies the files to the nodes and restarts the cluster. All of that is done without Broker. What does that look like with zeekcl + Broker? Let’s say I install the new version on the manager. If I then tell zeekcl to destroy the running instance, will that work, or will the newer zeekcl be incompatible with the Broker version of the running Zeek? Reading the script linked in [2], I notice that zeekcl would not support copying files from one node to another? Other features that would be missing that we routinely use are `zeekctl print` and `zeekctl exec`. I’m assuming `zeekcl` would be running in some uber-bare mode if it’s written in Zeek? –Vlad ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [June 18, 2020, 6:34am UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/3 "2020-06-18T06:34:00Z")
> Suggestion: `zeekcl`, Zeek (Command-Line) CLlient. "zeekcl" is very close to "zeekctl", which could lead to confusion. "zcl" maybe? > Is use of Python still desirable for other reasons? Otherwise, I lean > towards `zeekcl` being C++. No particular preference from my side, I can see either. Effort is probably about the same in this model, and C++ does have the advantage of less dependency issues. > Zeek's scripting language (e.g. `ctl.zeek`), but I don't suggest that Ack, agree. > I plan to have `zeekcl` code/tests live inside the main Zeek repo. Makes sense to me as well. Robin ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [June 18, 2020, 7:11am UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/4 "2020-06-18T07:11:41Z")
> As a concrete example, what does a cluster upgrade look like? The idea is to handle this more like other system services: you'll be in charge of getting the new Zeek version onto all your systems yourself, using whatever method you use for other software as well. For example, if you're installing through a package manager, you'd just run "update" on all systems. If you're installing from source, you'll either need to compile on each system, or copy the installation over manually. The underlying assumption is that people will already have a mechanism in place for administration of their systems, and we shouldn't be trying to reinvent the wheel, as ZeekControl oddly does. From a sysadmin perspective, ZeekControl is really doing a lot more right now that it should be doing; other tools don't work that way. We don't want it look like an APT anymore ([broctl looks like an APT · Issue #259 · zeek/zeek · GitHub](https://github.com/zeek/zeek/issues/259)). 🙂 > Today, that means install the new version on the manager, and then do > `zeekctl deploy`, which copies the files to the nodes and restarts the > cluster. All of that is done without Broker. There are two parts here: (1) deploying the Zeek installation itself, and (2) deploying any configuration changes (incl. new Zeek scripts). For (1), the above applies: we'll rely on standard sysadmin processes for updating. That means you'd use "zeekcl" to shutdown the cluster processes, then run "yum update" (or whatever), then use "zeekcl" again to start things up again. (The Zeek supervisor will be running already at that point, managaged through systemd or whatever you're using). (2) is still a bit up in the air. With 3.2, there won't be any support for distributing configurations automatically, but we could add that so that config files/scripts/packages do get copied around over Broker. Feedback would be appreciated here: What's better, having zeekcl manage that, or leave it to standard sysadmin process as well? > Reading the script linked in [2], I notice that zeekcl would not support > copying files from one node to another? Correct right now, (2) may or may not change that. > zeekctl print "print" will be supported (roadmap says not in 3.2 yet, but it should be easy to do, maybe we can get it in still). > zeekctl exec. "exec" will likely not be supported. We \*could\* support it, no technical reason for not doing that over Broker. It just s seems like another things that's better handled with different tools. Robin ---
**Author:** ![Vlad\_Grigorescu](https://avatars.discourse-cdn.com/v4/letter/v/a3d4f5/32.png) [@Vlad\_Grigorescu](https://community.zeek.org/u/Vlad_Grigorescu)\ **Post date:** [June 18, 2020, 2:45pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/5 "2020-06-18T14:45:33Z")
Thanks Robin, that helps. ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [June 18, 2020, 8:00pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/6 "2020-06-18T20:00:44Z")
I have a slightly different take: isn't it more common to expect "start" and "stop" operations here to be done by the service-manager rather than Zeek client? I'm assuming "update/deploy Zeek installation" could involve a change in the `zeek` binary and that implements the supervisor process itself, so you'd want, at the level of system services, to stop the entire Zeek process tree, including the root supervisor. That doesn't exclude the possibility of the client having operations like "start" (spawn `zeek -j \`), "stop" (kill the root `zeek` supervisor process), or even others that dynamically add/remove cluster nodes from the tree, but that's probably not the common/expected usage to prioritize since it's again back to model of the process tree being managed manually by the user, independent from a system's service-manager. - Jon ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [June 18, 2020, 8:15pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/7 "2020-06-18T20:15:14Z")
A clarification that may help you: the "orphaning" behavior isn't related to Broker connections, it's related to the parent-child relationship between processes. So there's a process tree here with `zeek` in supervisor-mode at the root and child processes that are individual cluster nodes (worker, manager, logger, proxy). The normal termination behavior for the supervisor process is to gracefully kill and wait for all children to exit. In the very exceptional case of the supervisor exiting/crashing without having cleaned up all children, those children will self-terminate upon noticing they are no longer parented to the supervisor. - Jon ---
**Author:** ![johanna](https://avatars.discourse-cdn.com/v4/letter/j/50afbb/32.png) [@johanna](https://community.zeek.org/u/johanna)\ **Post date:** [June 18, 2020, 8:34pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/8 "2020-06-18T20:34:14Z")
> > Suggestion: `zeekcl`, Zeek (Command-Line) CLlient. > > "zeekcl" is very close to "zeekctl", which could lead to confusion. > "zcl" maybe? > > > Is use of Python still desirable for other reasons? Otherwise, I lean > > towards `zeekcl` being C++. > > No particular preference from my side, I can see either. Effort is > probably about the same in this model, and C++ does have the advantage > of less dependency issues. I agree - I actually kind of like the idea that zeekcl does not have python as a dependency. > > I plan to have `zeekcl` code/tests live inside the main Zeek repo. > > Makes sense to me as well. Agreed here too. Johanna ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [June 19, 2020, 8:38am UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/9 "2020-06-19T08:38:10Z")
I believe we're pretty close to saying the same thing. I'm making a distinction between the supervisor Zeek process (which the service manager starts & stops), and the cluster's node processes (manager, workers, etc). The supervisor manages the latter and will by default shut them down when it gets the "stop" from its service-manager. But I think we also want their state controllable from the client as well, so that one can have an orderly shutdown of a multi-system cluster without loss of data (e.g., one probably wants to shutdown workers first to collect remaining log data). This what I meant above by "shutdown the cluster processes": "zeek-client stop" would tell the supervisors to shutdown their node processes (or rather: "zeek-client stop workers", or maybe "zeek-client" would now the order in which to stop nodes or systems). And I imagine one would do that before starting to a cluster-wide upgrade to the next Zeek version. That said, your note on Slack sounds right: let's figure out the single-system operation first and get that usable. I'm pretty confident that we will then be able to build the multi-system model on top of that without too much trouble, and it'll we easier to collect requirements for administration/management of multi-system setups once we got some experience with single-system setups. Robin ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [June 19, 2020, 6:46pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/10 "2020-06-19T18:46:14Z")
Ack, got it and agree that the distinction is likely helpful: the supervisor node implements the low-level "dirty work" of stopping processes and can ensure shutdown of its entire process tree if it really has to, but the client can carry out shutdown logic with a higher-level of insight into directing a shutdown process (possibly across many hosts) in orderly fashion. Also, based on "naming" feedback: plan to use `zeekc`. - Jon ---
**Author:** ![seth](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/seth/32/642_2.png) [@seth](https://community.zeek.org/u/seth)\ **Post date:** [June 30, 2020, 1:35pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/11 "2020-06-30T13:35:12Z")
Sorry for chiming in late on this... > Ack, got it and agree that the distinction is likely helpful: the > supervisor node implements the low-level "dirty work" of stopping > processes and can ensure shutdown of its entire process tree if it > really has to, but the client can carry out shutdown logic with a > higher-level of insight into directing a shutdown process (possibly > across many hosts) in orderly fashion. I think that the script we ship with zeek that effectively implements the supervisor behavior should understand the business logic of shutting down a cluster in the correct order. One way to think about it is that the supervisor script will presumably understand the business logic for starting a cluster in the right order so consequently it would seem that it should understand how to shut down the cluster as well. We talked about it recently and now that I've had some more time to think about it I'm really starting to think that the business logic for correctly starting and stopping a cluster should be fully implemented in the supervisor script. The zeekc tool could then just be a dumb tool that says to start and stop and doesn't end up causing us to spread our logic around to other tooling.    .Seth ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [June 30, 2020, 3:47pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/12 "2020-06-30T15:47:26Z")
How would that then work across multiple systems? Robin ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [June 30, 2020, 9:29pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/13 "2020-06-30T21:29:12Z")
Maybe the important observation is that the logic can be performed anywhere that has access to the Zeek-Supervisor process. \* The Supervisor process itself would be able to perform the logic via direct BIF access. \* External processes, like zeekc, have access to a Zeek-event interface to indirectly access those same BIFs, so they can also execute equivalent logic (either via multiple events, or a single "convenience" event that implements a sequence of BIF calls on remote) When we bring multi-hosting into the mix, it's still a similar situation, just with beefed up logic for orchestrating node-type-specific steps across many peers: anyone with access to the Zeek-event interface could implement this logic. You could pick zeekc to orchestrate, or you could pick a single Zeek-Supervisor process to orchestrate between other Supervisors, or you could pick a regular Zeek process, or you could write a Python script just using Broker Python bindings, etc. So where we put the logic at this point may not be important. If we can find a single-best-place for the logic to live, that's great, but if there's utility for others to have their own independent-yet-equivalent logic, I don't see a problem with that. - Jon ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [July 1, 2020, 9:02am UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/14 "2020-07-01T09:02:08Z")
> Maybe the important observation is that the logic can be performed > anywhere that has access to the Zeek-Supervisor process. Agree. > So where we put the logic at this point may not be important. If we > can find a single-best-place for the logic to live, that's great I believe that's what Seth is arguing for: have a Zeek-side script be the single point of that logic, rather than implement it multiple times and/or outside of Zeek. I can see doing that in Zeek but I think there's a trade-off here: if we want to do the singe-place approach with a multi-system setup, we'd need an authoritative place to run this logic and hence depend on \*that\* Zeek supervisor being up and running for performing the operation. That may be a reasonably assumption (say if we dedicated the supervisor running the manager to also be the cluster coordinator), but it's different from a world where the client can execute higher-level operations on its own. Robin ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:47pm UTC](https://community.zeek.org/t/zeek-supervisor-command-line-client/6158/15 "2022-05-06T15:47:19Z")