# broctl process tracking problems

**URL:** <https://community.zeek.org/t/broctl-process-tracking-problems/2230>\
**Category:** Development\
**Tags:** development\
**Created:** [February 17, 2012, 5:00pm UTC](https://community.zeek.org/t/broctl-process-tracking-problems/2230 "2012-02-17T17:00:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\
**Post date:** [February 17, 2012, 5:00pm UTC](https://community.zeek.org/t/broctl-process-tracking-problems/2230/1 "2012-02-17T17:00:28Z")

</div>

Has anyone else ever had trouble with broctl getting confused about the status of a process? I just ran into it a little bit ago where broctl thought that all of my workers were dead when I tried to do a restart command. They failed when they were trying to start again because with the myricom sniffer drivers you can only sniff the interface once.

We need to do some debugging on this, but it happens sporadically enough that it might be tough. I sort of wonder if there are issues with broctl.dat being written, I've run into problems in that file before where things wouldn't be written right. Would it make sense to maybe even move away from broctl.dat (which tracks cluster state) and toward something like an SQLite database?

&nbsp;&nbsp;.Seth

---

<div class="post-metadata">

**Author:** ![Aashish\_Sharma1](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Aashish\_Sharma1](https://community.zeek.org/u/Aashish_Sharma1)\
**Post date:** [February 17, 2012, 6:08pm UTC](https://community.zeek.org/t/broctl-process-tracking-problems/2230/2 "2012-02-17T18:08:07Z")

</div>

Yes. Incidently, I had same issue 3 days back. broctl analysis scan  
showed scan was disabled but cluster was still dropping host on scan.

So I started looking at broctl status which said cluster not running  
while tail -f conn.log was growing.

Ended up kill -s 9 on all bro worker nodes and restart with broctl.  
After which it has been fine. I was quite unusual.

Aashish

---

<div class="post-metadata">

**Author:** ![Will](https://avatars.discourse-cdn.com/v4/letter/w/90ced4/32.png) [@Will](https://community.zeek.org/u/Will)\
**Post date:** [February 17, 2012, 6:23pm UTC](https://community.zeek.org/t/broctl-process-tracking-problems/2230/3 "2012-02-17T18:23:35Z")

</div>

I had the same issue a time or two. Running ‘broctl ps.bro’ right after ‘broctl status’ has become part of my new ritual before stopping/starting or just restarting any of my clusters.

Will

---

<div class="post-metadata">

**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\
**Post date:** [February 17, 2012, 6:53pm UTC](https://community.zeek.org/t/broctl-process-tracking-problems/2230/4 "2012-02-17T18:53:41Z")

</div>

That's actually perfect! Sometime could you make a copy of your spool/broctl.dat before you restart?

I'd like two copies of that file. One before a restart and one after the restart (assuming the restart fails). It could point out if there is corruption entering the broctl.dat file at some point. This problem drives me crazy and I'd love to fix it.

&nbsp;&nbsp;.Seth

---

<div class="post-metadata">

**Author:** ![Aashish\_Sharma1](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Aashish\_Sharma1](https://community.zeek.org/u/Aashish_Sharma1)\
**Post date:** [February 17, 2012, 7:02pm UTC](https://community.zeek.org/t/broctl-process-tracking-problems/2230/5 "2012-02-17T19:02:23Z")

</div>

> \> I had the same issue a time or two. Running 'broctl ps.bro' right after 'broctl status' has become part of my new ritual before stopping/starting or just restarting any of my clusters.

Yes, but there could be other situations you can probably look for when  
the bro process is running:

- Your network interfaces may not see the data  
- Another possibility is that data is coming on the interfaces and bro  
&nbsp;&nbsp;is running but not processing the incoming data (hopefully won't happen)

so you might want to also check if the logs are growing regularly.

Additional complexities:

Your logs may be growing but above situation happens only on 1  
worker-node ( in which case you'd see manager logs growing but won't  
know that one node is consistently missing logs)

.... and so on for various other failures (I must say these are rare but  
on a production system you have to account for them and over period of  
time this has happened)

We have a cron script which watches for these conditions.

Aashish

---

<div class="post-metadata">

**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\
**Post date:** [February 20, 2012, 5:16pm UTC](https://community.zeek.org/t/broctl-process-tracking-problems/2230/6 "2012-02-20T17:16:13Z")

</div>

For the record, coincidentally I saw the same once early last week.

Robin

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\
**Post date:** [May 6, 2022, 3:40pm UTC](https://community.zeek.org/t/broctl-process-tracking-problems/2230/7 "2022-05-06T15:40:12Z")

</div>


