Bro stops logging to sqlite

Zeek
1

Hi Bro community,

Currently I have configured my Bro instance to send DNS logs to the sqlite database: /ust/local/bro/logs/current/dns.sqlite.I'm then reading these logs from a Python script and deleting the lines which were read. I'm facing the issue that Bro stops logging to the same sqlite file if the lines are deleted by my Python program.

Has someone faced similar issues in the past?

Thanks!

Kind regards,

Leonardo Mokarzel Falcon
@LMokarzel

2

Leonardo,

Yes, SQLite table locking is quite elementry. I have limited understanding of it but my impression is that when your Python program is making deletes its locking the table down and Bro cannot quite read it and BRO-SQLITE plugin gives up and terminates the connection.

YOu should see ERROR in reporter log similar to:

0.000000 Reporter::ERROR
/home/bro/<blhablbalhblah>/Log::WRITER_SQLITE: SQLite call failed: database table is locked: dns (empty)

You should be able to catch this reporter error in this event:

event reporter_error(t: time , msg: string , location: string )
{
        if (/WRITER_SQLITE/ in msg)
        {
                NOTICE([$note=WRITER_SQLITE_CRASH, $msg=msg]);
        }
}

And May be try to re-initialize the stream again. But that generally doesn't seem to work.

So second option is you might want to experiment with locking of SQLITE: http://www.sqlite.org/wal.html
  PRAGMA journal_mode=WAL;
        pragma synchronous=1;

and see if that helps.

Your Python program needs to not have contention with BRO writing basically.

I think using postgres is a better option if you have multiple read/writes going on since postgres does row level locks unlike SQLITE.

SQLITE DB is great if you have readonly or writeonly applications but again I have limited understanding here...

Hope this helps.
Aashish