I’d like to switch to writing both tab-delimited logs and JSON logs with my smaller bro cluster, but I would like the JSON logs to expire and get removed at a much shorter “LogExpireInterval” than my tab delimited logs. I see this is possible with the add-json package…
I’ve looked at both J-Gras’ add-json and Seth’s json-streaming-logs (both are great) but I’ve been looking more at add-json since it seems like it’s more along the lines of what I was thinking and I see I can set the rotation interval for the JSON writer by redefining the Log::default_rotation_interval option but I don’t see a way to extend add-json with a redef-able option for the log expire interval?
I also realize I could probably just script this with a shell script or python script to remove the archived JSON logs by leveraging the shorter rotation interval for JSON logs but I thought it would be nice to do right in the add-json package script.
Is a redef-able option for the log expire interval something that might be added in a future version of bro? Is there a way to do this now that I’m just missing? Is LogExpireInterval only available for broctl/broctl.cfg?
What you set with broctl is just the global filter. If you look at the json-streaming-logs package (link included below), you can see that I'm setting a custom rotation interval separately from the global default rotation interval. If you are looking to duplicate logging, you're going to be doing something similar to what json-streaming-logs is doing. I'm curious if json-streaming-logs doesn't do what you need to. It's possible that if what you need conceptually fits into that package I could just add it there.
From how you described your problem, it sounds like json-streaming-logs might already do what you need?
Thank you much Seth- in all honesty I probably didn’t dig into either package enough and just started exploring setting up JSON logging yesterday, certainly possible that I didn’t entirely understand what json-streaming-logs was doing/solving yet. I hadn’t tested either package, but just installed and enabled and have been testing your json-streaming-logs package.
I believe you may be correct that json-streaming-logs does most of what I’m wanting now that I look closer, however, what I’m still not sure about though is if there is a way to tell the JSON logs to auto “expire” and be removed off of disk (not just rotate) at a separate expire interval than the default tab delimited logs. So, for example- if I have a retention of say 15 days ( in broctl.cfg setting LogExpireInterval = 15) of archived logs for the default tab delimited logs. I want to be able to tell bro independently of the broctl.cfg global LogExpireInterval setting value that I want only all of my json_streaming_* logs to expire/be deleted/removed off of disk after say 1 day while the normal tab delimited logs still adhere to the 15 day archive retention.
In other words I do not want the JSON logs to eat up disk space since they will be getting shipped off box and my cold log retention on on box will be the archives of tab delimited logs.
I see you’re keeping iterations of the json_streaming versions of the logs around in the event a log shipper process or some process is still attached to the inode and that the creation of the .1, .2, json logs probably keys off the custom rotation interval (15 min) from what I can tell, which makes sense to me. Aside from that, in my testing I see that json_streaming logs are in fact being archived along with the default tab delimited logs so I’m assuming that as it stands now the json_streaming .gz log archives will stick around on disk just as long as my tab delimited archives unless I scripted something external of bro to remove them on a daily basis.
If this is all correct and I’m not missing anything else, I’m wondering if it would be possible for you to do something like I described above for removing the json_streaming_logs archives from disk more frequently with your package script? I think bro cron does this now? So not 100% certain how that may affect the plausibility of this, if at all.
The point here is that expiration of archived logs isn't done by bro but by broctl. Using add-json one thing that might work for you is to redef Log::path_json and write out your JSON logs into a different directory. For this you could setup a cron job or something to expire files using a different interval than you configured for the default logs.
Oh! That's a bug then. I was bad an never ended up running that script on a full cluster with Broctl, sorry about that. I'll do some more testing because that archiving was not the intent.
Jan I might try that in the meantime which is what I was originally thinking. Ideally I’d like the JSON logs to never get archived though then I don’t have to worry about maintaining another cleanup process/script.
Seth, thank you much for doing some more testing on a full cluster with broctl, if you can sort out the bug with the JSON logs getting archived when they were not intended to be I think I’ll be all set to deploy your json-streaming-logs : )
