I am working with the intel framework, using intel data from a file which is updated periodically. As far as I have seen in the documentation, it should be possible to update this file with new data and Zeek can adjust its behavior accordingly without restarting. The intel file must be loaded in mode=REREAD to achieve this.
However, I noticed that this works fine if new fields are added to the intel data file, but NOT if some fields are removed (for example if an ip address previously believed to be malicious is removed from the intel file because it was later realized to be safe). At the script level in the intel framework, intel data are stored into
global data_store: DataStore &redef;
and there are some functions for removing items from the record ( remove(item: Item, purge_indicator: bool), remove_meta_data(item: Item): bool ). But I am not sure they are really called anywhere.
Is anyone aware of this issue? Is it a work in progress? Thanks in advance.
However, I noticed that this works fine if new fields are added to the intel data file, but NOT if some fields are removed (for example if an ip address previously believed to be malicious is removed from the intel file because it was later realized to be safe). At the script level in the intel framework, intel data are stored into
global data_store: DataStore &redef;
and there are some functions for removing items from the record ( remove(item: Item, purge_indicator: bool), remove_meta_data(item: Item): bool ). But I am not sure they are really called anywhere.
that's correct. One has to keep in mind, that the files you provide to Zeek are just "input" for the data store. However, there are different approaches to remove Intel data from the internal data store.
1. You can use item expiration to expire items. If you update the intel file periodically, rereading items will reset the expiration timeout so that items removed from the file will eventually expire while the others are kept in the data store.
2. You could define which items to remove explicitly. Either by processing a file of items to delete, introducing a new meta data field indicating that an item should be removed or interfacing Zeek in any other way. This approach would require writing some custom script but shouldn't be too hard.