how to make time-machine write data to disk

Time-machine
1

Hi Guys,

I’ve run time-machine in centos 7 platform, but found that no data written to disk

Below are my configuration and some logs, would you please give some suggestions?

[timemachine.cfg]

main {

The device to sniff.

device “ens160”;
workdir “/usr/local/var/tm”;
indexdir “indexes”;
queryfiledir “queries”;

logfile “timemachine.log”;
daemon 1;
console 0;
log_interval 60;
conn_timeout 180;
rmtconsole 1;
rmtconsole_listen_addr 127.0.0.1;
rmtconsole_port 42042;
bro_listen 1;
bro_listen_port 47757;
bro_listen_addr 127.0.0.1;

index “connection4” disk;
index “connection3” disk;
index “connection2” disk;
index “ip” disk;
}

class “all” {
filter “”;
precedence 10;
cutoff 10k;
disk 50g;
filesize 128m;
mem 1g;
}

[timemachine.log]

1528773477.257997 main: TimeMachine version 0.1-4
1528773477.265212 main: Forking Daemon
1528773487.213428 main: capture started, capture thread
1528773487.244189 main: Index aggregation thread started
1528773487.244268 main: WARNING: Broccoli support not compiled in.

[classes.timemachine.log]

timestamp class stored_bytes stored_pkts cut_bytes cut_pkts mem_bytes mem_pkts mem_dt disk_bytes disk_pkts disk_dt
1528773487.24 class_all 0 0 0 0 0 0 0.00 0 0 0.00
1528774087.24 class_all 70733 530 250414 1556 70733 530 598.83 0 0 0.00
1528774687.25 class_all 121598 875 385579 2253 121598 875 1198.74 0 0 0.00

[indexes.timemachine.log]

timestamp index entries_mem qlen
1528773487.24 connection4 0 0
1528773487.24 connection3 0 0
1528773487.24 connection2 0 0
1528773487.24 ip 0 0
1528774087.24 connection4 61 10
1528774087.24 connection3 124 2
1528774087.24 connection2 40 10
1528774087.24 ip 42 4

[stats.timemachine.log]

timestamp conns dyn_class pcap_recv pcap_drop drop_ratio d_drop totPkts totBytes link_rate_mbit uncutPkts uncutBytes uncut_rate_mbit utime stime mem_os mem queryPkts query
Bytes
1528773487.24 0 0 1 0 0.00% 0 1 60 0.00 0 0 0.00 0.0 0.0 1158610944 103200 0 0
1528774087.24 37 0 7226 0 0.00% 0 7226 629295 0.01 530 70733 0.00 0.0 0.0 1159421952 174816 0 0
1528774687.25 21 0 14608 0 0.00% 0 14608 1195527 0.01 875 121598 0.00 0.0 0.0 1159557120 197488 0 0

Thanks
Best Regards,
Leaf

2

Leaf,

class "all" { filter ""; precedence 10; cutoff 10k; disk 50g; filesize 128m; mem 1g;}

You specified mem 1g;

Which basically means that TM which cache 1g of pcaps in memory before start to
write on the disk.

This functionality was there to help bro talk to time-machine and do a speedy
extraction for any notice/alert of interest.

If you run TM standalone (ie without bro integration), I'd advice put mem 1k or
so, and as soon as 1k buffer is filled, TM starts putting pcaps on disk.

Hope this helps,
Aashish

3

Hi Aashish,

That’s great! It works well under 1k-mem setting.
Thanks for your kindly help.

Best Regards,
Leaf

Leaf,

class “all” { filter “”; precedence 10; cutoff 10k; disk 50g; filesize 128m; mem 1g;}

You specified mem 1g;

Which basically means that TM which cache 1g of pcaps in memory before start to
write on the disk.

This functionality was there to help bro talk to time-machine and do a speedy
extraction for any notice/alert of interest.

If you run TM standalone (ie without bro integration), I’d advice put mem 1k or
so, and as soon as 1k buffer is filled, TM starts putting pcaps on disk.

Hope this helps,
Aashish

4

Hello Leaf,

Couple of other notes:

1) I see you've enabled indexes. Are you using them ? If not, I'd advice
disable indexes. These are implemented as AVL trees and we've noticed
packets dropped when these indexes are rebalanced.

If you are using indexes then only enable.

2) What branch of TM are you using ? We had a intern (Naoki Eto) work on IPv6
integration and he also did a huge performance improvements.

if you use topic/aashish/ipv6 you'd see much better results - this is
pretty much Naoki's branch with one 2 line change I made for vlan tag
stripping.

3) We (LBNL) has a couple of scripts to (i) extract TM logs if you
provide bro logs to the script (ii) create a 'human' timeline <-> class
files for some 'manual' or bot-controlled searches.

Both are to avoid problem of (1) we've disabled indexes. I don't quite
recall but removing indexes improves performance by 20+% - old data
point though.

4) we also have a concept of fat and shallow TMs.

Let us know if you have any questions/exchange tools technology etc.

Thanks,
Aashish