[Timemachine]cannot query previous data after timemachine restarting?

Time-machine
1

Dear Team,

Sorry for bothering you:)

I’ve launched timemachine in my server for one week and it worked well.

But yesterday after my server(timemachine) rebooting, I found the data captured before server rebooting cannot be queried anymore.

I tried many queries with parameters “IP” or “Connection”, the result always only contain the data captured from the time server/timemachine restarting.

Could anyone tell if timemachine have this limitation? Or how to work around?

Thanks

BR,

Leaf

2

Leaf,

Greetings!

[ Sorry for the delayed response ]

What you bring is a very well know issue with time-machine. Indexes aren't
persistent. So once you restart TM you'd loose the query capabilities. Sadly! My
understanding is that indexes are implemented as AVL trees. Those are kept in
memory and on the disk, but to be able re-read those after a new start is
tricky.

I know (I think) that Scott Sakai from SDSC has used some postgres/sqlite
backend to store indexes. Its specific to Scott's setup so never pushed upstream
to git repo.

We/LBNL just use workarounds:

1) A virtual timestamp based directory structure called TMquick:

Here it how it looks (basically a human readable timestamps appended to buckets.

[/TMquick/TODAY]$ ls

all-00:12:58 all-02:14:05 all-05:47:02 all-08:34:39 all-10:18:53 all-11:49:10 dns-03:04:48 dns-04:18:14 dns-06:30:58 dns-09:00:20 dns-10:36:25 dns-12:06:44 smtp-03:10:51 smtp-08:07:32 smtp-09:38:38 smtp-11:11:10 ssh-02:01:37

2) We have a tm-extract.sh script which if you give a bro log entry (conn, dns,
http, smtp etc), will go and find the right TM-bucket and extract the pcaps.

(1) is used by humans (2) is used by robot-controlled scripts.

For (1) If interested, I can share the TMquick scripts (Partha has ownership of those)

For (2) see extract-tm.sh here:

    https://github.com/initconf/timemachine-conf-scripts.git

Let me know if you have followup questions.

Aashish

ps - if you are not already use topic/aashish/ipv6 branch - its stable, has IPv6
support and good - Its Naoki Eto's branch with some little modifications related
to vlan tag stripping and FreeBSD 11 compilations.

Aashish