question: memory requirements of tables

Zeek
1

hi

sorry, but i have already another question.

i noticed that my bro (version 0.9a8.17) eats up my memory.
even with the reduce-memory.bro script it needed up to 2GB
or more after a while and swapped all the time.
when we reduced the traffic the memory usage decreased only
a little bit, but remained very high.

because we use &*_expire tags and scheduler to keep tables
small, we tried to reproduce our problem with a small script:

### memory_test.bro

# check all 10 secs if table entries are expired
redef table_expire_interval = 10 secs;

# When expiring, don't check more than this many table entries at a time.
redef table_expire_size = 10000;

global test_table: table[addr] of count &create_expire=30secs;

event print_table () {
        print "size of test_table in bytes", val_size(test_table);
        print "length of test_table", length(test_table);
        print "test_table", test_table;
        schedule 1 min { print_table() };
}

event bro_init () {
        schedule 1 min { print_table() };
}

event udp_request(u: connection) {
    test_table[u$id$resp_h] = 0;
}

the following is the output of our script when doing some udp scanning.
this logs shows that all entries in the table have been deleted, but the
table uses still a lot of memory! what happened.

we tried different constellations and noticed that it doesn't matter
if the table entries have been deleted with the "delete" function or
the expire tag. we also tested it over a longer period (let it running
through the night) but the memory will not be deallocated.

size of test_table in bytes, 304
length of test_table, 0
test_table, {

}
size of test_table in bytes, 760
length of test_table, 3
test_table, {
[x.x.228.20] = 0,
[x.x.229.229] = 0,
[x.x.229.5] = 0
}
size of test_table in bytes, 768
length of test_table, 1
test_table, {
[x.x.229.229] = 0
}
size of test_table in bytes, 840
length of test_table, 1
test_table, {
[x.x.229.229] = 0
}
size of test_table in bytes, 1675832
length of test_table, 12545
test_table, {
[93.172.138.23] = 0,
[60.114.97.75] = 0,
[4.103.6.3] = 0,
[51.164.85.16] = 0,
[85.83.252.215] = 0,
...
}
size of test_table in bytes, 4285296
length of test_table, 26532
test_table, {
[114.68.42.217] = 0,
[4.48.74.147] = 0,
[58.127.37.141] = 0,
...
}
size of test_table in bytes, 1526280
length of test_table, 3
test_table, {
[x.x.229.229] = 0,
[x.x.228.20] = 0,
[x.x.229.5] = 0
}
size of test_table in bytes, 1526384
length of test_table, 4
test_table, {
[x.x.201.77] = 0,
[x.x.201.145] = 0,
[x.x.201.49] = 0,
[x.x.201.52] = 0
}
size of test_table in bytes, 1526072
length of test_table, 1
test_table, {
[x.x.201.70] = 0
}
size of test_table in bytes, 1525968
length of test_table, 0
test_table, {

}
size of test_table in bytes, 1526072
length of test_table, 1
test_table, {
[x.x.229.229] = 0
}
size of test_table in bytes, 1526072
length of test_table, 1
test_table, {
[x.x.229.229] = 0
}

cheers
christoph

2

Hi Christoph,

i noticed that my bro (version 0.9a8.17) eats up my memory.
even with the reduce-memory.bro script it needed up to 2GB
or more after a while and swapped all the time.
when we reduced the traffic the memory usage decreased only
a little bit, but remained very high.

Have you verified that the memory is indeed consumed by your scripts?
Just load the profiling policy script. It outputs some very useful
resource consumption logs. The only problem: you do not want to run it
online "in operative mode" since it causes bro to use lots of CPU
every time it generates its statistics. But anyway, it works for
debugging purposes.

because we use &*_expire tags and scheduler to keep tables
small, we tried to reproduce our problem with a small script:
...
the following is the output of our script when doing some udp scanning.
this logs shows that all entries in the table have been deleted, but the
table uses still a lot of memory! what happened.

mmmh, I'm really not sure whether what you see in your script is
really your problem of running out of memory. The table first grows
but then again shrinks (0->4M->1.5M). I think that suggests that there
is not really a larger problem with table expiration.

Hope this helps,
  Holger

3

I put this back onto the mailing list, since I think we found again an
interesting memory issue...

i have no clue what most of the output of the profiling script means,
but this is interesting:

the test_table really seems to decrease its memory usage massively.
but why not proportionally to the containing entries?
when i have a lot of tables it is really a problem, that each table
consumes 3 MB even if there are (almost) no entries in it anymore...

we just quickly checked the code: tables are only resized when they
grow beyond a certain threshold. They are not resized when enough
elements are deleted. So far this was not a problem since there are
usually a few tables that grow large (and usually remain large). You
run into problems as soon as you have tables that grow and shrink only
once in their lifetime.

your right, there is a second problem:
even if the tables decrease its memory consumption massively bro does not
seem to deallocate this memory space.

have a look at this log file from the profiling script:

.....
> 1108468702.534722 Memory: total=28449K total_adj=27700K malloced: 4842K
.....

This line from the profiling.log file indicates that the system
accounts 28M memory allocated by the bro process (field total). On
the other hand the malloc library claims bro actually uses only 5M
(field malloced). The difference is memory that is not (yet) returned
to the system (by the malloc lib!) but will be reused as soon as you
reallocate memory. On some (all?) systems you can somehow configure
the malloc lib to return freed memory to the system.

  Holger