Do you have any other custom scripts loaded that are using sumstats?
With a dedicated logger process the manager doesn't really do anything other than sumstats.
Look in your cluster-layout.bro to see what port your manager process is assigned.. with 4 loggers I'd imagine it is around 47765/tcp
Then, run this command on the manager, on the interface that it talks to workers:
tcpdump -n -i em1 port 47765 -A | egrep -io '[A-Za-z_:-]{10,}'
That will output the names of the events that are bouncing between the workers and the manager
And see what you see.. It SHOULD be almost nothing, maybe a trickle of events.