Spicy script help for packet analyser #3505

IamTenacious · December 14, 2023, 9:59am

Please refer discussion spicy script help for packer analyser · zeek/zeek · Discussion #3220 · GitHub

Hi @Benjamin_Bannier, @biswa61 , I am totally new to the zeek world and so to spicy.
I have same requirement, I want to enable this analyser for Goose (IEC61850) actually mine is 802.1Q Virtual LAN (0*8100) , so I have few questions here kindly try to address them
1.Is there zeek version dependency for goose analyser?
2.If no then do we actually create new events using spicy or we use the existing events of zeek and write our own code / add our own functionalities on top of it?
3.and if Yes(1) then I am currently using zeek 5.0.3 version and SO 2.3.170 will it be possible to add the GOOSE analyser?
4.How can I enable the goose parser/spicy parser with the above specifications?
5.Whats the spicy analyser writing process?
I know I have asked a lot of questions here but please kindly try to address them ASAP.

This is the structure of my goose which is obviously same as in the above picture.

Frame 557: 259 bytes on wire (2072 bits), 259 bytes captured (2072 bits)
Ethernet II, Src: PcsCompu_01:5f:e5 (08:00:27:01:5f:e5), Dst: Iec-Tc57_01:00:01 (01:0c:cd:01:00:01)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 0
GOOSE
APPID: 0x03e8 (1000)
Length: 241
Reserved 1: 0x0000 (0)
Reserved 2: 0x0000 (0)
goosePdu
gocbRef: simpleIOGenericIO/LLN0$GO$gcbAnalogValues
timeAllowedtoLive: 500
datSet: simpleIOGenericIO/LLN0$AnalogValues
goID: simpleIOGenericIO/LLN0$GO$gcbAnalogValues
t: Aug 29, 2023 06:36:41.607999980 UTC
stNum: 1
sqNum: 0
test: False
confRev: 1
ndsCom: False
numDatSetEntries: 5
allData: 5 items

IamTenacious · December 15, 2023, 8:19am

I was able to register the goose spicy parser , following are my files and I am getting one error I am not sure what the issue is

main.zeek

@load base/misc/version

module spicy_GOOSE;
global goose_topic = "/topic/goose";

global begin_time: time;
global total_time: interval;

export {
        ## Log stream identifier.
        redef enum Log::ID += { spicy_GOOSE_LOG };

        ## Record type containing the column fields of the goose log.
        type Info: record {
                ## Timestamp for when the activity happened.
                ts: time &log &default=network_time();
                appid: count &log &optional;
                pkt_len: count &log &optional;
        };

        #global GOOSE::message: event(pkt: raw_pkt_hdr, appid: count, pkt_len: count);

        #global analyzer_confirmation: event(atype: AllAnalyzers::Tag, info: AnalyzerConfirmationInfo);

        global spicy_GOOSE::log_goose: event(rec: spicy_GOOSE::Info);

        global log_GOOSE: event(rec: Info);
}

redef record raw_pkt_hdr  += {
        spicy_GOOSE: Info &optional;
};


event zeek_init() &priority=20
	{
	suspend_processing();
	# TODO: Our example here models a custom protocol sitting between
	# Ethernet and IP. The following sets that up, using a custom ether
	# type 0x88b5. Adapt as suitable, some suggestions in comments.
	local analyzer = PacketAnalyzer::ANALYZER_SPICY_GOOSE;

	# Activate our analyzer on top of Ethernet.
	#PacketAnalyzer::register_packet_analyzer(PacketAnalyzer::ANALYZER_ETHERNET, 0x88b8, analyzer);
	if ( ! PacketAnalyzer::try_register_packet_analyzer_by_name("Ethernet", 0x88b8,"spicy_GOOSE") )
		print "cannot register GOOSE Spicy analyzer";

	# Activate IP on top of our analyzer. 0x4950 is our own protocol's
	# magic number indicating that IP comes next.
	#PacketAnalyzer::register_packet_analyzer(analyzer, 0x4950, PacketAnalyzer::ANALYZER_IP);

	# Alternative: Use this if your analyzer parses a link layer protocol directly.
	# const DLT_spicy_GOOSE : count = 12345;
	# PacketAnalyzer::register_packet_analyzer(PacketAnalyzer::ANALYZER_ROOT, DLT_spicy_GOOSE, analyzer);

	# Alternative: Use this if your analyzer parses a protocol running on top of
	# IPv4, using the specified IP protocol number.
	# PacketAnalyzer::register_packet_analyzer(PacketAnalyzer::ANALYZER_IP, 0xcafe, analyzer);

	# Alternative: Use this if you want your analyzer to run on top of UDP, activated on the specified well-known port.
	# const ports: set[port] = { 6789/udp } &redef;
	# PacketAnalyzer::register_for_ports(PacketAnalyzer::ANALYZER_UDP, analyzer, ports);
	#Log::create_stream(spicy_GOOSE::GOOSE_LOG, [$columns=Info, $ev=log_goose, $path="goose"]);
	}

#print this event per packet
event spicy_GOOSE::message(packet: raw_pkt_hdr, appid: count, pkt_len: count)
{
        local info: Info = [$ts=network_time(), $appid=appid, $pkt_len=pkt_len];
        print "Processing pcakets", packet;
       # Log::write(spicy_GOOSE::GOOSE_LOG, info);
}

#Example event defined in spicy_goose.evt.
event spicy_GOOSE::packet(packet: raw_pkt_hdr, appid: count, pkt_len: count)
	{
	# TODO: Consider just deleting this event handler if you don't need it.
	# For most packet analyzers, it's best to not do any script-level work
	# because the overhead could quickly become overwhelming.
	local info: Info = [$ts=network_time(), $appid=appid, $pkt_len=pkt_len];
        print "Processing pcakets", packet;
        #Log::write(spicy_GOOSE::GOOSE_LOG, info);
	}

spicy_goose.evt

import spicy_GOOSE;
import Zeek_spicy_GOOSE;

packet analyzer spicy_GOOSE:
    parse with spicy_GOOSE::GOOSEPacket;

#TODO: Connect Spicy-side events with Zeek-side events. The example just
#defines a simple example event that forwards the raw data (which in practice you
#don't want to do!). In fact, you should consider just deleting this event if
#you don't need it: For most packet analyzers, it's best to not do any
#script-level work because the overhead could quickly become overwhelming.
on spicy_GOOSE::GOOSEPacket -> event spicy_GOOSE::packet($packet, self.appid, self.pkt_len);

spicy_goose.spicy

#TODO: Define your analyzer here.

module spicy_GOOSE;

import zeek;

#TODO: Our example here models a simple example packet format of static size:
#19 payload bytes, followed by the protocol number for the next layer, for
#which the data then follows subsequently. (This is just what our test trace
#happens to contain). Adapt as suitable.
public type GOOSEPacket = unit {
    appid: uint8;
    pkt_len: uint16;
    payload: bytes &eod;

    on %done {
        # Feed into Zeek's next-layer packet analysis.
        zeek::forward_packet(self.protocol);
    }
};

zeek_spicy_goose.spicy

#Zeek-specific Spicy logic.

module Zeek_spicy_GOOSE;

import spicy_GOOSE;
import zeek;

#TODO: Add anything you need here.

whats wrong in here I am getting following error

$ /usr/local/zeek/bin/zeek -C main.zeek /home/ubuntu/Desktop/GOOSE.pcap 
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\xd4'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\xc3'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\xb2'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\xa1'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\x02'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\x04'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\xff'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\xff'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: '\x01'
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unrecognized character: ''
error in /home/ubuntu/Desktop/GOOSE.pcap, line 1: unknown identifier K, at or near "K"

Benjamin_Bannier · December 15, 2023, 8:36am

To read a PCAP you need to pass it with -r, e.g.,

$ /usr/local/zeek/bin/zeek -C main.zeek -r /home/ubuntu/Desktop/GOOSE.pcap

IamTenacious · December 15, 2023, 8:51am

Thanks, @Benjamin_Bannier
I tried it, now getting following error

/root@ubuntu-VirtualBox:/home/ubuntu/Desktop/zeek-spicy-goose/scripts# /usr/local/zeek/bin/zeek -C main.zeek -r /home/ubuntu/Desktop/GOOSE.pcap
in zeek_init
processing suspended
^Z
[13]+  Stopped                 /usr/local/zeek/bin/zeek -C main.zeek -r /home/ubuntu/Desktop/GOOSE.pcap
root@ubuntu-VirtualBox:/home/ubuntu/Desktop/zeek-spicy-goose/scripts# cat weird.log 
#separator \x09
#set_separator	,
#empty_field	(empty)
#unset_field	-
#path	weird
#open	2023-12-15-15-33-28
#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer	source
#types	time	string	addr	port	addr	port	string	string	bool	string	string
1216135243.118812	-	-	-	-	-	packet_error	Event parameter mismatch, expected 3 parameters, but got 2	F	zeek	-
root@ubuntu-VirtualBox:/home/ubuntu/Desktop/zeek-spicy-goose/scripts#

first I was getting type mismatch error so I have changed the type of both the variables as String

in main.zeek

@load base/misc/version

module spicy_GOOSE;
global goose_topic = "/topic/goose";

global begin_time: time;
global total_time: interval;
#Log::create_stream(spicy_GOOSE::GOOSE_LOG, [$columns=Info, $ev=log_goose, $path="goose"]);

export {
        ## Log stream identifier.
        redef enum Log::ID += { spicy_GOOSE_LOG };

        ## Record type containing the column fields of the goose log.
        type Info: record {
                ## Timestamp for when the activity happened.
                ts: time &log &default=network_time();
                appid: string &log &optional;
                pkt_len: string &log &optional;
        };

        #global GOOSE::message: event(pkt: raw_pkt_hdr, appid: count, pkt_len: count);

        #global analyzer_confirmation: event(atype: AllAnalyzers::Tag, info: AnalyzerConfirmationInfo);
        
        global spicy_GOOSE::packet: event(pkt: raw_pkt_hdr, appid: string, pkt_len: string);

        global spicy_GOOSE::log_goose: event(rec: spicy_GOOSE::Info);

        global log_GOOSE: event(rec: Info);
}

redef record raw_pkt_hdr  += {
        spicy_GOOSE: Info &optional;
};


event zeek_init() &priority=20
	{
	print "in zeek_init";
	suspend_processing();
	#TODO: Our example here models a custom protocol sitting between
	#Ethernet and IP. The following sets that up, using a custom ether
	#type 0x88b5. Adapt as suitable, some suggestions in comments.
	local analyzer = PacketAnalyzer::ANALYZER_SPICY_GOOSE;

	#Activate our analyzer on top of Ethernet.
	PacketAnalyzer::register_packet_analyzer(PacketAnalyzer::ANALYZER_ETHERNET, 0x88b8, analyzer);
	#if ( ! PacketAnalyzer::try_register_packet_analyzer_by_name("Ethernet", 0x88b8,"spicy_GOOSE") )
	#print "cannot register GOOSE Spicy analyzer";

	#Activate IP on top of our analyzer. 0x4950 is our own protocol's
	#magic number indicating that IP comes next.
	#PacketAnalyzer::register_packet_analyzer(analyzer, 0x4950, PacketAnalyzer::ANALYZER_IP);

	#Alternative: Use this if your analyzer parses a link layer protocol directly.
	#const DLT_spicy_GOOSE : count = 12345;
	#PacketAnalyzer::register_packet_analyzer(PacketAnalyzer::ANALYZER_ROOT, DLT_spicy_GOOSE, analyzer);

	#Alternative: Use this if your analyzer parses a protocol running on top of
	#IPv4, using the specified IP protocol number.
	#PacketAnalyzer::register_packet_analyzer(PacketAnalyzer::ANALYZER_IP, 0xcafe, analyzer);

	#Alternative: Use this if you want your analyzer to run on top of UDP, activated on the specified well-known port.
	#const ports: set[port] = { 6789/udp } &redef;
	#PacketAnalyzer::register_for_ports(PacketAnalyzer::ANALYZER_UDP, analyzer, ports);
	#Log::create_stream(spicy_GOOSE::GOOSE_LOG, [$columns=Info, $ev=log_goose, $path="goose"]);
	}

#print this event per packet
#event spicy_GOOSE::message(packet: raw_pkt_hdr, appid: string, pkt_len: string)
#{
#local info: Info = [$ts=network_time(), $appid=appid, $pkt_len=pkt_len];
#print "Processing packets", packet;
#Log::write(spicy_GOOSE::GOOSE_LOG, info);
#}

#Example event defined in spicy_goose.evt.
event spicy_GOOSE::packet(packet: raw_pkt_hdr, appid: string, pkt_len: string)
	{
	print "in packet";
	#TODO: Consider just deleting this event handler if you don't need it.
	#For most packet analyzers, it's best to not do any script-level work
	#because the overhead could quickly become overwhelming.
	print "Processing packets", packet;
	local info: Info = [$ts=network_time(), $appid=appid, $pkt_len=pkt_len];
        
        #Log::write(spicy_GOOSE::GOOSE_LOG, info, additional_parameter1, additional_parameter2);

	}

Christian · December 16, 2023, 1:12am

Hi there,

It would be more helpful if you could point us at a repository that has the whole analyzer than posting all those files, since you’d give us something reproducible to work with. From skimming I doubt that you are really running/building those files. A few quick tips:

Remove suspend_processing() — not sure why you put that there.
zeek::forward_packet(self.protocol) cannot work since you don’t have a protocol field in the GOOSEPacket unit.
count is the correct script-level type for the appid and pkt_len fields. You can’t just change types to string — there’s no implicit type conversion in Zeek, so the following was the correct event signature:
```
event spicy_GOOSE::packet(packet: raw_pkt_hdr, appid: count, pkt_len: count)
```
For script errors and when running Zeek directly on a pcap you should have more detailed output as to what’s wrong on stderr; no need to dig into logs.
When you run things repeatedly and look at log files, make sure you’re not accidentally looking at stale logs from previous invocations.

I think you are pretty close otherwise.

Best,
Christian

IamTenacious · December 16, 2023, 9:12am

Thank you so much @Christian , I have used all your suggestions.

I have created goose parser like this:
/usr/local/zeek/bin/zkg create --features spicy-packet-analyzer --packagedir zeek-spicy-goose
package name - GOOSE
analyzer name- spicy_GOOSE
parsing unit -File
/usr/local/zeek/bin/zkg install zeek-spicy-goose/
/usr/local/zeek/bin/zeek -NN | grep GOOSE

and this generated the repo with some default coding:

There I have added my code/modified the scripts.
Though I did the changes you mentioned I am getting the same error of conversion.
**cat weird.log **
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path weird
#open 2023-12-16-14-27-26
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer source
#types time string addr port addr port string string bool string string
1216135243.118812 - - - - - packet_error Event parameter mismatch, cannot convert Spicy value of type ‘string’ to Zeek value of type ‘count’ F zeek -
#close 2023-12-16-14-27-26

as I mentioned I am totally new to zeek , so can you please tell me what’s I am doing wrong also how can I execute and debug these parser.

Let me quickly tell you what my expectations are from this analyzer:
1.I want to parse the GOOSE traffic for now I am using PCAP.
2.My GOOSE Ethernet version is different from the version mentioned in the reference link I have provided.
3.For now I want to see the GOOSE traffic processed and the processing logs.
4.Once point 1 to 3 is done I wanna make the parser work on live traffic.

Christian · December 16, 2023, 9:35am

I have used all your suggestions.

Almost.

Please put your parser in a git repo somewhere so we can see the exact code and suggest diffs. Otherwise this involves too much guesswork as to what you’re running and margin for error.

Thanks!
Christian

IamTenacious · December 16, 2023, 9:41am

Thanks for the quick reply @Christian
I think so too, I have added all the parser files in my repo:

IamTenacious · December 18, 2023, 4:58am

Hi @Christian , @Benjamin_Bannier I have also added PCAP for reference.
This might help in better understanding.

Thanks.

Christian · December 18, 2023, 11:56am

Thanks! Take a look at GitHub - ckreibich/goose_parser. I took the subdirectory of your repo because a Zeek package needs to have zkg.meta in its toplevel directory. I then applied a few fixes, all minor.

I’m using Zeek 6.1 and the package now installs fine via zkg install. The tests run through, verifying that the parser triggers spicy_GOOSE::packet events.

I did not use your pcap since I’m not sure what’s in it.

Best,
Christian

IamTenacious · December 19, 2023, 5:33am

Thanks @Christian , I have applied same changes but still I am getting same type conversion error.
but my zeek version is 5.0.3 , could that be an issue?

Topic		Replies	Views
New spicy protocol with zeek broker Development development , spicy	3	356	August 3, 2023
Spicy packet analyzer Spicy	0	225	December 24, 2023
BGP/RSVP/OSPF Zeek	7	267	May 6, 2022
Packet processing doubt in spicy parser and zeek broker Zeek development , spicy	6	450	January 6, 2024
Analysing PCAPNG files from Wireshark traffic captured with Zeek or Spicy Zeek	6	1030	April 11, 2023

Spicy script help for packet analyser #3505

Related Topics