Network Packets#
Graphing network packets#
This notebook currently relies on HoloViews 1.9 or above. Run conda install holoviews to install it.
Preparing data#
The data source comes from a publicly available network forensics repository: http://www.netresec.com/?page=PcapFiles. The selected file is https://download.netresec.com/pcap/maccdc-2012/maccdc2012_00000.pcap.gz.
tcpdump -qns 0 -r maccdc2012_00000.pcap | grep tcp > maccdc2012_00000.txt
For example, here is a snapshot of the resulting output:
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.24.100.1038 > 192.168.202.68.8080: tcp 0
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.27.100.37877 > 192.168.204.45.41936: tcp 0
09:30:07.780000 IP 192.168.24.100.1038 > 192.168.202.68.8080: tcp 0
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
Given the directional nature of network traffic and the numerous ports per node, we will simplify the graph by treating traffic between nodes as undirected and ignorning the distinction between ports. The graph edges will have weights represented by the total number of bytes across both nodes in either direction.
python pcap_to_parquet.py maccdc2012_00000.txt
The resulting output will be two Parquet dataframes, maccdc2012_nodes.parq and maccdc2012_edges.parq.
NOTE: For your convenience this last step is captured in the anaconda-project run prepare_data command.
Loading data#
import holoviews as hv
from holoviews import opts, dim
import networkx as nx
import dask.dataframe as dd
from holoviews.operation.datashader import (
datashade, dynspread, directly_connect_edges, bundle_graph, stack
)
from holoviews.element.graphs import layout_nodes
from datashader.layout import random_layout
from colorcet import fire
hv.extension('bokeh')
keywords = dict(bgcolor='black', width=800, height=800, xaxis=None, yaxis=None)
opts.defaults(opts.Graph(**keywords), opts.Nodes(**keywords), opts.RGB(**keywords))
edges_df = dd.read_parquet('./data/maccdc2012_full_edges.parq').compute()
edges_df = edges_df.reset_index(drop=True)
graph = hv.Graph(edges_df)
len(edges_df)
21759
Edge bundling & layouts#
Datashader and HoloViews provide support for a number of different graph layouts including circular, force atlas and random layouts. Since large graphs with thousands of edges can become quite messy when plotted datashader also provides functionality to bundle the edges.
Circular layout#
By default the HoloViews Graph object lays out nodes using a circular layout. Once we have declared the Graph object we can simply apply the bundle_graph operation. We also overlay the datashaded graph with the nodes, letting us identify each node by hovering.
opts.defaults(opts.Nodes(size=5, padding=0.1))
circular = bundle_graph(graph)
datashade(circular, width=800, height=800, cmap=fire) * circular.nodes