Network Packets#
Graphing network packets#
This notebook currently relies on HoloViews 1.9 or above. Run conda install holoviews to install it.
Preparing data#
The data source comes from a publicly available network forensics repository: http://www.netresec.com/?page=PcapFiles. The selected file is https://download.netresec.com/pcap/maccdc-2012/maccdc2012_00000.pcap.gz.
tcpdump -qns 0 -r maccdc2012_00000.pcap | grep tcp > maccdc2012_00000.txt
For example, here is a snapshot of the resulting output:
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.24.100.1038 > 192.168.202.68.8080: tcp 0
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.27.100.37877 > 192.168.204.45.41936: tcp 0
09:30:07.780000 IP 192.168.24.100.1038 > 192.168.202.68.8080: tcp 0
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
Given the directional nature of network traffic and the numerous ports per node, we will simplify the graph by treating traffic between nodes as undirected and ignorning the distinction between ports. The graph edges will have weights represented by the total number of bytes across both nodes in either direction.
python pcap_to_parquet.py maccdc2012_00000.txt
The resulting output will be two Parquet dataframes, maccdc2012_nodes.parq and maccdc2012_edges.parq.
NOTE: For your convenience this last step is captured in the anaconda-project run prepare_data command.
Loading data#
import holoviews as hv
from holoviews import opts, dim
import networkx as nx
import dask.dataframe as dd
from holoviews.operation.datashader import (
datashade, dynspread, directly_connect_edges, bundle_graph, stack
)
from holoviews.element.graphs import layout_nodes
from datashader.layout import random_layout
from colorcet import fire
hv.extension('bokeh')
keywords = dict(bgcolor='black', width=800, height=800, xaxis=None, yaxis=None)
opts.defaults(opts.Graph(**keywords), opts.Nodes(**keywords), opts.RGB(**keywords))
edges_df = dd.read_parquet('./data/maccdc2012_full_edges.parq').compute()
edges_df = edges_df.reset_index(drop=True)
graph = hv.Graph(edges_df)
len(edges_df)
21759
Edge bundling & layouts#
Datashader and HoloViews provide support for a number of different graph layouts including circular, force atlas and random layouts. Since large graphs with thousands of edges can become quite messy when plotted datashader also provides functionality to bundle the edges.
Circular layout#
By default the HoloViews Graph object lays out nodes using a circular layout. Once we have declared the Graph object we can simply apply the bundle_graph operation. We also overlay the datashaded graph with the nodes, letting us identify each node by hovering.
opts.defaults(opts.Nodes(size=5, padding=0.1))
circular = bundle_graph(graph)
datashade(circular, width=800, height=800, cmap=fire) * circular.nodes
Force Atlas 2 layout#
For other graph layouts you can use the layout_nodes operation supplying the datashader or NetworkX layout function. Here we will use the nx.spring_layout function based on the Fruchterman-Reingold algorithm. Instead of bundling the edges we may also use the directly_connect_edges function:
forceatlas = directly_connect_edges(layout_nodes(graph, layout=nx.spring_layout))
datashade(forceatlas, width=800, height=800, cmap=fire) * forceatlas.nodes
Random layout#
Datashader also provides a number of layout functions in case you don’t want to depend on NetworkX:
random = bundle_graph(layout_nodes(graph, layout=random_layout))
datashade(random, width=800, height=800, cmap=fire) * random.nodes
Showing nodes with active traffic#
To select just nodes with active traffic we will split the dataframe of bundled paths and then apply select on the new Graph to select just those edges with a weight of more than 10,000. By overlaying the sub-graph of high traffic edges we can take advantage of the interactive hover and tap features that bokeh provides while still revealing the full datashaded graph in the background.
overlay = datashade(circular, width=800, height=800, cmap=fire) * circular.select(weight=(10000, None))
overlay.opts(
opts.Graph(edge_line_color='white', edge_hover_line_color='blue', padding=0.1))
Highlight TCP and UDP traffic#
Using the same selection features we can highlight TCP and UDP connections separately again by overlaying it on top of the full datashaded graph. The edges can be revealed over the highlighted nodes and by setting an alpha level we can also reveal connections with both TCP (blue) and UDP (red) connections in purple.
udp_opts = opts.Graph(edge_hover_line_color='red', node_size=20,
node_fill_color='red', edge_selection_line_color='red')
tcp_opts = opts.Graph(edge_hover_line_color='blue',
node_fill_color='blue', edge_selection_line_color='blue')
udp = forceatlas.select(protocol='udp', weight=(10000, None)).opts(udp_opts)
tcp = forceatlas.select(protocol='icmp', weight=(10000, None)).opts(tcp_opts)
layout = datashade(forceatlas, width=800, height=800, normalization='log', cmap=['black', 'white']) * tcp * udp
layout.opts(
opts.Graph(edge_alpha=0, edge_hover_alpha=0.5, edge_nonselection_alpha=0, inspection_policy='edges',
node_size=8, node_alpha=0.5, edge_color=dim('weight')))
Coloring by protocol#
As we have already seen we can easily apply selection to the Graph objects. We can use this functionality to select by protocol, datashade the subgraph for each protocol and assign each a different color and finally stack the resulting datashaded layers:
ranges = dict(x_range=(-.5, 1.6), y_range=(-.5, 1.6), width=800, height=800)
protocols = [('tcp', "Blues"), ('udp', "Reds"), ('icmp', "Greens")]
shaded = hv.Overlay([datashade(forceatlas.select(protocol=p), cmap=cmap, **ranges)
for p, cmap in protocols]).collate()
stack(shaded * dynspread(datashade(forceatlas.nodes, cmap=['white'], **ranges)), link_inputs=True)
Selecting the highest targets#
With a bit of help from pandas we can also extract the twenty most targetted nodes and overlay them on top of the datashaded plot:
target_counts = list(edges_df.groupby('target').count().sort_values('weight').iloc[-20:].index.values)
overlay = (datashade(forceatlas, cmap=fire[128:]) *
datashade(forceatlas.nodes, cmap=['cyan']) *
forceatlas.nodes.select(index=target_counts))
overlay.opts( opts.Nodes(size=8), opts.RGB(width=800, height=800))