The Role of Network Analyzers in AES67 Troubleshooting

AES67 is an Audio-over-IP (AoIP) interoperability standard that enables professional audio devices from different manufacturers to exchange high-quality, low-latency audio streams over standard IP networks. Its reliance on RTP (Real-time Transport Protocol), multicast addressing, and precise timing via PTP (Precision Time Protocol) makes it extremely robust under ideal conditions, but also exposes it to a unique set of network-related problems. When audio drops, crackles, or fails to start altogether, finding the root cause can be difficult without specialized tools.

Network analyzers (packet sniffers) such as Wireshark or tcpdump allow engineers to capture raw network traffic and inspect every packet. By examining the flow of RTP data, SAP announcements, PTP timing messages, and control protocols, you can isolate issues that would otherwise remain invisible. This article provides a structured approach to using network analyzers to diagnose AES67 audio streaming problems, from setup through resolution.

Prerequisites for Effective Network Analysis

Before you begin capturing traffic, ensure you have the following in place:

  • A network analyzer tool – The most common is Wireshark, which offers GUI and filtering capabilities. For headless servers, tcpdump combined with Wireshark for offline analysis works well.
  • Access to the network segment – Your computer must connect to the same VLAN or physical network as the AES67 devices. Use a managed switch with port mirroring (SPAN) if you cannot interrupt traffic.
  • Device IP addresses and multicast groups – Obtain the source and destination IPs, multicast group addresses (typically 239.x.x.x or 224.x.x.x), and RTP port (default 5004) used by your AES67 streams. SAP (Session Announcement Protocol) often runs on UDP port 9875 to advertise streams.
  • Sufficient capture permissions – Root/administrator rights are usually required to put the network interface in promiscuous mode.
  • A known good reference – If possible, capture a working stream from the same system or a controlled test source to compare against the faulty stream.

Setting Up Your Capture Environment

Connecting to the Right Network

For accurate diagnosis, connect your computer either directly to a switch port that mirrors the relevant traffic, or place it on the same subnet as the AES67 devices. Avoid connecting through a hub or unmanaged switch that may add noise or drop packets. When using port mirroring, ensure the switch does not strip VLAN tags or alter multicast behavior – configure the mirror port to preserve original frame headers.

Configuring the Network Analyzer Interface

In Wireshark, select the correct network interface (e.g., eth0, en0, or Wi-Fi – though wired is strongly preferred for reliability). Set a capture filter to reduce the volume of irrelevant traffic. For AES67, a useful capture filter is:

udp port 5004 or udp port 9875 or udp port 319 or udp port 320

This captures RTP (port 5004), SAP (9875), and PTP events/messages (ports 319 and 320). Adjust ports if your system uses different values.

Display Filters for Streamlined Analysis

After capturing, apply display filters to isolate AES67 traffic. Common display filters include:

  • rtp – Shows all RTP packets (Wireshark identifies them from the payload type).
  • ip.addr == 192.168.1.10 – Filter by device IP.
  • eth.dst == 01:00:5e:xx:xx:xx – Filter by multicast MAC (derived from IP).
  • ptp – Shows Precision Time Protocol messages.
  • sap – Displays Session Announcement Protocol packets.

You can combine filters with logical operators: rtp and ip.src==192.168.1.10.

Key Metrics to Monitor in AES67 Streams

Once you have captured traffic from a problematic AES67 stream, focus on these critical parameters:

Packet Loss and Sequence Numbers

RTP packets carry a 16-bit sequence number that increments by one for each packet sent. Wireshark's RTP analysis window (Telephony > RTP > Stream Analysis) calculates packet loss by detecting gaps in sequence numbers. Even a small number of lost packets (0.1% or more) can cause audible clicks or dropouts. Look for sudden jumps in sequence numbers or duplicate packets that indicate retransmissions (which shouldn't occur in standard AES67 unicast/multicast).

Jitter and Latency

RTP also includes a timestamp driven by the audio sample clock. Variation in the arrival time of packets (jitter) is buffered by the receiver, but excessive jitter exceeds the buffer depth and causes underruns. In Wireshark, the RTP analysis shows jitter in milliseconds. For AES67, jitter should typically remain below 1 ms on a well-designed network. Also monitor round-trip time if using unicast streams.

Synchronization and PTP

AES67 devices can use PTPv2 (IEEE 1588-2008) to synchronize sample clocks across the network. Without accurate PTP, devices may drift apart, leading to buffer overflows or underflows. In the capture, check that PTP messages (Sync, Follow_Up, Delay_Req, Delay_Resp) are arriving regularly – typically one Sync per second. Use Wireshark's PTP statistics (Statistics > PTP) to view clock offset and mean path delay. An offset greater than 1 microsecond may indicate a problem.

Stream Announcement and Discovery

Check SAP packets to ensure the stream is being announced correctly. Look for the stream's name, multicast address, port, and payload type. If SAP messages are missing or malformed, other devices cannot discover or join the stream.

Common AES67 Issues and How to Spot Them

Network Congestion and Bandwidth Saturation

High utilization on the switch port or uplink causes packet drops. Using a capture filter on the destination multicast address, observe if packets are arriving at the correct rate (e.g., 48 kHz / 24-bit stereo = ~2.3 Mbps plus overhead). Spikes in inter-arrival time above the packet period (typically 125 µs to 1 ms) indicate congestion. Check Wireshark's IO graph (Statistics > IO Graph) for large peaks.

Incorrect IGMP Snooping or Multicast Configuration

Multicast streams rely on IGMP (Internet Group Management Protocol) for switches to forward traffic only to interested ports. If IGMP snooping is misconfigured, the stream may not reach the receiver. Capture IGMP packets (igmp) to see if the receiver sends Join requests and if the switch responds. Also check for IGMP timeouts that cause the stream to stop after a few minutes.

Unidirectional Traffic or Firewall Blocks

Stateful firewalls may block RTP/UDP traffic if not explicitly allowed. In the capture, look for ICMP Destination Unreachable messages (icmp.type == 3) sent back to the source. Also ensure that PTP traffic (UDP ports 319/320) is permitted to cross VLANs if devices are on different subnets.

Switch QoS Misconfiguration

AES67 requires proper Quality of Service markings to prioritize audio packets over best-effort traffic. Switches should trust the DSCP value (often EF – 46 or CS7 – 56). In the capture, verify that RTP packets have the correct DSCP field (check IPv4 header DSCP bits). If they appear as zero or Best Effort, the switch may re-queue them and cause jitter.

Hardware or Cable Faults

CRC errors, runt frames, or collisions are visible in the capture at the Ethernet layer. Wireshark marks damaged frames with a "Bad checksum" or "Frame check sequence" error. Poor cabling often leads to intermittent errors that correlate with audio dropouts.

Step-by-Step Troubleshooting Example

Situation: An AES67 microphone array consistently loses audio for 2–3 seconds every 10 minutes. The system uses multicast on 239.10.10.1, port 5004.

  1. Capture for 20 minutes on the receiver side using the capture filter udp port 5004 or igmp.
  2. Analyze RTP sequence numbers near the dropout time. Observe a gap from sequence 25430 to 25470 (40 packets missing) – but no duplicate packets or retransmissions.
  3. Check IGMP messages at the same timestamp. Find an IGMP v2 General Query from the switch and a subsequent Leave message from the receiver – followed by no further audio packets for several seconds until the receiver re-joins the group.
  4. Conclusion: The receiver's IGMP Membership timeout is too short, and the switch is pruning the multicast stream while the receiver has not sent a refresh. The audio drop occurs because the stream stops forwarding.
  5. Resolution: Increase the IGMP query interval on the switch or adjust the receiver's IGMP robustness variable (if possible). Alternatively, change the switch's IGMP version or disable snooping temporarily for testing.

This example illustrates how a network analyzer can pinpoint a subtle IGMP timing issue that would be impossible to detect by listening alone.

Resolving Found Issues

Based on your analysis, take corrective actions:

  • Packet loss due to congestion – Reduce background traffic, implement QoS with priority queuing, or upgrade switch fabric bandwidth.
  • High jitter – Ensure all switches are configured with strict priority for the audio DSCP code, and that cables are not excessively long or of poor quality.
  • PTP instability – Check that all switches are PTP-aware (Boundary Clock or Transparent Clock) and that the Grandmaster clock is stable. Use PTP statistics to verify offset and delay.
  • Firewall blocking – Create static firewall rules allowing UDP ports 5004, 9875, 319, 320, and possibly SAP multicast addresses.
  • SAP mismatches – Correct stream announcement parameters on the source device.
  • Firmware/software bugs – Check the manufacturer's AES67 implementation guidelines and update device firmware as needed.

Preventative Network Monitoring

Proactive network analysis reduces downtime. Set up periodic captures (e.g., once a week) and compare key metrics like packet loss, jitter, and PTP offset. Use tools like Wireshark's IO Graph with trend analysis. Document baseline values so you can quickly spot anomalies. Additionally, monitor switch counters for CRC errors, dropped packets, and broadcast storms. Many professional audio networks now include a dedicated monitoring station running tcpdump in a loop, retaining the last few hours of captures for post-mortem analysis.

By mastering network analyzer techniques specific to AES67, you can significantly reduce troubleshooting time and ensure consistent, high-quality audio streaming in production environments.