Understanding AES67 and the Need for High Availability

AES67 is a standard that enables interoperability between various audio-over-IP systems, allowing different manufacturers’ equipment to communicate over a common network. In broadcast facilities, this standard is critical for transporting professional-quality audio with low latency, high channel density, and deterministic timing. High availability in an AES67 network means that the system continues to operate without interruption, even when individual components such as switches, cables, audio interfaces, or clock sources fail. For broadcasters, downtime is not merely an inconvenience; a single second of silence during a live news feed can erode audience trust, trigger contractual penalties, and damage brand reputation. Achieving high availability requires a comprehensive approach that addresses every layer of the network—from physical cabling and switching to device configuration and monitoring.

The core challenge of AES67 is that it relies on a standard Ethernet network that may carry both audio and non-audio traffic. Unlike dedicated analog or SDI infrastructures, IP networks introduce shared resources, congestion, and packet loss. The network must be engineered to guarantee that audio packets are delivered on time, in order, and without drops, even when other traffic surges. Therefore, designing for high availability means building resilience into the network fabric itself, plus ensuring that endpoints can recover gracefully from failures. This article expands on proven strategies to design and maintain a highly available AES67 network for broadcast production, live events, and mission-critical audio distribution.

Key Strategies for High Availability in AES67 Networks

1. Network Redundancy – The Foundation of Resilience

The physical network layer is where most single points of failure exist. A single cable cut, switch power supply failure, or module malfunction can take down an entire audio stream. The goal of network redundancy is to eliminate these vulnerabilities by providing multiple, independent paths for audio traffic.

  • Redundant Switches and Links: Deploy at least two core switches in a redundant configuration (e.g., MLAG, stacking with redundancy, or VPC). Connect each AES67 device to both switches using separate NIC ports or a dual-homed topology. This ensures that if one switch fails, the second switch continues to forward traffic without interruption. For devices with only one Ethernet port, consider using a failover relay or an external switching module that can sense link loss and reroute.
  • Link Aggregation (LAG) and Active/Active Paths: Use LACP (IEEE 802.3ad) to bundle multiple physical cables between switches into a single logical link. This provides both increased bandwidth and automatic failover if one cable fails. For AES67, ensure the LAG hash algorithm is symmetric for multicast traffic to prevent out-of-order packets. Some broadcast engineers prefer using active/standby LAG with fast failover timers to avoid jitter from load‑balancing anomalies.
  • Spanning Tree Protocol (STP) Tuning: While redundant switches create loops, the network must use a spanning tree variant like RSTP (802.1w) or MSTP (802.1s) to block redundant paths and prevent broadcast storms. Configure rapid transition times and tune timers to keep failover under 1 second for audio. In practice, RSTP will converge in 1–2 seconds, which may still cause a brief audio dropout; for sub‑50ms failover, consider protocols designed for industrial or carrier networks.
  • Physical Diversity: Run redundant fiber or copper paths via separate cable trays, risers, or even different physical routes through the facility. This protects against accidental cuts or building infrastructure failures. Document every cable run and its path; label all connections at both ends. A common pitfall is routing both redundant paths through the same conduit, so pull them through separate raceways or at least a few feet apart.

For broadcast facilities that require the highest uptime, consider a ring topology with Ethernet ring protection switching (ERPS) or Media Redundancy Protocol (MRP) which offer sub‑50ms convergence times. These protocols are especially useful in large facilities with many interconnections. However, be aware that ring topologies increase latency per hop, so they are best suited for smaller networks or fully deterministic switch fabrics. A more flexible approach is to build a spine‑leaf architecture with multiple parallel paths, using ECMP (Equal‑Cost Multipath) routing at Layer 3 to distribute AES67 flows across all available links. If one link fails, traffic is automatically redistributed without a convergence event.

2. Device Redundancy – Ensuring Endpoint Resiliency

Audio endpoints—AES67 transmitters, receivers, mixing consoles, and stage boxes—must also be protected against failure. A single power supply failure or main board crash can take a channel or even an entire studio off air. Device redundancy strategies include:

  • Active/Passive Redundancy (1+1): Deploy two identical devices with one in standby. Use a controller or health monitoring system to detect failure and switch traffic to the standby unit. For AES67, this means the standby device must join the same multicast groups and maintain the same session descriptions (via SAP) as the active unit. The switchover must be fast enough to prevent a gap in audio; some systems achieve sub‑frame switching by pre‑configuring both devices and using a common source identifier.
  • N+1 Redundancy: For larger systems, keep one spare device on the shelf that can be manually swapped or automatically brought online via a distributed controller. This is cost‑effective for environments where a short interruption (a few seconds) is acceptable, such as pre‑recorded programming or non‑critical monitoring. Define a clear procedure for swap‑out, and train staff on it.
  • Redundant Power Supplies and Fans: Choose equipment that supports dual, hot‑swappable power supplies (AC and DC optional). Connect each power supply to separate UPS feeds or building circuits to survive a single power distribution failure. Similarly, ensure cooling fans are redundant and monitored; an overheating switch can throttle performance, causing packet drops even if the audio traverses a live path.
  • Automatic Failover with Seamless Switching: Some AES67‑compatible devices support seamless redundancy using SMPTE ST 2022‑7 or similar methods. This sends the same audio packet over two separate network paths. The receiver selects the first packet that arrives and discards duplicates, providing hitless failover. When possible, enable this feature at the device level for the highest availability. However, be mindful that it consumes double the bandwidth and requires both paths to have similar latency. For facilities that cannot afford the extra overhead, consider using a controller to re‑route multicast traffic via IGMP leave/join on failure—this causes a brief gap but is simple to implement.

3. Timing and Clock Redundancy – Keeping Every Sample in Sync

AES67 relies on a common clock reference, typically Precision Time Protocol (IEEE 1588‑2008). If the grandmaster clock fails, all devices lose synchronization, resulting in sample slips, pops, clicks, or complete audio failure. To achieve high availability, you must engineer a redundant clock distribution:

  • Multiple Grandmasters: Configure at least two PTP grandmasters—one primary and one backup. The backup should be in “slave” or “hot standby” mode, but ready to take over if the primary disappears. Use the PTP alternate master (also called “master redundancy”) mechanism defined in the standard. In practice, this means both grandmasters send Announce messages; all clocks in the domain will select the best one using the Best Master Clock Algorithm (BMCA).
  • Best Master Clock Algorithm (BMCA): Enable BMCA on all boundary clocks and ordinary clocks. This algorithm automatically selects the best grandmaster based on clock priority, accuracy, and variance. Ensure your grandmasters have high priority (low numerical value) to avoid a less capable device taking over. For example, set priority1 to 128 on the primary and 129 on the secondary. Some manufacturers also allow setting a “holdover” clock that keeps the domain stable for minutes while the domain re‑elects a new grandmaster.
  • Redundant PTP Paths: Use multiple boundary clocks or transparent clocks to provide redundant paths for PTP messages. If one switch’s boundary clock fails, the PTP domain should reconverge quickly. Test this by physically removing the primary grandmaster and measuring the time to resync. In a well‑tuned domain, resync should happen within a few seconds without audible artifacts.
  • Holdover Capability: Some devices have built‑in holdover oscillators that maintain accuracy for several hours if the clock signal is lost. This provides a safety buffer while redundant paths are restored. For critical installations, specify devices with TCXO or OCXO oscillators; they cost more but can keep drift to less than 1 microsecond per hour of holdover.

For further reading on PTP redundancy best practices, refer to the IEEE 1588‑2019 standard which introduces improved redundancy mechanisms for telecom and broadcast. Also consider using a dedicated timing distribution network—some facilities run PTP over a separate fiber infrastructure to isolate it from audio‑path changes, though this adds cost.

4. Network Segmentation and QoS – Protecting Audio from Interference

AES67 audio traffic is sensitive to packet loss, latency, and jitter. In a broadcast facility, the network often carries file transfers, control traffic, video streams, and general office data. Without proper segmentation, a large file transfer can cause buffer overflow and drop AES67 packets. High availability requires isolating audio traffic logically and prioritizing it.

  • VLANs for Audio: Create a dedicated VLAN for all AES67 traffic. This prevents broadcast storms from other VLANs and allows you to assign QoS policies specifically to that VLAN. Use 802.1Q tagging on all trunk ports. For added isolation, consider a separate physical network for audio only if budget allows; it simplifies troubleshooting and eliminates contention entirely. However, most modern facilities prefer a converged network with VLANs and strict QoS.
  • Quality of Service (QoS): Mark AES67 packets with a high DSCP value (e.g., EF – 46 for audio, CS7 for clock). On switches, configure strict priority queues to ensure audio is always forwarded ahead of best‑effort traffic. Police or shape non‑audio traffic at the edge to prevent oversubscription. Additionally, use CoS (802.1p) to prioritize audio at Layer 2; many switches map CoS values to internal queues. Run a jitter test with and without background traffic to verify your QoS policies.
  • Multicast Management: AES67 typically uses IP multicast for point‑to‑multipoint audio distribution. Enable IGMP snooping on all Layer 2 switches to restrict multicast traffic only to ports that have joined the group. Without snooping, multicast floods all ports, which wastes bandwidth and increases jitter. Use IGMP querier on a redundant switch pair (with a virtual IP) to maintain the multicast group membership even if one querier fails. For larger networks, consider using PIM sparse mode at Layer 3 to build efficient multicast distribution trees.
  • Bandwidth Reservations: For high‑density systems, reserve bandwidth on uplinks for AES67 traffic. Using tools like LLDP‑MED or manually configured policers, ensure that non‑audio traffic never exceeds the reserved capacity. Some networks use “audio‑only” subnet ports where only AES67 devices are connected, further simplifying QoS. Calculate the total AES67 bandwidth: each 48‑kHz, 24‑bit channel is about 1.5 Mbps. A 128‑channel system requires at least 192 Mbps; leave headroom for growth and bursts.

5. Monitoring, Testing, and Proactive Maintenance

Even the best‑designed redundant network will eventually experience failures. The key to high availability is early detection and rapid recovery. Broadcast facilities should invest in ongoing monitoring and periodic testing.

  • Network Monitoring (SNMP and sFlow): Deploy an SNMP‑based monitoring system that tracks switch CPU load, port errors, temperature, power supply status, and fan health. Also monitor audio‑specific metrics: packet loss, jitter, and synchronization status. Tools like PRTG, SolarWinds, or open‑source solutions like LibreNMS can alert on interface errors or failures. Configure alerts for any interface that transitions to error‑disabled state or exceeds a threshold of CRC errors.
  • AES67‑Specific Monitoring: Use tools like Dante Controller or Livewire Designer that provide per‑stream statistics. Some third‑party software can map the entire AES67 network and report on stream uptime. Configure alerts for any stream showing packet loss above 0.1% or latency spikes. Also monitor the PTP domain: deviations beyond ±1 microsecond should trigger an alarm. Consider deploying a dedicated audio‑over‑IP monitor that continuously measures audio quality and detects dropouts.
  • Regular Failover Testing: Schedule quarterly or monthly tests where you manually failover one component at a time—pull a power supply, disconnect a fiber, or disable a grandmaster. Observe audio continuity and measure the time to restore. Document the results and fix any issues (e.g., a switch that takes too long to reconverge). Simulate worst‑case scenarios: simultaneous failure of an upstream switch and a grandmaster. Keep a log of test results and review them with the engineering team.
  • Firmware and Software Updates: Keep all switches, AES67 devices, and clock generators on the latest stable firmware. Before upgrading, test on a smaller isolated network. Use change management procedures to avoid introducing new failure points. Maintain a version spreadsheet with checksums and release dates. Some vendors provide beta firmware; avoid it for production networks unless a specific bug fix is critical.
  • Spare Parts Inventory: Maintain a stock of critical spares: power supplies, SFP optics, cables, and a spare switch. Store them in a known, clean location. For large facilities, keep a pre‑configured backup switch that can be swapped on‑air quickly. Label spares with the equipment they are meant to replace and include configuration notes. Test spares periodically to ensure they work; do not assume a shelf spare will boot correctly.

Building a Culture of Reliability

High availability is not just a technical checklist; it requires a cultural commitment to reliability. Each engineer, technician, and operator should understand the criticality of the AES67 network and follow best practices daily. This includes using proper cable strain relief, avoiding daisy‑chaining switches on the audio VLAN, and never bypassing QoS on test equipment. Conduct post‑mortems after every incident, no matter how minor, and update the design documentation accordingly. Involve the network team in broadcast operations so they can anticipate traffic patterns and upcoming events that may place extra load on the infrastructure.

Additionally, consider implementing an automated failover orchestration tool that can detect a switch failure and reconfigure IGMP groups or reroute PTP timing. While expensive, these tools can reduce mean time to repair from minutes to seconds. Start with small automation scripts that generate a syslog alert and then run a pre‑defined remediation command—for example, issuing a clear igmp snooping groups on a backup switch if the primary is unreachable.

Conclusion

Designing a high‑availability AES67 network for broadcast facilities is not a one‑time project but a continuous process. It requires careful planning at the physical network layer, device configuration, clock distribution, and ongoing operational discipline. By implementing redundant switches and diverse paths, using active/passive device failover and seamless redundancy protocols, building a redundant PTP timing domain, segmenting and prioritizing audio traffic, and maintaining vigilant monitoring and testing, broadcasters can achieve the reliability expected in mission‑critical environments. The effort invested in redundancy and failover readiness pays off every time a component fails without interrupting the audience’s listening experience.

For more detailed guidance on implementing AES67 networks, refer to the AES67 standard document and the Audinate approach to AES67 interoperability. Remember that high availability is not just about equipment—it is about a mindset of proactive engineering, systematic testing, and continuous improvement.