Inside Meta's 102.4T IceCube and Flattened Planar Fabric for AI Scale-Out

The Core Update

Meta unveiled its next tier of open networking hardware and transport architecture at the OCP Global Summit. The drop targets gigawatt-scale AI clusters that span multiple data hall halls and cross-campus regions.

The hardware release centers on two main pieces: IcePack, Meta's liquid-cooled network rack built on the OCP ORV3-HPRv1 standard, and IceCube, a 102.4 Tbps switch blade offering 128x800G non-blocking throughput. IceCube runs either Broadcom Tomahawk 6C or NVIDIA Spectrum-6 ASICs on top of Meta's open FBOSS network operating system via the OCP Switch Abstraction Interface (SAI).

Alongside the silicon, Meta introduced the Flattened Planar Fabric (FPF) backend architecture, an open-source transport protocol dubbed MetaRoCE, and the Open Switch Platform Architecture (OSPA) working group.

Official Source: Meta Announcement

Technical Impact & Mechanism

Scaling transformer models across tens of thousands of GPUs breaks conventional Clos networks. Air cooling fails when switch ASICs pull sustained peak wattage across 128 ports of 800G optical modules. Traditional multi-tier topologies also introduce excessive tail latency during all-reduce collective communications.

IcePack addresses the thermal wall by running direct liquid cooling down to the switch tray level. This gives the 102.4T platform double the throughput of older 51.2T switches within the same thermal footprint.

At the topology layer, Flattened Planar Fabric (FPF) alters how accelerator nodes link to network fabrics. Instead of bundling GPUs into standard hierarchical spine-leaf clusters, FPF decouples endpoints into parallel horizontal planes. Each rail from an accelerator node connects directly into a dedicated network plane. This setup eliminates intermediate routing tiers, drops packet hop counts, and simplifies route reflection.

MetaRoCE sits on top of this physical layer to solve RDMA transport collapse. Standard RoCEv2 relies on Priority Flow Control (PFC) and Explicit Congestion Notification (ECN). At scale, this causes PFC deadlocks and buffer bloat. MetaRoCE integrates end-to-end telemetry directly into packet headers, letting senders throttle flows before switch buffers saturate.

# Conceptual FBOSS/SAI Port and Queue Profile for MetaRoCE on IceCube
switch_ports:
  - id: eth1/1/1
    speed_gbps: 800
    fec: RS544
    cooling_mode: liquid_monitored
    asic_driver: broadcom_th6c
    traffic_class_mapping:
      lossless_rdma_pool: 3
      control_plane_pool: 7
    congestion_control:
      engine: meta_roce
      ecn_marking:
        min_threshold_kb: 128
        max_threshold_kb: 512
        probability: 0.10
      pfc:
        enabled: false # Mitigated via proactive telemetry throttling

Decoupling the hardware layer through FBOSS and SAI means operators are not locked into one chip vendor. You can run Tomahawk 6C in one row and Spectrum-6 in another without changing your orchestration scripts or metric scrapers.

Action Plan for Developers & Businesses

1. Audit switch telemetry for tail latency bottlenecks. If your distributed training clusters still use standard RoCEv2 with hard PFC triggers, profile your pause frame frequency. Inspect whether buffer saturation during gradient synchronization is idling compute nodes.
2. Evaluate OCP ORV3-HPR rack compatibility. Higher bandwidth ASICs will demand liquid cooling at the switch level. If you plan private infrastructure for 800G or 1.6T networking, prepare your data center facilities for liquid loops directly to network racks, not just compute nodes.
3. Decouple control software using OCP SAI. Avoid proprietary network stacks that marry your telemetry code to specific switch vendors. Build on open network operating systems like SONiC or FBOSS to keep switch silicon swappable as Broadcom and NVIDIA trade performance leads.