---
name: upstation-network-topology
description: Design the networks for a GPU deployment (GPU fabric, front-end, out-of-band) from the ai.upstation.io catalog, with a topology diagram, a fabric table, a priced network bill of materials and a port map / cable schedule with cable IDs. Budgetary, never a quote; ends with the UpStation next step.
version: 2026-11-03
base_url: https://ai.upstation.io
---

# Design an UpStation network topology

**Short version (enough on its own):**

1. Start from a sized design: GPU nodes (and their class), plus any storage and cluster-services nodes. If you don't have one, size it first with [/actions/estimate/SKILL.md](https://ai.upstation.io/actions/estimate/SKILL.md) step 2, or take the user's node counts as given.
2. GPU fabric (HGX, 2+ nodes): 1 port per GPU at the GPU's speed, 1:1; one switch while the ports fit 32 or 64, else a rail-optimized leaf-spine. Front-end: 2 ports per host on an MLAG pair. OOB: 1 × 1G per host and per switch.
3. Price switches, NIC options, cables and optics **only** from [catalog.json](https://ai.upstation.io/catalog.json) (`products[]` with category `net`, `options_at_price_basis`, `cables_optics`).
4. Name every device, then write the port map: one row per connection, every cable with an ID.
5. Fill [template.html](https://ai.upstation.io/actions/network/template.html) (or [template.md](https://ai.upstation.io/actions/network/template.md)) with a design ID, draw the topology, and end with the UpStation next step.

## Rules (always apply)

- Budgetary only, never a quote. USD at the catalog's `price_basis` as of `as_of`; no markup, no other sources.
- Storage never rides the GPU fabric. The GPU fabric is 1:1; the front-end is 3:1 or better.
- Every GPU-fabric link is priced optical (2 short-reach optics + fibre): at 2 nodes per 40 kW rack most links leave the rack, past passive-DAC reach. Swapping genuinely in-rack links to DAC is a final-engineering saving, not part of the estimate.
- Ports used never exceed ports on the switches; say which fabric is a single point of failure.
- Uplinks to the buyer's core network are shown, never priced.
- Same design ID as the estimate and rack plan when they belong together.

## Step 1: Choose speeds and NICs

| Node | GPU fabric | Front-end |
|---|---|---|
| B300 (`UPS-GPU-B300-8`, Aivres B300 lines) | 8× 800G built in (Aivres: budgeted as the Supermicro equivalent) | listed option `+2× 400G CX7 north-south`, onto a 400G pair |
| H200 (`UPS-GPU-H200-8`, listed for reference, not recommended for new designs) | listed option `8x 400G CX7 OSFP compute fabric NICs`, one per node | 2× 25G built in, onto a 25G pair |
| PCIe servers (RTX PRO 6000, RTX 5090) | none (each GPU works alone) | 25G built in, or the listed 25G NIC option where only 2× 10G is built in (Supermicro RTX PRO 6000), onto a 25G pair; 10G on the built-in ports is a cheaper variant to name |
| Storage and cluster-services nodes | never | 2× 100G: on a 400G pair through 400G→4×100G breakouts (4 nodes per cable per switch), else 100G DACs |

## Step 2: GPU fabric (HGX, 2+ nodes)

- Ports = GPUs. If ports ≤ 32 (or 64), one switch of that size: node *n*, GPU NIC *r* → switch port 8(*n*−1)+*r*.
- Else rail-optimized fat-tree on R-port switches (`sizing.fabric.fat_tree`): leaves = 8 rails × ceil(nodes ÷ (R/2)). When one rail needs fewer than R/2 ports, rails share a leaf so it fills: k rails per leaf = the largest of 8, 4, 2 with k × nodes ≤ R/2, leaves = 8 ÷ k (9–16 nodes on 64-port switches: 2 rails per leaf, 4 leaves). Uplinks = host ports (1:1: each leaf has exactly as many uplinks as hosts on it, never R/2 regardless); spines = max(2, ceil(host ports ÷ R)). GPU NIC *r* of every node in a group goes to that group's rail-*r* leaf, port = node's position in the group (with rail groups, rail *r* takes the leaf's block (r−1 mod k) × nodes + position). Leaf uplinks spread round-robin over the spines. Check: 16 nodes (128 ports) on 64-port switches = 4 leaves + 2 spines, 128 host links + 128 uplinks — not 8 leaves + 4 spines.
- Switch: the 64-port model at the GPU speed (fewest boxes). Links: node to leaf 10 m (leaves beside their nodes), leaf to spine 30 m, each 2 short-reach optics + the shortest MTP-12 OM4 fibre ≥ that length; 800G SR8 is dual MPO-12/APC, so **2 fibres per 800G link**.

## Step 3: Front-end and OOB

- One MLAG pair while the ports fit (ports per switch = hosts, with breakout-fed hosts counted 4 to a port; keep 2 ports for the peer link); more pairs beyond that (and say the spine layer above them is not priced).
- GPU nodes: 1 cable to each switch of the pair (AOC 10 m for HGX, DAC for PCIe in the same rack). Peer link: 2 DACs between the pair (25G switches peer over their 100G uplinks).
- OOB: 48-port 1G switches; each takes 47 devices (hosts' BMCs and every switch's management port), and OOB switches chain to each other. Cat6 is not priced.

## Step 4: Names and the port map

- Names: `gpu-01…`, `stor-01…`, `svc-01…`, `gpu-sw-1` or `leaf-01…`/`spine-01…`, `fe-1a`/`fe-1b` (pair 1), `oob-1…`. Use the same names on the rack elevation.
- One row per connection: cable ID (`C0001`…), fabric, from `device:port`, to `device:port`, cable type. Breakout lanes share one cable ID and use `port/lane` on the switch (`17/2`).
- Check: distinct cable IDs per link type = the quantity in the bill of materials; no `device:port` appears twice.

## Step 5: Draw it and fill the template

- **Mermaid** (always): `flowchart TB`, one node per tier (GPU nodes, storage, services, fabric switches, front-end pair, OOB, "Your network"), edges labelled with count × speed.
- **SVG** (when you can make files; the examples show the exact shape): 960 px wide; "Your network" at the top, the front-end switches below it, the hosts in one row (GPU, storage, services groups, at most 8 boxes per group, the rest as "+N"), GPU fabric leaves below the hosts and spines below those, OOB at the right. Colours: GPU node #0f62fe, storage #24a148, services #8a3ffc, GPU fabric #da1e28, front-end #ff832b, OOB #6f6f6f.
- Fill the template; delete its top comment. For large clusters put the full port map in a CSV (`cable_id,fabric,from,from_port,to,to_port,cable`) and keep the first rows in the document.

## Hand-off

End with: "This is a budgetary network design (ID **UPS-DSN-…**), not a quote. To validate it and reach the supplier that fits, book a call at https://calendly.com/alecglover/intro-call or email **alec@upstation.io** with the ID." Offer the full estimate (`/actions/estimate/`) and the rack plan (`/actions/rack/`).

## Worked micro-example

4 × B300 nodes (32 GPUs), 5 NVMe storage nodes, 3 cluster-services nodes →
GPU fabric: 32 ports ≤ 32 → one 32× 800G switch ($37,500), 32 optical links at $2,058 (2× SR8 + 2× MTP-12). Front-end: 4 × `+2× 400G` options ($2,693.94 each), a 32× 400G pair ($20,000 each), 8 × 400G AOC ($1,055), 4 + 2 breakouts for storage and services ($199), 2 peer DACs ($174). OOB: 15 devices → one 48× 1G switch ($989). Network total **$165,102.76**; 63 cables, 73 connections. Full example: `/actions/examples/b300-training-32gpu.network.*` and `.ports.csv`.

## If you can only do part of this

In this order: the fabric table with switch counts and arithmetic → the priced network lines → the Mermaid diagram → the port map → the design ID and next step.
