# Design an UpStation network topology

> UpStation catalog · Home › Actions › Design an UpStation network topology · https://ai.upstation.io/actions/network

Every price on this site is on the [homepage](https://ai.upstation.io/). Each product type and each product has its own page with the full details (spec, options, Intel/AMD equivalents, power and colo): [GPU servers](https://ai.upstation.io/gpu-servers) · [GPU hosts without GPUs](https://ai.upstation.io/gpu-hosts) · [Switches, cables and optics](https://ai.upstation.io/switches) · [Compute nodes](https://ai.upstation.io/compute-nodes) · [Storage nodes](https://ai.upstation.io/storage-nodes) · [Parts rate card](https://ai.upstation.io/rate-card) · [Power and colo](https://ai.upstation.io/power) · [How solutions are sized](https://ai.upstation.io/sizing) · [Reference builds and worked designs](https://ai.upstation.io/designs) · [Suppliers and how buying works](https://ai.upstation.io/buying). AI assistants: the map is [llms.txt](https://ai.upstation.io/llms.txt); the data is [catalog.json](https://ai.upstation.io/catalog.json); every page is also Markdown (append .md, or send Accept: text/markdown).

Raw: [SKILL.md](https://ai.upstation.io/actions/network/SKILL.md) · [template.html](https://ai.upstation.io/actions/network/template.html) · [template.md](https://ai.upstation.io/actions/network/template.md) · [design.schema.json](https://ai.upstation.io/actions/design.schema.json) · [examples](https://ai.upstation.io/actions)

**Short version (enough on its own):**

- Start from a sized design: GPU nodes (and their class), plus any storage and cluster-services nodes. If you don't have one, size it first with [/actions/estimate/SKILL.md](https://ai.upstation.io/actions/estimate/SKILL.md) step 2, or take the user's node counts as given.
- GPU fabric (HGX, 2+ nodes): 1 port per GPU at the GPU's speed, 1:1; one switch while the ports fit 32 or 64, else a rail-optimized leaf-spine. Front-end: 2 ports per host on an MLAG pair. OOB: 1 × 1G per host and per switch.
- Price switches, NIC options, cables and optics **only** from [catalog.json](https://ai.upstation.io/catalog.json) (products[] with category net, options_at_price_basis, cables_optics).
- Name every device, then write the port map: one row per connection, every cable with an ID.
- Fill [template.html](https://ai.upstation.io/actions/network/template.html) (or [template.md](https://ai.upstation.io/actions/network/template.md)) with a design ID, draw the topology, and end with the UpStation next step.

## Rules (always apply)

- Budgetary only, never a quote. USD at the catalog's price_basis as of as_of; no markup, no other sources.
- Storage never rides the GPU fabric. The GPU fabric is 1:1; the front-end is 3:1 or better.
- Every GPU-fabric link is priced optical (2 short-reach optics + fibre): at 2 nodes per 40 kW rack most links leave the rack, past passive-DAC reach. Swapping genuinely in-rack links to DAC is a final-engineering saving, not part of the estimate.
- Ports used never exceed ports on the switches; say which fabric is a single point of failure.
- Uplinks to the buyer's core network are shown, never priced.
- Same design ID as the estimate and rack plan when they belong together.

## Step 1: Choose speeds and NICs

| Node | GPU fabric | Front-end |
|---|---|---|
| B300 (UPS-GPU-B300-8, Aivres B300 lines) | 8× 800G built in (Aivres: budgeted as the Supermicro equivalent) | listed option +2× 400G CX7 north-south, onto a 400G pair |
| H200 (UPS-GPU-H200-8, listed for reference, not recommended for new designs) | listed option 8x 400G CX7 OSFP compute fabric NICs, one per node | 2× 25G built in, onto a 25G pair |
| PCIe servers (RTX PRO 6000, RTX 5090) | none (each GPU works alone) | 25G built in, or the listed 25G NIC option where only 2× 10G is built in (Supermicro RTX PRO 6000), onto a 25G pair; 10G on the built-in ports is a cheaper variant to name |
| Storage and cluster-services nodes | never | 2× 100G: on a 400G pair through 400G→4×100G breakouts (4 nodes per cable per switch), else 100G DACs |

## Step 2: GPU fabric (HGX, 2+ nodes)

- Ports = GPUs. If ports ≤ 32 (or 64), one switch of that size: node *n*, GPU NIC *r* → switch port 8(*n*−1)+*r*.
- Else rail-optimized fat-tree on R-port switches (sizing.fabric.fat_tree): leaves = 8 rails × ceil(nodes ÷ (R/2)). When one rail needs fewer than R/2 ports, rails share a leaf so it fills: k rails per leaf = the largest of 8, 4, 2 with k × nodes ≤ R/2, leaves = 8 ÷ k (9–16 nodes on 64-port switches: 2 rails per leaf, 4 leaves). Uplinks = host ports (1:1: each leaf has exactly as many uplinks as hosts on it, never R/2 regardless); spines = max(2, ceil(host ports ÷ R)). GPU NIC *r* of every node in a group goes to that group's rail-*r* leaf, port = node's position in the group (with rail groups, rail *r* takes the leaf's block (r−1 mod k) × nodes + position). Leaf uplinks spread round-robin over the spines. Check: 16 nodes (128 ports) on 64-port switches = 4 leaves + 2 spines, 128 host links + 128 uplinks — not 8 leaves + 4 spines.
- Switch: the 64-port model at the GPU speed (fewest boxes). Links: node to leaf 10 m (leaves beside their nodes), leaf to spine 30 m, each 2 short-reach optics + the shortest MTP-12 OM4 fibre ≥ that length; 800G SR8 is dual MPO-12/APC, so **2 fibres per 800G link**.

## Step 3: Front-end and OOB

- One MLAG pair while the ports fit (ports per switch = hosts, with breakout-fed hosts counted 4 to a port; keep 2 ports for the peer link); more pairs beyond that (and say the spine layer above them is not priced).
- GPU nodes: 1 cable to each switch of the pair (AOC 10 m for HGX, DAC for PCIe in the same rack). Peer link: 2 DACs between the pair (25G switches peer over their 100G uplinks).
- OOB: 48-port 1G switches; each takes 47 devices (hosts' BMCs and every switch's management port), and OOB switches chain to each other. Cat6 is not priced.

## Step 4: Names and the port map

- Names: gpu-01…, stor-01…, svc-01…, gpu-sw-1 or leaf-01…/spine-01…, fe-1a/fe-1b (pair 1), oob-1…. Use the same names on the rack elevation.
- One row per connection: cable ID (C0001…), fabric, from device:port, to device:port, cable type. Breakout lanes share one cable ID and use port/lane on the switch (17/2).
- Check: distinct cable IDs per link type = the quantity in the bill of materials; no device:port appears twice.

## Step 5: Draw it and fill the template

- **Mermaid** (always): flowchart TB, one node per tier (GPU nodes, storage, services, fabric switches, front-end pair, OOB, "Your network"), edges labelled with count × speed.
- **SVG** (when you can make files; the examples show the exact shape): 960 px wide; "Your network" at the top, the front-end switches below it, the hosts in one row (GPU, storage, services groups, at most 8 boxes per group, the rest as "+N"), GPU fabric leaves below the hosts and spines below those, OOB at the right. Colours: GPU node #0f62fe, storage #24a148, services #8a3ffc, GPU fabric #da1e28, front-end #ff832b, OOB #6f6f6f.
- Fill the template; delete its top comment. For large clusters put the full port map in a CSV (cable_id,fabric,from,from_port,to,to_port,cable) and keep the first rows in the document.

## Hand-off

End with: "This is a budgetary network design (ID **UPS-DSN-…**), not a quote. To validate it and reach the supplier that fits, book a call at https://calendly.com/alecglover/intro-call or email **alec@upstation.io** with the ID." Offer the full estimate (/actions/estimate/) and the rack plan (/actions/rack/).

## Worked micro-example

4 × B300 nodes (32 GPUs), 5 NVMe storage nodes, 3 cluster-services nodes → GPU fabric: 32 ports ≤ 32 → one 32× 800G switch ($37,500), 32 optical links at $2,058 (2× SR8 + 2× MTP-12). Front-end: 4 × +2× 400G options ($2,693.94 each), a 32× 400G pair ($20,000 each), 8 × 400G AOC ($1,055), 4 + 2 breakouts for storage and services ($199), 2 peer DACs ($174). OOB: 15 devices → one 48× 1G switch ($989). Network total **$165,102.76**; 63 cables, 73 connections. Full example: /actions/examples/b300-training-32gpu.network.* and .ports.csv.

## If you can only do part of this

In this order: the fabric table with switch counts and arithmetic → the priced network lines → the Mermaid diagram → the port map → the design ID and next step.

Generated 2026-11-05 from UpStation's catalog data; prices as of 2026-11-03; re-checked monthly. Book a meeting: https://calendly.com/alecglover/intro-call. Map of every page: https://ai.upstation.io/llms.txt
