Skip to content

wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host - #13758

Open
wido wants to merge 1 commit into
apache:mainfrom
wido:direct-routed-network
Open

wip: Direct Routed (L3) guest networks: route public IPv4/IPv6 to VM from Host#13758
wido wants to merge 1 commit into
apache:mainfrom
wido:direct-routed-network

Conversation

@wido

@wido wido commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Description

This Pull Request adds a new guest network type in which the hypervisor performs L3 routing for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance receives a public IPv4 address as a /32 and/or an IPv6 address as a /128, with a shared, host-independent gateway (169.254.0.1 and fe80::1) that every hypervisor carries on the network's bridge. All addressing reaches the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on the host (FRR, BIRD, ...) advertises the addresses to the fabric and is deliberately out of scope for CloudStack.

Management server

  • GuestType.L3; the guest_type column is char(32), so no schema change.
  • Offering validation: UserData via ConfigDrive is mandatory, Dns optional but ConfigDrive-only, SecurityGroup permitted (now allowed for L3 alongside Shared), Dhcp rejected as not supported and not needed. Network mode, specifyVlan and VPC use are rejected.
  • DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the Shared-network address lifecycle. canHandle() selects on the offering's guest type alone; design() produces a Native broadcast domain with no isolation id. After allocation the NicProfile is forced into host-route form, which is also the signature by which the agent and ConfigDrive recognise these NICs.
  • createNetwork treats L3 like Shared for the subnet: explicit IP range mandatory, vlan/IP-range row created at network creation, IPv6 accepted without the /64 restriction, aclType Account.
  • Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share one host routing table and one fabric, so an overlap is an address conflict. The IPv6 vlan check was already zone-wide.

ConfigDrive

  • Network data is always generated for a direct routed NIC; the historical gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR but would leave these NICs with no addressing at all. Route generation itself is unchanged: cloud-init detects an IPv4 gateway inside 169.254.0.0/16 and sets on-link on the rendered route by itself.

KVM agent

  • One uplink-less bridge per network, brdr-, created and removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove a bridge still in use). The bridge carries the gateway addresses, forwarding and strict rp_filter; separate bridges make isolation between networks topological rather than a filtering concern.
  • BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs the existing modifymacip.sh hook per NIC to install the static neighbour entry and host route, regardless of the host-wide EVPN property, whose meaning is unchanged.

The design document, including the decision log and the verification notes behind each choice, is added under docs/design/.

This implements issue #12210

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Testing is still ongoing on real hardware and this PR currently (July 2026) exists to gain initial feedback.

… Instances

Adds a new guest network type in which the hypervisor performs L3 routing
for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance
receives a public IPv4 address as a /32 and/or an IPv6 address as a /128,
with a shared, host-independent gateway (169.254.0.1 and fe80::1) that
every hypervisor carries on the network's bridge. All addressing reaches
the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on
the host (FRR, BIRD, ...) advertises the addresses to the fabric and is
deliberately out of scope for CloudStack.

Management server:
- GuestType.L3; the guest_type column is char(32), so no schema change.
- Offering validation: UserData via ConfigDrive is mandatory, Dns optional
  but ConfigDrive-only, SecurityGroup permitted (now allowed for L3
  alongside Shared), Dhcp rejected as not supported and not needed.
  Network mode, specifyVlan and VPC use are rejected.
- DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the
  Shared-network address lifecycle. canHandle() selects on the offering's
  guest type alone; design() produces a Native broadcast domain with no
  isolation id. After allocation the NicProfile is forced into host-route
  form, which is also the signature by which the agent and ConfigDrive
  recognise these NICs.
- createNetwork treats L3 like Shared for the subnet: explicit IP range
  mandatory, vlan/IP-range row created at network creation, IPv6 accepted
  without the /64 restriction, aclType Account.
- Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share
  one host routing table and one fabric, so an overlap is an address
  conflict. The IPv6 vlan check was already zone-wide.

ConfigDrive:
- Network data is always generated for a direct routed NIC; the historical
  gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR
  but would leave these NICs with no addressing at all. Route generation
  itself is unchanged: cloud-init detects an IPv4 gateway inside
  169.254.0.0/16 and sets on-link on the rendered route by itself.

KVM agent:
- One uplink-less bridge per network, brdr-<network id>, created and
  removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove
  a bridge still in use). The bridge carries the gateway addresses,
  forwarding and strict rp_filter; separate bridges make isolation between
  networks topological rather than a filtering concern.
- BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs
  the existing modifymacip.sh hook per NIC to install the static neighbour
  entry and host route, regardless of the host-wide EVPN property, whose
  meaning is unchanged.

The design document, including the decision log and the verification notes
behind each choice, is added under docs/design/.
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.40%. Comparing base (4f11707) to head (bf3d8a3).

❗ There is a different number of reports uploaded between BASE (4f11707) and HEAD (bf3d8a3). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (4f11707) HEAD (bf3d8a3)
unittests 1 0
Additional details and impacted files
@@              Coverage Diff              @@
##               main   #13758       +/-   ##
=============================================
- Coverage     19.65%    3.40%   -16.25%     
=============================================
  Files          6368      488     -5880     
  Lines        574881    41958   -532923     
  Branches      70351     7927    -62424     
=============================================
- Hits         112970     1429   -111541     
+ Misses       449639    40329   -409310     
+ Partials      12272      200    -12072     
Flag Coverage Δ
uitests 3.40% <ø> (-0.01%) ⬇️
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant