📌 Overview

This project applies the Secure Networking domain of the AZ-500 (Azure Security Engineer Associate) exam — at 20–25%, the largest single domain on the exam — to a real environment rather than a standalone lab. The hub-spoke topology itself isn't new: it's the same VNets, VPN Gateway, and hybrid connectivity built for the AZ-700 project, renamed and reused. What's new is everything layered on top of it: NSGs and ASGs, a centralized Azure Firewall, user-defined routes, and a Network Watcher validation pass that ended up being the most instructive part of the whole build.

Environment:

  • Hub VNet (JPVNetHub) with a Virtual Network Gateway and a centralized Azure Firewall
  • Two spoke VNets (JPVNetSpoke1, JPVNetSpoke2) peered to the hub, each running one workload VM
  • Site-to-Site VPN to an on-prem FortiGate, Point-to-Site VPN authenticated through Microsoft Entra ID
  • Three VMs (JPAZVM11 hub, JPAZVM12 Spoke1, JPAZVM13 Spoke2), each behind its own NSG and Application Security Group

🔧 Objectives

  • Apply defense-in-depth network controls — NSGs/ASGs, centralized firewall, UDRs — to an existing topology instead of a disposable lab
  • Get hands-on with the specific AZ-500 Secure Networking skills: segmentation, egress control, VPN connectivity, and Network Watcher diagnostics
  • Validate every control live rather than assuming configuration equals enforcement
  • Produce a portfolio-ready case study, including the troubleshooting that didn't go as expected

🔒 Segmentation: NSGs & ASGs

The first surprise of this project showed up before any new resources were even deployed: peering two VNets together does not segment them. Azure's platform-default AllowVnetInBound rule (priority 65000) quietly allows everything inside the peered address space, and it can't be deleted — only out-prioritized. Phase 1 added an explicit Deny-VirtualNetwork-Inbound rule (priority 4000) and Deny-Internet-Inbound (4010) to all three NSGs, then layered narrow allows above them for exactly what's needed: RDP and ICMP from on-prem, plus RDP from the hub as a jump-host path.

JPNSGSpoke1 inbound security rules showing Deny-VirtualNetwork-Inbound and Deny-Internet-Inbound ranked above the platform default AllowVnetInBound

JPNSGSpoke1's full rule set, platform defaults included — the explicit denies at 4000/4010 sit well above AllowVnetInBound at 65000. (Click to enlarge.)

Application Security Groups (JPASGHub, JPASGSpoke1, JPASGSpoke2) sit between the NSG rules and the VMs so rules reference a role instead of a hardcoded IP. With one VM per role today that's not saving much, but it means the rules don't need to change if a role ever grows past one VM.

JPASGSpoke1 Application Security Group overview showing it tied to JPAZVM12-nic (10.1.0.4), attached to JPAZVM12

JPASGSpoke1's membership — tied to JPAZVM12's NIC, the same object referenced as a destination in the NSG rules above. (Click to enlarge.)

🔥 Centralized Firewall & Egress Control

Segmentation handles north-south trust at the NSG layer, but spoke-to-spoke traffic over VNet peering has no inspection point in the path at all by default — peering just connects, it doesn't route through anything. Phase 2 deployed an Azure Firewall (Standard SKU) into the hub and built a policy with a network rule permitting HTTPS egress from both spokes and an application rule allowing Windows Update by FQDN tag, with Threat Intelligence in Alert mode to start.

JPAZFW01-Policy rule collections showing rcg-spoke-egress with net-allow-required (network, priority 100) and app-allow-updates (application, priority 200)

The policy's two rule collections: baseline HTTPS egress, and Windows Update by FQDN tag. (Click to enlarge.)

JPAZFW01-Policy Threat Intelligence setting showing Alert Only mode

Threat Intelligence set to Alert Only — validate against real lab traffic before tightening to Alert and Deny. (Click to enlarge.)

⚠️ The UDR Precedence Gotcha

This phase had its own gotcha, and it's the one worth remembering: Azure resolves routing by longest-prefix match, and a user-defined route only beats a system route at equal or greater specificity. The automatic peering route between the two spokes and the explicit UDR pointing spoke-to-spoke traffic at the firewall were originally both /16 prefixes — so adding only a 0.0.0.0/0 → Firewall default route wasn't enough. Without an explicit /16 → Firewall route on Spoke1's route table (and the mirror on Spoke2's), the automatic peering route kept winning and spoke-to-spoke traffic silently bypassed the firewall entirely. Nothing errored. Nothing warned about it. The only way to catch it was to check Next Hop and see VirtualNetwork instead of VirtualAppliance — which is exactly what Phase 4 is for.

Those routes have since been tightened to the spokes' actual, resized address spaces (/23 instead of /16), so they now win on specificity alone rather than relying on the UDR-over-system tiebreak — same underlying lesson either way.

RT-Spoke1 routes showing to-Internet (0.0.0.0/0) and to-Spoke2 (10.2.0.0/23), both Virtual Appliance to 10.0.2.4

RT-Spoke1: both routes pointing at the firewall's private IP. (Click to enlarge.)

RT-Spoke2 routes showing to-Internet (0.0.0.0/0) and to-Spoke1 (10.1.0.0/23), both Virtual Appliance to 10.0.2.4

RT-Spoke2, mirrored. (Click to enlarge.)

Effective routes on JPAZVM12-nic showing the User route to 10.2.0.0/23 Active via Virtual appliance, and the Default 0.0.0.0/0 Internet route marked Invalid

The UDR actually winning in practice — and a nice side effect: the platform's own 0.0.0.0/0 route shows as Invalid, superseded by the custom one. (Click to enlarge.)

A related gap surfaced later, while deploying Azure Bastion into the hub for secure, browser-based VM access during a same-night troubleshooting push. Unlike the spoke UDR issue above — a precedence contest the explicit route eventually won — this wasn't a contest at all: JPAZVM11's subnet had no route table associated with it whatsoever. Effective Routes showed its 0.0.0.0/0 going straight to next hop Internet, bypassing the firewall entirely. The firewall policy's egress rule had also only ever been scoped to the two spoke address ranges, never the hub's own subnet — a second, independent gap stacked on top of the missing route. Fixed both: added RT-Hub (0.0.0.0/0 → Virtual Appliance → 10.0.2.4), associated only to the hub's VM subnet — deliberately not to GatewaySubnet, AzureFirewallSubnet, or the Bastion subnet, each of which needs its own direct path rather than being forced through the firewall — and extended the firewall policy's allowed sources to include the hub.

In hindsight, this should have been part of the original design. A hub-and-spoke model that centralizes egress inspection only works if every subnet that needs internet actually routes through the firewall — the spokes had that from day one, and the hub's own VM subnet was the one left out.

🔐 Secure VPN Connectivity

No new build here — the Site-to-Site (custom IPsec/IKE policy, AES256/SHA256/DH Group 14/PFS2048) and Point-to-Site (Entra ID authentication, OpenVPN) connections already existed from the AZ-700 project. This phase was about reviewing what was already there and mapping it explicitly to this domain's requirements: a custom cipher suite instead of accepting Azure's default proposal, and identity-based P2S auth instead of a shared certificate.

JPVPNGW connections showing JPAZ-to-JPHome, Site-to-site (IPsec), status Connected

The Site-to-Site connection to the on-prem FortiGate, Connected. (Click to enlarge.)

JPAZ-to-JPHome connection configuration showing custom IPsec/IKE policy: AES256 encryption, SHA256 integrity, DHGroup14, PFS2048

The custom cipher suite instead of Azure's default proposal. (Click to enlarge.)

JPVPNGW Point-to-site configuration showing address pool 172.16.20.0/24, OpenVPN (SSL) tunnel type, Microsoft Entra ID authentication

Point-to-Site: Entra ID authentication instead of a shared certificate. (Click to enlarge.)

Azure VPN app registration in Microsoft Entra ID showing status Activated

The Entra ID side of P2S auth — the Azure VPN Client app registration, active. (Click to enlarge.)

🖥️ Azure Bastion: A Browser-Based Workaround

The Point-to-Site VPN above works — but not reliably from my actual work laptop. RDP over P2S from my work laptop connected fine, ping succeeded, and then RDP itself timed out every time. ping -f -l 1400 confirmed a Path MTU black hole: something on the tunnel path was silently dropping oversized packets instead of returning the ICMP message that tells a client to shrink them, so OpenVPN's encapsulation overhead was quietly killing RDP's larger negotiation packets while small ICMP echoes sailed through. On my personal computer, RDP over P2S VPN worked perfectly.

Digging further surfaced one factor: Global Secure Access, a Zero Trust client enrolled on the my work laptop, independent of anything in this lab's design. A workaround found on a Microsoft forum (editing the Azure VPN Client's exclude-routes profile) didn't resolve it, and a FortiClient SSL-VPN double-hop through my home firewall ran into its own dead end.

With a deadline, I deployed Azure Bastion into the hub VNet as a workaround rather than a permanent architecture change. Bastion only needs outbound HTTPS from the browser to portal.azure.com — no VPN client, no MTU-sensitive tunnel, nothing for Global Secure Access to intercept. I chose Basic SKU (~$0.19/hr) over the free Developer SKU for the sake of a deadline-night deployment that was known to work end-to-end; Developer SKU also doesn't support VNet peering, so it could only have reached the hub VM directly regardless. Bastion gave direct browser-based RDP into JPAZVM11 (hub); reaching the spoke VMs meant RDPing into the hub via Bastion first, then using Remote Desktop Connection (RDP) from inside that session to JPAZVM12 / JPAZVM13.

🔍 Visibility & Validation

This phase deploys nothing — it's a validation pass against the earlier phases, run through five Network Watcher tools: Effective Security Rules, Next Hop, IP Flow Verify, NSG Diagnostics, and Connection Troubleshoot.

Flow logs were evaluated for this phase first, and dropped. Both NSG Flow Logs and Virtual Network Flow Logs produce deeply nested JSON with no simple allow/deny field — flow state gets folded into a single value instead — and critically, they never capture the IP Flow Verify or NSG Diagnostics tests, since neither of those tools sends a real packet; they're rule-evaluation simulations. Network Watcher's other diagnostics validate the same enforcement more directly and read far better live, so flow logging was left out of the build rather than bolted on as a sixth method.

Network Watcher Effective Security Rules tool targeting JPAZVM12, network interface JPAZVM12-nic, showing Associated NSGs: JPNSGSpoke1

Targeting JPAZVM12-nic — the tool confirms its associated NSG before showing the combined rule set. (Click to enlarge.)

Effective security rules on JPAZVM12-nic showing the combined inbound and outbound rule set, with Deny-VirtualNetwork-Inbound and Deny-Internet-Inbound ranked above AllowVnetInBound

The combined rule set as actually applied on the NIC — not just what the NSG's own definition says. (Click to enlarge.)

IP Flow Verify configured for JPAZVM12, inbound TCP, local port 3389, remote IP 192.168.20.10

Simulating inbound RDP from an on-prem address against JPAZVM12. (Click to enlarge.)

IP Flow Verify result showing Access allowed, matched by Allow-RDP-From-Home on JPNSGSpoke1

Inbound RDP from an on-prem source: Allowed, matched by Allow-RDP-From-Home. (Click to enlarge.)

NSG Diagnostics configured for JPAZVM12, outbound traffic from 10.1.0.4 to 10.2.0.4 on port 3389

Tracing outbound Spoke1-to-Spoke2 RDP traffic through every NSG in the path. (Click to enlarge.)

NSG Diagnostics result showing Traffic status Allowed across every NSG evaluated for outbound Spoke1 to Spoke2 traffic

NSG Diagnostics names every NSG in the path and the specific rule that matched — useful when a result needs explaining, not just confirming. (Click to enlarge.)

Next Hop result from JPAZVM12 to 10.2.0.4 showing Next hop type VirtualAppliance, IP address 10.0.2.4

Spoke-to-spoke traffic routing through the firewall, confirmed directly. (Click to enlarge.)

🚧 The Three-Layer RDP Mystery

The last test in the validation plan was Connection Troubleshoot — real traffic, not a simulation — from JPAZVM12 (Spoke1) to JPAZVM13 (Spoke2) on port 3389. By design this was expected to come back Unreachable, since Spoke2's NSG only allows RDP from the hub subnet and on-prem, not from Spoke1 directly. It did:

Connection Troubleshoot showing Unreachable, 316 probes sent, 316 failed

Connection Troubleshoot: Unreachable, 316/316 probes failed — the expected result. (Click to enlarge.)

That was the expected result — right up until I modified Allow-RDP-From-Hub to add JPAZVM12's IP as an explicit source, intending to test opening that one host through. The NSG rule now clearly allowed it. Connection Troubleshoot still said Unreachable, 316/316 failed.

Troubleshooting approach: work outward from the NSG, since that's the layer I'd just changed. First, confirm outbound wasn't the problem — the Spoke1 NSG's outbound rules were untouched, just the platform defaults:

NSG outbound rules showing only the default AllowVnetOutBound, AllowInternetOutBound, and DenyAllOutBound rules

Spoke1's outbound rules, unmodified — the default AllowVnetOutBound already covers this. (Click to enlarge.)

The VirtualNetwork service tag includes peered VNets, so outbound from Spoke1 to Spoke2 was never the issue. With NSG outbound on Spoke1 allowed, NSG inbound on Spoke2 allowed (after my edit), and still 100% probe failure, there was exactly one checkpoint left on the path: Azure Firewall. UDRs route all Spoke1↔Spoke2 traffic through the firewall, and its policy only had a network rule for TCP 443 — nothing for 3389. Azure Firewall default-denies anything with no matching rule, so it was silently dropping every SYN regardless of what either NSG said.

Adding a network rule fixed it — permitting TCP 3389 from Spoke1 to Spoke2:

Firewall policy rule collection showing the new net-allow-rdp rule, permitting TCP 3389 from 10.1.0.0/24 to 10.2.0.0/24

The missing firewall rule, added — TCP 3389, Spoke1 to Spoke2, Allow. (Click to enlarge.)

Re-running Connection Troubleshoot: Reachable, 316/316 probes passed.

Connection Troubleshoot now showing Reachable, 316 probes sent, 0 failed

Reachable — once both the NSG and the firewall agreed. (Click to enlarge.)

Except the full diagnostic (Connectivity + NSG diagnostic + Next Hop + Port Scanner, all in one pass) told a more complete story — four checks passed clean, and a fifth came back stuck:

Full diagnostic results: Connectivity Reachable, Outbound and Inbound NSG diagnostic Allow, Next hop Success via Virtual Appliance, and Destination port accessible showing Timeout

Everything upstream passes — but "Destination port accessible" times out. (Click to enlarge.)

"Destination port accessible" is a different question than "can a packet get there" — it's asking whether something is actually listening on 3389 from the destination VM's own perspective. A timeout there, with everything upstream green, points past Azure entirely and into the guest OS. Sure enough: these VMs had never been RDP'd into, so Windows Defender Firewall on JPAZVM13 was still blocking RDP at its default, out-of-the-box state — a layer Network Watcher can diagnose the symptom of but can't reach to fix.

That's three independent checkpoints in one troubleshooting pass — NSG, Azure Firewall, and guest OS firewall — each one caught something real, each one needed a different tool and a different fix. I scripted the last one using Invoke-AzVMRunCommand so it can flip fDenyTSConnections and enable the Windows Firewall's Remote Desktop rule group over the control plane, without needing RDP to already work to get in and fix it:

Set-ItemProperty -Path "HKLM:\System\CurrentControlSet\Control\Terminal Server" `
  -Name "fDenyTSConnections" -Value 0
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

📈 Results

  • Earlier phases confirmed enforcing via live diagnostics, not just reviewed configuration — Effective Security Rules, Next Hop, IP Flow Verify, and NSG Diagnostics all returned expected results
  • Traced a real "it should work but doesn't" failure through three independent enforcement layers (NSG → Azure Firewall → guest OS) to its actual root cause at each step, rather than guessing
  • Flow logs evaluated and deliberately left out of the final build, in favor of Network Watcher's diagnostic tools, which cover the same ground and read far better live
  • VPN connectivity from the AZ-700 build reviewed and explicitly mapped to AZ-500's Secure Networking requirements, with no changes needed
  • A reusable Run Command script added to the project for enabling RDP on fresh VMs ahead of a live demo, without needing the network path open first

📝 Notes / Lessons Learned

  • Passing one layer doesn't mean the request passed every layer — an NSG allow, a UDR pointing at the firewall, and the firewall's own rule set are each an independent decision point. "I allowed it" only answers one of those three questions at a time
  • Azure Firewall fails closed and silently — with no matching rule, it just drops the packet, no NSG-style "Deny" result to point at. The only visible symptom from the client side is a timeout indistinguishable from a dozen other causes, which is why a tool that sends real traffic is worth running even after the simulated tools say everything's fine
  • The UDR precedence issue is the kind of bug that doesn't announce itself — two routes at the same prefix length, one silently winning over the other, with no error on either side. Worth validating routing explicitly instead of assuming a UDR took effect just because it exists
  • Not every "bug" during testing is actually in the lab. Network Watcher's IP Flow Verify started failing against JPAZVM12 and JPAZVM13 with TargetNicResourceBelongToDifferentVm, quoting two different casings of the same resource group — rg-az500-doit-cus for the NIC, RG-AZ500-DOIT-CUS for the VM. Azure resource group names are case-insensitive and can't actually exist twice like that; this turned out to be a documented Azure Resource Manager quirk where different resource providers (Microsoft.Compute, Microsoft.Network) can independently cache a resource's casing, and Microsoft's own guidance is to always compare names case-insensitively rather than expecting the platform itself to stay consistent. The actual fix needed no redeploy: de-associating and re-associating the NIC's Application Security Group forced a write on the specific resource holding the stale reference, which resolved the casing cleanly