Network Detective

This activity puts into practice the concepts from the Networking Fundamentals lecture. Instead of simulating a network, you will investigate a real one: the network your machine is connected to right now. By the end, you will be able to sit at any Linux machine, answer the question “what is this machine’s network situation?”, and trace a packet from the shell prompt to the public internet and back.

The exercises work best from inside the Arch Linux VM you built in the previous activity, connected from your laptop using the NAT port-forwarding and SSH setup you configured there. Running commands over SSH replicates exactly what you will do on EC2.

Tip

Open two terminal windows: one connected to your Arch VM via SSH (ssh -p 2222 archuser@127.0.0.1), and one on your laptop. Don’t forget the -i flag if you have a custom key name. Running the same commands on both gives you a concrete comparison between “inside the VM” and “on the host machine.”

The Network You Are Working In

There are at least two NAT translations between your VM’s shell prompt and the public internet. Your hypervisor’s NAT adapter assigns the VM a private address (likely 10.0.2.15) and translates its outbound traffic to your laptop’s address. Your laptop is itself behind your home or campus router, which performs a second translation before packets reach the internet. This is a hierarchical topology: a point-to-point link from the VM to the hypervisor’s virtual router, which connects to a star network of devices on your home network, which connects upstream to the ISP. Tracing the packet path through all of those hops is one of the things this activity is designed to show you.

What You Will Need

• Your Arch Linux VM from the previous activity (with networking and SSH configured)

Where To Run Commands

• Unless a step explicitly says host machine, run commands in the VM terminal (your SSH session to ssh -p 2222 archuser@127.0.0.1).
• Use the host machine terminal only for steps that explicitly call it out (for example, external nmap against forwarded port 2222).
• Do not expect host and VM networking output to match exactly. The host usually has extra interfaces (Wi-Fi/Ethernet, VPN, Docker/Hypervisor adapters), while the VM usually has a much simpler table.

Windows Guidance

For this activity, Windows users should run the shell commands in WSL so the Linux commands match the instructions. PowerShell equivalents exist for some commands, but not all behavior and output formats are the same, which makes troubleshooting inconsistent.

Tool Preflight

Before starting, verify required tools are installed where you will run commands.

• On the Arch VM:

  sudo pacman -S --needed iproute2 iputils curl traceroute bind nmap openssl

• On host machine (only needed for host-side nmap steps):

  • macOS:

    brew install nmap

  • Ubuntu/Debian (or WSL Ubuntu):

    sudo apt update && sudo apt install -y nmap traceroute dnsutils

  • Windows without WSL:

    • Install Nmap from nmap.org
    • Run host-side commands in PowerShell only where explicitly noted

---

Layer 1-3: Link, Address, and Neighbors

Before diagnosing anything else, establish that the machine has a physical link, an IP address, and a populated ARP cache. These steps cover layer 1 (is the interface up?), layer 2 (can the machine resolve its gateway’s MAC address?), and layer 3 addressing (what IP and subnet is this machine on?). Layer 3 reachability, meaning whether traffic can actually flow to another host, is tested in the next section using ping and the routing table.

1. Check whether the network interface has a link:

   ip link show

   Every interface has an administrative state and an operational state. In ip link output, confirm your main interface (likely enp0s3 or similar) shows state UP for operational link. If it shows state DOWN, no higher-layer tools will work: the cable is disconnected, the virtual NIC is unconfigured, or the interface was manually brought down.

2. Show all network interfaces and their IP addresses:

   ip addr show

   You will see at least two interfaces: lo (loopback, always 127.0.0.1) and one or more real interfaces (likely enp0s3 or similar). Focus on the real interface.

   Look at the output and identify:

   • Interface name
   • IP address (the number after inet, before the /)
   • Prefix length (the number after the /)
   • Broadcast address (the number after brd)

3. Compare the VM’s address to your host machine’s address. They are likely on different subnets because the hypervisor’s default NAT networking creates its own virtual network for each VM.

   Host-side quick checks:

   • macOS: ifconfig
   • Linux: ip addr show
   • Windows PowerShell: ipconfig

4. Identify your subnet. Given a prefix length of /24, the network has 254 usable addresses. Given /25, it has 126. Given /16, it has 65,534.

   Which private range does your address fall in?

   • 10.0.0.0/8
   • 172.16.0.0/12
   • 192.168.0.0/16

5. Inspect the layer-2 neighbor (ARP) table:

   ip neighbor show

   This is the ARP cache. For every IP address the VM has communicated with on the local subnet, the cache stores the corresponding MAC address so that frames can be delivered correctly at layer 2. You should see at least one entry for the hypervisor gateway.

   A typical entry looks like:

   10.0.2.2 dev enp0s3 lladdr 52:54:00:12:35:02 REACHABLE

   The state field tells you how fresh the mapping is:

   • REACHABLE: confirmed recently via ARP reply
   • STALE: not confirmed recently but still usable; the kernel will send a unicast probe to revalidate it on next use
   • DELAY: a packet was sent to a stale entry and the kernel is waiting a short time (usually a few seconds) for a confirmation before sending a full ARP probe; this is a normal transient state between STALE and REACHABLE
   • FAILED: ARP request was sent but no reply came back; the host is unreachable at layer 2

   A FAILED entry for your default gateway is an important signal during troubleshooting: it means the layer-2 path is broken (wrong VLAN, bad cable, or gateway down) even if the IP configuration looks correct.

Think About It

Your ARP table shows an entry for the hypervisor gateway (10.0.2.2) but nothing for 8.8.8.8. ARP is a broadcast protocol: a device sends “who has this IP?” to the entire local network and waits for a reply. Why does that mechanism mean you can never get an ARP entry for a public server like 8.8.8.8?

---

DHCP: Where Your Address Came From

You have an IP address because systemd-networkd requested one from the hypervisor’s built-in DHCP server when the VM booted. DHCP hands out more than just an address; it also provides the default gateway and a DNS server. This section shows you how to see what DHCP actually provided versus what your system is configured to use.

1. Check which network interfaces systemd-networkd is managing:

   networkctl

   You will see a table of interfaces. The enp0s3 (or similar) interface should show routable and configured, confirming that systemd-networkd received a DHCP lease and applied it.

2. Inspect the DHCP lease details for your interface:

   networkctl status enp0s3 # substitute your interface name if different

   In the output, find:

   • Address: the IP and prefix length assigned by DHCP
   • Gateway: the default gateway DHCP provided
   • DNS: the DNS server address DHCP offered (likely 10.0.2.3, the hypervisor’s internal resolver)

3. Now compare that DNS server to what your system actually uses for name resolution:

   cat /etc/resolv.conf

   You will see 1.1.1.1 and 1.0.0.1: Cloudflare’s public resolvers that you set manually during the Arch Linux install. These are different from what DHCP offered.

   This is an important distinction: DHCP offers configuration, but the system only uses it if nothing overrides it. When you wrote /etc/resolv.conf by hand, you overrode the DHCP-provided DNS server.

   How Production Systems Handle This

   On servers where you want DHCP to manage DNS automatically, you use systemd-resolved and symlink /etc/resolv.conf to the stub resolver (as in the optional step at the end of the Arch install). systemd-resolved reads the DNS server from the DHCP lease and keeps /etc/resolv.conf up to date. On Ubuntu servers, Netplan and systemd-resolved handle this for you.

4. Look at the persistent network configuration file you created:

   cat /etc/systemd/network/20-wired.network

   This is the file that tells systemd-networkd to use DHCP on this interface. The ip addr and ip route commands you have been running show the current state derived from this configuration (the transient values that disappear on reboot). The .network file is the persistent source of truth that recreates that state at every boot.

Think About It

Production servers (web servers, databases, DNS servers) typically have their addresses assigned via DHCP reservations rather than static IP configuration in the OS. A reservation is a DHCP rule that always assigns the same IP to a specific MAC address. What operational advantage does a DHCP reservation have over hard-coding the IP directly in the server’s network configuration file?

---

Routing and the Default Gateway

Knowing your IP address tells you where you are. The routing table tells you where traffic goes.

1. Display the routing table:

   ip route show

   You will see at least two important kinds of entries:

   • A default via entry: traffic for any unknown destination goes to this IP (your default gateway).
   • A direct-link route for your subnet: traffic to addresses in your local subnet is delivered directly, without going through the gateway. This route is often shown with scope link.

   You may also see additional routes (for example, extra scope link entries, host-specific routes, or routes from VPN/container adapters). That is normal. For this activity, focus on identifying the default route and the connected subnet route.

   Record:

   • Default gateway address
   • Interface used for the default route

   Why is the gateway .2 and not .1?

   The hypervisor’s NAT virtual network reserves .1 for the host machine itself: from inside the VM, 10.0.2.1 reaches your laptop directly. That leaves .2 as the first address available for the gateway, and .3 for the built-in DNS server. The normal subnet rules still apply: .0 is the network address and .255 is the broadcast address, so neither can be assigned to a host. The range does start at .1, but .1 is already spoken for.

   Optional host comparison:

   • macOS: route -n get default
   • Linux: ip route show
   • Windows PowerShell: Get-NetRoute -AddressFamily IPv4

2. Can you reach the gateway?

   ping -c 4 <your-gateway-address>

   A response confirms layer 1 through 3 are working between the VM and its gateway (the hypervisor’s virtual router). No response suggests a networking problem in the VM configuration.

3. Can you reach the public internet by IP address, bypassing DNS?

   ping -c 4 8.8.8.8

   If this works, the gateway is routing traffic outbound through NAT.

4. Does DNS work?

   ping -c 4 google.com

   If the previous step worked but this fails, the result is ambiguous: DNS might be broken, or the target might simply not reply to ICMP from your path.

   If this fails, try a different hostname (for example, oregonstate.edu) and then run dig google.com to confirm whether name resolution is working.

---

Tracing the Path to the Internet

ping tells you whether a host is reachable. traceroute shows the path packets take to get there, by sending probes with progressively higher TTL values and collecting the ICMP “time exceeded” replies from each router along the way.

Running traceroute from inside the VM is a useful way to understand how hypervisor NAT works and why it makes certain tools behave unexpectedly. The hypervisor’s NAT adapter is implemented as a userspace network stack rather than real kernel-level NAT, and that distinction changes the behavior of traceroute in two different ways depending on which protocol it uses.

1. Run the default UDP traceroute from inside the VM:

   traceroute 8.8.8.8

   You will see one real hop (the hypervisor gateway at 10.0.2.2), then * * * for every subsequent hop until the trace stops at 30 hops. The hypervisor’s NAT forwards UDP outbound, but when real internet routers send back ICMP “time exceeded” messages (the mechanism traceroute depends on), the hypervisor’s userspace stack swallows them instead of relaying them to the VM. The VM never sees them.

2. Try TCP traceroute:

   sudo traceroute -T -p 443 8.8.8.8

   You will see only one hop: the destination itself. TCP traceroute also fails to reveal the path, but for a different reason: the hypervisor does not just forward TCP connections through NAT, it acts as a transparent TCP proxy. It intercepts the SYN, establishes the real TCP connection to 8.8.8.8 on your behalf, and bridges the data. From the VM’s perspective, the remote host answered directly with no intermediate routers, because slirp was already the remote end. There are no TTL-decrement hops to observe.

3. Run traceroute from your host machine where it works correctly:

   On your host terminal (not inside the VM):

   traceroute 8.8.8.8 # tracert on Windows

   Your host goes through real kernel-level NAT, which is transparent to traceroute. You should see the full path in 10 to 15 hops.

   Record:

   • How many hops until you reach 8.8.8.8?
   • Where does the address space transition from private to public? At which hop?

   Now compare this to the two outputs you got from inside the VM. The same destination, three completely different results, all caused by where the network stack lives (inside the VM’s view of the hypervisor proxy vs. the host’s real kernel routing).

   Scattered * * * entries within an otherwise-complete trace are normal: some routers are configured not to respond to probes as a matter of policy. An unbroken run of * * * continuing to the end usually means traffic stopped there.

4. Compare a nearby host to a distant one, from the host machine:

   traceroute oregonstate.edu
   traceroute sydney.edu.au

   Does the path to Australia diverge from the path to Oregon State at some point? At which hop? What does that tell you about the structure of the internet?

   Note that you might never reach your destination in a reasonable amount of hops in these cases.

   mtr: a better traceroute

   mtr continuously probes the path and gives you a live view of latency and packet loss at each hop. On your host: brew install mtr (macOS), sudo apt install mtr (Debian/Ubuntu), or sudo pacman -S mtr (Arch). Run it with mtr 8.8.8.8. It has the same hypervisor NAT limitations as traceroute when run inside the VM.

5. Find your own public IP address. Ask a public server to reflect it back, from inside the VM:

   curl -s https://ifconfig.me

   The comparison that reveals NAT is between the VM’s local IP (the address from ip addr show, such as 10.0.2.15) and the public IP that ifconfig.me reflects back. Those two addresses will be different, and that gap is NAT: one or more translators rewrote the source address as your packet traveled toward the internet.

   Do not compare the VM’s public IP to your host machine’s public IP. Both share the same home or campus router, so they exit the internet through the same public address. Seeing the same public IP on both machines is expected and is not a sign that NAT is absent.

   Note that ifconfig.me prefers IPv6 when available, and IPv6 addresses are globally routable without NAT, so the VM’s local IPv6 address and its public address may be identical. To observe IPv4 NAT specifically, force IPv4:

   curl -s -4 https://ifconfig.me

   Compare that result to your local IPv4 address from ip addr show. They should differ, confirming the IPv4 NAT path through the hypervisor and your router.

Think About It

Your VM has a private address (e.g., 10.0.2.15), but curl https://ifconfig.me returned a different public IP. How many separate NAT translations are happening between this VM and the internet? Trace each one.

---

What Is Running on This Machine?

A server exposes services through TCP and UDP ports. ss shows you which ports are open from the inside; nmap shows you what is visible from the outside. The difference between the two tells you what a firewall is blocking.

1. Show all listening TCP sockets with process names:

   sudo ss -tlnp

   Flags: -t TCP, -l listening only, -n numeric (no DNS resolution), -p process info.

   Record every listening port and the process next to it.

2. Check UDP listening sockets:

   sudo ss -ulnp

   UDP has far fewer listening services on a minimal server than TCP. On this VM you may see very little or nothing, depending on which optional services you enabled during setup. If you enabled systemd-resolved, you will see it listening on port 53 (DNS). If you have a DHCP client socket open, it may appear on port 68.

   This contrast is intentional: most application protocols (SSH, HTTP, databases) run over TCP because they need the reliability guarantees. DNS and DHCP often use UDP because they favor speed and simplicity; a single small request and reply, with retries handled in the application layer rather than the transport layer. DNS can also use TCP (for example, large responses or zone transfers).

3. Find sshd. You should see:

   LISTEN  0  128  0.0.0.0:22  0.0.0.0:*  users:(("sshd",pid=...,fd=...))

   0.0.0.0:22 means sshd is listening on all interfaces. If you saw 127.0.0.1:22 instead, it would only accept connections from localhost, which is why SSH from your host machine would fail.

4. Now scan this machine from the outside. On your host machine (not inside the VM), open a terminal and run:

   nmap -sV -Pn -p T:2222 127.0.0.1

   This scans the forwarded host port that maps to SSH on the VM. If this port shows open, the host-to-guest forwarding path is working.

   Port Closed or Filtered?

   If nmap reports the port as closed or filtered, verify your VM still has the NAT forwarding rule from the Arch activity: host 127.0.0.1:2222 to guest :22. Then verify sshd is running in the VM with sudo systemctl status sshd.

5. Scan the VM’s localhost from inside:

   sudo nmap -sV -Pn -p T:1-1024 127.0.0.1

   This scans from within the machine, bypassing upstream network controls such as router ACLs or cloud security groups. Host-local firewall policy can still affect what appears open. Does the output match ss -tlnp? If they differ, what would explain the discrepancy?

How To Interpret Inside vs Outside Results

The ss (inside view) vs. nmap (outside view) relationship is exactly what you will encounter on EC2. ss confirms your service is running and listening. nmap from outside (or a failed browser connection) tells you whether path controls are allowing traffic through. If ss shows the port listening but external connections fail, check the security group, host firewall rules, and whether the service is bound to the expected interface.

Think About It

Under what circumstances would ss show a port as listening but nmap from outside would not see it as open? Name two different possible reasons.

---

DNS Investigation

DNS translates names to addresses. Understanding how your machine resolves names and how to query DNS manually is essential for diagnosing the most common category of “the internet is down” complaints.

1. Confirm which resolver your machine sends queries to:

   cat /etc/resolv.conf

   The nameserver line is the address dig and all other tools will use by default when you do not specify @. On this VM you will see 1.1.1.1 (Cloudflare), the value you set manually. Compare this to the DHCP-offered DNS you saw with networkctl status: your system is bypassing the DHCP-provided resolver entirely.

2. Perform a full DNS lookup and read the output:

   dig google.com

   Identify:

   • ANSWER SECTION: the IP address(es) returned
   • TTL: how many seconds this record can be cached before a fresh query is required
   • Query time: how fast did your resolver respond?

3. Ask two different resolvers the same question:

   dig +short google.com @8.8.8.8
   dig +short google.com @1.1.1.1

   Do you get the same IP? Large sites like Google often return different addresses from different resolvers in different regions, routing users to the nearest data center.

4. Explore CNAME (alias) records:

   dig www.oregonstate.edu
   dig oregonstate.edu

   Look at the ANSWER SECTION. If the response contains a CNAME record, the name is an alias pointing to another name, which is then resolved to an A record. For example:

   www.oregonstate.edu.    300  IN  CNAME  oregonstate.edu.
   oregonstate.edu.        60   IN  A      128.193.4.130

   The resolver followed the chain automatically. CNAMEs are commonly used to point www.example.com at the apex domain, or to delegate a hostname to a CDN provider without exposing the CDN’s IP directly.

   If the response shows only an A record with no CNAME, that hostname is configured directly. Try a few others to find a CNAME chain in the wild.

5. Look up a mail server record:

   dig MX oregonstate.edu

   The MX records show which servers accept email for this domain. The number before the hostname is the priority; lower is preferred. If you have multiple MX records, the sender tries the lowest priority first and falls back if it is unreachable.

6. Do a reverse lookup on your public IP address:

   dig -x $(curl -s -4 https://ifconfig.me)

   This performs a PTR lookup on your actual public IPv4 address, the one your packets arrive from when they reach the internet. PTR records are only registered for public addresses; private addresses like your gateway (10.0.2.2) or local subnet address will return NXDOMAIN because they have no meaning outside the local network and nobody registers DNS for them.

   You may see one of two outcomes. If a PTR record exists, it appears in the ANSWER section and the hostname usually reveals your ISP and sometimes your approximate location (for example, c-73-250-58-101.hsd1.or.comcast.net). On campus or cloud networks it often identifies the institution or provider directly.

   More commonly for residential IPs, you will get NXDOMAIN with an SOA record in the AUTHORITY section instead of a PTR record in the ANSWER section. That SOA is still useful: it names the authoritative server for the reverse zone (the nameserver field) and the administrative contact (the email field, with the first . replaced by @). The ISP’s name is usually visible in both. What it means operationally is that the zone for your IP block is delegated to your ISP, but they chose not to register a specific PTR for your individual residential IP. ISPs typically do register PTR records for their infrastructure and business-grade IPs, which is why traceroute hops sometimes resolve to hostnames and sometimes do not.

7. Simulate broken DNS. Ask for a hostname that does not exist:

   dig nonexistent.example.invalid

   Note the response code (NXDOMAIN). Now ask a real domain but point at a non-existent resolver:

   dig google.com @10.255.255.1

   Note how long the query takes before timing out. This is what “DNS is broken” feels like from an application’s perspective: every network operation that requires a hostname hangs for several seconds before failing.

Think About It

google.com typically has a TTL of 5 minutes or less. What is the operational reason Google prefers a short TTL over something like 86400 seconds (one day)?

---

HTTP and Application-Layer Connectivity

curl reaches into layer 7 and talks directly to a web server. Combined with the tools above, it completes the diagnostic chain from physical link to application response.

1. Fetch just the HTTP headers from a web server:

   curl -I https://oregonstate.edu

   Record:

   • The HTTP status code (200 OK, 301 Moved Permanently, etc.)
   • The Server: header (what web server software is running?)
   • The Strict-Transport-Security: header if present (what does it mean?)

2. Measure request timing broken down by layer:

   curl -o /dev/null -s -w \
     "DNS:     %{time_namelookup}s\nConnect: %{time_connect}s\nTLS:     %{time_appconnect}s\nTotal:   %{time_total}s\nHTTP:    %{http_code}\n" \
     https://oregonstate.edu

   Interpret each line:

   • DNS: time to resolve the hostname (layer 7 service)
   • Connect: time for the TCP three-way handshake (layer 4)
   • TLS: time for the TLS handshake on top of TCP (layers 4-7)
   • Total: wall-clock time for the complete request and response

   Slow DNS shows up in the first number. A slow or distant server shows up in Connect and Total. A TLS misconfiguration shows up in the TLS line.

3. See what happens when you bypass DNS entirely and connect to the IP directly:

   curl -I https://$(dig +short oregonstate.edu A | head -n 1)

   This will fail with a TLS error. The certificate was issued for oregonstate.edu, not for a bare IP address, so the hostname validation fails. To bypass the validation check:

   curl -kI https://$(dig +short oregonstate.edu A | head -n 1)

   The -k flag disables certificate verification. The server responds, but you are no longer verifying the identity of who you are talking to. --resolve solves this properly: it routes the request to a specific IP while still sending the correct Host header and validating TLS against the real hostname:

   curl -I --resolve oregonstate.edu:443:$(dig +short oregonstate.edu A | head -n 1) https://oregonstate.edu

   This time the TLS handshake succeeds because curl is using oregonstate.edu as the hostname for certificate validation, even though it is connecting to the IP directly. You get all the security checks without going through DNS. This matters when you need to test a new server before pointing DNS at it, or when you suspect DNS is returning a wrong IP and want to verify a specific backend directly.

4. Inspect the TLS handshake directly with openssl s_client:

   openssl s_client -connect oregonstate.edu:443 -servername oregonstate.edu

   You will see several hundred lines of output. Look for:

   • Certificate chain: the chain of certificates from the server’s cert up to the root CA
   • subject: the hostname the certificate was issued for
   • notBefore / notAfter: the certificate’s validity window
   • Verify return code: 0 (ok): this line at the end confirms the certificate validated successfully

   If the verify return code is anything other than 0 (ok), you have a TLS problem. The exact numeric code and wording can vary by OpenSSL version and options, but common causes are expired certificates, missing intermediates, and hostname mismatch. These are the same classes of errors curl reports as TLS failures, but openssl s_client shows you where validation failed.

Think About It

A user reports “I can reach 8.8.8.8 but I cannot open any website.” Based on what you practiced in Parts 1 through 6, what is the most likely cause, and what command would you run to confirm it?

---

Network Summary

You have used the full diagnostic toolkit from layer 1 through layer 7. Produce a clean terminal snapshot that captures your machine’s identity and services in one place:

printf "Hostname: %s\nLocal IP: %s\nGateway:  %s\nPublic IP: %s\nDNS:      %s\n\n" \
  "$(cat /etc/hostname)" \
  "$(ip -4 addr show scope global | awk '/inet /{print $2}' | head -1)" \
  "$(ip route show default | awk '{print $3}')" \
  "$(curl -s -4 https://ifconfig.me)" \
  "$(awk '/^nameserver/{print $2; exit}' /etc/resolv.conf)"
echo "Listening TCP services:"
sudo ss -tlnp

Your hostname is the personal identifier here: it is the name you set during the Arch Linux install and it appears in your shell prompt. You should see your local address (likely 10.0.2.x with hypervisor NAT), a different public IPv4 address confirming NAT is working, your manually configured DNS resolver, and at minimum SSH on port 22 in the services table.

Going Further

• Mess with DNS!
• Read man ip-route and man ip-address for the full reference on iproute2 commands. Most of what ifconfig and netstat used to do lives here now.
• Try mtr 8.8.8.8 (install with sudo pacman -S mtr) for a live view of packet loss and latency at each hop along the path. It is more informative than a single traceroute run.
• Use dig +trace google.com to watch the full DNS resolution chain from root servers down to the authoritative nameserver. It shows how delegation actually works.
• Explore /proc/net/ on the VM. Files like /proc/net/tcp, /proc/net/arp, and /proc/net/route are what tools like ss and ip read under the hood. They show that network state on Linux is just files.