Why are you here? Because someone told you "the internet is slow" and you believed you could fix it in an afternoon. You cannot. Out of the jokes, QoS is one of those things that everybody talks about, nobody configures, and when you finally do it you discover that half of the documentation on the internet is from 2015 and the other half is wrong.

This document covers how we fixed the WAN congestion on the Netgate 6100. It is written as a story and not as a recipe, because if you only copy the final config you will not understand why it is like that, and the day it breaks you will be screwed.

If you only want the working config and you do not care about my suffering, jump directly to The Final Solution. I will judge you, but the config is there.

The Problem

Users complaining. The classic ones:

• "Teams cuts off when I talk"
• "The internet goes slow in the afternoon"
• "It works fine at 8am"

And the classic answer from us: can't reproduce. Because of course, when you test it at 6pm from your desk, everything is fine.

What we actually had, measured and not guessed:

• ~9% packet loss at the firewall during peak events
• Latency going from 14 ms idle to 545 ms under load
• Waveform bufferbloat grade: C/D
• WAN saturating to 1 Gbps and staying there for ~20 seconds every time someone pulled an LLM model from Hugging Face

That last one was the trigger. Somebody downloads a 70 GB model, and for 10 minutes the entire company has garbage calls.

The name of this disease is bufferbloat. Short version: the firewall has a big dumb FIFO buffer on the WAN. TCP fills it because TCP always fills whatever you give it. Once it is full, latency explodes and packets get dropped randomly — including the Teams audio packets that actually matter.

The Search For Solutions

Before touching anything, measure. If you configure QoS without measuring first you are just doing astrology with cables.

Verify the physical layer

Because if you have a broken cable and you deploy QoS, you have a broken cable and QoS.

Checked on the Netgate WAN interface and the SX550X uplink:

• In/out errors: 0/0
• Collisions: 0
• FCS errors, CRC errors, late collisions: 0
• Received/transmitted pause frames: 0 ← this one means the link was never congested at L2

Everything clean. Cabling and optics are not the problem. Good, one suspect eliminated.

Find where the queue actually forms

This is the part that everybody gets wrong, so pay attention:

QoS only does something where a queue forms. A queue only forms where ingress rate > egress rate.

Our topology:

Internet ─[1 Gbps]→ Netgate ─[10 Gbps]→ SX550X ─[10 Gbps ring]→ DGS-1510 ─[1 Gbps]→ user

Everything internal is 10 Gbps. The WAN is 1 Gbps. So:

• The SX550X has 10G in and 10G out → no queue → QoS there does absolutely nothing
• The DGS-1510 same story
• The Netgate has 10G on the LAN side and 1G on the WAN side → this is the only place in the whole network where a queue exists

So all the QoS goes on the firewall. If someone comes to you saying "let's configure QoS on the switches", send them this paragraph.

Check if anything marks DSCP

The textbook design says "trust the DSCP marks from the endpoints". Cute. Let's verify.

Two packet captures:

• On the firewall, Diagnostics → Packet Capture, ~1000 packets. Filter in Wireshark: ip.dsfield.dscp != 0 → 2 packets matched (noise)
• On a laptop, Wireshark, ~13000 packets. Same filter → 0 packets matched

Conclusion: nothing in this network marks DSCP. Not the endpoints, not the applications, not the upstream. So any design based on trusting DSCP would match exactly nothing.

This killed a whole branch of the design before we wasted time on it. Do this capture. It takes 10 minutes and saves you days.

Before You Touch Anything: The Rollback

You are going to deploy on a production firewall with 200 users behind it. Set up your escape routes first.

Serial console

Verify it before you need it. Discovering that your console cable is charge-only after you locked yourself out of the GUI is a genuinely terrible afternoon.

The Netgate 6100 has a USB Micro-B console port. From Linux:

sudo picocom -b 115200 /dev/ttyUSB0

Press Enter, you get the pfSense console menu. Do not select anything. Exit with Ctrl-A Ctrl-X.

The option you care about in an emergency is 15) Restore recent configuration. That is your "oh shit" button, and it works without internet, without DNS, without the GUI.

The three backup layers

Config History is the important one and it is already enabled by default. I did not know it existed until I went looking. Now you know, so you have no excuse.

Attempt 1: FQ-CoDel On The WAN

The official Netgate recipe for bufferbloat. Two limiters, FQ-CoDel scheduler, one floating rule. No classification, no HFSC, no DSCP.

The idea: put a cap slightly below your real ISP speed, so the queue forms on your firewall (where you can manage it) instead of at the ISP (where you cannot touch it).

The traps in the pfSense UI

Three of them, and you will hit all three.

Trap 1: FQ_CODEL is not where you think

The limiter form has two dropdowns:

• Queue Management Algorithm: Tail Drop, CoDel, PIE, RED, GRED
• Scheduler: WF2Q+, FIFO, QFQ, Round Robin, PRIO, FQ_CODEL, FQ_PIE

FQ_CODEL is a Scheduler, not a Queue Management Algorithm. Yes, even though the name literally contains CoDel and there is a CoDel option right there in the other dropdown. Leave QMA on Tail Drop.

Trap 2: The parameters are hidden until you save

You select FQ_CODEL and the page says:

Save this limiter to see algorithm parameters.

So you have to save the limiter first, then edit it again to fill the parameters. Every single limiter. Twice the clicks, forever.

Trap 3: The parameters come pre-filled BROKEN

This is bug #16037 and it affects pfSense Plus 23.09.1. When the parameter fields finally appear, they come with:

• target: 0
• interval: 0
• quantum, limit, flows: blank

Those are not defaults. Those are invalid values and FQ-CoDel does not work with them. You must fill all five manually:

And check ECN.

Trap 4 (bonus): In / Out pipe is backwards from what you expect

On a rule on the LAN interface with direction in:

• In pipe = traffic entering the firewall from the LAN = upload to the internet
• Out pipe = traffic going back to the LAN = download from the internet

Yes, "Out pipe" means download. The help text under the field explains it correctly, which nobody reads. I got it backwards on the first try and only caught it because I screenshotted the form before saving.

The result

It worked:

Teams calls stopped dying. Problem solved, victory declared, beers.

Except: the gateway packet loss was still bouncing between 0% and 8%. Which for a production network is garbage. A proper network sits at 0. So, not done.

Attempt 2: Throttling LLM Downloads (This One Failed)

The idea was reasonable: the original problem was people pulling huge models from Hugging Face. So classify traffic by destination and throttle it to 300 Mbps.

The plan:

• An alias LLM_sources with the Hugging Face / Ollama / Civitai hostnames
• A second pair of limiters at 300 Mbps
• A rule above the main one sending that traffic to the slow pipe

Problem 1: half the hostnames did not even exist

Built the alias with the hostnames everybody lists in tutorials, applied, and checked Diagnostics → Tables to see what resolved.

Result: 8 IPs, all of them Cloudflare (104.x, 172.66-67.x, 2606:4700::).

Two things wrong there:

1. Five of the six hostnames did not resolve at all. Hugging Face moved their CDN from .huggingface.co to .hf.co and nobody updated the tutorials.
2. The only two that resolved (Ollama, Civitai) are behind Cloudflare. Throttling Cloudflare IPs means throttling half the internet. If I had applied that rule, I would have created an incident far worse than the one I was fixing.

Lesson: always check Diagnostics → Tables after creating an FQDN alias. Look at what actually resolved before you point a rule at it.

Problem 2: the real one

Fixed the hostnames (the current Hugging Face CDN hosts are cdn-lfs.hf.co, cdn-lfs-us-1.hf.co, cdn-lfs-eu-1.hf.co, and the xethub.hf.co ones). They resolved to AWS CloudFront, dedicated infra, safe to throttle.

Deployed. Then triggered an actual model download and checked Diagnostics → States filtered by the client IP to see where it was really going.

The download was going to 18.154.29.83. Which was not in my alias. Limiter Info confirmed it: the LLM pipes were empty, and the normal pipe was carrying the download at full speed.

Why: a pfSense FQDN alias stores a DNS snapshot. Whatever IPs the firewall's resolver got at that moment. CloudFront has hundreds of edge IPs worldwide, and the client does its own DNS lookup and gets a different edge than the firewall did. My alias caught maybe 5% of the traffic.

Destination-based QoS classification does not work for CDN-hosted services. Period.

The era when you could identify "Netflix traffic" or "Hugging Face traffic" by destination IP ended around 2015. Everything lives on shared CDNs with dynamic IP pools now. FQDN aliases work for a small server with 2 static IPs. They do not work for CloudFront, Cloudflare, Akamai or GCP.

Rolled it back in 60 seconds thanks to the rollback layers we set up at the beginning. See? Worth it.

The lesson that hurts: I should have run that download-and-check-States test before building the aliases, limiters and rules. 30 minutes of testing would have saved three days of building something that did nothing.

Attempt 3: Tuning The Cap (Also A Disaster, But A Funny One)

Back to the 0-8% packet loss. The 950 Mbps cap was a guess. If the real sustained ISP speed is lower than the cap, the queue forms at the ISP and FQ-CoDel is decorative.

So: measure properly. A script with speedtest running every 30 minutes between 1am and 5:30am, logging to CSV, for three nights.

#!/bin/bash
LOGFILE="$HOME/speedtest_results.csv"
RESULT=$(speedtest --accept-license --accept-gdpr -f json 2>/dev/null)
DOWNLOAD=$(echo "$RESULT" | jq -r '.download.bandwidth' | awk '{printf "%.2f", $1*8/1000000}')
UPLOAD=$(echo "$RESULT" | jq -r '.upload.bandwidth' | awk '{printf "%.2f", $1*8/1000000}')
echo "$(date -Iseconds),$DOWNLOAD,$UPLOAD" >> "$LOGFILE"

Cron:

*/30 1-5 * * * /home/user/speedtest_log.sh >/dev/null 2>&1

31 samples, very low variance, script worked perfectly. Results:

Beautiful data. Consistent. Completely useless.

Because on a 1 Gbps symmetric line, with a Waveform test the month before showing 831 Mbps download... 270 Mbps makes no sense. And upload higher than download on a symmetric line makes even less sense.

The reason: the test PC was using a USB-to-Ethernet adapter because its onboard port was dead. And ~270 Mbps is exactly the practical ceiling of a USB 2.0 Ethernet adapter.

I had spent three nights measuring a USB dongle with great precision. If I had set the firewall cap from that data, I would have capped 200 users at the speed of one broken adapter.

Every measurement has a chain. The result tells you about the slowest link in that chain, which might not be the one you care about.

Check your test client before building a methodology on top of it. ethtool <iface> | grep -i speed should say 1000Mb/s. If it does not, you are measuring the wrong thing.

The Final Solution

While looking, Marc, the senior of the team found this Netgate forum post:

https://forum.netgate.com/topic/200595/this-is-how-i-used-limiters-to-fairly-share-bandwidth-among-all-devices-on-lan

And it turns out the whole thing was simpler than what I had built, and better for this problem.

The concept

A pfSense limiter has two levels:

• The limiter (pipe) — holds the bandwidth cap and the scheduler
• The child queue — nested inside, and this is where the magic is

If you give the child queue a /32 mask, pfSense automatically creates one dynamic queue per IP address. All the child queues share the parent's bandwidth equally.

The difference with plain FQ-CoDel:

• FQ-CoDel alone = fairness between flows. A user with 50 connections gets 50 times more bandwidth than a user with 1.
• Masked child queues = fairness between hosts, and then FQ-CoDel inside each host. Every IP gets the same share no matter how many connections it opens.

For 200 users with wildly different applications, the second one is obviously what you want. RFC 8290 even says it: "an ISP may be more interested in ensuring fairness between customers than between flows".

The config

Step 1: The INTERNAL_NETWORKS alias

Firewall → Aliases → IP → Add

• Name: INTERNAL_NETWORKS
• Type: Network(s)
• Entries: your RFC1918 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)

This is so internal traffic does not go through the limiters. Internal traffic does not cross the WAN, so shaping it is pointless.

Step 2: The parent limiters

Firewall → Traffic Shaper → Limiters → New Limiter

One for download, one for upload. Identical except the name:

Save, edit again, fill the parameters (target 5, interval 100, quantum 1514, limit 10240, flows 1024), save again. Remember trap 2 and trap 3.

Step 3: The child queues

Inside each limiter, at the bottom of the page, the green button + Add new Queue. NOT New Limiter in the sidebar.

Destination for download, Source for upload. The rule sits on the LAN, so the LAN host is the source when it uploads and the destination when it downloads. Get this backwards and instead of one queue per IP you get one queue for everything, which is the same as not having done any of this.

Save the queue, then save the parent limiter again. The forum post warns about this and it is true — sometimes the child does not stick otherwise.

The tree should end up looking like this:

DownloadLimiter
└── DownloadQueue
UploadLimiter
└── UploadQueue

Step 4: The rule

Firewall → Rules → LAN → Add. On the LAN tab, not floating.

And in Display Advanced → In / Out pipe:

Select the child queues, not the parent limiters. The dropdown shows all four and it is very easy to pick the wrong one. If you select the parents you get a flat 950 Mbps cap with no per-IP fairness, which is exactly what we are trying to get away from.

And do not forget the Invert match on the destination. Without it the rule matches only internal traffic, the exact opposite of what you want.

Step 5: Apply and verify

Diagnostics → Limiter Info should show something like this:

Schedulers:
00001: 950.000 Mbit/s
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN

Queues:
q00001  50 sl. 0 flows (256 buckets) sched 1
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000   ← masked on destination
q00002  50 sl. 0 flows (256 buckets) sched 2
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000   ← masked on source

The 0xffffffff on the right is the destination mask, on the left is the source mask. If both are on the same side, you configured something wrong.

Heads up: with FQ_CODEL you will NOT see the per-IP queues in Limiter Info. The forum post says it explicitly: the scheduler hides them, but it is working. To verify it actually works, use the real test below.

Results

And the test that actually proves the per-IP fairness: three PCs running speedtest at the same time → ~300 Mbps each. Split evenly. That is what you want to see.

How To Verify It (And Not Lie To Yourself)

Four checks, in order of usefulness:

1. Three simultaneous speedtests from three different machines. Should split evenly. This is the only real proof of per-IP fairness.
2. Waveform bufferbloat (https://www.waveform.com/tools/bufferbloat) from a wired client. Target A or A+. From wifi it does not count, wifi adds its own latency and ruins the measurement.
3. Status → Gateways — loss should be 0%.
4. Diagnostics → Limiter Info — traffic flowing through both pipes, and drops appearing under load (drops are good, that is FQ-CoDel telling TCP to slow down).

About the "1% loss"

dpinger sends one ping every 500 ms over a 60 second window. That is ~120 samples. One lost packet = 1/120 = 0.83%, which the dashboard shows as 1%.

So "sometimes it shows 1%" means "occasionally one ping got lost", which on the internet is completely normal. There is no way to show 0.1% with that config — the smallest non-zero value the gauge can produce is ~0.8%.

If you want a real number, from a wired client:

ping -c 2000 -i 0.2 1.1.1.1

2000 packets, ~7 minutes, resolution of 0.05%. The summary at the end gives you the real loss.

What I Would Tell Past Me

• A pfSense change commits when you click Apply Changes, not Save. Use that gap: stage several changes, verify, apply everything at once.
• Set up the rollback before touching anything. Console + Config History + manual XML. It took an afternoon and it paid for itself the first time something broke.
• Trigger the real traffic and look at Diagnostics → States BEFORE building anything based on destinations. The IPs the client actually connects to are the truth. What your firewall resolves via DNS is not.
• Check Diagnostics → Tables after creating any FQDN alias. If Cloudflare IPs appear, stop and rethink, because you are about to throttle the internet.
• Verify your test equipment before trusting the measurements. ethtool <iface> | grep -i speed. A USB dongle will happily give you three nights of consistent, precise, worthless data.
• The forums know more than the official documentation. The masked child queue pattern is not in the Netgate recipe page. It is in a forum post. Most of the real knowledge lives in forums and Reddit, not in vendor docs.
• The simple solution was better than the sophisticated one. I spent weeks designing a five-class HFSC structure with DSCP, application classification and time-of-day schedules. The thing that actually works is two limiters, two child queues and one rule.

Now, about the 950 Mbps cap that I picked half at random and works perfectly even though my (broken) measurements said it should not... that is a problem for future me.