Fix Network Performance: Optimize MTU, iperf3, and mtr Tools

Introduction

To resolve network performance issues effectively, it’s essential to optimize tools like iperf3, mtr, and MTU settings. These powerful diagnostic tools help identify and fix network asymmetry, which can lead to slow speeds and unreliable connections. In this tutorial, we’ll walk through the process of using iperf3 and mtr to diagnose network issues caused by MTU misconfigurations, TCP pooling, and other factors. By adjusting MTU settings and implementing precise testing techniques, network administrators can significantly improve performance, streamline troubleshooting, and ensure smoother operations.

Step 1 – Identifying the Issue

Alright, let’s jump right in! You’re dealing with network performance problems, and your first job is to establish a solid starting point—a baseline. It’s like setting the starting line in a race. You’ve got to know where you’re starting from before you can figure out how far you’ve come or if you’ve hit any bumps along the way. Here, we’re talking about checking how your network is doing and whether it’s hitting the expected speeds.

Picture this: you’re working with a bare metal GPU node and a Premium Dedicated Cloud Server (PDS), both in the same AMS (Amsterdam) region. Your goal is to measure the maximum throughput between these two network nodes. Now, here’s the twist: the expected throughput should be 10 Gbps. That’s the speed you’re aiming for since that’s the max that a Premium Dedicated Cloud Server can handle. By running a few tests and comparing the actual throughput to this target, you’ll get a pretty good idea of whether anything’s wrong with the network.

Test Procedure:

1. First, set up a bare metal GPU node in the AMS region.
2. Next, deploy your Premium Dedicated Cloud Server (PDS) in the same AMS region.
3. Then, fire up an iperf3 server instance on your bare metal GPU node.
4. Finally, set up an iperf3 client instance on the PDS.

Using iperf3 for Bandwidth Testing

Now, let’s talk about how to measure that bandwidth and get a real sense of what’s going on. The tool you’ll be using is iperf3 —think of it like the Swiss Army knife for network testing. It’s going to help you measure real-time bandwidth and give you a snapshot of how much data can flow between these two points on the network.

Here’s how you do it:

• On the bare metal GPU node (this is where your iperf3 server will live), run this command to start the server:

$ iperf3 -s

• On the Premium Dedicated Cloud Server (PDS) (acting as the iperf3 client), run this command to start testing the connection:

$ iperf3 -c <server-b-ip> -P 10 -t 30

Let’s break down what’s happening here:

• -c  : This tells iperf3 to connect to the server. You’ll swap out <server-b-ip> with the actual IP address of the server running on your bare metal GPU node.
• -P 10 : This tells iperf3 to use 10 parallel streams. Imagine this as simulating 10 people trying to access the server at the same time. This mimics real-world conditions where multiple connections are happening all at once.
• -t 30 : This sets the test duration to 30 seconds. It’s enough time to really measure the bandwidth and catch any potential bottlenecks or issues in the network.

Once you run these commands, you’ll get a detailed view of the available bandwidth between the two nodes. This gives you valuable insight into whether the network is performing as expected or if something’s slowing it down. If the bandwidth doesn’t meet your expectations, it could point to a deeper issue that needs further investigation. Either way, these tests give you the starting point for a more thorough diagnostic process, helping you zoom in on the root cause of any performance hiccups.

RFC 791: Internet Protocol

Step 2 – Running Advanced Network Diagnostics

Imagine this: You’ve set up everything for a smooth-running network, but something’s still off. Maybe the speeds are slower than expected, or the connection keeps dropping. You know there’s an issue somewhere, but it’s hidden, like trying to find a needle in a haystack. So, what do you do? You bring in the big guns—My Traceroute, or mtr , a tool that tracks your network’s every move and helps you uncover any hidden problems.

Think of mtr as your network detective. It’s like sending out a team of probes down the path your data takes, carefully looking for clues along the way. The great part about mtr is that it gives you real-time feedback on each hop, showing you exactly where delays or packet loss might occur. So, if something’s slowing you down or going wrong, mtr will point you straight to the problem.

Let’s say you’re tracing the route from a bare metal GPU node to a Premium Dedicated Cloud Server (PDS). To do this, run the following command:

$ mtr -rwbzc100 <server-a-ip>

Here’s what each part of the command does:

• -r : This activates “report mode,” meaning mtr collects all the data and then exits. It doesn’t just keep running interactively. Instead, you get a clean snapshot of the network’s performance right then and there.
• -w : This is the “wide report format,” and it adds extra details to the output. It gives you more insights, like additional stats, so you get a fuller picture of what’s going on.
• -b : This shows both IP addresses and hostnames for each hop. It doesn’t just give you the raw numbers; it tells you which routers (or hosts) are involved, which is super useful if you need to identify specific devices or services along the way.
• -z : This shows the Autonomous System (AS) numbers for each hop. An AS number tells you which network the hop belongs to, so you can tell if your data is traveling through different ISPs or cloud providers. This might give you clues if traffic is taking a weird detour.
• -c100 : This sends 100 probe packets to each hop, making sure you gather a solid set of data. The more data you collect, the better your averages will be, helping you get a clearer view of the network’s performance.

Once you run this command, you’ll start to see the results. The key thing to look for is any sign of packet loss or high latency. If a hop shows delays or packet loss, that’s your red flag. It might be caused by congestion or even misrouting in the network—both of which can seriously mess with your network’s speed and reliability.

Now, let’s say you want to run a similar trace, but this time, you’re going from your Premium Dedicated Cloud Server (PDS) back to your bare metal GPU node. The command is the same, just swap the destination IP:

$ mtr -rwbzc100 <server-b-ip>

If, along the way, you notice packet loss at a specific hop, and it doesn’t get better further down the route, it’s time to zoom in on that hop. That’s probably where the problem is—a router struggling to keep up, or maybe it’s misconfigured, causing the traffic to get stuck.

The cool thing about using mtr is that it helps you pinpoint exactly where the issue is. You can’t fix what you don’t see, right? So, with mtr , you’ll focus your troubleshooting on the exact hop or router causing the issue. That’s how you start solving network performance problems—one hop at a time!

Step 3 – Diagnosing the Root Cause

Diagnosing network asymmetry is like solving a mystery. The clues are there, but the cause can sometimes be tricky to find, hidden behind a few layers of complexity. You might have one or more issues that are all contributing to the problem, making it tough to pinpoint just one thing. But don’t worry, I’m going to guide you through some of the most common causes of network asymmetry so you can start figuring it out and get your network back to smooth sailing.

TCP Connection Pooling and Thread Limits

First up, let’s talk about one of the trickiest suspects: how some tools handle TCP connections. You know how it is when you’re testing the speed of your network, and something feels off? One common issue here is that many speed testing tools treat downloads and uploads differently. It’s like the system is playing favorites.

Take speedtest-cli , for example. By default, this tool uses 34 parallel TCP connections for downloads but only 6 for uploads. Think about it—34 streams for downloads and only 6 for uploads. It’s no surprise that your downloads seem much faster than your uploads, even if your network is fully capable of handling both at the same speed. This difference in the number of connections can make your speed readings a bit misleading, giving downloads an unfair advantage. It’s a classic case of the tool not treating both directions the same way, and that can throw off your measurements.

MTU (Maximum Transmission Unit) Misconfiguration

Next, let’s talk about MTU—the Maximum Transmission Unit. MTU controls the maximum size of a data packet that can travel through the network. It’s like trying to send a package through the mail: if it’s too big, it won’t fit through the system, and it’ll get broken down into smaller pieces. If the MTU is set too high, it leads to packet fragmentation, meaning the packets get split up and have to be reassembled at the other end.

This fragmentation isn’t just a hassle—it also slows everything down. Why? Because those smaller pieces take extra time to reassemble. So, if your MTU is too high, you’re adding unnecessary weight to the network, reducing its efficiency and causing performance issues.

Now here’s the kicker: if you’re using VPN tunnels, like WireGuard , this issue can get worse. VPNs add extra overhead because they’re encrypting and tunneling the data. So, if your MTU is already too big, the encryption process just makes things slower. Lowering the MTU ensures the data fits properly into the network, so it can be transmitted more efficiently. This cuts down on fragmentation and helps your network run much smoother.

Network Provider Policies & Peering Agreements

But sometimes, the problem might not even be in your hands. External factors like traffic shaping and peering agreements between ISPs or cloud providers can mess with your network’s performance. It’s like having a traffic light in the middle of your data’s path, stopping it for no good reason.

Traffic shaping is when an ISP or cloud provider deliberately prioritizes certain types of traffic over others. They might give download traffic more priority, which could explain why your downloads are flying, but uploads are dragging. This technique is meant to help manage network resources, but it can throw off the balance of your network’s performance.

And then, there’s peering agreements. These are deals between ISPs and cloud providers on how they route traffic between networks. If these agreements aren’t optimized or are misconfigured, your data might take a less-than-ideal route, causing delays or performance issues.

Wrapping It All Together

So, what does all this mean for you? Now that you know what might be causing the issues—whether it’s TCP connection pooling, MTU misconfigurations, or external network policies—you’ve got a much better idea of where to start troubleshooting. By addressing these issues, you’ll be able to get your network running more smoothly and efficiently. It’s like tuning up a car’s engine: you just need to tweak the right parts to make everything work properly again. Once you’ve sorted out the underlying causes, you can be pretty sure your network will perform better, and along the way, you’ll have learned a thing or two!

Step 4 – Implementing Fixes

Fix 1: Adjust MTU Settings

Here’s the thing: one of the best ways to fix network performance issues is by adjusting the MTU, or Maximum Transmission Unit. It’s like adjusting the size of the pipes in your plumbing system—if they’re too small, water (or data) gets stuck; if they’re too big, things get messy and inefficient. In networking, MTU defines the maximum size of a data packet that can travel through your network interface.

If you’re using VPN tunnels, like WireGuard, you might run into problems when the MTU is set too high. This can cause packet fragmentation—basically, the data gets split into smaller chunks because it’s too big for the network to handle all at once. And that’s not good for performance. So, by lowering the MTU, you can ensure smoother data flow and prevent this fragmentation.

Here’s how to adjust the MTU for WireGuard on both the bare metal GPU node and the Cloud Server (PDS):

Setting MTU to 1400

To start, try setting the MTU to 1400. This is a safe size that ensures packets don’t exceed the limit, reducing fragmentation and improving performance.

$ ip link set dev wg0 mtu 1400

Lowering MTU to 1300

If you’re still having trouble, or if your network needs a smaller MTU to work more efficiently, you can lower the MTU to 1300. VPNs like WireGuard add extra encryption overhead, which can mess with the network’s ability to send large packets. Lowering the MTU helps ensure that these packets fit comfortably into the network’s “pipes.”

$ ip link set dev wg0 mtu 1300

You should always start with the optimal MTU settings, then fine-tune from there. If the MTU is too high, packet fragmentation can slow everything down, especially when using VPNs that already add some overhead. Lowering the MTU can make a huge difference in reducing delays and improving overall network performance.

Fix 2: Use an Alternative Testing Tool (iPerf3)

If you’re serious about diagnosing network issues, it’s time to ditch the old tools like speedtest-cli and bring in the big guns—iperf3. Unlike speedtest-cli, which uses public speed test servers, iperf3 lets you run tests on your own network, providing a more accurate and controlled measurement of performance.

To get started with iperf3, follow these steps:

On the Client (Cloud Server – PDS):

Run the following command to start testing the connection. This will connect to the iperf3 server and begin measuring the bandwidth:

$ iperf3 -c <server-b-ip> -P 10 -t 60

Let’s break that down:

• -c <server-b-ip> : This tells iperf3 which server to connect to. You’ll plug in the IP address of the server you set up on the bare metal GPU node.
• -P 10 : This option uses 10 parallel streams to simulate multiple simultaneous connections. This is key to saturating the network link and getting a more accurate measurement.
• -t 60 : This sets the test duration to 60 seconds, giving you enough time to gather enough data for a stable and reliable measurement.

On the Server (bare metal GPU node):

On the server side, run this command to start the iperf3 server and wait for connections from the client:

$ iperf3 -s

By using iperf3 instead of speedtest-cli, you gain much more control over the testing environment. It’s like having your own private testing lab, ensuring that you’re measuring the true performance of your internal network, free from the interference of external factors like public speed test servers.

Fix 3: Contact ISP and Cloud Provider

But what if the problem isn’t on your end? Sometimes, the network asymmetry comes from external factors—like your ISP or cloud provider. Both of these can implement traffic shaping and routing policies that affect the balance of download and upload speeds.

Traffic Shaping:

Think of it like a traffic cop directing the flow of cars on the highway—except in this case, the “cop” is your ISP or cloud provider. Some providers give priority to download traffic over uploads, which can cause your uploads to crawl while downloads zoom by. If this is the case, it might be worth reaching out to your ISP or cloud provider to ask about their policies and whether they’re affecting your upload performance.

Route Optimization:

Another thing to consider is how your data is being routed. If your network is experiencing slowness, it could be because the traffic isn’t taking the most efficient route. You can contact your ISP or cloud provider and request a route optimization check. This will ensure that your traffic takes the fastest, most direct path possible, reducing delays and improving network performance.

By working with your ISP or cloud provider, you can often resolve issues that are outside your control and get your network back on track. If traffic shaping or inefficient routing is at play, addressing these factors can make a significant difference in balancing your network’s performance across the board.

Now that you have a toolkit full of fixes—adjusting the MTU, using iperf3, and working with your provider—you’re well-equipped to tackle network asymmetry. Just remember: the key is to dig deeper, test your assumptions, and make adjustments as needed. You’ll have your network running smoothly in no time!

How Traffic Shaping Affects Network Performance

Step 5 – Re-testing Performance

Now that you’ve rolled up your sleeves and made all the necessary changes to fix those tricky network performance issues, it’s time to take a deep breath and check how things are going. Think of it like taking a car for a spin after some engine repairs. You want to make sure the adjustments worked and see if your network is running smoothly. Re-testing is your chance to measure how much better things are and make sure everything is working at top speed.

Here’s what you should be looking for during the re-testing phase:

Upload Speeds Should Improve Significantly

You’ve made some changes, like adjusting the MTU or using tools like iperf3 to improve performance. Now, the goal is to see those upload speeds pick up. You should notice a noticeable increase in how fast your uploads are, especially if they’ve been slow before. Using iperf3 or other bandwidth testing tools will help you confirm this improvement. The real win here is when your upload speeds either match or at least come close to what you expected—this means your network is now supporting symmetrical data transfer, both ways.

Packet Loss in mtr Results Should Be Minimized

Let’s talk about mtr now. If you’ve been using mtr to trace network routes, it’s time to check whether those earlier signs of packet loss have been cleared up. You should notice fewer packets getting lost along the way, especially at those intermediate hops. Packet loss earlier could have been a sign of network congestion or misrouting. Now that you’ve applied your fixes, those issues should be gone, leading to a more stable and reliable network. This will improve the overall communication of your network, making it more dependable.

Network Asymmetry Should Be Reduced

Lastly, the ultimate goal of all these tweaks, tests, and changes is to reduce network asymmetry. Network asymmetry is that frustrating situation where your download speeds are blazing fast, but your upload speeds lag behind, right? After adjusting the MTU, running more precise tests with iperf3 , and applying other fixes, you should see a significant reduction in that speed gap. The network should now behave more evenly, with less difference between your upload and download speeds. The result? A more balanced and efficient network that works smoothly in both directions.

By carefully analyzing the results of your re-tests, you’ll be able to confidently figure out whether the network performance issues have been resolved. If the improvements aren’t quite what you expected, don’t panic! You may just need to revisit some of the previous steps—maybe tweak the settings further or look into external factors like traffic shaping from your ISP. Whatever it takes, you’ll have that network running like a well-oiled machine in no time!

Network Performance Troubleshooting Guide

Conclusion

In conclusion, optimizing network performance with tools like iperf3, mtr, and adjusting MTU settings is essential for diagnosing and resolving issues related to asymmetric performance. By following the steps outlined in this tutorial, network administrators can effectively identify common causes of network slowdowns, such as MTU misconfigurations and TCP connection pooling. With precise adjustments and the right diagnostic tools, users can enhance network performance, reduce troubleshooting time, and ensure smoother, more efficient operations. As network demands continue to grow, mastering these tools and techniques will be crucial for maintaining optimal infrastructure. Moving forward, keep an eye on emerging tools and practices that will further improve network reliability and speed in increasingly complex environments.

Master MySQL: Create Tables and Insert Data with SQL Commands