BOVPN to BOVPN Slow Transfer Rates
Good afternoon! I'm quite new to the world of Network Admin (I'm the deskside guy who has been tasked to fix the network). My company has 3 locations nationwide (USA) and we have an M370 with Symmetrical Gigabit Fiber in each location. One of these locations has complained about horrendously slow network connectivity since we upgraded to gigabit fiber and upgraded their firewall from a T10 to the M370.
When I check our systems on the web interface for our watchguards, I noticed that our bandwidth on the dashboard > Front Panel indicates our external bandwidth and our IPSec VPN rates. Both of these are under 32 Mbps, which indicates I've overlooked something during configuration preventing my connections from taking advantage of the gigabit speeds I should have. Is there anything I have indeed missed with what little description I've provided? Do I need to set up routing or should it be automated typically? Is there a handy guide for configuration? I've found what the systems CAN do, but not necessarily how to do it - short of spending time on youtube.
Thank you for your input!
Comments
When the firewall was upgraded from a T10 to the M370, was the T10 config moved to the M370 or was a new config created from scratch?
Any specific type of transfer or access is slow over the gigabit link?
Is Internet traffic speed OK?
Check if the firewall external interface is set to Auto or a fixed speed/duplex.
A mismatch between the external interface and the gigabit connection device can cause collisions and/or errors on that connection, which will slow down BOVPN and Internet traffic.
. One can see errors or collision counts in WSM System Manager -> Firebox System Manager -> Status Report -> Interfaces section
. You can see the current Link Speed in the Web UI -> Dashboard -> Interfaces or in FSM Status Report -> Interfaces section
. You can see the External interface NIC Config setting in the Web UI -> Network -> Interfaces
Your gigibit connection device may have lights which indicate the speed/duplex setting for the Ethernet port.
If only BOVPN traffic is slow, you can try setting the Don't Fragment (DF) Bit for IPSec to Clear on the External interface -> Advanced settings, and see if that helps
Set DF Bit for IPSec
https://www.watchguard.com/help/docs/help-center/en-US/Content/en-US/Fireware/networksetup/df_bit_set_c.html
Also, if you have Traffic Management enabled, it is possible to set a max Outgoing Interface Bandwidth. This value should be set to 0.
Set Outgoing Interface Bandwidth
https://www.watchguard.com/help/docs/help-center/en-US/Content/en-US/Fireware/qos_trafficmanagement/outgoing_interface_bandwidth_config_c.html
When the firewall was upgraded from a T10 to the M370, was the T10 config moved to the M370 or was a new config created from scratch?
Created from scratch, but similar settings.
Any specific type of transfer or access is slow over the gigabit link?
Everything within the network is slow coming from my server to the problem location. I actually have quite fantastic speeds while traveling from the location back to the server though.
Is Internet traffic speed OK?
Internet speeds in general appear to operate as expected, roughly 800 Mbps on average.
External NIC Config is set as auto negotiate, link speed is 1000Mb/s, Full Duplex
I don't have WSM installed - currently using the web interface, i'll take a look at setting this up.
DF is currently set to copy, i'll give that a shot
Traffic Management is not enabled
Thanks for your assistance!
On your server, you can set it to detect the path MTU, which may help a lot.
A BOVPN reduces the data portion of a packet because of the encryption header addition. This can be up to 100 bytes. This will result in packet fragmentation which will reduce throughput.
Options to address this:
. You can manually change the MTU on your server. Try a value of 1400.
You can use DrTCP or other tools to do this.
. Use Alan's PMTU script to change Windows so that it automatically
identifies the correct MTU to use for a session.
I have listed that script in this topic:
IKEV2 sound
https://community.watchguard.com/watchguard-community/discussion/comment/5280#Comment_5280
The script makes a few changes to the Registry.
Switching DF doesn't appear to have made any significant changes. When you say server in your last reply, are you meaning the server itself or server side firewall?
The server. You want it to identify that it has a connection through a BOVPN tunnel, and thus needs to dynamically reduce its MTU for that session.
Or just set a lower value for the MTU - 1400, and see if that helps.
Okay, I've tested the MTU and received no difference in download speeds. I have adjusted every setting I can think of, and again, I receive Full connection speeds from the office to our server but not from the server to the office. (They can upload at gigabit speeds but not download - while accessing our network drives). I'm going to go bald trying to figure this out lol.
At this point, I'm not even sure it's a watchguard issue, and I'm looking further into the server itself. Thanks again Bruce, if you have any more insitefulness let me know!
"I've tested the MTU" - which means what?
You can do packet captures on the firewall using TCP Dump, which may show something. With the Advanced options, you can specify IP addrs, interfaces etc.
Run Diagnostic Tasks on Your Firebox
https://www.watchguard.com/help/docs/help-center/en-US/Content/en-US/Fireware/system_status/stats_diagnostics_tasks_web.html
You could open a support incident and see if a WG rep has any suggestions.
Review the TCP-Window-Size info here:
How to Calculate TCP throughput for long distance WAN links
http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/
I'm sorry. That means I adjusted the MTU settings as you suggested and then let them run for a day or so. Thanks for the follow-up again!
I'm looking into TCP Window Size as well as Bandwidth Delay Product to see what else I can learn from this. I'm afraid starting a support incident will get me only to "your device is working correctly, I'm closing the ticket." That's the answer i've received from everyone else regarding this situation.
Also, regarding TCP Window sizes, i've noticed there that my transmittal rate is extremely low using these formulas. But I'm still able to upload from the external office to the server at nearly 100 MB (800Mb). Why does it come to the server so well, but not similar when leaving the server?
What file transfer method is used for each direction?
Just a network share. I can move it from a machine remotely to the network drive with great success, but trying to move it back from the network drive to the machine caps the transfer rate below 1MB.
@Orracle
Network shares (SMB) is likely the biggest culprit here. SMB sends traffic in blocks of a specific size and will effectively wait for an ACK with a hash of that data from the receiving side before sending more data. While SMB can be a quick way to transfer files, due to how it works, latency is its biggest enemy. The longer and narrower the tunnel gets (distance, latency) the longer it'll take to transfer a file. It'll also be limited by the slowest upload speed of the two connections even if the slower one isn't what's sending due to those ACKs.
We have a KB article that goes over this in more detail, along with some possible troubleshooting steps:
(Why are SMB/CIFS file transfers so slow over my VPN?)
https://techsearch.watchguard.com/KB?type=Article&SFDCID=kA10H000000g33SSAQ&lang=en_US
If possible, using a different protocol to send the traffic that streams the data (like FTP, or SFTP) will generally be faster due to the limitations of SMB.
-James Carson
WatchGuard Customer Support
Ah, ok. I believe I misunderstood Bruce's last question. SMB2 is what i have set up. I'll look at the above KBA and see where I can go from there. Thanks!
Well, I went out to the remote site, verified that the hardware was setup correctly and then bypassed the firewall. The results were immediately apparent that my cable company is provided a solid internet connection, and by using my VPN IKEv2 I was able to connect to my network drives, and had incredible speeds (Revit opened the file within 1 minute, as opposed to 20 to 30 minutes while going through the firewall).
I ran several wireshark tests and have been reviewing them, but - am I correct in assuming that there's a simple setting that is preventing these speeds when going through the firewall? (It's done this on a T10 and now my M370)
My users are able to run speedtests on various sites and pull gigabit, but transfer rates are still incredibly low. I've adjusted nearly every setting I can think of (and that has so kindly been suggested), but for a network novice i'm dying.
Could the list of Firewall Policies interfere with my speeds? (Right now the BOVPN allow out and in are 10 and 11 in that list, and that's pretty much the only thing I haven't tried changing.) Is there anything else I'm missing?
SMB (MS network file copies) is very slow over higher latency links, such as many a BOVPN.
Always has been, always will.
Did you read the (Why are SMB/CIFS file transfers so slow over my VPN?) article linked above, and then the MS Slow SMB files transfer speed article?
For more help, consider opening a support incident.
I believe that IKEv2 is faster than IKEv1
Are you using IKEv2 for your BOVPN?
I read that, yes. We're at a 50ms latency on average - that's not where I want to be, but is that included in the "higher latency links" you mention? I'm seeing no SMBv1 getting in the way either.
Why, though, would this cause slow downloads but not prevent the uploads from slowing? (I think that's my real stumper, because I know that in some capacity it's working.)
Please explain the type of file or files that are being transferred for both the upload & download cases.
Few/many small/large files.
Is AV file checking involved on either/both ends?
Has anyone tests other file copy methods besides Windows file copies?
Review this:
Performance tuning for SMB file servers
https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/file-server/smb-file-server
In this post, the site had a 50 ms latency to a remote site, and was getting just 2-4 MBytes/sec transfer rates.
https://www.reddit.com/r/networking/comments/74hvv5/best_file_transfer_protocol_for_high_latency_wan/
Every type of file. Revit files trying to load within the software, 3 GB ISO files that are zipped, smaller 3MB photos, as well as a 2GB folder with small and large files contained. In every instance where I'm able to send the package from the remote office to the server, the upload speed is reliably quick (Normally 15MB but it's been up to 100MB on the iso), but the exact same file in the opposite direction is painfully slow(355KB).
Also, yes, we're IKEv2 for the BOVPN
As for AV, it doesn't seem to be interfering with transfers, as I completely removed it from a client machine and experienced the same transfer rates.
About 15 years ago, I had a similar difference between push & pull of a file transfer in a LAN environment - 100 Mbps transfers.
I never sorted out the reason - I just ended up choosing the faster method when I needed to copy a very large file to/from my Windows PC and the Windows server.
re. AV - I can see that an AV would inspect incoming files and may not inspect outgoing files - thus the reason for the question.
No more thoughts
I'm still thinking that it is a Windows server issue, but I can't think of a way to prove or disprove it.
Bruce, I'm in the 90% convinced category that it's a server issue as well. The only thing preventing the other 10% is that it works through the same network when I remove the firewall and use a normal VPN. All I know is that there's a setting somewhere and I haven't been trained on this part of IT, just thrown in the deep end.
I appreciate all of your answers, and assistance (and you!) If I figure this out without additional hardware I'll post here.
Try posting on some other forum.
Perhaps a MS server oriented one, on maybe on the Spiceworks WG forum:
https://community.spiceworks.com/networking/watchguard
Okay, I've received no luck anywhere, but I did stumble upon a subscribed AV from Watchguard (that I wasn't aware I had). Turned it off, speeds skyrocketed to gigabit speeds (from 355 KB/ps to 100MB/ps). After about 5 minutes of glorious speeds and believing a fix to have been found... speeds went back to normal.
I can't for the life of me tell if the removal of the subscribed "trial" AV fixed it or I somehow had a very coincidental burst of data.
An AV subscription should only impact traffic allowed by proxy policies which have AV inspection enabled