Mac OS Ventura smb/smb2 extremly slow

Robert_Vilhelmsen

Hi,

This might not be directly related Watchguard but here it goes.

I have a branch office vpn, ikev2, connection from a M370 (1Gbit connection) to a FireboxV (10Gbit connection). No issues with the vpn, it has been stable for a long time.

This weekend we moved our last onsite servers (VMs) to our datacenter. All Windows clients has a very good and stable smb2/3 connection to our Windows 2019 fileservers where upload/download speeds is around 3-4-5-600Mbit/s all depending on file size and numbers.

But we have a few Mac OS Ventura 13.2 and after moving the file servers from local lan to remote brach the file upload/download speeds is extremly slow. Infact so slow FINDER most of the time halts or stops responding for a long time.
Downloading 10GB file takes over 2 hours where it takes a very short time on a Windows machine.

I have tried different settings in /etc/nsmb.conf without any luck or difference at all. Tried setting the MTU sixe to 1280, 1400 and 1500, same same. Disabled ipv6, disabled wifi to make sure only wired lan was active.

With Wireshark from the Mac, i see a ton of TCP retransmission and duplicates when the file copy starts, but all SMB2 communication right before seems to be normal without issues.

I have ExtremZ-IP installed on the file servers and even with AFP it is the same slow transfers.

If i instead try to use ftp then download from the server is very fast indeed and upload very good also.

All firewall rules for the smb/afp traffic has no IPS/AV or other inspection enabled.

Any good clues why i see so bad file performance from Ventura and normal speeds with everyting else (http/https/ftp)?

Regards
Robert

Bruce_Briggs

Have you seen this?
https://support.storagecraft.com/s/article/Slow-Transfer-speeds-from-SMB-shares-Mac-OSX-10-11-5-and-later?language=en_US

james.carson

In addition to what Bruce mentioned I'd also suggest checking what version of SMB the mac is trying to use -- if it's an older version, it will tend to be slower.

Your testing points out that the protocol is likely the issue if other protocols work well via that same connection.

SMB was never designed to be sent across WANs, and relies on the receiving machine ACKing every packet it gets. The sending machine only sends a certain amount of traffic at a time, so the longer the ACKs take, the slower the connection will be. This is usually how you end up with huge pipes having very slow transfer speeds -- latency plays a key role with SMB.

While the VPN does introduce a small amount of latency, the latency added by the VPN specifically will be relatively consistent across any device that connects to the VPN. Distance, ISP, upload throughput at the remote end, and other factors all play a part in the appreciable speed you see with SMB.

You can find more info here:
https://techsearch.watchguard.com/KB?type=Article&SFDCID=kA10H000000g33SSAQ&lang=en_US

The see also links point to some Microsoft articles that go over troubleshooting SMB itself, as well.

Robert_Vilhelmsen

Hi both

I have tried Bruce suggestion already plus running only smb1, smb2 and smb3. As the AFP protocol also are extremly slow, i suspect it acn berelated to either how FINDER hanle browsing and/or some tcp settings in Mac OS.

I will try disable tdc ack´s and see it this does anyting to the better.

/robert

Robert_Vilhelmsen

Now i have a very good download speed from the server. What i have done is:

• netsh int tcp set global rss=disabled
• disabled SMB signing via GPO on the server
• TcpAckFrequency=1 on the server
• TCPNoDelay=1 on the server

On the Mac´s i have added this:

• net.inet.tcp.delayed_ack=0
• signing_req_vers=6
• signing_required=no
• netBIOS_before_DNS=no
• port445=no_netbios
• defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE

But upload still takes forever. Trying to start upload a 200MB file FINDER simply stall for a very long time before uploading begins with a extremly slow speed.

Bruce_Briggs

Perhaps Finder is trying to render thumbnail previews for all the files on the remote share?
If so, this article suggests to disable thumbnail previews for SMB shares.

https://lemp.io/why-does-mac-os-finder-slow-on-smb-shares/

Robert_Vilhelmsen

@Bruce_Briggs
I´ll check it out.

Any good inputs are appreciated.

During upload of file(s) to the SMB server, i see tons of TCP retransmission and TCP DUP ACK on the Mac, but very little on the SMB server side compared to the Mac client. A factor 1 to 1000 i guess.

Uploading fra a Windows client shows no TCP errors at all. As in none!

Downloading from the SMB server from Mac is quite fast and shows only very few TCP errors.

Could this be related to how Apple handles congestion even though there is none.

Bruce_Briggs

1st - I know noting about Macs, including their TCP stack implementation.

There is (to me) an interesting post here, which may give you some insight on your problem and could be a place to post a question:

macOS SMB uploads to Windows Server share hang for dozen of seconds
https://ask.wireshark.org/question/22062/macos-smb-uploads-to-windows-server-share-hang-for-dozen-of-seconds/

Robert_Vilhelmsen

Thanks, i did se this article on wireshark and it the issue looks very much as the same, i have.
I can see Apple Mac uses Cubic as congestion tool which many other OS´es also use.

I have opened a support case at my hosting provider, i hope they maybe can bring some light on this issue, as i see the exact same behavior from my home connection also going through a vpn, but via a high speed cable provider using another ISP than we do at work.

This pretty mutch rule out the ISP´s but maybe the issue is the the hosting centers VMWare hosts, we are running on or their spine and leaf backbone.

Robert_Vilhelmsen

@james.carson

If i send the Apple traffic through a firewall filter instead of the bovpn tunnel, it works! Full speed upload and download without any packets missing.

I will open a support case. It makes no sence.

Bruce_Briggs

Maybe a MTU issue when going thought the VPN tunnel?
Which can cause long packets to be split in 2, and may result in a processing error.

Robert_Vilhelmsen

Nope, already checked that. Have even set Apple mtu to 1280 and can clearly see no fragmentation is happening.

Robert_Vilhelmsen

Case opened 01832830

Robert_Vilhelmsen

This is from a tcpdump directly on the firebox on the external interface capturing Apple traffic going via ipsec. Tons of tcp errors and it is the same on both firebox ends.

Robert_Vilhelmsen

I feel very sure, it is how WG kernel handle the tcp congestion (ECN) when the apple traffic are send over a tunnel. Nearly all tcp packets are marked with a ECN-echo when going through a ipseoc tunnel, but via a firewall filter, no packets carry the ECN echo bit.

Robert_Vilhelmsen

It is a bug, FBX-24611.

Disable TCP TSO on the fireboxes VIF solves it.

Victor_Renard

Hello Robert,
we have the same issue with M370 on a side and T55 on another side.
already checked all the MTU settings without success.

Fireware are 12.9.x actually, FBX-24611 suggests it is solved since Fireware 12.8.3 ...

Can you explain what you did to solve it ?
Thank you

Bruce_Briggs

FYI - I'm not finding any reference to FBX-24611 other than in this post.
Where are you seeing anything suggesting that it have been fixed?

Robert_Vilhelmsen

@Victor_Renard

I had WG disable TCP TSO on the fireboxes VIF. The second tso was disabled selective ack´s started working normal again. This setting does not survive a firebox reboot and i had this running for quite some time.
End the end i upgraded the firmware to a newer version and have not seen this issue since.

In the same timeframe the underlaying VMWare was also upgraded with a minor version.

I have no clue what solved it, but it was a fact Tcp TSO had something to do with it.

james.carson

@Victor_Renard FBX-24611 is confirmed as fixed, so that issue is likely not what you're running into.
The biggest factor that is usually an issue with SMB connections is latency due to how that protocol works. BOVPNs (and BOVPN virtual interfaces) tend to induce some latency due to their physical distance from each other. If you have access to test with any other protocol, I would suggest trying that first to see if your problem is based on SMB.

Bruce_Briggs

V12.10 officially fixes this:

This release resolves an issue that caused slow VPN traffic from Apple devices to Windows servers. [FBX-24611]