I spent a lot of time trying to figure why our Netbackup connections were failing. We knew we had a lots of packet loss but could never get to the bottom of that. We found that disabling SACK on the Win03 systems fixed it but that wasn’t an option for the Win08 hosts. So, in a effort to just get it fixed and forget about root cause we upped the TCPMaxDataRetransmissions to 25 to see what would happen. Well, the backup job ran for 4 days but finally finished. We rolled out this change to 4 more servers and they all finished thier backup.
In the end we set the TCPMaxDataRetransmissions to 10 on the servers that consistently had failing backups. The default is 5 but in our case it was being set to 3 with Group Policy, this is a common security hardening change.