Multi-threaded copy / download tools?

[Dieselboy]

I have a problem, I need to transfer about 60GB of data to Sri Lanka from my Australia office, over the internet / VPN. I'm still in the process of getting parties together to discuss a Riverbed Steelhead quote, so at the moment I don't have any appliances that could do this work for me.

I do have Windows 2012 server and Windows 7 / Windows 8 computers - I don't know if anything exists here that would help me, although I recall back in the UK one of the server guys was telling management we no longer need to use Riverbed appliances at all; since Windows now has these fancy protocols built in. I was calling BS on this one, but at the same time I wasn't doubting there were some improvements to SMB or something similar, that would enhance data access over high latency links. If anyone can shed any light on that I'd be grateful. 

I'm now looking for a tool that will allow me to do something like an FTP transfer with something like filezilla, that will allow me to say "use X number of threads", basically use multiple TCP connections to get this one file.
I've found an add-on for Google Chrome which may do it. But browsing to ftp://my-computers-IP is only giving me an error in Chrome, this used to work before (think the latest update has a bug). And my laptop doesn't yet have a web server on it, so I can access the large files through Chrome at the remote computer in Sri Lanka.

Does any of you, know of a tool that would allow mutiple threads for one file download?

[Dieselboy]

Just found this: https://whatbox.ca/wiki/Multi-threaded_and_Segmented_FTP

will see how something from the above link goes.

[wintermute000]

SMBv3.x has various speed optimisation improvements, and there's some kind of remote caching feature called branch-cache.
I believe most of these features need Win8+ clients to take full advantage.

I'd be very, very surprised if it comes close to a real steelhead, though it may be a cheap fascimile (otherwise Riverbed has been comfortably nicking a living for the last 3 years without anyone noticing?!?!?!). Our company sells riverbed by the truckload

[Reggle]

What exactly would be the advantage of multiple TCP streams over a single with extended TCP Windowing and Selective ACK support?

[Otanx]

What exactly would be the advantage of multiple TCP streams over a single with extended TCP Windowing and Selective ACK support?

The first thing that comes to mind is being able to use multiple links in a bond. Not sure if that is in play here or not.

-Otanx

[Dieselboy]

There's an under sea fibre cable break and internet traffic between Sri Lanka and Western Australia is re-routed via Singapore (I think) and then through the Easten Australian states. Remember Australia is a gnats cock bigger than USA. The result is now an average of 340ms latency between our offices and without any WAN optimisation in play, single TCP throughput is getting me roughly 180kbps. So to transfer one file which is 17GB this = a bloody long time. I've managed to use CuteFTP which can run multiple TCP connections to download a single file. The result is multiples of 180kbps which considerably reduces the download time of the 17GB file.

TCP windowing is not enough on its own to increase the throughput. How would Selective ACK come into play here on it's own, between Windows computers? I assume that it would already be implemented in the TCP protocol. 340ms latency and 64kbyte window equates to 192kbyte/s. So this is roughly what I am achieving per TCP connection with 180k data throughput across the VPN tunnel. With S ACK I would need the client to request enough sections of data to saturate the 10M link. Is this possible?

So in short, multiple TCP connections from the FTP client, allows me to saturate the 10MB internet link at the Sri Lanka office and reduces total time to download a single file, compared to a single FTP connection.

I did try windows SMB from Sri Lanka to my laptop by using Windows explorer and browsing like: \\192.168.x.x\c$\folder\file.tib but the transfer kept crapping out. tbh aside from high latency and the occasional lost packet in the internet, I really don't know why it kept crapping out instead of just being very slow. When Windows showed the file as being copied though, the speed was similar to a single FTP.

With a Riverbed appliance, it does all these fancy things, and more as well as caching and proprietary stuff.

[SimonV]

How about just putting it on a thumbdrive/hdd and shipping it via DHL?

[Dieselboy]

It was discussed yesterday But via the internet is quicker. Last time we shipped something from here to SL it was a couple of weeks.
FYI it generally takes a week or two to get items to Western Australia from the Eastern states.

[wintermute000]

360ms is poor but even @ non-optimised 64k window I think your bandwidth delay product calculation is off... now that's not to say that something else is causing speed to top out @ 180kb but its not just the BDP. Don't forget modern OS (even windoze lol) will scale pretty effectively... Win7 apparently can scale up to 16MByte windows.


based on below calc's at your given parameters I'd be thinking there's some loss somewhere as well that's causing the window not to scale up.

https://www.switch.ch/network/tools/tcp_throughput/

Assuming 10^6% packet loss (standard) I'm getting 2.91Mbit/s with 64k window and 180ms latency

Theoretical network limit

rough estimation: rate < (MSS/RTT)*(C/sqrt(Loss)) [ C=1 ] (based on the Mathis et.al. formula)
network limit (MSS 1460 byte, RTT: 360.0 ms, Loss: 10-08 (10-06%)) : 324.44 Mbit/sec.
Bandwidth-delay Product and buffer size

BDP (100 Mbit/sec, 180.0 ms) = 2.25 MByte
required tcp buffer to reach 100 Mbps with RTT of 180.0 ms >= 2197.3 KByte
maximum throughput with a TCP window of 64 KByte and RTT of 180.0 ms <= 2.91 Mbit/sec.

[Reggle]

... 340ms latency and 64kbyte window equates to 192kbyte/s...

Although I follow your arguments here and even agree, a small remark: I said Extended Windowing. 64kB is the basic Windowing limit. Extended increases it to 4 GB (I think). I've never seen 4GB of course but I did see negotiations into the 300-400kB mark, which should increase the speed about 5 times.

[Dieselboy]

Wintermute I used the same website:

Bandwidth-delay Product and buffer size

BDP (100 Mbit/sec, 340.0 ms) = 4.25 MByte
required tcp buffer to reach 100 Mbps with RTT of 340.0 ms >= 4150.4 KByte
maximum throughput with a TCP window of 64 KByte and RTT of 340.0 ms <= 1.54 Mbit/sec.

so, 1540000 bit /8 = 192.5kbyte/s total bit rate.
Add on VPN headers, the odd packet loss (as some packets do get lost), as well as TCP / FTP headers I expect this is why I'm getting around 180k, roughly.

Although I have seen single connections sometimes getting quite high throughput yesterday, 400k.

What's the deciding factors to go above 64k window size? I'm going to install wireshark and see what is going on.

[Dieselboy]

... 340ms latency and 64kbyte window equates to 192kbyte/s...

Although I follow your arguments here and even agree, a small remark: I said Extended Windowing. 64kB is the basic Windowing limit. Extended increases it to 4 GB (I think). I've never seen 4GB of course but I did see negotiations into the 300-400kB mark, which should increase the speed about 5 times.

Thanks, wasn't aware this was possible under normal operation with OS's now.

[wintermute000]

I'm not aware of the exact algos windows uses, but if its not scaling > 64k then you've probably got signifcant packet loss or something like that.
Be interested in your wireshark result.

BTW you used small k so I assumed you mean kbits!

[Reggle]

BTW you used small k so I assumed you mean kbits!

k is always small. I used a capital B though

[Dieselboy]

I always get confused with k K b B. I've looked it up so many times. Still confuses me about which way round it goes.