Performance of network traffic through bare-metal vs. virtualization

[ggnfs000]

i was contemplating whether there is a differentiation in performance when comparing:
- traffic test (i.e iperf) between two VMs across different ESXi hosts.
- traffic test (i.e. iperf) between two baremetal on same servers, same net adapters only this time through baremetal OS not ESXi.

Reading through the VTEP topics, I think I already found the answer where the VM traffic will be less in terms of performance due to added encapsulation VTEP header.

In fact I am thinking to do the actual test on this one to see the performance comparison. Any thoughts.

[deanwebb]

Interested in the results... and while the baremetal will likely be faster, will the difference be noticeable?

[that1guy15]

x86 (or CPU) processed traffic has slower performance than ASIC (hardware) based processing. Simply put software can not process traffic as well as hardware. Encapsulation does play a role in performance though.
http://blog.ipspace.net/2012/07/does-cpu-based-forwarding-performance.html

Now that might change in the near future. With Intel leading the charge on DPDK technologies. This group is getting some amazing speeds out of x86 gear. Worth checking out.
http://techfieldday.com/appearance/intel-presents-at-networking-field-day-12/

yes thats me! But here is one with burynd so...
http://techfieldday.com/appearance/intel-presents-at-networking-field-day-10/

[ggnfs000]

this had me wondered for quite some time, will it be equal? Lot of factors seems at play. First packet is being processed by hypervisor unless direct pass through is utilized.
But I was reading through SDN book and section of the SDN explains how the VM traffic happens through what is called VTEP aka Virtual Tunnel Endpoint. Through VTEP packet sent by one VM to another VM gets through another layer of encapsulation. They also call it variant of native/pure SDN implementation dubbed as: "SDN via hypervisor".

In normal bare-metal we got IP normal packet  ( remember MAC addr field + IP addr field + payload)

Well, in hypervisor VM network, this packet is assigned another IP address for VM end host so that packet becomes
IP address (MAC addr + IP addr + payload). The external IP address is used by Virtualization hosts and VM hosts will maintain the VTEP ID-to-IP addr mapping table. This external IP address is therefore used for the packet to travel through the actual physical network involving VM hosts. Once the packet travels through physical network and reaches the destination VM hosts that external IP is stripped leaving the normal baremetal-like ip packet which is processed within the VM network within the host. Not sure this is the very clear explanation, if not let me know.

So I presume that the very fact that there is add'l IP header is involved in the VM traffic will eat away at least some of the bandwidth from the line-rate bandwidth and expect it to be somewhat lower.

I will see about it. I will have two servers within the chassis which will be equipped with 1G cards and make sure it is 9.9Gbit/s through bare-metal and will repeat it with installing ESXi on each hosts and initiate traffic from VM using iperf.

[NetworkGroover]

As pipelines get bigger, and applications handle network conditions better, it doesn't really matter. Got plenty of very large customers running VXLAN without issue.

[ggnfs000]

OK here is what i got for VM traffic:
iperf clocked 8.25Gbits/sec from VM to VM traffic.
I set the time to 30 sec-s and got 9.25 Gbits/sec
Now 30 sec interval was broken down to three phases:
0-10 sec: 7.19 Gbits/sec
10-20 sec: 9.67 Gbtis/sec
0-30 sec: 9.26 Gbits/sec

VM is residing on same ESXi host with Cisco UCS blade equpped with one Cisco VIC1240 card. I am not sure if the VIC1240 has anything to do with it when VM-to-VM is concerned.

Next I will setup baremetal rhel installation on two separate ESXi host on Cisco UCS blade with VIC1240 card with 10Gbits/sec linecard.
I did it few years back and got about close to 9.9Gbits/sec rate. I am expecting the same.

For the VM-to-VM traffic, I have attempted to capture the inter VM traffic from ESXi host using pktcap-uw just out of curiosity but so far no success yet. It got too much over my head.

[ggnfs000]

As pipelines get bigger, and applications handle network conditions better, it doesn't really matter. Got plenty of very large customers running VXLAN without issue.

Yeah kinda,  I am just doing it for fun and out of curiosity. I hope it will not kill me.

[NetworkGroover]

As pipelines get bigger, and applications handle network conditions better, it doesn't really matter. Got plenty of very large customers running VXLAN without issue.

Yeah kinda,  I am just doing it for fun and out of curiosity. I hope it will not kill me.

No doubt - always a good thing to do to increase the knowledge base - and to possibly discover things others have missed.

[ggnfs000]

struggled with pktcap-uw. I was thinking to use normal rhel host to act as hypervisor and just launch tshark from linux command line.
with that i have to have two hosts managed by same management interface. Anything available out there? I know I should just google but i am just being lazy here.

[Dieselboy]

I'm keen to know this but my bet is under the best conditions possible that there will be barely noticeable difference.

For test 1 you need to set the baseline = = install an OS on the physical system so you can run the iperf tests. Best OS would be RHEL 7. This will then tell you what the hardware and associated network are capable of. If you have two identical hardware such as a Server chassis with 2 blades, interconnected via 10GB (or higher) internal network then this would be good.

For test 2 you would then install KVM on one piece of hardware and then install a VM running RHEL 7 and repeat the same test. If the tests slow down then it can only be due to virtualisation.

Note;
Both pieces of hardware mentioned above should have good specs. Decent RAM and CPU. iperf does all tests in memory so make sure there's enough RAM to do the tests without swapping.

Note-2: when you do the tests on a VM, only run 1 VM on that hardware. Again make sure the VM has enough cpu and ram. But doing it this way you ensure you're not taxing the physical hardware which would skew the tests.

As a final test you could try and saturate the resources (or at least get business close to 75% capacity) of the Hypervisor and see how much the tests drop off!

Great thread!