Cisco UCS, iSCSI storage. SAN NIC seen in OS :(

[Dieselboy]

I'm deploying OpenStack. The B200 M3 blades have three NICs:
eth0 - management / server VLAN
eth1 - SAN/ iSCSI
eth2 - openstack provisioning

I have configured Cisco UCS to use eth1 to access the iscsi LUN for the server.

When I load an OS, and look at the network adapters it shows eth1 in the OS with the same IP address as configured at the hardware layer. I can ping and resolve ARP on the SAN VLAN. Apart from this risk, OpenStack is seeing this network interface and trying to do discovery on it.

I've raised a support case with Cisco and provided explanations a bit more in depth than I have done here. I had a call from them shortly after to say they are busy and will look at it tomorrow

Is it possible to terminate eth1 at the hardware layer, and not allow the OS to know about eth1? So even though the UCS profile has the three nics configured, the OS would only see two, eth0 and eth2. I can understand that the OS may re-number the nics to eth0 and eth1 which might be confusing, but will be documented if possible!

[wintermute000]

I'm deploying OpenStack.

<runs for cover>

[deanwebb]

You want to hide hardware from the OS? What are you, Microsoft?

I would think that this would have to be done as the device does the POST, and you'd need to have modified the hardware itself physically with a jumper switch or something like that... I can't think of any way to do this that doesn't involve violating the warranty and/or resulting in the Cisco TAC person making a face like the one below when they discover what you've done.

[NetworkGroover]

Heh - so first I'll say I don't know the answer, sorry.

Second - I wonder if this why more folks run into walls with OpenStack - they're Cisco shops and Cisco is telling them they are "too busy" to help.... 

[wintermute000]

not really, its just that openstack is a complete monster.

Take it from me, even the prebuilt packstack redhat VM - I had no end of issues as soon as I started getting under the hood. Its a beast.

As a point of reference, I managed to VCP vsphere and VCIX NSX by myself with zero formal training in under a year including standing up a homelab several times in several different topologies. But Openstack, I was completely lost (and I am comfortable with basic linux). Even the documentation is all over the place, even someone like RH compared to what you get from vmware, MS or cisco.

Anyone that can master Openstack, I tip my hat to you

[NetworkGroover]

Yeah, true, I know of only a few deployments out there and they are larger companies so probably have more resources to throw at it.

[Dieselboy]

Haha Thanks for the replies. I've still not deployed openstack because there's been a few issues and a huge learning curve for me but I think I have it figured out About to try this now...

Red Hat have plenty of documentation on their website. Admittedly there are holes. Also, they (red hat) have messed up the DHCP portion of things and because of this the deployment fails as their implementation of DHCP is a bit strange, let me know your thoughts on this one!?

To explain:

There's a provisioning network which is where the director controls the nodes and PXE boots them. You need to configure the parameters for this to work such as the DHCP scope which will be issued from the director. I configured DHCP to be in the range of:
192.168.18.100-192.168.18.200/24

The config also asks for an "Introspection DHCP" range. This is a range which can be allocated (by DHCP) to the nodes (again from the director), but only when the hardware is first added to openstack and you initiate the hardware inspection. This must be done before you can assign a role to the hardware (such as compute). I have allocated an inspection dhcp range of 192.168.18.79-192.168.18.99/24 This is what is causing me the next problem:

Without going into a ton of detail, we all know how DHCP works, right? If a client obtains a DHCP address and the lease is for 8 days, if that client boots up again and the lease is still valid then it will try and request the same IP.

So, what we see is- during the deployment process the node boots up, sends the dhcp broadcast. We checked TCP dump on the director NIC to make sure that this broadcast reaching the director DHCP server. Red Hat informed me that the director has allocated ip 192.168.18.111 but the node is trying to request 192.168.18.79 (from the introspection range!).

Because the node is trying to request a lease which is outside of the normal DHCP scope, the director does not respond or do anything or even log an error! So deploy fails. Annoying.

Why there's 2 DHCP scopes I've no idea. Would be easier and simpler if they just had the one scope on that subnet. DHCP doesn't do roles. One scope would mean that this would have worked first time. They've over-engineered this part! (Dean - can I blog this stuff?)

Going back to the original post - Cisco TAC don't know how to achieve this. In my mind, if you give hardware 2 nics (one for LAN and one for the iSCSI storage HDD only) then you should be able to hide the storage nic from the OS (for reasons like security). This is a Cisco UCS system so I expect this is achievable. If not, then why not?

Dean there is a BIOS here but the entire system is managed through the Cisco UCSM or the CLI. In my network the CLI and UCSM run on the switches called Fabric Interconnects. These then connect to the Cisco UCS chassis where the blades are. I think this management connectivity works similarly to how the Cisco Nexus switches work with the Fabric Extenders, where the management plane is set up for that specifically.

[Dieselboy]

Yeah, true, I know of only a few deployments out there and they are larger companies so probably have more resources to throw at it.

We have 100 VMs running and it's 90% for the dev guys. They ask me to bring up VMs urgently! and then I'm never told when they're no longer needed... Hope to give them OpenStack and let them manage it themselves.

[deanwebb]

Please do blog, if it involves an IP address, blog about it.

As for those scopes... are you able to enter the same range for both scopes? Or would the logic in that DHCP blow its mind?

Another thought - if there's a way to automate a new DHCP request, then you solve the issue. And the quickest way to get a new DHCP request is to shut/no shut the port, so if there's a way to bounce the port after it leaves the inspection mode, you're in the clear.

[Dieselboy]

So at the moment the dhcp failure looks like it's due to filtering being done by the KVM hypervisor which is where the dhcp server is running on. About to look at that now. Last night I run sniffers in a few places and could not see all broadcast traffic; then Red Hat came back and said that the Red Hat Virtualisation will be filtering.

For the scopes, the documentation says the scopes must not clash and must be separate.
After posting the below I set up a test and found that the DHCP lease time is actually 2 minutes already so this is not the issue. I posted a new thread on the dhcp issue though.

For this thread, I worked with Cisco yesterday. iSCSI boot on UCS uses HBA. So as I understand it, you configure the NIC / iSCSI vNIC in Cisco UCS. This then writes a HBA boot table to memory, which is then picked up by the OS during installation and during boot up time. You cannot sleep or suspend the OS for this reason. Also for this reason, the nic must be presented to the OS because after booting, the OS will maintain the connection even though it's booted from it.

I think it's like time travel to be honest (how can you boot an OS from iSCSI when the OS manages the iSCSI, catch 22 but no not really)


So, if this were a multi-tenanted network (and I've seen this before), Windows will broadcast out of that storage nic and find other systems, possibly other customers systems which would not normally be able to see each other because of security but also can see each other because of the shared storage network. Private VLAN might fix this and stop this from happening. 

[ggnfs000]

I'm deploying OpenStack.

<runs for cover>

openstack is this bad? What are the bad aspect of it? Stability, documentation? Learning curve, difficulty? Installation troubles? When last one is coupled with improper documentation it is nightmare.

[wintermute000]

Try standing it up yourself

[Dieselboy]

I tried standing it up. Myself. From scratch. All I can advise is, don't do that 

Best advice - start small. I have two separate OpenStack "environments" running now and I do like it a lot. Your possibilities are almost endless with different architectural designs.

I've deployed a packstack all-in-one, using cinder for storage on the local HDD, however the HDD is an iSCSI LUN with flash storage. I'm waiting for the nimble driver to be updated so I can go about integrating this so volumes can be created and destroyed (if necessary) from within openstack. So far I have it set up so that behind openstack, are tenant networks. I have a few static routes on our core switches pointing to the virtual router in openstack for this to work.

Next challenge is that I'd like to create separate "projects" for dev-ops, dev, QA etc and allow Active Directory to authenticate users who do the builds. I think it will be a much better way to manage the hardware resources.

Networking was a bit of a challenge but I think I either hit a bug or caused a db "inconsistency" when I was trying to get it working. Learnt a lot in a short period of time, though 

[wintermute000]

Yep it is an absolute beast. Way more complex than a simple hyper visor stack