Weird DHCP packets

[Dieselboy]

I have captured some DHCP packets to try and find out why DHCP is not working between a PXE boot client and a DHCP server. The weirdness is explained in point 3. below.

The DHCP server is part of OpenStack which means I've not specifically configured this DHCP server. How it works is, you configure a .conf configuration file with things like "dhcp range" and some other things but the installation scripts take care of the packages install and component config.

I've attached the capture. In this capture which is taken on the DHCP server itself which is RHEL 7 and this is achieved using tcpdump. We can see the following:

1. The first packet is the dhcp server itself. I expect the dhcp server is making sure that there are no other dhcp servers on this network. Note the source mac address begins 001a.... This mac address is correct for the interface in question on the dhcp server

2. There are two separate dhcp "DISCOVER" because I have two physical machines attempting to boot via PXE boot. They are Cisco UCS servers. Their source mac addresses begin 0001.af...... This is correct.

3. DHCP OFFER packets look strange to me. The source mac of the OFFER packets are fa16.3e53.f454 and this mac address has the following issues:

- this mac is not assigned or configured to any interface on the dhcp server
- the mac address does not exist in my network whatsoever
- the mac address cannot be found and is not assigned to any manufacturer

Seems like the dhcp server is using a random mac address to source this traffic from and doesn't seem normal to me.

Additionally, the source IP address of the OFFER packets is 192.168.18.100. This has the following issues:

- the IP is the fist IP in the dhcp range on this server (range is 192.168.18.100 - 192.168.18.200)
- the IP is not assigned to any system whatsoever in my network (theres no leases or DHCP client with this IP)
- the IP is not assigned to any interface on the dhcp server

Seems like the dhcp server is using a random IP address to source this traffic from but again does not seem normal.

Now because this process has the broadcast flag set, then all communication (client and server) are using broadcast packets, which should be received each time on both systems. So with that this "issue" looks to be benign at the moment.

What's the forums take on this? I have asked Red Hat for more info. Seems a bit odd to me.

[wintermute000]

Welcome to openstack... I told you this would happen [emoji14]

[Dieselboy]

Well it's wrong.

What "legalities" are there with this config / build? Apart from it being dodgy and open to other problems, can I make them fix it? My response to the support person this morning was again "please explain the ip and mac address issues". I asked them yesterday a number of times, that part of the email was ignored.

The DHCP packets were being filtered out by Red Hat Virtualisation hypervisor. When I looked as to why it mentions that unknown macs are filtered out. So I've had to remove this "feature" from RHV on the Openstack provisioning network. I want to put this feature back in. The dhcp server sends a packet sourced from a dodgy mac address.

Basically, Red Hat have broken their own products. 

So I've deployed openstack, the nodes are built. BUT deployment is stuck at 52%. If I get time over the weekend I'll cancel / delete deployment if that option is still available and then re-deploy. If that fails then it's back to support.

Do they do this to guarantee customers re-purchase support / subscriptions or something? Seems a bit crazy unless this is the reason why. Because it only took me a few hours to get to the point which I am at now if you remove the needed support case and remote support time, which I've mostly fixed myself anyway. On this dhcp issue, the 1st support guy was doing everything except debugging the dhcp problem. The 2nd support person (yesterday) started poking holes in the self-signed SSL cert saying the "generate local self signed cert = true" has been changed from the default of "false" and that we should change it back. My response was "No problem, if we feel that this cert is breaking dhcp then we'll change it immediately". Didn't change it. So since beginning reading the first doc, it's been 3 or 4 weeks. Like I said above, only a few actual hours of work, the rest are working with support fixing issues.

But on the flip side, it's not really a bad thing for me because I've learnt so much more than I would have if everything went smoothly. So got to take the rough with the smooth I suppose.

[Dieselboy]

I'm going to push for them to raise a bugzilla... unless they can explain / prove to me that this has been configured this way purposely for actual reasons then I'm pushing for a bug to be raised. Even if their reason is that "we must source from a random MAC address for reasons X, Y, Z," then I will say fine, raise a bugzilla because you've not used the OUI MAC address scope specifically for that purpose.

[deanwebb]

A cert needed for DHCP? And how can the IP-less device validate that cert beyond a huge "Trust me!" from the DHCP server? That's total nonsense if a guy says you need a cert to do DHCP.

Someone needs to get into that code and find out why it can't use a perfectly normal MAC address that actually exists and is assigned to the server.

[Dieselboy]

 

I'll keep you updated but it's pretty comical

[deanwebb]

I'll keep you updated but it's pretty comical

It's funny until it's a facepalm.

Good:

Bad: