SimonV's Blog-Outage Report – SRX ALG failure – ‘application failure or action’ logs

[SimonV]

Outage Report – SRX ALG failure – 'application failure or action' logs

The initial complaint came from one of our branch offices, which was having issues reaching internal applications and internet sites. After some troubleshooting, we found that we could no longer query against the remote DNS server, and they could not reach their internal forwarders. First, I wrote it off as a DNS server issue but … Continue reading Outage Report – SRX ALG failure – ‘application failure or action’ logs →

The initial complaint came from one of our branch offices, which was having issues reaching internal applications and internet sites. After some troubleshooting, we found that we could no longer query against the remote DNS server, and they could not reach their internal forwarders. First, I wrote it off as a DNS server issue but after more querying to different VPN networks it was clear that all UDP DNS traffic was being lost in transit.

To verify if the packets were going through, I started a packet capture on the LAN interface of the remote firewall and did some DNS queries. Fact, there were no UDP/53 packets arriving over the tunnel. No issues with TCP/53 though. So I enabled logging on this specific tunnel policy and while checking the logs for the DNS, I found these lines instead of the usual RT_FLOW_SESSION_CREATE entries.

2015-09-25 14:19:53	User.Info	10.164.3.70	1 2015-09-25T14:19:53.047 VPN_Box_A RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason="application failure or action" source-address="10.164.242.1" source-port="58358" destination-address="10.10.132.40" destination-port="53" service-name="junos-dns-udp" nat-source-address="10.164.242.1" nat-source-port="58358" nat-destination-address="10.10.132.40" nat-destination-port="53" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="17" policy-name="FW-Policy100" source-zone-name="trust" destination-zone-name="untrust" session-id-32="51106" packets-from-client="0" bytes-from-client="0" packets-from-server="0" bytes-from-server="0" elapsed-time="1" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.0" encrypted="No "]

This is not a standard event: RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason=”application failure or action”

When going through the entire log file for other entries with the “application failure or action” message, I found many more related to RPC and FTP. This immediately pointed to the ALG engine.

2015-09-25 14:19:54	User.Info	10.164.3.70	1 2015-09-25T14:19:53.846 VPN_Box_A RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason="application failure or action" source-address="10.164.110.223" source-port="9057" destination-address="10.104.12.161" destination-port="21" service-name="junos-ftp" nat-source-address="10.9.1.150" nat-source-port="58020" nat-destination-address="10.xx.70.1" nat-destination-port="21" src-nat-rule-name="SNAT-Policy5" dst-nat-rule-name="NAT-Policy10" protocol-id="6" policy-name="FW-FTP" source-zone-name="trust" destination-zone-name="untrust" session-id-32="24311" packets-from-client="0" bytes-from-client="0" packets-from-server="0" bytes-from-server="0" elapsed-time="1" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.0" encrypted="No "]

2015-09-25 14:19:51	User.Info	10.164.3.70	1 2015-09-25T14:19:51.444 VPN_Box_A RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason="application failure or action" source-address="10.164.243.110" source-port="50801" destination-address="10.10.132.50" destination-port="135" service-name="junos-ms-rpc-tcp" nat-source-address="10.164.243.110" nat-source-port="50801" nat-destination-address="10.10.132.50" nat-destination-port="135" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="6" policy-name="FW-Policy100" source-zone-name="trust" destination-zone-name="untrust" session-id-32="2317" packets-from-client="0" bytes-from-client="0" packets-from-server="0" bytes-from-server="0" elapsed-time="1" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.0" encrypted="No "]

Coincidentally, we had already received a ticket related to FTP traffic over a different VPN, but the tunnel was up and all other services were open, so we didn’t immediately relate the two cases.

As a quick workaround for DNS, I disabled the DNS ALG. If you want to know what the ALG exactly does, Bart Jansens has a good write-up here. We could live without those features for a while.

{primary:node0}
netprobe@VPN_Box_A> show configuration security alg dns 
disable;

After disabling the ALG, all our DNS queries were going through again. Instead of immediately closing the session when a response is received, the DNS sessions get a 60 second timeout. This gave a few more sessions in the flow table but nothing the SRX couldn’t manage.

The same workaround worked for FTP. Unfortunately, I couldn’t find a quick way to restart the ALG process so I have rebooted the cluster nodes over the weekend. After rebooting, all the services inspected by the ALGs worked as designed again.

*Hostnames, IP addresses and Policy names were changed

Source: Outage Report – SRX ALG failure – 'application failure or action' logs