Re: OT: ASA Failover & Monitored Interfaces

Joe, what version of code are in production that had the issue?

Regards,
 Joe Sanchez

( please excuse the brevity of this email as it was sent via a mobile device. Please excuse misspelled words or sentence structure.)

On Mar 25, 2013, at 8:58 AM, Joe Astorino <joeastorino1982_at_gmail.com> wrote:

> Well when I did the test and both sides went unknown I think I know why
>
> Guide states these 4 tests happen
>
> 1) link status test. If this passes we do network tests
> 2) network traffic test
> 3) arp test
> 4) broadcast ping
>
> If both sides fail all tests they both go unknown. In this case I
> think both sides would have failed because there is literally nothing
> on the VLAN except the two firewalls.
>
> So it was unknown for like 15 minutes before I fixed it...now this is
> still a different problem though than what happened in production. Wha
> happened there was both sides continued to show "normal" when there is
> no way hellos could be received on several data interfaces. Only when
> I manually failed over did those interfaces show failed which is
> disturbing
>
> Sent from my iPhone
>
> On Mar 25, 2013, at 8:43 AM, Jay McMickle <jay.mcmickle_at_yahoo.com> wrote:
>
>> Thanks!
>>
>> How long did this interface remain as unknown? To Carlos' point, you need to make sure the timer expired (I'm sure you did, and I think it's 15 seconds). Can you tweak the poll time? Can you post your sh run fail output?
>>
>> Regards,
>> Jay McMickle CCIE #35355
>> Sent from my iPhone
>>
>> On Mar 24, 2013, at 10:56 PM, Joe Astorino <joeastorino1982_at_gmail.com> wrote:
>>
>>> Thanks for the reply Jay and good luck on your lab man! To answer your
>>> questions and give you some more info:
>>>
>>> - This actually occurred on two different pairs. One pair is 8.4 and
>>> the other is 8.2
>>>
>>> - I did some testing and here is what I found. I don't understand it,
>>> but here is what I found : )
>>>
>>> One of the sub-interfaces we have on one of these pairs is not used
>>> for anything. Being a sub-interface it is of course 802.1Q and the
>>> switch connection on the other side is a trunk. So to simulate the
>>> issue, I simply pruned this VLAN from the trunk on the primary side so
>>> that the monitoring hello's on that interface would not reach the
>>> other side
>>>
>>> Interestingly, when running "show failover" the interface I was
>>> playing with does not show "failed" it shows "unknown". Because it is
>>> not "failed" failover will never occur. The only thing I can find
>>> about status "unknown" in the documentation is that it is the initial
>>> state and that the status cannot be determined.
>>>
>>> WTF is unknown? I mean, if I lose IP connectivity on a monitored
>>> interface shouldn't that be considered a failure? I've read all the
>>> ASA documentation about failover, failover triggers and health
>>> monitoring and I just don't get this.
>>>
>>> On Sun, Mar 24, 2013 at 11:44 PM, Jay McMickle <jay.mcmickle_at_yahoo.com> wrote:
>>>> It should indeed, but I agree, I too had this issue March 8, 2013 (yes, I remember the exact date as it was catastrophic).
>>>>
>>>> We have 5585-20's in HA A/S, with IPS 4270's inline with fail-close as well.
>>>>
>>>> I came unglued when this occurred, and Cisco stated this was a "software" issue but without any attributing bug. We are were running 9.0.1 on the ASA's (now 9.1.1-4), and could not reproduce the issue on our 5585-20/4270 lab hardware with the same IOS. The default failover is set for 1 interface, but to your point, if it doesn't mark it down, it won't failover. In the lab, it worked as expected. In production, it did not, but now does.
>>>>
>>>> May I ask what code on the ASA's your running?
>>>>
>>>> BTW- the IPS's locked up due to a bug in the 7.1(6)E4 IPS engine when signature 694 was pushed. Interesting to see if any of these variables were constants in your environment.
>>>>
>>>> I'm taking the Security IE lab on Wednesday of this week, and these little nuances made me a bit nervous!
>>>>
>>>> Cheers.
>>>>
>>>> Regards,
>>>> Jay McMickle CCIE #35355
>>>> Sent from my iPhone
>>>>
>>>> On Mar 24, 2013, at 10:17 PM, Joe Astorino <joeastorino1982_at_gmail.com> wrote:
>>>>
>>>>> I ran into an interesting situation tonight, and am trying to piece
>>>>> together what happened and what should happen. Pretty simple setup:
>>>>>
>>>>> 2 ASA's running in active standby. These ASA's have a number of
>>>>> interfaces and sub-interfaces all of which are monitored in the
>>>>> failover configuration. There are IPS units physically inline between
>>>>> these ASA's on most of the interfaces. The failover interface itself
>>>>> is of course a straight connection between the ASA's
>>>>>
>>>>> So, the IPS on the primary side "locked up" and it is set to
>>>>> fail-closed. From the inside network, the interfaces on the primary
>>>>> ASA were completely unreachable. I was unable to ping the interfaces
>>>>> at all. I was expecting that when this happened, the ASA's would
>>>>> trigger failover because several of the monitored interfaces were not
>>>>> reachable but they didn't. When logging into the standby unit,
>>>>> everything showed "normal" ...all the monitored interfaces were
>>>>> normal. As soon as I manually failed it over, that changed and the
>>>>> interfaces that were unreachable on the other side showed up as
>>>>> "failed".
>>>>>
>>>>> So basically, the ASA's were unable to communicate with each other
>>>>> over several of the monitored data-interfaces, but the status still
>>>>> showed "normal" until a manual failover was done. If the failover link
>>>>> is fine, but the ASA's cannot communicate via the monitored data
>>>>> interfaces shouldn't that trigger a failover event?
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Joe Astorino
>>>>> CCIE #24347
>>>>> http://astorinonetworks.com
>>>>>
>>>>> "He not busy being born is busy dying" - Dylan
>>>>>
>>>>>
>>>>> Blogs and organic groups at http://www.ccie.net
>>>>>
>>>>> _______________________________________________________________________
>>>>> Subscription information may be found at:
>>>>> http://www.groupstudy.com/list/CCIELab.html
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Joe Astorino
>>> CCIE #24347
>>> http://astorinonetworks.com
>>>
>>> "He not busy being born is busy dying" - Dylan
>>>
>>>
>>> Blogs and organic groups at http://www.ccie.net
>>>
>>> _______________________________________________________________________
>>> Subscription information may be found at:
>>> http://www.groupstudy.com/list/CCIELab.html
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html

Blogs and organic groups at http://www.ccie.net
Received on Mon Mar 25 2013 - 09:33:19 ART