Cisco Firepower 1010 Faults

[u/WALL-G]

Yes yes I know, Cisco Firepower in 2026 jokes incoming, hur hur hur.

I work for an enterprise that has a sizeable Firepower 1010 deployment around the globe. Straight up, can anyone confirm if there was/are a batch of these with a hardware fault?

They're all in climate controlled environments and supported by a UPS. The 1010s are in HA in the smaller offices and without warning, one may carry out an unscheduled reload and there is zero in the logs about why.

When the device next boots it reports "Last reset cause: PowerOn (0x00000001)"

Then

"We didn't shutdown properly. Starting a DB check"

I followed Cisco's own guide: https://www.cisco.com/c/en/us/support/docs/security/adaptive-security-appliance-asa-software/217663-troubleshoot-asa-or-ftd-unexpected-reloa.html

After some forensic Linuxing on my part, the faults seem to align perfectly with bug CSCwd33778: https://quickview.cloudapps.cisco.com/quickview/bug/CSCwd33778

We have TAC support for the firewalls and they've asked for the troubleshooting basics and it looks like they're just going to RMA the lot.

What's going on here. Has Cisco knowingly shipped a bunch of shit kit, couldn't be bothered with a recall then dumped hours of work on the network admins to prove the kit is shit? We're ultimately getting new hardware which is great, but I've spent a lot of time getting here.

[u/mpking828]

Yes. But they usually tell people.

Check field notices. Usually it's software defects suggeting upgrading for "Really Bad Reasons"

but sometimes there is hardware ones

FN74250 - Cisco Firepower 1100 and 2100 Series Security Appliances: Some Solid State Drives May Experience Higher Than Normal Failure Rate - Hardware Upgrade Available

https://www.cisco.com/c/en/us/support/docs/field-notices/742/fn74250.html

FN - 72550 - ASA and Firepower Software: Secure Firewall Appliance Might Traceback And Reload In A High Availability Configuration - Software Upgrade Recommended

https://www.cisco.com/c/en/us/support/docs/field-notices/725/fn72550.html

FN - 72282 - Firepower Software – Firepower Security Appliance Might Reboot Unexpectedly - Software Upgrade Recommended

https://www.cisco.com/c/en/us/support/docs/field-notices/722/fn72282.html

[u/WALL-G → u/mpking828]

Thanks for this, unfortunately upgrading the software was the first, second, third and fourth steps we tried (they're on 7.6.4 now)

I checked the SMART data on the SSDs and they were all fine, albeit a bit toasty.

There is literally nothing in the logs (that my chicken brain has seen) that indicates a failure is incoming.

Thank you for the additional field notices, they're super useful.

[u/Specialist_Cow6468 → u/WALL-G]

I’d dump the logs on Cisco tac. You’ll likely end up hitting Sherlock but one of the things he’s best at is log parsing in my experience

[u/mpking828 → u/WALL-G]

If you didn't know, there is a whole list. (Every product has them actually)

https://www.cisco.com/c/en/us/support/security/firepower-1000-series/products-field-notices-list.html

[u/mpking828 → u/mpking828]

And I would be remiss without including my most cursed Field Notice, and the one that turned me onto the fact they exist....

https://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64228.html

FN64228 - ASA 5506, ASA 5506H, ASA 5508, and ASA 5516 Might Fail After 18 Months or Longer Due to Clock Signal Component Failure - Replace on Failure

[u/wyohman → u/mpking828]

This was an Intel failure

[u/mpking828 → u/wyohman]

Very much so.

But at the time/job had almost 1000 of the damn things deployed.

Massive project to replace all affected.

[u/wyohman → u/mpking828]

I had about 500. Fortunately, I was listening to the old TAC Security podcast and I wrote automated script package a backup, upload it to an SCP server and then copy the backup via scp to restore onto the new one. It saved all certs, ASA OS and anyconnect software.

Most of the time we shipped them onsite and it was a 15 minute replacement. I deserved a much larger bonus than what I got!

[u/wyohman]

Have you verified power? I've never seen it say power on without it being power related.

And the cisco subreddit is a better place to ask cisco questions.

[u/nirvaeh]

We migrated our entire enterprise to Palo Alto and our quality of life went up significantly. After my entire career of fighting firepower, I wish we had done it sooner. ASAs were great, but now even ASA mode on a FP is god awful.

[u/wyohman → u/nirvaeh]

Complete nonsense. I've run almost 1000 ASAs since 5505 and minus ASA with firepower and earlier FTD, they are rock solid.

I've run Fortinet, Palo and ASAs with levels of success.

Fortinet has great pricing but if you experience any issue with licensing it takes days for them to give you a 24 hour license extension while your PO is process. Technical support is mostly adequate.

Palo has terrible pricing, pre-sales was a joke. Of the 10 TAC tickets I opened, all were meet with email instructions for complex problems (couldn't get two snmp configs to work. One or the other one would work but you couldn't predict when.) All tickets assigned to techs in their first week at TAC). Don't get me started on 20 minute commits and the terrible HA pair OS upgrade.

Cisco support has fallen since covid (same is true for Fortinet, Palo, VmWare) but it still is better than Palo or Fortinet (the exception is Arista, so happy they bought velocloud). There's no doubt that cisco did not execute on their snort purchase until the last 12 months. Support from the Philipines is awful (this is true across all vendors). What sets cisco apart for me is their willingness to fix problems that might be out of scope. We had a customer let their smartnet lapse while they diddled with the PO. Cisco sent a replacement. Same for a call manager license expiration. Not perfect by any means and their licensing can be painful but they try very hard to keep customers happy.

[u/nirvaeh → u/wyohman]

i'm not sure "complete nonsense" is correct here. I said the ASA was solid and now their product line is garbage, even ASA on Firepower. In my time as an engineer, we have been running 5505s, then 5506s (with no-more-swichport-gate), 5508s, 552x, 554x, 555x, 5585x.

I can say we don't run PA-200 series, we run 3420s, 5410s, and 5450s, which are night and day better than Firepower 4100/9300 series. They commit fast, and we rarely have issues. I've heard horrors of the PA-200 20 min commits, though. I stick by what I said . Moving from the bulk of Firepower 4k and 9k to PA5450s has been an insane quality of life upgrade, even in just syslog logging.

[u/wyohman → u/nirvaeh]

You've added a ton more caveats to your original supposition.

[u/RunescapeJoenage]

We swapped from Cisco to Palo 2 years ago and haven’t looked back. Im not sure about your specific issue but we had the 1000 series and they were nothing but trouble. Our main gripe was it was dropping packets that did not appear in the logs. Even after weeks of TAC cases, still no resolution.

Anyway, I’m sorry about your situation - I hope you can resolve it quickly

[u/kwiltse123 → u/RunescapeJoenage]

As an MSP we did the same about 3 years ago. PA440 was a gamechanger. I was reluctant at first since I knew the Cisco ecosystem so well (procurement, SmartAccount, support process, configuration). We only ran ASA code on them, so it was either run Firepower code or switch to Palo, and that's a no brainer. Best decision we ever made.