Missing Trusted CA Certificates (mostly intermediates)

Hello everyone,

we have been using https deep packet inspection for years now.

Recently, we have been getting more and more feedback that websites cause certificate warnings, but only from the company network. The reason is always a missing intermediate certificate on the Watchguard's certificate store. If I add these manually, everything is ok again.
Now I have updated to the latest firmware, but unfortunately the manually added certificates are lost!?
Has anyone had similar experiences at the moment?

Thanks in advance,

Joerg

Comments

  • james.carsonjames.carson Moderator, WatchGuard Representative

    Hi Joerg,

    Updating the firewall itself should not erase any intermediary (or any other keys) for that matter. They're stored in a different place on the system's flash storage.

    In general, if websites are presenting their certificates appropriately, loading an intermediary should not be needed -- the firewall should be able to verify based on the chain and the root cert it has.

    If this continues to be an issue, I'd suggest opening a support case so our team can look into what might be going wrong, and get a bug started if that is the case.

    -James Carson
    WatchGuard Customer Support

  • @JXB said:
    Hello everyone,

    we have been using https deep packet inspection for years now.

    Recently, we have been getting more and more feedback that websites cause certificate warnings, but only from the company network. The reason is always a missing intermediate certificate on the Watchguard's certificate store. If I add these manually, everything is ok again.
    Now I have updated to the latest firmware, but unfortunately the manually added certificates are lost!?
    Has anyone had similar experiences at the moment?

    Thanks in advance,

    Joerg

    Same here. I've been updating the firmware more than 3 times and it loses intermediate cert on every update. I have to manually add them back after each update. I ended up saving the intermediate certs on my PC so I remember to reload them after firmware updates. I only have to deal with 2 intermediate certs as of now. I won't bother contacting anyone. Reloading is a lot faster than dealing with support.

  • james.carsonjames.carson Moderator, WatchGuard Representative

    Hi @morpheus27
    If you're running into this issue, I'd still recommend opening a case -- the only reports I have of this happening in the wild are in this forum post.

    If I attempt to upgrade a firewall with intermediary certs loaded, they survive an upgrade, so something else is happening here that I don't quite understand yet.

    -James Carson
    WatchGuard Customer Support

  • Just to be sure we add certificates the same way... here's how I did it on mine:

    from Firebox System Manager, I click View - Certificates... Then I click Update Trusted CA Certificates - Add an additional Trusted CA cert (Based64 PEM) - Load from file.

  • james.carsonjames.carson Moderator, WatchGuard Representative

    @morpheus27
    I am adding them the same way. I've also tried adding them manually via certificates in FSM.

    -James Carson
    WatchGuard Customer Support

  • james.carsonjames.carson Moderator, WatchGuard Representative

    It's also worth noting that there are a few cert issues fixed in 12.8.2 -- Release notes are still being edited a bit, so they may not be on there quite yet.

    I'd suggest opening a case, even if you just want to report the issue and do nothing else. We can (for example) copy your config over to a lab firewall and work out what specifically is causing the firewall to dump the certs on upgrade.

    I certainly don't doubt the issue is happening (a few people have popped up to note that it's happening) but getting the specific details together on what needs to be in place for it to happen are needed at this stage.

    On my side, I'm going to get a new firewall and set it up, and try this on it -- just to see if it's an issue that needs a new config, or a less-established firewall to occur.

    -James Carson
    WatchGuard Customer Support

  • Submitted it about 2 hours ago. No replies yet.

  • james.carsonjames.carson Moderator, WatchGuard Representative
    edited August 2022

    @morpheus27 What's the case number? I can pull it into the bug I'm working on now. I found it, 01754030.

    I was able to get it to happen with a godaddy cert going from 12.7.1 -> 12.8.2, so I'm making headway.

    -James Carson
    WatchGuard Customer Support

  • james.carsonjames.carson Moderator, WatchGuard Representative

    For anyone else (like @JXB ) that might want to follow along, I created a bug, FBX-23777, to get this corrected.

    If you'd like to follow that bug, please create a support case and mention FBX-23777 somewhere in the case. The technician assigned to the case can set that up for you.

    -James Carson
    WatchGuard Customer Support

  • Thanks. How do we look at bug reports (and their investigation progress)?

  • A limited number of bugs are listed on the support site - select Known Issues.
    https://techsearch.watchguard.com/#t=KB&sort=relevancy&f:@sfarticle_type__c=[Known Issues]

    Otherwise, you should get notified via your support incident when the bug is fixed, indicating the software version which addresses it.

  • I did search before asking but FBX-23777 isn't found. Oh well, I'll just wait for update notification.

  • james.carsonjames.carson Moderator, WatchGuard Representative

    @morpheus27 It doesn't exist there yet, it was entered in on Aug 25.
    As it's just been entered it has to go through an entire triage process where the developers determine what the root cause of the bug is, propose fix(es), and check to see if the proposed fix can potentially cause any other issues. The KI (known issue) generally gets made somewhere near the end of the process.

    The case you created will be updated if there's any change in the status of the bug (for example, if there's a beta release with that bugfix, or if the development team has an early access build with a potential bugfix.) At minimum you'll be alerted when the bug is officially fixed via that case.

    -James Carson
    WatchGuard Customer Support

  • Hello friends,
    sorry for the very late reply, I was a bit longer OOF for reasons.
    Case was open, I ve reopened it, with the request for updates on FBX-23777

    Thanks and kind Regards...

  • james.carsonjames.carson Moderator, WatchGuard Representative

    Hi @JXB

    I spoke with the development team that is working on FBX-23777 and they said they're currently targeting this for the 12.9 release. That target can move as additional issues are discovered or things blocking it from being fixed are fixed.

    When the release is pushed out, or if there are any pre-release builds available to test with you'll be notified via your case.

    -James Carson
    WatchGuard Customer Support

  • Thank you, James.

  • edited January 2023

    Update

    January 20, 2023

    Case 01754030: [Bug FBX-23777] Intermediate Certificates Lost.

    The defect/enhancement you reported is:
    FBX-23777

    The current status is:
    BUG/RFE is open and awaiting Engineering review

    The above is taken from Watchguard email I receive on Dec 27, 2022. Bug fix isn't in 12.9 and 12.9 update 1 release.

    I just went from 12.9 to 12.9 update 1. Lost intermediate certs, again. I have to manually add them back.

Sign In to comment.