Windows Fabric certificate error – Front End service fails to start

My team and I were stuck on an error no one had ever seen before and one that was proving very difficult to determine the root cause! This post is to help you avoid the pain and suffering we went through late into the night.

Issue

After rebooting the 2x Skype for Business Standard Edition servers conferencing no longer worked on either pool. Errors seen in the ‘Lync’ event log:

“Skype for Business Server MCUFactory exception was handled Exception: System.AggregateException: One or more errors occurred. —> System.Fabric.FabricException: A communication error caused the operation to fail. —> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BBC”

We also were seeing lots of Fabric errors (occurring every second):

  • 0x1C55350FD40: authorization failure
  • 1c55350fd40: cert chain trust status is in error: 0x40
  • Disconnected: <FE-SERVER>:5092 ([ nodeInstance=0:0 ]) error=E_ABORT

With the Fabric errors in mind, we tried a quorum reset (reset-cspoolfabricstate -resettype Quorumlossrecovery) on one of the Front End servers which caused the Front End service to stop running altogether!

“The Skype for Business Server Front-End service terminated with the following service-specific error: General access denied error”

Now we are in real trouble 🙂 So we tried reinstalling Windows Fabric Host service, no luck.

It seemed very suspect to us that both servers would be affected at the same time and our feeling was that this was environment related namely certificates. However, the certificates all looked OK but we re-issued them just in case. Nope, that didn’t help!

We kept looking all over but kept circling back to certificates. On deeper inspection, we noticed that the Intermediary CA that issued the certificates had a valid and published CRL, however, the Root CA that issued the Intermediary CA certificate didn’t. The CRL was also expired and because the Root CA was offline, it was not re-generated. We verified this using the certutil (C:\Windows\System32\certutil.exe) and entering the DN for the CRL:

You can double click the entries in the above tool to see more detail. In our case you can see the CRL is expired:

In summary:

  • Root CA Offline – No CRL
  • Intermediary – No CRL as Root not publishing one
  • Certs issue by Intermediary – Has CRL
  • Root CA CRL expired

Interestingly the Root CA’s CRL was 20 months expired, so Fabric must have a damn long grace period!? A little weird this issue didn’t surface a long time ago. In fact, this Skype for Business deployment was not even 20 months old, yet it installed and worked happy for 6 months, go figure….

Resolution

It was the middle of the night and we couldn’t resolve the CRL issue right there and then so we looked for a workaround. The temporary fix was to change the ClusterManifest.Xml.Template (C:\Program Files\Skype for Business Server 2015\Server\Core) value to <Parameter Name=”CrlCheckingFlag” Value=”0″ />. Right after doing this, the services started and conferencing came up on both servers.

A big thanks to Luis Ramos for pointing us to this fix!!!

Interestingly the default for <Parameter Name=”IgnoreCrlOfflineError” Value=”true” /> was true, so one would have thought that the inaccessibility wouldn’t matter, but maybe because there was no published CRL point, it broke it before it got to the offline check.

The long term fix was to renew the CRL and make sure a process was put in place to ensure that it is manually renewed in the future.

 

Andrew Morpeth
Andrew Morpethhttps://ucgeek.co/author/amorpeth/
Andrew is a Modern Workplace Consultant specialising in Microsoft technologies based in Auckland, New Zealand; Andrew is a Director and Professional Services Manager at Lucidity Cloud Services and a Microsoft MVP.

Related Articles

NZ to get native Microsoft Teams calling

New Zealand (NZ) is set to get native Microsoft Teams calling by July 2021 - this feature allows you to replace your telephony solution with Microsoft Teams. Microsoft Teams calling for NZ further reduces the barrier to using Microsoft Teams as a telephony replacement solution for New Zealand businesses.

Microsoft Teams Calling Options for Telephony Replacement

Microsoft Teams Calling provides telephony replacement capability that could replace your PABX or any other solution you may have. This article takes you through the options and when to use them, and the licencing you will require to activate this feature for your users.
00:13:43

Skype for Business Response Groups Made Easy

Call Flow Manager for Skype for Business and Lync makes creating and managing Response Groups easy! No more bouncing between interfaces to configure a simple call flow. Call Flow Manager brings all the functionality of the Response Group service into a single user interface

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Andrew Morpeth
Andrew Morpethhttps://ucgeek.co/author/amorpeth/
Andrew is a Modern Workplace Consultant specialising in Microsoft technologies based in Auckland, New Zealand; Andrew is a Director and Professional Services Manager at Lucidity Cloud Services and a Microsoft MVP.

Latest Articles

NZ to get native Microsoft Teams calling

New Zealand (NZ) is set to get native Microsoft Teams calling by July 2021 - this feature allows you to replace your telephony solution with Microsoft Teams. Microsoft Teams calling for NZ further reduces the barrier to using Microsoft Teams as a telephony replacement solution for New Zealand businesses.

Microsoft Teams Calling Options for Telephony Replacement

Microsoft Teams Calling provides telephony replacement capability that could replace your PABX or any other solution you may have. This article takes you through the options and when to use them, and the licencing you will require to activate this feature for your users.
00:13:43

Skype for Business Response Groups Made Easy

Call Flow Manager for Skype for Business and Lync makes creating and managing Response Groups easy! No more bouncing between interfaces to configure a simple call flow. Call Flow Manager brings all the functionality of the Response Group service into a single user interface
00:13:30

Office 365 Backup with Synology NAS

There are many reasons that you may want to back up your Office 365 data. The most common reasons I see are data sovereignty...

Microsoft 365 Feature Roundup Dec 2020

Feature Roundup Presentation Microsoft 365 key feature releases and announcements for June to December 2020, in an easy to digest PowerPoint format.