Lync 2013 Front End Service Stuck Starting

A common issue which can be encountered, is a Front End Service that get’s stuck starting. No amount of restarting will fix this problem, and there are a number of reasons I have come across that may cause this. Quite often there will be nothing in the event log, most likely because the service start will never time-out, and never actually fails to start. I have seen the service in this state for over 24 hours!!

1. Certificates
Microsoft documentation states that in Server 2012 to increase security, certificates that are not self signed (i.e. where the issue by and subject name are the same e.g. a Certificate Authorities certificate) should not in in the “Trusted Root Certificate Authorities” container in the Certificate store. Certificates that are NOT self signed should go in the “Intermediate Certificate Authorities”.

Microsoft states the following here:

This issue occurs because a certificate that is not self-signed was installed in the Trusted Root Certification Authorities store. This is an incorrect configuration that can cause HTTP communication between Lync servers to fail with an untrusted root certificate error. Lync Server 2013 deployments in Windows Server 2012 may experience this issue because Windows Server 2012 implements checks for a higher level of trust for certificate authentication.

Note: A self-signed certificate is defined as a certificate in which the “Issuer” property and the “Subject” property on the certificate are the same. This is normal and expected for Root Certification Authorities.
To resolve this issue, use one of the following methods:

1. If you use group policies to deploy certificates, make sure that the Trusted Root Certification Authorities store only contains self-signed certificates (certificates in which the certificate property “Subject” is the same as the certificate property “Issuer”). Move any certificates that are not self-signed certificates from the Trusted Root Certification Authorities store to the Intermediate Certification Authorities store.
2. If you import new certificates manually, make sure that you select the computer’s Trusted Root Certification Authorities store for the self-signed certificates, and the computer’s Intermediate Certification Authorities store for the certificates that are not self-signed certificates.

You can use a Windows PowerShell command to find certificates that are put in the Trusted Root Certification Authorities store incorrectly on the local computer. The following command compares the “Issuer” property and the “Subject” property of each certificate in the store, and then outputs details of certificates that do not meet the criteria of a self-signed certificate:

2. Quorum knickers in a twist
In order to fix this issue open the Lync Management Shell and try one the following commands in order, then reboot the Front End server:

IMPORTANT: Use this next one with caution! It’s safe to use this during deployment before you have live users, however it could take you out of service for a considerable amount of time for large pools. If in doubt, it’s recommended that you contact Microsoft support first.

3. Patch RTM to latest update release
First install the cumulative update and reboot. Once the server comes back up, don’t forget to run the SQL update script as per the update documentation. Once this is done you may need to forcefully stop and start the service and/or reboot the server.

To forcefully stop the stuck service run “sc queryex RTCSRV” from Windows Command Prompt. Take note of the process number then run “taskkill /f /pid <process number>”.

4. Remove Lync Server Front End Component and Windows Fabric 

• Run Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn <SqlFqdn> | Select-Object databasename,installedversion,expectedversion and verify backend databases are at the same patch level as expected
• Remove Lync updates from add and remove programs
• Attempt to start service again (it will probably fail with event viewer event id 1000 and 1001)
• Remove “Lync Server Front End Component” and “Windows Fabric” from add remove programs
• Reboot Server
• Run step 2 in “Lync Server Deployment Wizard Tool”
• Attempt to start service again (If it gets stuck starting you may need to re-install updates)
• Run “LyncServerupdateinstaller.exe” to install updates
• Reboot server