This error appeared on a Customer’s Central Admin site recently, basically when trying to do anything useful such as view the Service Applications, change Diagnostic Logging levels, in fact trying to do anything useful with the farm. It also manifested itself in PowerShell and if you viewed the Running Timer Jobs you noticed that they were ALL paused… NOT a good situation.
After running through a number of recommended fixes or approaches on the forums, our team pretty much came to a standstill on this one. Was it an error masking something else or was it actaully the correct error message?
So after a deal of looking around and finding nothing of use on the interweb, I contacted my fellow MVPs asking if anybody had had a similar message, and luckily one of the http://www.sharepointpt.org founders Rodrigo Pinto has experienced it in a test system using SharePoint 2007. He said it really was the real error and referred to a corrupted entry in the SharePoint Config database.
So we dived into SQL Management Studio and looked at the [dbo].[Objects] table and with the help of Notepad++ and it’s extraordinary searching capabilities, we tracked down an entry relating to the User Profile Sync service which when put into XML Notepad displayed the exact same error as we were getting through the web UI and through PowerShell.
Here’s what I found :
<sFld type="Int32" name="profileStoreLanguage">1033</sFld>
<sFld type="String" name="profileStoreLanguagePacksApplied"></sFld>
<sFld type="Int32" name="profileStoreCollationId">25</sFld>
<sFld type="Int32" name="daysWorthOfEventsToKeep">7</sFld>
<sFld type="Int32" name="nextSynchronizationStage">2</sFld>
<fld type="Microsoft.Office.Server.Administration.UserProfileApplication+ILMInstallStage, Microsoft.Office.Server.UserProfiles, Version=188.8.131.52, Culture=neutral, PublicKeyToken=71e9bce111e9429c" name="ilmInstallStage">C>0</sFld>
Now I don’t know about you, but I could see form the go get, that this was plainly the cause of the error. Position 3037 is highlighted in red.
How on earth did this happen? Well we never found out, but the customer reported that a Virtual Host had failed the night the Timer Jobs stopped running, so I can only assume some weird write error to the Config database occurred as we later discovered that the system was part way through a User Profile Sync.
How did we fix it? Well, we didn’t want to have to rebuild the farm, but as it stood we couldn’t actually de-provision the User Profile Service , as neither the UI or PowerShell were working enough to allow that, so that was looking likely, but we opted instead to carry out what is strictly an “Unsupported Action”, i.e updating a SharePoint database directly, but we felt in this case it was worth it for the sake of the customer, and ourselves.
We had a good backup of the Config database form before the incident, so I could actually extract the entry before the incident occurred and indeed we examined the entry and it was perfectly formed XML unlike the above. So after backing up all the essential databases again, shutdown the WWW service, Timer Service, Administration Service, Search Service, even FIM then I ran a simple SQL update statement to replace the above rubbish with the entry from the Config database backup and lo and behold, the farm sprung into life, timer jobs restarted, I could navigate around Central Admin and even view the User Profile Service management screen. UPS Sync even had restarted up.
Needless to say the customer was pleased as punch.
So sometimes you have do “Unsupported Actions” for the sake of the business. Just don’t make a habit of them.