Troubleshooting a SharePoint 2010 Service Pack 1 install

I was handed a SharePoint 2010 farm with some weird issues going on, specifically around PowerPivot.

First thing I checked was the status of any updates. Sure enough, there was an upgrade available (as seen in Central Admin > Manage Servers in this Farm). My continual troubleshooting of PowerPivot is in this other post.

I jumped to the SharePoint Management Shell and ran

psconfig -cmd upgrade -inplace b2b -force

And much to my surprise (NOT) I ran into the following error

An exception of type Microsoft.SharePoint.Administration.SPUpdatedConcurrencyExc eption was thrown.  Additional exception information: An update conflict has occ urred, and you must re-try this action. The object SPUpgradeSession Name=Upgrade -20120329-125109-783 was updated by domain\sp_admin, in the PSCONFIG (8484) process, on machine SHAREPOINT.  View the tracing log for more information abou t the conflict.

and from the upgrade log

03/29/2012 12:52:08  12  ERR            Task upgrade has failed with an unknown exception 03/29/2012 12:52:08  12  ERR            Exception: Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException: An update conflict has occurred, and you must re-try this action. The object SPUpgradeSession Name=Upgrade-20120329-125109-783 was updated by domain\sp_admin, in the PSCONFIG (8484) process, on machine SHAREPOINT.  View the tracing log for more information about the conflict.    at Microsoft.SharePoint.Administration.SPConfigurationDatabase.StoreObject(SPPersistedObject obj, Boolean storeClassIfNecessary, Boolean ensure)    at Microsoft.SharePoint.Administration.SPConfigurationDatabase.Microsoft.SharePoint.Administration.ISPPersistedStoreProvider.PutObject(SPPersistedObject persistedObject, Boolean ensure)    at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdate()    at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Update()    at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ContinueOnLocalThread(Guid id, Boolean consoleOutput)    at Microsoft.SharePoint.Upgrade.SPManager.ContinueSessionOnLocalThread(Guid id, Boolean consoleOutput)    at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()    at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

After some research online, I came across KB939308, which instructs to clear the timer job cache. I did that, and received the same error. I cleared the timer cache again and then it progressed past this error. I’ve had this “clearing of the timer cache twice” occur elsewhere as well, for some reason my initial attempt at clearing the timer cache never works well for me.

Rerunning the psconfig command progressed a little further than last time, but then a new error occurred.

An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfig urationTaskException was thrown.  Additional exception information: The upgrade command is invalid or a failure has been encountered. Failed to upgrade SharePoint Products.

and from the upgrade logs

03/29/2012 13:28:54  12  ERR            An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown.  Additional exception information: The upgrade command is invalid or a failure has been encountered. Failed to upgrade SharePoint Products. Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException: Exception of type ‘Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException’ was thrown.    at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()    at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

I found a lot of posts online, this issue is due to service applications’ associated services being stopped. I reviewed this and saw that most of the services were stopped on the server. I confirmed what was needed and removed the service applications for what was not needed, and started the services for what was needed.

Ran psconfig again, went a little further but ended up getting the same errors again. I then checked out the Upgrade-DATE-error.log file (didn’t see that one before) and that revealed a lot.

[OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [INFO] [3/29/2012 2:31:41 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [ERROR] [3/29/2012 2:31:41 PM]: Application Web Config for this IIS site (52113925) could not be found at C:\inetpub\wwwroot\wss\VirtualDirectories\site.domain.local80\web.config. [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [INFO] [3/29/2012 2:31:41 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [ERROR] [3/29/2012 2:31:41 PM]: Unexpected error updating SafeControl entries. Object reference not set to an instance of an object. [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [INFO] [3/29/2012 2:31:41 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [ERROR] [3/29/2012 2:31:41 PM]: Application Web Config for this IIS site (52113925) could not be found at C:\inetpub\wwwroot\wss\VirtualDirectories\site.domain.local80\web.config. [OWSTIMER] [SPIisWebSiteWssSequence] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPIisWebSiteWssSequence] [ERROR] [3/29/2012 2:31:42 PM]: Action 4.0.2.0 of Microsoft.SharePoint.Upgrade.SPIisWebSiteWssSequence failed. [OWSTIMER] [SPIisWebSiteWssSequence] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPIisWebSiteWssSequence] [ERROR] [3/29/2012 2:31:42 PM]: Exception: Object reference not set to an instance of an object. [OWSTIMER] [SPIisWebSiteWssSequence] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPIisWebSiteWssSequence] [ERROR] [3/29/2012 2:31:42 PM]: at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.EnsureAssemblyRedirection(XmlDocument webConfig) at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.Upgrade() at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: Upgrade [Microsoft.SharePoint.Administration.SPIisWebSite] failed. [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: Inner Exception: Object reference not set to an instance of an object. [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.EnsureAssemblyRedirection(XmlDocument webConfig) at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.Upgrade() at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: Exception: Action 4.0.2.0 of Microsoft.SharePoint.Upgrade.SPIisWebSiteWssSequence failed. [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: No context object [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: Upgrade Timer job is exiting due to exception: Microsoft.SharePoint.Upgrade.SPUpgradeException: Action 4.0.2.0 of Microsoft.SharePoint.Upgrade.SPIisWebSiteWssSequence failed. —> System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.EnsureAssemblyRedirection(XmlDocument webConfig) at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.Upgrade() at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() — End of inner exception stack trace — at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPUpgradeJobDefinition.Execute(Guid targetInstanceId)

As you’ll see it mentions the web.config file for the web application, and that it cannot be found. I went and looked myself and discovered that was true, the web.config file was no longer present. After further review, it appears that most of the web applications on this server were stopped in IIS, and have been moved to another farm. This farm remained as a PowerPivot play ground only.

I spoke with the customer to confirm the sites were moved, and I also ran this quick script that loops through the web applications and pings each one, confirming the IP address was not this farm’s IP.

$webapps = Get-SPWebApplication
foreach($web in $webapps)
{
ping $web.url.replace(“http:”,””).replace(“https”,””).replace(“/”,””) -n 1
}

One web app came back as still on this server, and it was the web app used to run PowerPivot. I went ahead and deleted all of the other web applications and reran psconfig.

SUCCESS!

The upgrade finally finished and the server status was back to No Action Required. Now on to troubleshooting PowerPivot.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s