Troubleshooting a SharePoint 2010 Service Pack 1 install

I was handed a SharePoint 2010 farm with some weird issues going on, specifically around PowerPivot.

First thing I checked was the status of any updates. Sure enough, there was an upgrade available (as seen in Central Admin > Manage Servers in this Farm). My continual troubleshooting of PowerPivot is in this other post.

I jumped to the SharePoint Management Shell and ran

psconfig -cmd upgrade -inplace b2b -force

And much to my surprise (NOT) I ran into the following error

After some research online, I came across KB939308, which instructs to clear the timer job cache. I did that, and received the same error. I cleared the timer cache again and then it progressed past this error. I’ve had this “clearing of the timer cache twice” occur elsewhere as well, for some reason my initial attempt at clearing the timer cache never works well for me.

Rerunning the psconfig command progressed a little further than last time, but then a new error occurred.

An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfig urationTaskException was thrown.  Additional exception information: The upgrade command is invalid or a failure has been encountered. Failed to upgrade SharePoint Products.

and from the upgrade logs

03/29/2012 13:28:54  12  ERR            An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown.  Additional exception information: The upgrade command is invalid or a failure has been encountered. Failed to upgrade SharePoint Products. Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException: Exception of type ‘Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException’ was thrown.    at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()    at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

I found a lot of posts online, this issue is due to service applications’ associated services being stopped. I reviewed this and saw that most of the services were stopped on the server. I confirmed what was needed and removed the service applications for what was not needed, and started the services for what was needed.

Ran psconfig again, went a little further but ended up getting the same errors again. I then checked out the Upgrade-DATE-error.log file (didn’t see that one before) and that revealed a lot.

[OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [INFO] [3/29/2012 2:31:41 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [ERROR] [3/29/2012 2:31:41 PM]: Application Web Config for this IIS site (52113925) could not be found at C:\inetpub\wwwroot\wss\VirtualDirectories\site.domain.local80\web.config. [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [INFO] [3/29/2012 2:31:41 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [ERROR] [3/29/2012 2:31:41 PM]: Unexpected error updating SafeControl entries. Object reference not set to an instance of an object. [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [INFO] [3/29/2012 2:31:41 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [UpgradeWebConfig (4.0.2.0)] [ERROR] [3/29/2012 2:31:41 PM]: Application Web Config for this IIS site (52113925) could not be found at C:\inetpub\wwwroot\wss\VirtualDirectories\site.domain.local80\web.config. [OWSTIMER] [SPIisWebSiteWssSequence] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPIisWebSiteWssSequence] [ERROR] [3/29/2012 2:31:42 PM]: Action 4.0.2.0 of Microsoft.SharePoint.Upgrade.SPIisWebSiteWssSequence failed. [OWSTIMER] [SPIisWebSiteWssSequence] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPIisWebSiteWssSequence] [ERROR] [3/29/2012 2:31:42 PM]: Exception: Object reference not set to an instance of an object. [OWSTIMER] [SPIisWebSiteWssSequence] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPIisWebSiteWssSequence] [ERROR] [3/29/2012 2:31:42 PM]: at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.EnsureAssemblyRedirection(XmlDocument webConfig) at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.Upgrade() at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: Upgrade [Microsoft.SharePoint.Administration.SPIisWebSite] failed. [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: Inner Exception: Object reference not set to an instance of an object. [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.EnsureAssemblyRedirection(XmlDocument webConfig) at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.Upgrade() at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: Exception: Action 4.0.2.0 of Microsoft.SharePoint.Upgrade.SPIisWebSiteWssSequence failed. [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: Microsoft.SharePoint.Administration.SPIisWebSite [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) [OWSTIMER] [SPUpgradeSession] [INFO] [3/29/2012 2:31:42 PM]: No context object [OWSTIMER] [SPUpgradeSession] [ERROR] [3/29/2012 2:31:42 PM]: Upgrade Timer job is exiting due to exception: Microsoft.SharePoint.Upgrade.SPUpgradeException: Action 4.0.2.0 of Microsoft.SharePoint.Upgrade.SPIisWebSiteWssSequence failed. —> System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.EnsureAssemblyRedirection(XmlDocument webConfig) at Microsoft.SharePoint.Upgrade.UpgradeWebConfig.Upgrade() at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() — End of inner exception stack trace — at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade() at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPPersistedUpgradableObject.Upgrade(Boolean recursively) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.ReflexiveUpgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Upgrade.SPUpgradeSession.Upgrade(Object o, Boolean bRecurse) at Microsoft.SharePoint.Administration.SPUpgradeJobDefinition.Execute(Guid targetInstanceId)

As you’ll see it mentions the web.config file for the web application, and that it cannot be found. I went and looked myself and discovered that was true, the web.config file was no longer present. After further review, it appears that most of the web applications on this server were stopped in IIS, and have been moved to another farm. This farm remained as a PowerPivot play ground only.

I spoke with the customer to confirm the sites were moved, and I also ran this quick script that loops through the web applications and pings each one, confirming the IP address was not this farm’s IP.

$webapps = Get-SPWebApplication

foreach($web in $webapps)

{

ping $web.url.replace(“http:”,””).replace(“https”,””).replace(“/”,””) -n 1

}

One web app came back as still on this server, and it was the web app used to run PowerPivot. I went ahead and deleted all of the other web applications and reran psconfig.

SUCCESS!

The upgrade finally finished and the server status was back to No Action Required. Now on to troubleshooting PowerPivot.