How to troubleshoot High CPU Usage / Memory Leaks / Performance Issues in Sitecore - Part 1

 

Summary

The blog is intended to provide guidance on how to troubleshoot any High CPU Usage /  slow performance / memory leaks / hangs by a process in Internet Information Services (IIS). Also this could be used across any applications hosted in IIS but not limited to Sitecore CMS. 

Details

Symptoms of High CPU Usage would include latency in web page response times or slow performing server. There are different ways and various tools and technologies to troubleshoot this issue however I will limit it to default Microsoft Debug Tool. 

First step to troubleshoot this is to create a Memory Dump of the IIS Process when IIS is still processing the client requests. In a Sitecore / .Net hosted app typically this process would be w3wp process. 

Microsoft's Debug Diagnostics Tool comes in very handy and can be used to capture, analyze memory dump files when IIS processes client requests.

How to Configure Debug Diagnostics Tool: 

• Once the tool is downloaded open DebugDialog.Collection.exe from the configured folder ( default folder - C:\Program Files\DebugDiag) and navigate to Tools - Options and Settings - Performance Log - Enable Performance Counter Data Logging as shown below 

• Next step is to create a dump file which can be done using two routes such as a configuration of a hang rule or using manual method. In this series I'll explain the automated process.  

Method 1 - How to create and configure Rules: 

• Navigate to the rules tab and add a new rule. For this series let us assume that it's a performance issue and select & configure the Performance option as below: 

• As explained in the screenshot below this can be done based on different triggers either using the Counters Threshold limit or latency in the Response Times. Assuming we are experiencing very slow responses I will select HTTP Response Times option.

• Configure the website URL here where we are experiencing the slowness and update the timeout or ping options based on the application. Default setup looks at any host that includes with or without https and both ports. There is an option here to specify a specific folder / path / virtual directory as well. 

• Prerequisite - If you choose the ETW option make sure to have tracing enabled on your server. Depending on whether it's a Windows 10 or Windows Server here are few instructions from Microsoft on how to enable it. 

• Last step is to configure the dump target, location and frequency of the dump files. While creating the targets select application pool or w3wp process. 

• Once the rule is activated browse the configured site / web service in any browser. IIS will start tracking and based on the configurations provided earlier dumps will be generated in the configured folder. 

In the next part of this blog I'll show how to read / analyze these dumps. 

Sitecore XConnect Operation #0, AlreadyExists, Contact and XConnect FacetOperationException: Operation #1, ReferenceNotFound Issue

Description: We have been seeing below issues when trying to save or retrieve XConnect Data

2022-02-17 13:56:51.743 -05:00 [Error] Sitecore.XConnect.Operations.AddContactOperation: Sitecore.XConnect.Operations.EntityOperationException: Operation #0, AlreadyExists, Contact 2022-02-17 13:56:51.743 -05:00 [Error] Sitecore.XConnect.Operations.SetFacetOperation`1[Sitecore.XConnect.Facet]: Sitecore.XConnect.Operations.FacetOperationException: Operation #1, ReferenceNotFound, Contact, Classification 2022-02-17 13:56:51.743 -05:00 [Error] ["XdbContextLoggingPlugin"] XdbContext Batch Execution Exception Sitecore.XConnect.Operations.EntityOperationException: Operation #0, AlreadyExists, Contact 2022-02-17 13:56:51.743 -05:00 [Error] ["XdbContextLoggingPlugin"] XdbContext Batch Execution Exception

Inorder to resolve this issue there are multiple checks one has to go through :

Scenario 1: 

Check with Sitecore as Sitecore already has a hot fix for below issue: 

Sitecore XP 9.0.0: SC Hotfix 316493-1.zip

Sitecore XP 9.0.1: SC Hotfix 307306-1.zip

Sitecore XP 9.0.2: SC Hotfix 307348-1.zip

Sitecore XP 9.1.0: SC Hotfix 329879-1.zip

Sitecore XP 9.1.1: SC Hotfix-343592-1.zip

Sitecore XP 9.2.0: SC Hotfix 490409-1.zip

Sitecore XP 9.3.0: SC Hotfix 449095-1.zip

Here is the reference to the Knowledge Center Article for Sitecore Support. 

Note : Make sure to apply this fix in Standalone instances and test it thoroughly before proceeding to  CM and CD instances. Also this fix should be applied first in CM before any modifications in CD Instances.  

Tip: You can verify this dll fix by navigating to the bin folder and look at the properties : 

Once the above hotfix is applied it should fix the issue. 

If the issue still persists continue below..

Scenario 2: 

Verify the shared session state settings and make sure that: 

• Session State Stores such as Private and Shared are in sync between the config files. Tip: Use instance's show admin page where you can view all the configurations ( including the patch files ).
• Similarly verify Session State Processes ( In-Proc , Out-of-Proc / Custom ) are in sync. 
• Finally check the Load Balancer configuration as the settings for above would depend on whether if Sticky Sessions are configured. Reminder - We can only configure one Session State Server for one CD Cluster. 

Refer to this Knowledge Center Article to tune these settings. 

Scenario 3: 

In our scenario the above fix partially fixed the issue however CPU was running high and Sessions were locked and ended up in a Deadlock Sate. 

After thorough research and help from Sitecore from we found out : 

• SQL Sessions on the server were blocked. We had to end the sessions, recycle App Pool. 
• Removed orphaned Contact which had high number of Interactions using Sitecore ADM Module. This module comes in very handy to maintain Analytical Data, rebuild indexes and purge Contact Data when not needed. Here is a link to the module - Sitecore ADM Module

Hope this helps. Feel free to share any similar experiences and resolution paths. 

Sitecore Security Patch for Sitecore 8.2, 9.0 and 9.1

*** IMP: Below should be installed only if recommended by Sitecore Support. Do check with them if it's needed for your Environment Setup. ***

We were recently contacted by Sitecore who identified and recommended a patch for a security issue (SC2019-001-302938). This being a security issue I highly suggest technical or admins to check with your local Sitecore Rep and apply below fix in your Sitecore Environments if needed. 

Again make sure to verify with Sitecore before making any changes.

Versions Affected: 

All versions of Sitecore XP 8.2, all versions of Sitecore XP 9.0, and Initial Release of Sitecore XP 9.1.

Versions not affected

Sitecore CMS/XP versions 6.3—8.1 are not vulnerable.

Sitecore xDB Cloud environments are not affected.

In order to fix this issue apply the following patch (compatible with all affected versions): Sitecore.Support.302938-9.0.1.1 All related instructions are in the read me file.

Additional details about the issue are found here - https://kb.sitecore.net/articles/760963

Sitecore 9 High CPU Usage Issue

Recently I came across a strange issue where 100% of the CPU was consumed as soon as I started running Sitecore 9 instance and Sitecore website in my local environment.

Per Sitecore's suggestion we had a laptop with decent configuration (Windows 10 64 Bit, 16GB RAM with SSD an so on). As I debugged into log files I couldn't notice any obvious reasons. However when I looked into task manager I found out that the JAVA Binary was consuming 95% of the CPU. Once I stopped SOLR, CPU usage was back to normal. This may all sound rather obvious but I was banging my head against the desk for about an hour before I tracked it down.:( 

I found out that the path that we had in the Environment variables wasn't valid anymore and was the culprit for the RAM and MEMORY consumption!!!  Not sure if the java automatically updated to newer version which caused the changes or something else.. Either way once I changed the path everything was back up to normal. 

Also one another step was to restart the Sitecore XConnect services. 

Hope this helps someone!

How to fix Sitecore View Rendering Visual Studio Intellisense issues

While working on a Helix implementation or even regular Sitecore implementation I came across various intellisense issues in Visual Studio. Some of them are descriptive and some of them aren't. Here are few sample errors that you might come across:

• The type or namespace name 'Helpers' does not exist in the namespace 'System.Web' .
• The type or namespace name 'xyz' does not exist in the namespace 'System.Web.Mvc'.
• 'System.Web.Mvc' doesn't contain a definition for Sitecore and no extension method Sitecore accepting first argument of type.....

You should be able to resolve this issue by following one or more of these steps: 

1. Update your view's Web.config file with Sitecore's Web.config file as below. Make sure to clean and rebuild the solution once done. 
2. Once you add the config file and if you are still experiencing issues make sure that the referenced dll is added to the solution. For ex: As above error if it complain about System.Web.dll make sure that it's referenced properly by Nuget. 
3. Make sure that the dll exists in the bin directory. If not set the Copy Local property in Visual Studio for the dll. Specially for Sitecore make sure that Sitecore.Kernel, Sitecore.Mvc, System.Web, System.Web.Mvc, Glass Mapper dlls(if you are using glass) are referred in the solution. 
4. In VS 2017 when I experienced this issue I resolved it by updating the DotNetCompilerPlatform by adding one more compiler option in the web.config. The default is 1659;1699;1701; I added 1702 as below:
   

These are few of the common scenarios and solutions which fixed my issues. If you came across a different solution please feel free to share.

Sitecore 9 Experience Platform Installation Steps using SIFLESS - A Step by Step Guide

Loooong List of Prerequisites:

1.  Visual Studio 2017
2.  Sql Server 2016 SP1
3.  Sql Server Management Studio - – (** Make sure to set up above sql server user ( You can create any username. For local work stations I use sa account. ) with SQL   Server Authentication. Make sure to uncheck enforce password policy and select sysadmin   role)
4. Run below script in SQL Server ( This came straight from Sitecore ) : 
sp_configure ‘contained database authentication’, 1;
GO
RECONFIGURE;
GO
5. .Net Framework 4.6.2 or later
6.  IIS(v10)
7.  Microsoft Web Platform Installer - Once you download this you should be able to download below by searching and installing it directly from Web Installer.
1. Web Deploy 3.6
2.  URL Rewrite 2+
3.  Microsoft SQL Server Data-Tier Application Framework(17.1)
4.  Microsoft SQL Server Transact-SQL ScriptDom 
8.     JDK and Java Runtime Environment
9.   Windows Powershell 5.1+
10.   Microsoft® SQL Server® Data-Tier ApplicationFramework
11.   Sitecore Installation Framework - Refer to Download options for On Premises deployment.
12.   Sitecore XP Package for XP Single 
13.   Microsoft Visual C++ 2015 Redistributable Update3
14.   SOLR 6.6.2
15.   SIFLESS Package
16.   Download Non-sucking service manager (NSSM) tool
17.   Once you have IIS enabled make sure to enable below below settings (screenshot below) :

a.       HTTP Redirection

b.      Web DAV Publishing
c.       .NET Extensibility

d.      ASP.NET 4.6

e.       IIS Hostable Web Core

f.        SMB 1.0/CIFS File Sharing Support

Step 1 - Set up Installation Directories:

• Create a folder by name “Sitecore” and another folder by name “install ( inside Sitecore ) ” in your C: Drive. ( C:\Sitecore\install ). This could be any available drive. 
• Copy license file inside install folder.
• From step 11 above you should have downloaded the Sitecore packages. Open Sitecore 9.0.2 rev. 180604 (WDP XP0 packages).zip and copy Sitecore 9.0.2 rev. 180604 (OnPrem)_xp0xconnect.scwdp.zip and Sitecore 9.0.2 rev. 180604 (OnPrem)_single.scwdp.zip into C:\Sitecore\install.
• Copy only files of XP0 Configuration files 9.0.2 rev. 180604.zip into C:\Sitecore\install.
• Your folder should now look like:

Step 2 - Configure SOLR:

• Make sure to install java libraries before proceeding.   
• Next Step is to set up Java Environment Variables. Go to View Advanced Settings in Windows 10 and click on Environment Variables. Add new one as "JAVE_HOME" as below: 

Edit the Path variable and add a new one as below:

• Extract the contents of SOLR 6.6.2 into C:\Sitecore\folder. 
• Quick Check - Navigate to bin folder and start solr to see if it's working.

If you see any issues while running the above step it is due to a bug in solr cmd. Make changes in cmd as described here - https://trayek.wordpress.com/2018/06/06/installing-solr/

Step 3 - Configure SOLR as Service using NSSM:

• Download Non-sucking service manager (NSSM) tool and extract contents under C:\Sitecore folder. Rename the folder as nssm. 
•  Open CMD prompt as admin and navigate to nssm bin folder and run nssm install command as below.

• Once you enter all info click Install Service which should install it as a service and you should now be able to start it as a windows service from services.msc. 
•  Verify if you can browse solr at http://localhost:8983/solr. 

Step 4 - Sitecore 9 is all about Security.... Time to add SSL to SOLR and make it Secure

•  Instead of creating this manually sitecore community has some very good powershell scripts to do the same. You can download the script from here and drop it into Sitecore installation folder.
•  Script uses Java KeyTool and we need to update the script with local paths. Open the script and replace keyTool ( found in line 34 or so ) Path as below:
• try { $keytool = (Get-Command 'C:\Program Files\Java\jre-10.0.2\bin\keytool.exe').Source }

** Make sure to verify your path.

• Run the powershell window as admin and navigate to installation directory. As an argument you have to pass in the solr directory ( server \ etc folder ) while running the command. See below:

• You can confirm above step by navigating to the folder and verify if the keys are created.
• Few more things before we wrap this up - Open solr.in.cmd from bin folder and remove the comments in SSL Section as below:

• Save the above file and restart the service. You should now be able to navigate via https as this - https://localhost:8983/solr/#/

Step 5 - Configure SIF

• In Windows, launch PowerShell as an administrator. 
• To register the repository, in a PowerShell command line, run the following cmdlet: 

Register-PSRepository -Name SitecoreGallery -SourceLocation https://sitecore.myget.org/F/sc-powershell/api/v2

• Install the PowerShell module by running the following cmdlet: Install-Module SitecoreInstallFramework 
• When prompted to install, press Y, and then press ENTER. 
• Install Sitecore Framework using Install-Module SitecoreInstallFramework 
• Install Sitecore Fundamentals Module using Install-Module SitecoreFundamentals. 
• Import both modules – Import-Module SitecoreInstallFramework, Import-Module SitecoreFundamentals. 
• See below if needed.

Last Step – Install Sitecore using SIFLESS

• Extract SIFLESS zip into your C: Drive and run SIFLESS.exe as an Admin from extracted folder and then fill in all the details as below.
• Click Test and you shouldn’t see any issues. If you have any fix them before proceeding. Run Generate Files and Install. That’s it!!! 

How to Enhance Sitecore Content Editor Performance

This post is written to provide few tips to improve Sitecore Content Editor performance for Sitecore users. These tips have been uncovered during the development phases.

1. Number of items under any parent item affects how quickly that parent renders. As Sitecore suggests try to limit these items to 25 and avoid items more than 100 items under one parent item. This will have a huge impact on Sitecore performance.                    
2. While creating a data template organize fields into different sections rather than having all of them in a single section. Also remember to collapse sections that aren't used frequently.                                                                                                                          
3. For any item hide the sections and fields inherited from Standard Template. In order to do the same login into Sitecore Content Editor. Click view in the ribbon and uncheck Standard Fields option.

    4.  Application Options will also have a minimal impact on client performance. 

         Disable the options that aren't used frequently. 

         Login into Sitecore using content editor and click Sitecore icon on the top left                          corner and select Application Options.

       5. While creating a list type field types consider using TreeListEx field type since it only             processes and displays options when a user clicks the Edit command to update the               field value.                                                                                                                        

       6. If your site is configured with Page editor then I would suggest using Page Editor                   than Content Editor.                                                                                                            

       7. Encourage smart or Incremental publishing for publishing the changes to web                        database.                                                                                                                               

      8. Enable static and dynamic HTTP compression in IIS. This speeds up both the                       Sitecore client and the public web site and is especially beneficial in environments                 with a low bandwidth.

     9. Set the ContentEditor.RenderCollapsedSections setting to False. This is a hidden                  setting that is set to True by default. Changing this setting can help a lot.                           

   10.  Sitecore recommends to have no more than 5-10 versions per item in Master                         Database.