Sitecore XConnect Operation #0, AlreadyExists, Contact and XConnect FacetOperationException: Operation #1, ReferenceNotFound Issue

Description: We have been seeing below issues when trying to save or retrieve XConnect Data

2022-02-17 13:56:51.743 -05:00 [Error] Sitecore.XConnect.Operations.AddContactOperation: Sitecore.XConnect.Operations.EntityOperationException: Operation #0, AlreadyExists, Contact 2022-02-17 13:56:51.743 -05:00 [Error] Sitecore.XConnect.Operations.SetFacetOperation`1[Sitecore.XConnect.Facet]: Sitecore.XConnect.Operations.FacetOperationException: Operation #1, ReferenceNotFound, Contact, Classification 2022-02-17 13:56:51.743 -05:00 [Error] ["XdbContextLoggingPlugin"] XdbContext Batch Execution Exception Sitecore.XConnect.Operations.EntityOperationException: Operation #0, AlreadyExists, Contact 2022-02-17 13:56:51.743 -05:00 [Error] ["XdbContextLoggingPlugin"] XdbContext Batch Execution Exception

Inorder to resolve this issue there are multiple checks one has to go through :

Scenario 1: 

Check with Sitecore as Sitecore already has a hot fix for below issue: 

Sitecore XP 9.0.0: SC Hotfix 316493-1.zip

Sitecore XP 9.0.1: SC Hotfix 307306-1.zip

Sitecore XP 9.0.2: SC Hotfix 307348-1.zip

Sitecore XP 9.1.0: SC Hotfix 329879-1.zip

Sitecore XP 9.1.1: SC Hotfix-343592-1.zip

Sitecore XP 9.2.0: SC Hotfix 490409-1.zip

Sitecore XP 9.3.0: SC Hotfix 449095-1.zip

Here is the reference to the Knowledge Center Article for Sitecore Support. 

Note : Make sure to apply this fix in Standalone instances and test it thoroughly before proceeding to  CM and CD instances. Also this fix should be applied first in CM before any modifications in CD Instances.  

Tip: You can verify this dll fix by navigating to the bin folder and look at the properties : 

Once the above hotfix is applied it should fix the issue. 

If the issue still persists continue below..

Scenario 2: 

Verify the shared session state settings and make sure that: 

• Session State Stores such as Private and Shared are in sync between the config files. Tip: Use instance's show admin page where you can view all the configurations ( including the patch files ).
• Similarly verify Session State Processes ( In-Proc , Out-of-Proc / Custom ) are in sync. 
• Finally check the Load Balancer configuration as the settings for above would depend on whether if Sticky Sessions are configured. Reminder - We can only configure one Session State Server for one CD Cluster. 

Refer to this Knowledge Center Article to tune these settings. 

Scenario 3: 

In our scenario the above fix partially fixed the issue however CPU was running high and Sessions were locked and ended up in a Deadlock Sate. 

After thorough research and help from Sitecore from we found out : 

• SQL Sessions on the server were blocked. We had to end the sessions, recycle App Pool. 
• Removed orphaned Contact which had high number of Interactions using Sitecore ADM Module. This module comes in very handy to maintain Analytical Data, rebuild indexes and purge Contact Data when not needed. Here is a link to the module - Sitecore ADM Module

Hope this helps. Feel free to share any similar experiences and resolution paths.