Skip to content

Here is the complete, raw-text document for docs/identity/gmsa/troubleshooting.md.

\============================================================ TARGET FILE: docs/identity/gmsa/troubleshooting.md

Title: gMSA Troubleshooting, Diagnostic Engineering, and Failure Modes

  1. The Diagnostic Philosophy: Isolating Identity Plumbing When a Group Managed Service Account fails in an enterprise environment, administrators frequently mistake protocol-level breakdown for application defects. Applications running under gMSAs rarely fail due to internal application errors during cutover; they fail because the underlying operating system authority cannot construct an authenticated security token.

Troubleshooting a gMSA requires dissecting the identity lifecycle into four distinct operational layers: Layer 1: Directory Storage Layer (Active Directory attributes, security descriptors, and schema consistency) Layer 2: Key Derivation Layer (Domain Controller KDS service, root keys, and synchronization cycles) Layer 3: Transport & Channel Layer (RPC communication, Kerberos ticket acquisition, and mutual machine trust) Layer 4: Host Execution Layer (LSA injection, OS user rights assignments, and process token generation)

Every diagnostic effort must proceed sequentially from Layer 1 through Layer 4. Jumping directly to Layer 4 application logs while ignoring Layer 2 key derivation creates blind troubleshooting loops.

  1. Primary Diagnostic Matrix

Symptom 1: Error 1069 - The service did not start due to a logon failure Event Log: System Event Log, Event ID 7000 or 7005 from Service Control Manager. Underlying Mechanism: The Windows Service Control Manager (SCM) instructed the Local Security Authority (LSA) to log on the process as the target identity. The LSA queried the local Domain Controller via the MS-GKDI protocol, but the DC returned STATUS_ACCESS_DENIED (0xC0000022). Root Cause: The host member server's computer account (HOST$) is not listed in the security descriptor stored in the gMSA's msDS-GroupMSAMembership attribute, or the host's Kerberos machine token has not refreshed after being added to the authorized Active Directory security group. Remediation Sequence: Step 1: Inspect group membership on the DC to verify the computer account belongs to the authorized group. Step 2: On the target member host, purge the SYSTEM logon session ticket cache: klist -li 0x3e7 purge Step 3: Test key derivation locally via Test-ADServiceAccount -Identity . If it returns True, start the service.

Symptom 2: Test-ADServiceAccount returns False with Event ID 4001 Event Log: Microsoft-Windows-Group-Policy/Operational or Microsoft-Windows-KDS/Operational, Event ID 4001. Event Payload: "The Microsoft Key Distribution Service failed to retrieve the master root key." Underlying Mechanism: The member server successfully contacted the Domain Controller, but the Domain Controller cannot compute the derived password. Root Cause: Either no KDS Root Key has been generated in the Active Directory forest, or the KDS Root Key's EffectiveTime attribute is set to a future timestamp that has not yet elapsed (the standard 10-hour intra-forest synchronization buffer). Remediation Sequence: Step 1: On a Domain Controller, query all active root keys: Get-KdsRootKey Step 2: Inspect the EffectiveTime attribute. If the current UTC time is earlier than EffectiveTime, password computation will fail by design on all Domain Controllers until that boundary is crossed. Step 3: Verify the Key Distribution Service is running on all domain controllers: Get-Service -Name kdssvc

Symptom 3: Intermittent Logon Failures Across Multiple Domain Controllers Event Log: Microsoft-Windows-Security-Kerberos, Event ID 787 or 788. Event Payload: "The Key Distribution Center (KDC) encountered an error during gMSA password computation." Underlying Mechanism: Member servers query different Domain Controllers within an Active Directory site. Some DCs successfully return passwords while others return derivation errors or stale password strings. Root Cause: Active Directory replication latency or partition isolation. The Configuration Partition containing the Master Root Keys container (CN=Master Root Keys,CN=Group Key Distribution Service,CN=Services,CN=Configuration,DC=...) has failed to replicate to one or more Domain Controllers in the site. Remediation Sequence: Step 1: Check replication status across all Domain Controllers in the topology: repadmin /syncall /A /e /P /q Step 2: Force explicit replication of the Configuration partition: repadmin /syncall /C /e /d Step 3: Query the specific Domain Controller returning failures directly: Get-KdsRootKey -Server "DC02.lab.lan"

Symptom 4: The Clock Skew Trap (Desynchronization Failure) Event Log: System Event Log, Event ID 4 (Kerberos-Key-Distribution-Center) or Event ID 787. Event Payload: "The time difference between the client and server is greater than the allowed maximum." Underlying Mechanism: gMSA password calculation relies on an algorithm where the current time interval is an explicit mathematical input to the Pseudo-Random Function. Root Cause: Kerberos permits a maximum clock skew of five minutes (300 seconds). If a member server's clock drifts beyond five minutes relative to the domain controller, Kerberos authentication fails entirely. Furthermore, if a Domain Controller's clock drifts across a password interval boundary (msDS-ManagedPasswordInterval), it computes a different derived password than the rest of the forest, causing catastrophic authentication mismatches across clustered workloads. Remediation Sequence: Step 1: Check local time offset against the domain hierarchy: w32tm /monitor Step 2: Force local clock synchronization against the Primary Domain Controller (PDC) Emulator: w32tm /resync /force Step 3: Ensure virtualization guest time synchronization integrations (e.g., hypervisor time sync) do not fight or override domain-based NTP time propagation.

Symptom 5: Service Starts but Terminates Immediately (Missing OS Privileges) Event Log: Application Event Log or System Event Log, Event ID 7034 ("The service terminated unexpectedly"). Underlying Mechanism: The operating system successfully derived the gMSA password and initialized the process token under the gMSA's SID. However, the application process crashed during initialization when attempting to interact with the local operating system kernel. Root Cause: The gMSA lacks critical User Rights Assignments within Local Security Policy (secpol.msc). While standard accounts configured via the GUI often receive basic rights automatically, programmatic assignment requires explicit local policy configuration. Required User Rights Assignments: Right 1: SeServiceLogonRight (Log on as a service) - Required for all Windows background services. Right 2: SeNetworkLogonRight (Access this computer from the network) - Required if the service hosts RPC, SMB, or web endpoints. Right 3: SeBatchLogonRight (Log on as a batch job) - Required for Task Scheduler operations. Right 4: SeAssignPrimaryTokenPrivilege (Replace a process-level token) - Mandatory for multi-process engines such as Microsoft SQL Server Analysis Services or specialized worker pools. Right 5: SeIncreaseQuotaPrivilege (Adjust memory quotas for a process) - Mandatory for engines managing their own memory sub-allocations. Remediation Sequence: Step 1: Export local security policy: secedit /export /cfg C:\secpol.txt Step 2: Inspect lines beginning with SeServiceLogonRight, SeAssignPrimaryTokenPrivilege, and SeIncreaseQuotaPrivilege. Step 3: Add the gMSA's SID to the authorized lists and reapply policy: secedit /configure /db secedit.sdb /cfg C:\secpol.txt /areas USER_RIGHTS

  1. Low-Level Diagnostic Tooling and Commands

Inspection Command 1: Auditing the Security Descriptor (SDDL) To determine exactly which computer accounts have access to query the gMSA password, decode the binary descriptor stored on the Active Directory object: $gMSA = Get-ADServiceAccount -Identity "svc_sql_prod" -Properties msDS-GroupMSAMembership $RawSD = New-Object Security.AccessControl.RawSecurityDescriptor($gMSA."msDS-GroupMSAMembership", 0) $RawSD.GetSddlForm("All")

Inspection Command 2: Querying the Local Kerberos Ticket Cache for SYSTEM To view the cached tickets held by the computer identity (Logon ID 0x3e7): klist -li 0x3e7

Inspection Command 3: Testing SPN Uniqueness Forest-Wide To ensure a registered SPN is not duplicated on a legacy user or computer account: setspn -X -F

Inspection Command 4: Verifying Local MSA Registry State Windows caches service account registration metadata under the local registry. If an account becomes corrupted, inspect: HKLM:\SAM\SAM\Domains\Account\Users\Names Note: Access requires elevated SYSTEM context (e.g., via PsExec -s).

  1. Recovery Runbook: Total gMSA Credential Desynchronization If an operational disaster occurs where a gMSA becomes unstartable across all cluster nodes due to an administrative error or schema corruption:

Step 1: Stop all dependent services across the node cluster. Step 2: Remove the gMSA binding from member hosts: Uninstall-ADServiceAccount -Identity "svc_app" Step 3: Purge host ticket caches: klist -li 0x3e7 purge Step 4: On a Domain Controller, verify the gMSA object is enabled and inspect msDS-GroupMSAMembership. Step 5: Verify the KDS Root Key is valid and effective: Get-KdsRootKey Step 6: Re-register the gMSA on each member host: Install-ADServiceAccount -Identity "svc_app" Step 7: Perform deterministic derivation verification: Test-ADServiceAccount -Identity "svc_app" Step 8: Once Test-ADServiceAccount returns True on all cluster nodes, restart the service instances sequentially.