Monitor & Respond - Risky Sign-Ins Queue

Role Requirements

Procedure Scope: Administrators

Required Group Membership: Admin.SecurityOperator

Handbook Reference

Package: Identity Security

Domain: Risky Sign-Ins Queue

Modifies: Identity Risky Sign-Ins Queue

When to Perform this Operation

Twice a day: Key times such as 8am and 2pm.

Analyst Description and Importance

The risky sign-ins queue is integral to validating detections that indicate if a single sign-in event could contribute to the likelihood of account compromise. It complements identity protection’s automated remediation efforts by providing analysts the opportunity to apply human judgment to individual sign-in anomalies. While many risky sign-ins can be remediated by conditional access and risk policies, certain edge cases benefit from the analyst’s contextual understanding. By reviewing entries in this queue twice a day, analysts ensure that sign-in-based risk indicators remain accurate, timely, and effective at preventing unauthorized access. This proactive validation helps maintain the organization’s identity security posture by ensuring user risk remediation risk events maintain accuracy, heightening the effectiveness of user identity security controls.

Security Importance:

Monitoring and responding to the risky sign-ins queue ensures that identity protection controls can rely on accurate assessments of individual sign-in events, helping to mitigate the security risk of advanced threat actors performing singular sign-in based attacks that could go unverified or misclassified.

Business Importance:

Monitoring and responding to the risky sign-ins queue ensures legitimate user sign-ins are validated, helping to mitigate the business risk of an errant sign-in risk classification from enforcing actions which would prevent a user with legitimate business need from accessing their account.

Covered in this Operation

In the context of managing the risky sign-ins queue, training extends beyond simply acknowledging that a sign-in appears suspicious. Analysts must be able to interpret nuanced risk signals tied to a single authentication event. This often involves understanding environmental context, correlating IP addresses and device states, recognizing unusual sign-in patterns, and leveraging knowledge of normal user behavior. By gaining proficiency in examining the details of each sign-in event and the associated identity signals, analysts evolve from passive validators into active decision-makers. They ensure that identity protection controls remain both accurate and actionable, effectively reducing the likelihood of letting a compromised sign-in slip through or halting a legitimate business process unnecessarily.

This skillset enables analysts to carefully filter out false positives and reinforce the reliability of identity protection’s automated measures. By aligning decisions with the known-good baselines—such as compliant device states, trusted IP ranges, or normal time-of-day usage—analysts help maintain both security integrity and user productivity. This informed, context-driven approach results in a more refined security posture, focusing analysts’ efforts on genuine threats and ensuring that legitimate sign-in attempts proceed unimpeded.

Understanding Risky Sign-Ins Security Information

When a unique risky sign-in entry is clicked into, useful information becomes available. This can be used for the analysis of primary indicators.

Basic Info section:

• Request ID: Identifies the unique request associated with this sign-in, such as “f7c9b5e8-xxxx-xxxx-xxxx-xxxxxxxxxxxx”. This helps analysts correlate the event with other logs and diagnostics.
• Correlation ID: Connects this sign-in to related activities, such as “d2a1bd9c-xxxx-xxxx-xxxx-xxxxxxxxxxxx”. This helps analysts correlate the event with other logs and diagnostics.
• Sign-in Type: Indicates how the user signed in, such as “Interactive” or “Non-Interactive”. This helps analysts determine if a human user or an automated process initiated the event.
• User: Shows the display name of the user associated with the risk, such as “John Doe”. This helps analysts confirm the specific user impacted by the detection.
• Username: Lists the user’s login identifier, such as “jdoe@example.com”. This assists analysts in identifying the account associated with the risk.
• User ID: A unique identifier for the user, such as a GUID or directory object ID. This helps analysts locate the account in logs or directory searches.
• Application: Lists the application accessed during the sign-in, such as “SharePoint”, “Outlook”, or “Teams”. This helps analysts determine which service was targeted.
• Application ID: Displays the application’s unique identifier, such as “00000003-0000-0ff1-ce00-000000000000”. This helps analysts differentiate between multiple instances of similar apps and confirm exactly which resource was targeted.
• Resource: Identifies the resource requested, such as “Microsoft Graph”. This helps analysts understand the service source involved in the sign-in beyond an interactive session or application being accessed.
• Resource ID: Provides the resource’s unique identifier, such as “e1a5c9ac-xxxx-xxxx-xxxx-xxxxxxxxxxxx”. This could help an analysts correlate activity against a specific resource.
• IP address: Displays the IP address used during the sign-in, such as “192.168.1.1”. This helps analysts identify suspicious or unrecognized locations.
• Location: Indicates the geographical location of the sign-in, such as “New York, USA” or “Lagos, Nigeria”. This helps analysts detect anomalous or high-risk regions.
• Date: Shows when the sign-in occurred, such as “December 24, 2024, at 10:00 AM”. This helps analysts correlate the sign-in with other events or known travel schedules.
• Status: Describes the outcome of the sign-in, such as “Success” or “Failure”. This helps analysts quickly see if the suspicious activity actually gained access.
• Sign-in error code: Provides an error code if the sign-in failed, such as “50053”. This helps analysts identify the specific reason for failure as it pertains to Azure sign-in error codes.
• Failure reason: Explains why the sign-in failed, such as “Invalid credentials” or “Blocked by Conditional Access”. This helps analysts determine if the failure was expected or indicative of malicious behavior.
• Client app: Indicates the application or protocol used to authenticate, such as “Browser” or “Legacy Auth Client”. This helps analysts identify if the sign-in method aligns with typical user patterns or if a risky legacy method was used.

Device Info section:

• Device ID: Lists a unique identifier for the device, such as “EXP-291”. This helps analysts determine if the sign-in originated from a recognized, managed endpoint or an unknown device.
• Browser: Shows which browser was used, such as “Chrome” or “Edge”. This helps analysts identify unexpected browser usage that might indicate suspicious activity.
• Operating System: Indicates the OS of the device, such as “Windows 11” or “iOS 17”. This helps analysts see if the platform aligns with the user’s normal environment or if it’s anomalous.
• Compliant: States if the device meets compliance policies, such as “Compliant” or “Non-Compliant”. This helps analysts quickly assess if the device is meeting the organizations Intune compliance requirements or poses additional risk due to being uncompliant.
• Managed: Reveals whether the device is managed by the organization, such as “Managed” or “Unmanaged”. This helps analysts determine if corporate security policies are enforced on this device.
• Join Type: Describes how the device is joined to the directory, such as “Azure AD Joined” or “Hybrid Azure AD Joined”. This helps analysts understand the type of joining that occurred on the device.

Risk Info section:

• Time Detected: Indicates when the detection occurred, such as “December 3, 2024, at 11:00 PM”. This helps analysts track the timeline of the suspicious activity.
• Risk state: Describes the risk status, such as “At Risk”, “Dismissed”, or “Remediated”. This state allows analysts to determine if the risk is active or already resolved.
• Risk level: Displays the severity of the detection, such as “Low”, “Medium”, or “High”. This helps analysts prioritize responses to high-risk detections.
• Risk detail: Provides additional context for the detection, such “Unusual API enumeration”. This provides additional details and specifics about the detection type.
• Source: Specifies which identity provider or source performed the activity, such as “Microsoft Entra ID”. This provides data for analysts on which system the risk was detected in.
• Detection last updated: Shows the last time the risk state was updated, such as “December 24th, 2024, at 10:00 AM”. This ensures analysts work with up-to-date information.
• Sign-in time: Shows when the sign-in occurred, such as “December 24, 2024, at 9:15 AM”. This helps analysts pinpoint the timeframe of the risky event.
• IP address: Displays the IP address used during the sign-in, such as “192.168.1.1”. This helps analysts identify suspicious or unrecognized locations.
• Sign-in location: Indicates the geographical location of the sign-in, such as “New York, USA” or “Lagos, Nigeria”. This helps analysts detect anomalous or high-risk regions.
• Sign-in client: Specifies the client used during the risky event, which can include browser version and device platform. This helps analysts identify if a particular client could be at risk for vulnerabilities.
• Token issuer type: Indicates the token issuer involved, such as “Microsoft Entra ID”. This helps analysts understand if the sign-in token came from a trusted source or if something unusual occurred in token issuance.

Multifactor Authentication Info section:

• Multifactor authentication result: Indicates if MFA succeeded or failed, such as “Passed” or “Failed”. This helps analysts determine if MFA acted as an effective safeguard or if it was bypassed.
• Multifactor authentication auth method: Shows the type of MFA used, such as “Push notification” or “SMS code”. This helps analysts verify if the chosen MFA factor aligns with policy and user behavior.
• Multifactor authentication auth detail: Provides additional info about the MFA event, which can change from event to event. This helps analysts detect suspicious MFA interactions.

Conditional Access section:

• Policy Name: Identifies which conditional access policies were triggered for the request, such as “Require MFA for External Logins”. This helps analysts understand which policies applied to the sign-in.
• Grant Controls: Shows what conditions must be met to grant access, such as “Require compliant device”. This helps analysts see a quick summary of enforced security measures.
• Session Controls: Specifies session-level restrictions applied, such as “Sign-In Frequency”. This helps analysts see a quick summary of enforced security measures.
• Result: Indicates the outcome of applying the policy, such as “Success”, “Failure”, or “Not Applied”. This helps analysts determine which controls passed or failed for the sign-in event.

Report-Only section:

• Policy Name: Identifies which conditional access policies were triggered but in report-only mode for the request, such as “Require MFA for External Logins”. This helps analysts understand which policies applied but did not enforce a control to the sign-in.
• Grant Controls: Shows what conditions must be met to grant access, such as “Require compliant device”. This helps analysts see a quick summary of potentially enforced security measures.
• Session Controls: Specifies session-level restrictions applied, such as “Sign-In Frequency”. This helps analysts see a quick summary of potentially enforced security measures.
• Result: Indicates the outcome of applying the policy, which will be “Report Only: Not Applied”. This helps analysts confirm the conditional access policy did not get enforced.

Understanding Risky User Detection Technologies

Understanding that a sign-in is generating possible risk is the starting point for understanding the scope. The risk info section specifies the risk detection technology which triggered the entry. Use this section to determine if the activity aligns with being a feasible allegation and determine investigation goals for different detections. For Microsoft’s verbatim definitions of these, view the knowledge article here.

Tuning Sign-In Risk Detection Technologies

Some detection technologies can create large amounts of false positives, resulting in alert fatigue and unnecessary analyst workload. By refining what sanctioned behaviors look like—defining known-good patterns and trusted baselines—or employing compensating controls such as conditional access policies, organizations can reduce these non-actionable alerts. This allows analysts to focus their efforts where it truly improves the security posture.

Recognize When to Confirm Sign-In Compromised

Confirming a sign-in as compromised is appropriate when evidence strongly indicates unauthorized access occurred at that single event. This action is used to validate risk that informs the course of action to take within the user risk operation (e.g., prompting user password reset, blocking sign-in). Situations where analysts may observe this include:

• Successful Sign-In with True Positive: True positive malicious indicators combined with a successful login require a response, as opposed to a detection which does not result in a successful login.
• High-Value Account: A compromised sign-in to a high-privilege account increases the likelihood of the account being a legitimate target and worth taking precaution on.
• Indication of Control Bypass: Detection technologies which pertain to ways an attacker could circumvent security controls or protocols, such as “Token Issuer Anomaly” and “Anomalous Token”.

Recognize When to Confirm Sign-In Safe

Confirming a sign-in as safe applies when initial suspicion is overridden by a business use case, and contextual evidence shows it was a legitimate event. This action is used to confirm the detection would otherwise be correct, but in the context of the organization the behavior is permitted. Situations where analysts may observe this include:

• Successful Sign-In with False Positive: False positive malicious indicators combined with a successful login require a response, as opposed to a detection which does not result in a successful login.
• Validated Repeat Internal Use Case: Indicators which can be justified with an internal business use case, such a user utilizing a sanctioned VPN service or traveling for work.
• Validated Repeat External Use Case: Indicators which can be justified with an external business use case, such a guest user using a non-managed device in a way which is in accordance with organizational policy. 

Recognize When to Dismiss Sign-In Risk

Dismissing sign-in risk occurs when the sign-in does not warrant contributing to the user risk rating, as it does not qualify as a true positive or negative. This action is used to ensure user risk is not modified to confirm safety or compromise on detections which do not hold significance. Situations where analysts may observe this include:

•  Unsuccessful Sign-In: Events where the sign-in was unsuccessful indicates identity security controls worked as intended. However, it may be in the interest of some organizations to alert on specific detections which could hold significance outside of single attempts and signal that an attack on an account is taking place. 
• Validated One-Off Use Case: Indicators that can be justified by the business but do not occur at a frequency which should warrant a behavior baseline change, such as a single atypical travel alert explained by temporary business arrangements.
• Self-Remediation: If the user just completed MFA or a password reset and no further suspicious activity exists or the risk has been automatically addressed, the risk can be dismissed.

In managing the Identity Protection risky sign-ins queue, monitoring focuses on validating potential sign-in security risks, along with invalidating risks which warrant dismissal.

By evaluating relevant security data alongside any available contextual information, analysts can accurately decide on the validity of sign-in risks. This approach ensures the instances where response is warranted are identified and actioned upon, while also ensuring that the wider range of false positives are dismissed to keep sign-in risks which flow into overall user risk accurately maintained.

Review the Risky Sign-Ins Queue

Identifying risks requiring validation which have not yet reviewed by an analyst is the primary goal of monitoring. Use the training details to determine if a sign-in is compromised, safe, or if the risk should be dismissed.

In the context of risky sign-in queue management, a response action is always warranted.

This fact is due to the nature of how risky sign-in detections impact user risk scores, wherein, an item in the queue indicates the system is awaiting validation for a potential sign-in risk. When confirmed as a true negative or positive, the actions required from a user to potentially remediate their risk are updated, and as a result the efficacy of complementary identity security controls are indirectly affected. The core scope of this operation is focused the validation of these entries within the queue, abstracting away the risk reduction actions themselves which are performed using automatic remediation and/or response actions within the user risk queue.

Respond with Confirm Sign-In Compromised Action

This action applies to risky sign-in entries which have been confirmed for compromise by an analyst, and a contribution of confirmed threat actor activity will be reflected for the users’ risk.

Respond with Confirm Sign-In Safe Action

This action applies to risky sign-in entries which have been confirmed for legitimate and recurring business by an analyst, and a contribution of confirmed safe activity will be reflected for the users’ risk.

Respond with Dismiss Sign-In Risk Action

This action applies to risky sign-in entries which have been either confirmed for legitimate but uncommon business or for not being a true negative or positive by an analyst, and no contribution of activity will be reflected for the users’ risk.

Need Assistance?