This blog explains how you can detect malicious OAuth applications lurking in your organization. It is not simply from a static posture perspective, but goes way deeper into behavior and usage as well. While malicious OAuth apps are inherently risky, consented OAuth apps that later go rogue are often far more difficult to detect which is why a strong security strategy must address both.
Let’s start with the basics. What are OAuth applications? OAuth applications are third-party or internal integrations that use the OAuth authorization framework to access data or perform actions on behalf of a user without ever seeing the user’s password. A few examples are:
- In platforms like Microsoft 365 / Entra ID, an OAuth app may request permissions such as reading mailboxes, accessing files in OneDrive, or querying directory data through Microsoft Graph
- In Salesforce, connected apps can obtain tokens to read or modify CRM records, run APIs, or automate workflows
- In GitHub, OAuth apps can access repositories, issues, or organization data depending on granted scopes
While OAuth enables secure and seamless integrations, it also introduces risk if one does not have proper detections in place. Once a user or admin grants consent, the app receives persistent token-based access, meaning a malicious or compromised app can operate quietly in the background, often with privileges that extend far beyond what users realize.
What is the risk?
There are two broader types of risks associated with OAuth applications.
1. Malicious OAuth App
A significant threat involves initial access, where an attacker tricks a user into granting permission to a rogue OAuth application through phishing. Identifying whether the application is malicious is crucial. Often, there are clear indicators of malicious intent, such as an application impersonating a well-known OAuth App.
Detection often relies on scrutinizing the application's metadata, such as its name, redirect URL, developer website, and other details. A thorough check of this information can expose a malicious imposter masquerading as the legitimate OAuth application.
With sufficient metadata and well-designed prompts, large language models (LLMs) can be invaluable tools. They can analyze naming conventions, contextual clues, and subtle anomalies to determine if an OAuth application is legitimate or an attempt to impersonate a trusted App.
This risk is easily mitigated by disconnecting and remediating the rogue application. Taking these steps can prevent an attacker from causing damage with the permissions granted to the app.
2. OAuth App going Rogue
OAuth applications can become rogue due to several potential scenarios:
- A Third-Party Breach: This involves a security compromise affecting an external service, such as the breach that occurred at SalesLoft.
- System Compromise: An internal OAuth application running on a machine that is compromised, allowing a malicious actor to gain access to the application's tokens.
- Malicious Insider Activity: An authorized internal user exploits the permissions granted to an existing OAuth application for unauthorized purposes.
A legitimate, functioning OAuth application going rogue is a particularly complex challenge to detect. The detection difficulty varies by threat model, while attacker-driven scenarios are already challenging, insider-driven scenarios raise the complexity even further due to trusted access and familiarity with normal operational patterns.
An application is considered "going rogue" when its behavior deviates suddenly from its established patterns. This could manifest as:
- Geographic Deviation: Refreshing its tokens from an entirely new, never-before-seen location.
- Client Deviation: Calling APIs using client applications that were not previously associated with it.
- Operational Anomaly: Performing operations in terms of type, speed, and frequency that are drastically different from its historical activity.
Steps for detecting OAuth apps gone rogue
- Baselining the application
- Simple rule-based detections
- Machine learning based detections
- Agentic triaging of detections
Baselining the application
Before baselining the application, the most important part is understanding the different signals and events associated with the OAuth activities.
From app to app, the place where the signals are present and their availability differ. For example:
- For Microsoft, you need to rely on both interactive and non-interactive logs. In some cases, we need to look at the events coming from legacy APIs alongside Graph API logs.
- For Google, you need to look at OAuth Token Audit Activity Events, Login Audit Activity Events, and every Google service-specific Audit Log.
- Similarly, for GitHub, you need GitHub Audit Logs as well as the GitHub API logs, which are available when you stream logs from GitHub.
Every single app has its own nuances regarding where the audit logs are located and how they are to be used and interpreted.
Using LLMs effectively can help classify, catalog, enrich, and tag the audit logs well at scale. Unfortunately, verification of LLM based classifications is necessary because some applications' audit log behavior does not align with their documented capabilities, and without proper classification, the effectiveness of detections will be low. Once the context and semantics of the logs are clear, the baselining can begin.
At the higher level, we can baseline the basic vitals for (not an exhaustive list):
- IP address
- The Client & its version
- The Operating System of the Client.
- Location
- ASN
- Activity pattern:
- Time of the Day
- Day of the Week
- Operations
- Type (Privileged/Non-Privileged)
- Sensitivity
Depending on the Type of SaaS application, the baselining can include several different attributes. Such as:
- Microsoft events can have a dimension of the application ID, application name, whether it is a first-party app, etc.
- GitHub events could have details on the different values of Programmatic Access Type.
Inferring whether an OAuth application is co-located with the user or is a confidential application running in a different data center is a key element derived from baselining data. This crucial piece of information significantly helps minimize false positives within the detection process.
Simple rule-based detections
Baselining over a very short period of time provides enlightening insights into the app usage. Then, simple rules can be created to detect basic deviations from the baseline. Rules covering anomalies around:
- Infrequent & Unseen ASNs
- Client Version downgrades
- Unseen & Infrequent Operations
- Unseen Client Operating Systems
- Usage deviations from activity patterns of time of day and day of week
- And more!
These rules cover a wide range of anomalous OAuth app usage and can, in fact, detect both malicious and rogue OAuth apps.
Traditional Machine Learning-based detections
Machine learning models enhance simple rule-based detections. They function by learning and continuously adapting to typical OAuth application usage patterns, enabling them to identify and flag anomalous or malicious behavior as app activity evolves.
Instead of relying solely on static thresholds, ML models can combine multiple weak signals, such as ASN rarity, client version changes, unseen operations, operating system shifts, and activity deviations across time-of-day or day-of-week, to identify behavioral drift that may otherwise go unnoticed.
Algorithms such as Isolation Forest work well for spotting outliers. Clustering approaches like DBSCAN help identify usage patterns that don’t fit known behavioral groups. Plus, lightweight time-series models like Holt-Winters can capture temporal anomalies.
Where historical labels exist, supervised models such as XGBoost can further improve precision by learning combinations of signals that typically indicate malicious or rogue behavior.
The end result is a detection layer that moves beyond static checks into adaptive behavioral intelligence capable of catching both malicious OAuth apps and legitimate apps that slowly go rogue.
Agentic Triaging of detections
To reduce false positives and even further reduce human involvement, a simple agent with the right set of tools does wonders. In deciding the risk associated with the detections, as well as discarding false positives with the right explanation and evidence.
Funneling a very small set of high confidence detections through agentic triage also keeps the overall token usage lower and helps with fast and quicker triaging of detections.
In an upcoming series, we will cover what is the right way to build agents for triaging high-quality detections.
FAQs
What are OAuth applications? OAuth applications are third-party or internal integrations that use the OAuth authorization framework to access data or perform actions on behalf of a user without ever seeing the user’s password.
What does “going rogue” mean? An application is considered "going rogue" when its behavior deviates suddenly from its established patterns.
How do malicious OAuth applications differ from "rogue" OAuth applications? A malicious OAuth application is typically an attacker-controlled app that uses phishing or impersonation to trick users into granting initial access. In contrast, an OAuth application going "rogue" is a previously legitimate, consented application that begins to exhibit sudden behavioral deviations.
