Implementing Ingestion Delay Correction in Microsoft Sentinel

Improve your analytic rule detection rate with a quick change.

Microsoft Sentinel analytic rules do not correct for ingestion delay on their own, so this prompted the Microsoft Tech Community to post a blog about creating your own ingestion delay solution. The solution seems a bit convoluted, but it makes sense once you wrap your head around it.

In this blog, we will expand upon what was written in the original blog so that the concepts can be more easily understood and conveyed to others. We will also show you the easiest way to implement the solution in all your analytic rules so that you can improve your analytic rule detection rate.

The Solution Can be Stated in 3 Slides

Instead of making everyone read through the Tech Community article, the summary of the article can be explained simply in 3 slides using the 3rd, 6th, and 8th images from the blog. Once each of these images are explained most analysts should be able to understand why they are making these changes to their analytic rules.

Understand Delay in Your Workspace

The delay between the time generated and ingestion time is what will determine how long you should make your delay. You can determine how much delay you need by using the query below and comparing it to a sample set of incidents in the security incidents table. The delay will be projected in a sortable column that can be used to learn about the existing delay in your Microsoft Sentinel workspace.

SecurityIncident
| extend IngestionTime = ingestion_time()
| extend Delay = ingestion_time() - TimeGenerated

The Solution Stops Working After the Delay

If an incident has a delay that is longer than the time that the query looks back, the analytic rule will not be able to detect the incident. Analysts at CyberMSI discovered that if they used a 10-minute delay they would be able to catch 98.8% of incidents within the allotted delay time.

This is a sensible balance between having a high detection rate and having an efficient query that will not take up resources by querying too far back. The model below helps describe this concept by showing how the lookback system would manage an extreme scenario like a 2-hour delay. The solution is not perfect, and analysts need to be aware of that.

The Solution is Implemented in 2 Steps

The solution described in the original blog can be simplified to only 2 quick steps. Microsoft Sentinel analytic rules operate using time generated so there is no need to add time generated restrictions to the query. Instead, the time generated restriction can be made in the Query Scheduling section. All you need to do to update existing analytic rules is to copy a where statement and add 10 minutes to a section.

Solution Can Be Deployed With JSON

If your organization has a quick way of deploying and updating analytic rules like a CI/CD pipeline, this change can be made even faster by updating JSON files. The same method in the previous section can be implemented in the “query” and “queryPeriod” value pairs.

We will continue to share best practices and lessons learned in future posts on creating optimal Microsoft Sentinel analytic rules. CyberMSI is committed to making sure that the analytic rules we produce are as effective as possible and we plan on sharing other techniques with the community in the future.

In closing, consider these three questions when using analytic rule delay corrections in your organization:

1. Is there someone on the team that is good with explaining difficult concepts and that we could assign to these types of projects?
2. How do we strike the right balance between a large delay window and analytic rule query time?
3. Is there a convenient way to update our existing analytic rules like using JSON files?