Why Isn’t My Microsoft Sentinel CI/CD Pipeline Working?
Using DevOps to automate management of Microsoft Sentinel features.
The “Deploying and Managing Microsoft Sentinel as Code” Tech Community article is a popular blog that describes how to use Microsoft DevOps to create a CI/CD pipeline for Microsoft Sentinel features. The tool described in the article is very powerful, but it is also so complex that it caused 139 comments (as of the publishing of this blog) from experienced security admins and engineers asking how to get some of the basic components at the start of the project to work.
This is not the fault of the developers or the blog authors for that matter. The blog authors put as much information as they reasonably could in the blog. The reason that so many experienced security professionals were struggling with this project was because the blog could not explain all the concepts that the users would need to understand to finish the implementation.
In this blog, we go over some of the concepts that were not in the blog but are necessary to finish deploying the Microsoft CI/CD pipeline in Microsoft DevOps. This is not a comprehensive guide to deploying the CI/CD pipeline, but it will help other users get past roadblocks that CyberMSI cybersecurity engineers ran into while creating the pipelines that we now use in production.
Service Connection Permissions
The blog discusses creating a service connection and giving it permissions, but they do not discuss which permissions. The Sentinel Contributor role is required to deploy all Microsoft Sentinel features and the Logic App Contributor is also required to deploy playbooks. Not having permissions in a tenant is one of the most common causes for errors.
If a security admin is planning on using a Lighthouse connection to deploy Microsoft Sentinel features, they will have to both add this service connection to the Lighthouse connection and create another service connection in the other tenant to avoid issues with accepting tokens.
“Success” Does Not Mean Success
Something many CI/CD pipeline users discovered quickly is that the green checkmark icon does not indicate that the pipeline succeeded. The pipeline will claim it ran successfully but no Microsoft Sentinel features will be deployed. This is because DevOps does not tell you that there actually was an error that prevented the feature from deploying.
Instead of trusting DevOps to show errors, users should instead check for common areas where error messages will appear, like the “Create and Update” job. By locating where the error is, users can address the issue described in the red error message.
Improvised Coding May Be Required
None of the CI/CD pipeline components from the GitHub worked “out of the box”. They all caused a series of complex errors when they were run. We are not saying that the code from the GitHub is broken, but it did not run in our environment, and many others had similar issues.
We were fortunate to have a former software developer on the team that could make edits to the YAML, JSON, and PS1 files that would circumvent the errors. This resulted in some strange fixes that involved adding new lines of code and creating new variables that were not in the original CI/CD pipeline instructions.
YAML and PS1 Files Need Matching Variables
When either adding more of your own variables or using pipelines that require more variables like multi-subscription deployments, the variables need to match in the YAML and PS1 files. This is because the YAML file is the only one that can take variables from your DevOps library and give that variable to the script.
In the example images below, we show you what a pair of matching variables looks like in the YAML and PS1 files. The YAML pipeline declares all the variables it wants to give to the script in the “Script Arguments” and the PS1 file uses the “Param” section to check for the variables that the YAML pipeline it had given it.
Multiple Subscription Files are in the GitHub Repo
One of the features discussed in the blog was the ability to deploy Microsoft Sentinel features across multiple subscriptions using the CI/CD pipeline. There are no instructions for setting up the multi-subscription deployment, but the files needed to create a multi-subscription deployment for analytic rules are available in the GitHub repo.
The files are named unintuitively, which is why it was hard for some users to find them. The YAML file can be found in a subfolder titled “Old” in the pipelines folder. The PS1 file can be found in the scripts folder with the name “AnalyticRuleAPI.PS1”. Once these folders are downloaded, the user can edit them for the other 3 types of Microsoft Sentinel features.
Comments Help with Future Errors
Fixing DevOps pipelines relies heavily on understanding what the error messages means. Error messages normally have something inside of them that implies where the issue happened. If the code is commented, finding the sections of the code that the error describes becomes easier.
The following example image shows what well commented code looks like. Comments like these can be used to identify parts of the function when troubleshooting complex issues like tokens being rejected by a subscription that the service connection has permissions in.
We will continue to share best practices and lessons learned in future posts on Microsoft Sentinel automation in customer environments. Managing an automation like DevOps is a sizable task, and we hope this article will make it easier for anyone else that is attempting to implement it.
In closing, consider these three questions when using Microsoft Sentinel CI/CD pipelines in your organization:
- Have we done a cost benefit analysis that will tell us if it is worth going through the CI/CD pipeline implementation?
- Is there a “User Access Administrator” available in each subscription we are using that can help with the service connections?
- Do we have anyone on staff that is knowledgeable enough on software development to repair the CI/CD pipeline?