Use the Amazon Web Services (AWS) connectors to pull AWS service logs into Microsoft Sentinel. These connectors work by granting Microsoft Sentinel access to your AWS resource logs. Setting up the connector establishes a trust relationship between Amazon Web Services and Microsoft Sentinel. This is accomplished on AWS by creating a role that permits Microsoft Sentinel to access your AWS logs.
This connector is available in two versions: the legacy connector for CloudTrail management and data logs, and the new version that can ingest logs from the following AWS services by pulling them from an S3 bucket:
AWS CloudTrail - Management and data events
Configure AWS S3 Connector
This document explains how to configure the new AWS S3 connector. The process of setting it up has two parts: the AWS side and the Microsoft Sentinel side.
In your AWS environment
o, Configure your AWS service(s) to send logs to an S3 bucket.
o, Create a Simple Queue Service (SQS) queue to provide the notifications.
o, Create an assumed role to grant permissions to your Microsoft Sentinel account (external ID) to access your AWS resources.
o, Attach the appropriate IAM permissions policies to grant Microsoft Sentinel access to the appropriate resources (S3 bucket, SQS).
In Microsoft Sentinel
o, Enable and configure the AWS S3 Connector in the Microsoft Sentinel portal. See the instructions below.
Each side's process produces information used by the other side. This sharing creates secure communication.
We have made available, in our GitHub repository, a script that automates the AWS side of this process. See the instructions for automatic setup later in this document.
Architecture overview
This graphic and the following text show how the parts of this connector solution interact.
AWS services are configured to send their logs to S3 (Simple Storage Service) storage buckets.
The S3 bucket sends notification messages to the SQS (Simple Queue Service) message queue whenever it receives new logs.
The Microsoft Sentinel AWS S3 connector polls the SQS queue at regular, frequent intervals. If there is a message in the queue, it will contain the path to the log files.
The connector reads the message with the path, then fetches the files from the S3 bucket.
To connect to the SQS queue and the S3 bucket, Microsoft Sentinel uses AWS credentials and connection information embedded in the AWS S3 connector's configuration. The AWS credentials are configured with a role and a permissions policy giving them access to those resources. Similarly, the Microsoft Sentinel workspace ID is embedded in the AWS configuration, so there is in effect two-way authentication.
Global prerequisites
You must have written permission on your Microsoft Sentinel workspace.
Automatic setup
To simplify the onboarding process, Microsoft Sentinel has provided a PowerShell script to automate the setup of the AWS side of the connector - the required AWS resources, credentials, and permissions.
The script takes the following actions:
Creates an IAM assume ed role with the minimal necessary permissions, to grant Microsoft Sentinel access to your logs in a given S3 bucket and SQS queue.
Enables specified AWS services to send logs to that S3 bucket, and notification messages to that SQS queue.
If necessary, create that S3 bucket and that SQS queue for this purpose.
Configures any necessary IAM permissions policies and applies them to the IAM role created above.
Prerequisites
You must have PowerShell and the AWS CLI on your machine.
Instructions
To run the script to set up the connector, use the following steps:
1. From the Microsoft Sentinel navigation menu, select Data connectors.
2. Select Amazon Web Services S3 from the data connectors gallery, and in the details pane, select the Open connector page.
3. In the Configuration section, under 1. Set up your AWS environment, and expand Setup with PowerShell script (recommended).
4. Follow the on-screen instructions to download and extract the AWS S3 Setup Script (the link downloads a zip file containing the main setup script and helper scripts) from the connector page.
5. Before running the script, run the AWS configure command from your PowerShell command line, and enter the relevant information as prompted. See AWS Command Line Interface | Configuration basics for details.
6. Now run the script. Copy the command from the connector page (under "Run script to set up the environment") and paste it into your command line.
7. The script will prompt you to enter your Workspace ID. This ID appears on the connector page. Copy it and paste it at the prompt of the script.
8. When the script finishes running, copy the Role ARN and the SQS URL from the script's output (see example in the first screenshot below) and paste them in their respective fields on the connector page under 2. Add connection (see second screenshot below).
9. Select a data type from the Destination table drop-down list. This tells the connector which AWS service's logs this connection is being established to collect, and into which Log Analytics table it will store the ingested data. Then select Add connection.
Manual setup
Microsoft recommends using the automatic setup script to deploy this connector. If for whatever reason you do not want to take advantage of this convenience, follow the steps below to set up the connector manually.
Prerequisites
You must have an S3 bucket to which you will ship the logs from your AWS services - VPC, GuardDuty, or CloudTrail.
o, Create an S3 storage bucket in AWS.
You must have an SQS message queue to which the S3 bucket will publish notifications.
o, Create a standard Simple Queue Service (SQS) queue in AWS.
Instructions
The manual setup consists of the following steps:
Create an AWS assure med role and grant access to the AWS Sentinel account
1. In Microsoft Sentinel, select Data connectors and then select the Amazon Web Services S3 line in the t, able, and in the AWS pane to the right, select Open connector page.
2. Under Configuration, copy the External ID (Workspace ID) and paste it aside.
3. In your AWS management console, under Security, Identity & Compliance, select IAM.
4. Choose Roles and select Create role.
5. Choose Another AWS account. In the Account ID field, enter the number 197857026523 (you can copy and paste it from here). This number is Microsoft Sentinel's service account ID for AWS. It tells AWS that the account using this role is a Microsoft Sentinel user.
6. Select the Require External ID check box, and then enter the External ID (Workspace ID) that you copied from the AWS connector page in the Microsoft Sentinel portal and pasted aside. Then select Next.
7. Enter a Role name.
8. Add Permissions and enter a Tag (optional). Then select Create Role.
9. Apply IAM permissions policies. For information on these and additional policies that should be applied for ingesting the different types of AWS service logs, see the AWS S3 connector permissions policies page in our GitHub repo.
10. In the Roles list, select the new role you created.
11. Copy the Role ARN and paste it aside.
12. In the AWS SQS dashboard, select the SQS queue you created, and copy the URL of the queue.
13. In the AWS S3 connector page in the Microsoft Sentinel portal, under
2. Add connection:
Paste the IAM role ARN you copied two steps ago into the Role ARN field.
Paste the URL of the SQS queue you copied in the last step into the SQS URL field.
Select a data type from the Destination table drop-down list. This tells the connector which AWS service's logs this connection is being established to collect, and into which Log Analytics table it will store the ingested data.
Select Add connection.
