Azure Event Hub
  • 22 May 2022
  • 3 Minutes to read
  • Dark
    Light

Azure Event Hub

  • Dark
    Light

Overview

Azure Event Hub is a fully-managed, real-time data ingestion service that supports the ability to receive many types of event logs from various Azure services. The Graylog Azure Event Logs input supports the ability to retrieve Event Hub events and process them within Graylog.

Prerequisites

In order to use the Azure Event Logs input, it is assumed that you have an existing Azure subscription with a properly configured Event Hub. Please see the Azure Event Hub documentation if you need help setting up Event Hub. You may also find this overview of features and terminologies helpful.

Azure Event Hub Configuration

Assuming your Azure Event Hub is set up and receiving log events from various sources, you will need to perform the following configuration steps for the Graylog Azure Log Events input to connect to and read events from your event hub.

Steps

  1. Add a Shared Access Signature policy in order for the Graylog Azure Logs input to access and communicate with your Event Hub. Please consult the Azure documentation for security and management best practices before creating a policy.

  2. To create a policy, click the Shared access policies option in the left Event Hub navigation bar. Then, click the New button at the top to create the policy.

azure_sas_policy_config

  1. Choose the Listen permission — only — since Graylog will only need to read events from Event Hub.

azure_sas_policy_name

  1. Note either the primary or secondary connection string, once a policy is defined. This will be needed when configuring the input within Graylog.

Consumer Groups

A Consumer group is required for the Azure Logs input to read events from Event Hub. Azure creates a $Default consumer group, which is sufficient for Graylog to read and ingest logs. If you have defined a custom consumer group, it may also be specified within the Graylog configuration.

The Graylog Azure Logs input currently only supports running on a single Graylog node, so there is no need currently to configure a consumer group with additional concurrent readers at this time.

Plugin Configuration

Review this table defining plugin configuration parameters.

Parameter Description
Input Name Provide a unique name for your new Azure Event Logs input.
Azure Event Hub Name The name of your Event Hub within the Azure console.
Connection String The primary or secondary connection string as defined in the Shared Access Signature policy above in the configuration.
Consumer Group The consumer group from which to read events. Use $Default if you have not defined a custom consumer group for your event hub.
Polling Interval (minutes) How often to query the Azure Event hub for new events. We suggest using the default of 5 minutes to avoid hitting Azure rate limits.
Maximum Batch Size The maximum batch size to wait for when the input reads Event Hub. The input will block and wait for the specified batch size to be reached before querying the event hub.
Maximum Wait Time The maximum time to wait for the Maximum Batch Size above to be reached.
Store Full Message Stores the entire message payload received from Azure Logs.

Store Full Message

Introduced in Graylog 4.3, your Azure Event Hub supports the option to store full messages from your Azure log data. This option will allow you to manually parse data received from all Azure log message types utilizing processing pipelines. To enable this option ensure that you have selected "Store Full Message" in the Azure Event Hub Integrations menu.

Azure_Store_Full_Message

Azure Event Hub Event Sources

This input currently supports parsing and ingesting the following types of Azure event logs. Please see the Azure documentation for instructions for how to Forward events from these services to Event Hub.

  • Azure Active Directory (Audit and Sign-in logs)
  • Azure Audit
  • Azure Network Watcher
  • Azure Kubernetes Service
  • Azure SQL

Was this article helpful?