- 09 Aug 2022
- 2 Minutes to read
Operations Google Cloud BigQuery Output
- Updated on 09 Aug 2022
- 2 Minutes to read
Google Cloud BigQuery Output allows you to send data to your Google Cloud BigQuery tables. Each message in the stream is inserted as a new row in the configured BigQuery table.
Unlike the Raw TCP and TCP Syslog Outputs, which require a payload formatter to work, the BigQuery Output does not rely on a payload formatter. Because the
Outbound Payload Format is required when setting up any Enterprise Framework Output, we have provided a
No-op Formatter specifically for use with the BigQuery Output.
The BigQuery Output uses the key-value pairs in each Graylog message to build a row to be inserted into the BigQuery table with the Graylog message keys mapping to the BigQuery table’s columns. Any Graylog message key that does not have a corresponding column in the BigQuery table will be dropped by Google when the insert is performed. You can use a processing pipeline or the
Excluded Fields list in the BigQuery Output configuration to prevent the inclusion of unwanted fields when each row is sent to the BigQuery table.
Required Google Cloud Setup
To use the Google Cloud BigQuery Output, a working Google Cloud account and access to the Google Cloud console are required. See the steps to create an account below.
Create Service Account
- Log into the Google Cloud console.
- Navigate to
IAM & Adminfrom the left-hand menu.
Service Accountsfrom the left-hand menu.
+ CREATE SERVICE ACCOUNTat the top of the right-hand pane.
- Create a new service account.
- Provide a name for the service account (i.e. “Graylog Data”).
- Enter a description for the service account.
- Click the
- Select appropriate permissions for the new service account. At a minimum, the service account needs the ability to write to your BigQuery table.
- Click the ‘’CONTINUE’’ button.
- If desired, grant other users access to the service account.
- Click the ‘’DONE’’ button to finish service account creation.
Generate and Download Service Account Credentials
- Click on the newly created service account in the list of service accounts.
- In the
Create new keyfrom the
ADD KEYdrop-down menu.
JSONas the key type.
- Click on the
- Save the generated JSON file.
- Copy the downloaded JSON credentials file to Graylog host(s). Store the credentials file in the same location on each host, and take appropriate steps to protect the credentials file (e.g., assigning ownership of the file to the account that runs your Graylog server and setting file permissions to 400).
The Google Cloud BigQuery Output supports all of the standard Operations Output Framework configuration options.
- Google Cloud Project ID.
- Output BigQuery Dataset.
- Output BigQuery Table.
- A comma-separated list of fields is filtered out when data is sent to BigQuery.
Credentials File Location
- Path to the Service Account credentials file located on Graylog Node(s).