Best Practices: BindPlane Logs for Stackdriver

How to organize data in Stackdriver

Divide GCP projects by user access

This ensures only users who have permission to view the logs have access to those logs in Google Stackdriver

Ensure logs are parsed as much as possible before sending them to Google Stackdriver

This allows for more filtering and exclusion options to define what logs make it into Google Stackdriver.

Ensure logs are marked by service, host, severity, and log type.

These fields will allow for consistently intuitive searching of logs and log fragments within Google Stackdriver. To learn how to mark logs with custom fields, please read Creating custom tags in Google Stackdriver Logs

How to filter log events during ingestion to Stackdriver

📘

Generic Node Resource

All logs sent from the BindPlane log agent will be available under the Generic Node Resource

Under Stackdriver Logging:

  1. Select Logs Ingestion
  2. Find the row labeled Generic Node.
  3. Select Create exclusion filter based on this resource.
  4. Define your filter for the exclusion being made and provide a Name, Description, and Percentage to Exclude
  5. Click the Create exclusion button.
2782

Steps 1-3

The following provides examples of filters and some best practices around creating exclusion filters.

A common starting filter

The following filters incoming data by log level, commonly excluding non-error logs, unless debugging.
resource.type="generic_node" and jsonPayload.severity=("INFO" OR "DEBUG" OR "TRACE")

2780

An optional best practice filter with BindPlane

With BindPlane, we recommend making exclusions for each log type. This allows you to enable debug logging for an application that is being debugged without enabling debugging for unnecessary log types.

The following example creates an exclusion for the MySQL log type:
resource.type="generic_node" and jsonPayload.severity=("INFO" OR "DEBUG" OR "TRACE") and jsonPayload.bindplane_source_type="mysql"

2780

How to filter the view of logs using BindPlane

Best way to structure logs

  1. Parse out all possible fields.
  2. Know what specific attributes you can consistently search on.
  3. Logically organizing log-based metrics based on deployments, or services.
  4. Parse all numbers without units.
    All numeric values should be parsed to leave only the number as the value. This will significantly decrease the amount of effort needed to build metrics within Stackdriver.

Anatomy of a BindPlane log event:

BindPlane provides, in every event, the following fields for consistency across logging types.
####Resource labels

  • namespace
    Set as bindplane
  • node-id
    The hostname where the agent is installed (also known as the agent name)
  • project-id
    The GCP project ID. This is set by the service role that is used to upload logs.
  • location
    The GCP location the logs are sent to. The configuration is available in the out-stackdriver.conf file.
  • tag
    This value can be set to anything, but BindPlane sets this value equal to the type of log information (JBoss, Tomcat, etc).
2782

Example of a tag in Stackdriver Logging for MySQL logs

  • jsonPayload
    This includes all fields that are parsed by regex, or other means, the severity level of the log, and the message of that log that will be shown in Stackdriver as the base message.

Filtering Log Events

How to see all logs for a specific host:

In the filter for BindPlane logging, set the resource.labels.node_id equal to the hostname you want to filter logs for.

2782

Example of filtering by the node-id

How to see all logs for specific application

To filter for logs related to specific application there are two methods.

  1. Use the Tag Drop Down to filter for specific application logs.
2780

Tag drop-down menu

  1. Using the advanced filter for Stackdriver Logging, use the jsonPayload.bindplane_source_type resource to filter for specific application logs.
2780

Example of using the advanced filter to filter by jsonPayload.bindplane_source_type