Best Practices: BindPlane Logs for Stackdriver
How to organize data in Stackdriver
Divide GCP projects by user access
This ensures only users who have permission to view the logs have access to those logs in Google Stackdriver
Ensure logs are parsed as much as possible before sending them to Google Stackdriver
This allows for more filtering and exclusion options to define what logs make it into Google Stackdriver.
Ensure logs are marked by service, host, severity, and log type.
These fields will allow for consistently intuitive searching of logs and log fragments within Google Stackdriver. To learn how to mark logs with custom fields, please read Creating custom tags in Google Stackdriver Logs
How to filter log events during ingestion to Stackdriver
Generic Node Resource
All logs sent from the BindPlane log agent will be available under the Generic Node Resource
Under Stackdriver Logging:
- Select Logs Ingestion
- Find the row labeled Generic Node.
- Select Create exclusion filter based on this resource.
- Define your filter for the exclusion being made and provide a Name, Description, and Percentage to Exclude
- Click the Create exclusion button.
The following provides examples of filters and some best practices around creating exclusion filters.
A common starting filter
The following filters incoming data by log level, commonly excluding non-error logs, unless debugging.
resource.type="generic_node" and jsonPayload.severity=("INFO" OR "DEBUG" OR "TRACE")
An optional best practice filter with BindPlane
With BindPlane, we recommend making exclusions for each log type. This allows you to enable debug logging for an application that is being debugged without enabling debugging for unnecessary log types.
The following example creates an exclusion for the MySQL log type:
resource.type="generic_node" and jsonPayload.severity=("INFO" OR "DEBUG" OR "TRACE") and jsonPayload.bindplane_source_type="mysql"
How to filter the view of logs using BindPlane
Best way to structure logs
- Parse out all possible fields.
- Know what specific attributes you can consistently search on.
- Logically organizing log-based metrics based on deployments, or services.
- Parse all numbers without units.
All numeric values should be parsed to leave only the number as the value. This will significantly decrease the amount of effort needed to build metrics within Stackdriver.
Anatomy of a BindPlane log event:
BindPlane provides, in every event, the following fields for consistency across logging types.
####Resource labels
- namespace
Set asbindplane
- node-id
The hostname where the agent is installed (also known as the agent name) - project-id
The GCP project ID. This is set by the service role that is used to upload logs. - location
The GCP location the logs are sent to. The configuration is available in theout-stackdriver.conf
file. - tag
This value can be set to anything, but BindPlane sets this value equal to the type of log information (JBoss, Tomcat, etc).
- jsonPayload
This includes all fields that are parsed by regex, or other means, the severity level of the log, and the message of that log that will be shown in Stackdriver as the base message.
Filtering Log Events
How to see all logs for a specific host:
In the filter for BindPlane logging, set the resource.labels.node_id
equal to the hostname you want to filter logs for.
How to see all logs for specific application
To filter for logs related to specific application there are two methods.
- Use the Tag Drop Down to filter for specific application logs.
- Using the advanced filter for Stackdriver Logging, use the
jsonPayload.bindplane_source_type
resource to filter for specific application logs.
Updated over 4 years ago