Apache Hadoop

Logs Collected

The information being sent to Google Stackdriver Logging is collected from

DataNode log files
ResourceManager log files
NameNode log files
Below are a few examples of Apache Hadoop events being sent to Google Stackdriver Logging using the BindPlane Log Agent.

hadoop.namenode.general.png — Hadoop General NameNode Logs Example

Click to close... — Hadoop General NameNode Logs Example

hadoop.resourcemngr.general.png — Hadoop General Resource Manager Logs Example

Click to close... — Hadoop General Resource Manager Logs Example

hadoop.resourcemgr.yarn.png — Hadoop Yarn Resource Manager Logs Example

Click to close... — Hadoop Yarn Resource Manager Logs Example

Supported Versions

Apache Hadoop: 2.7.x+

Log Collection Setup

Follow these steps to setup log collection from Apache Hadoop.

Pre-requisites

Locate the file path for General and Yarn DataNode log files.
Locate the file path for General and Yarn ResourceManager log files.
Locate the file path for General and Yarn NameNode log files

Configure an Apache Hadoop Source

Install the BindPlane Log Agent on the host system.
Login to BindPlane and select the Logs tab.

Screen Shot 2019-07-03 at 9.54.53 AM.png — Logs Tab

Click to close... — Logs Tab

Select the Sources tab.

Screen Shot 2019-07-03 at 9.56.25 AM.png — Sources Tab

Click to close... — Sources Tab

In the top-right portion of the screen, click on the Add Source Configuration button

Screen Shot 2019-07-03 at 10.00.25 AM.png — Add Source Configuration Button

Click to close... — Add Source Configuration Button

Choose Apache Hadoop
Fill out the Apache Hadoop log configuration options.

Apache Hadoop Log Configuration Form

Click to close... — Apache Hadoop Log Configuration Form

Updated about a year ago