Google Cloud Dataproc

Monitor Google Spark and Hadoop Hosted Services

🚧

Not Included in the BindPlane with Google Cloud Monitoring offering

All of the Google Cloud Platform sources listed within this documentation are not included with the BindPlane with Google Cloud Monitoring offering.

For more information on how to use the below LPU and other Google Cloud Data Collection setup. See the Google Cloud Platform Sources

Least Privileged User

A user role with the at least the following permissions is required:

📘

Deploying a Least Privileged User

To learn more about how to deploy a role with these permissions to a GCP Organization, or a GCP Project, please refer to this documentation:
Deploy an Individual LPU role to a GCP Project, or GCP Organization

- cloudnotifications.activities.list
- monitoring.alertPolicies.get
- monitoring.alertPolicies.list
- monitoring.dashboards.get
- monitoring.dashboards.list
- monitoring.groups.get
- monitoring.groups.list
- monitoring.metricDescriptors.get
- monitoring.metricDescriptors.list
- monitoring.monitoredResourceDescriptors.get
- monitoring.monitoredResourceDescriptors.list
- monitoring.notificationChannelDescriptors.get
- monitoring.notificationChannelDescriptors.list
- monitoring.notificationChannels.get
- monitoring.notificationChannels.list
- monitoring.publicWidgets.get
- monitoring.publicWidgets.list
- monitoring.timeSeries.list
- monitoring.uptimeCheckConfigs.get
- monitoring.uptimeCheckConfigs.list
- resourcemanager.projects.get
- resourcemanager.projects.list
- stackdriver.projects.get- 
- compute.machineTypes.get
- compute.regions.get
- compute.regions.list
- compute.zones.get
- dataproc.clusters.get
- dataproc.clusters.list
- dataproc.jobs.get
- dataproc.jobs.list
- dataproc.operations.get
- dataproc.operations.list
- dataproc.workflowTemplates.get
- dataproc.workflowTemplates.list

Connection Parameters

NameRequired?Description
Private Key JSONRequiredThe contents of the private key JSON file created when setting up a service account.
Metric CollectionControls which metrics get requested from GCP's Stackdriver API.
ProjectsRequiredA comma separated whitelist of project IDs. If the wildcard "*" is used, resources will be collected from all available projects.
RegionsRequiredA comma separated whitelist of regions. At least one region must be specified.
Connection TimeoutThe number of seconds to allow for connecting to the target.

Metrics

Cloud Dataproc Cluster

NameDescription
Cluster NameThe name of the cluster.
Cluster UuidThe unique ID of the cluster.
Configuration BucketA Cloud Storage staging bucket used for sharing generated SSH keys and config. If you do not specify a staging bucket, Cloud Dataproc will determine an appropriate Cloud Storage location (US, ASIA, or EU) for your cluster's staging bucket according to the Google Compute Engine zone where your cluster is deployed, and then it will create and manage this project-level, per-location bucket for you.
Failed JobsIndicates the number of jobs that have failed on a cluster.
Failed OperationsIndicates the number of operations that have failed on a cluster.
GCE Cluster Configuration Internal Ip OnlyIf true, all instances in the cluster will only have internal IP addresses. By default, clusters are not restricted to internal IP addresses, and will have ephemeral external IP addresses assigned to each instance. This internalIpOnly restriction can only be enabled for subnetwork enabled networks, and all off-cluster dependencies must be configured to be accessible without external IP addresses.
GCE Cluster Configuration MetadataThe Compute Engine metadata entries to add to all instances (see Project and instance metadata).
GCE Cluster Configuration Network UriThe Compute Engine network to be used for machine communications. Cannot be specified with subnetworkUri. If neither networkUri nor subnetworkUri is specified, the "default" network of the project is used, if it exists. Cannot be a "Custom Subnet Network" (see Using Subnetworks for more information).
GCE Cluster Configuration Service AccountThe service account of the instances. Defaults to the default Compute Engine service account.
GCE Cluster Configuration Service Account ScopesThe URIs of service account scopes to be included in Compute Engine instances.
GCE Cluster Configuration Subnetwork UriThe Compute Engine subnetwork to be used for machine communications. Cannot be specified with networkUri.
GCE Cluster Configuration TagsThe Compute Engine tags to add to all instances (see Tagging instances).
GCE Cluster Configuration Zone UriThe zone where the Compute Engine cluster will be located. On a create request, it is required in the "global" region. If omitted in a non-global Cloud Dataproc region, the service will pick a zone in the corresponding Compute Engine region. On a get request, zone will always be present.
HDFS Capacity (Gigabytes)Indicates capacity of HDFS system running on cluster in GB.
HDFS DataNodesIndicates the number of HDFS DataNodes that are running inside a cluster.
HDFS Storage Utilization (%)The ratio of HDFS storage currently used.
LabelsThe labels to associate with this cluster. Label keys must contain 1 to 63 characters, and must conform to RFC 1035. Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035. No more than 32 labels can be associated with a cluster.
Master Configuration Accelerators Accelerator CountThe number of the accelerator cards of this type exposed to this instance.
Master Configuration Accelerators Accelerator Type UriFull URL, partial URI, or short name of the accelerator type resource to expose to this instance.
Master Configuration Disk Configuration Boot Disk Size (Gigabytes)Size of the boot disk (default is 500GB).
Master Configuration Disk Configuration Boot Disk TypeType of the boot disk (default is "pd-standard"). Valid values: "pd-ssd" (Persistent Disk Solid State Drive) or "pd-standard" (Persistent Disk Hard Disk Drive).
Master Configuration Disk Configuration Local SSD CountNumber of attached SSDs, from 0 to 4 (default is 0). If SSDs are not attached, the boot disk is used to store runtime logs and HDFS data. If one or more SSDs are attached, this runtime bulk data is spread across them, and the boot disk contains only basic config and installed binaries.
Master Configuration Image UriThe Compute Engine image resource used for cluster instances. It can be specified or may be inferred from SoftwareConfig.image_version.
Master Configuration Instance CountThe number of VM instances in the instance group. For master instance groups, must be set to 1.
Master Configuration Instance NamesOutput only. The list of instance names. Cloud Dataproc derives the names from clusterName, numInstances, and the instance group.
Master Configuration Is PreemptibleSpecifies that this instance group contains preemptible instances.
Master Configuration Machine Type UriThe Compute Engine machine type used for cluster instances.
Master Configuration Managed Group Configuration Instance Group Manager NameOutput only. The name of the Instance Group Manager for this group.
Master Configuration Managed Group Configuration Instance Template NameOutput only. The name of the Instance Template used for the Managed Instance Group.
Metrics Hdfs MetricsThe HDFS metrics.
Metrics Yarn MetricsThe YARN metrics.
Project IdThe identifier of the project the cluster belongs to.
RegionThe region of the cluster these metrics belong to.
Running JobsIndicates the number of jobs that are running on a cluster.
Running OperationsIndicates the number of operations that are running on a cluster.
Secondary Worker Configuration Accelerators Accelerator CountThe number of the accelerator cards of this type exposed to this instance.
Secondary Worker Configuration Accelerators Accelerator Type UriFull URL, partial URI, or short name of the accelerator type resource to expose to this instance.
Secondary Worker Configuration Disk Configuration Boot Disk Size (Gigabytes)Size of the boot disk (default is 500GB).
Secondary Worker Configuration Disk Configuration Boot Disk TypeType of the boot disk (default is "pd-standard"). Valid values: "pd-ssd" (Persistent Disk Solid State Drive) or "pd-standard" (Persistent Disk Hard Disk Drive).
Secondary Worker Configuration Disk Configuration Local SSD CountNumber of attached SSDs, from 0 to 4 (default is 0). If SSDs are not attached, the boot disk is used to store runtime logs and HDFS data. If one or more SSDs are attached, this runtime bulk data is spread across them, and the boot disk contains only basic config and installed binaries.
Secondary Worker Configuration Image UriThe Compute Engine image resource used for cluster instances. It can be specified or may be inferred from SoftwareConfig.image_version.
Secondary Worker Configuration Instance CountThe number of VM instances in the instance group. For master instance groups, must be set to 1.
Secondary Worker Configuration Instance NamesOutput only. The list of instance names. Cloud Dataproc derives the names from clusterName, numInstances, and the instance group.
Secondary Worker Configuration Is PreemptibleSpecifies that this instance group contains preemptible instances.
Secondary Worker Configuration Machine Type UriThe Compute Engine machine type used for cluster instances.
Secondary Worker Configuration Managed Group Configuration Instance Group Manager NameOutput only. The name of the Instance Group Manager for this group.
Secondary Worker Configuration Managed Group Configuration Instance Template NameOutput only. The name of the Instance Template used for the Managed Instance Group.
Software Configuration Image VersionThe version of software inside the cluster. It must be one of the supported Cloud Dataproc Versions, such as "1.2" (including a subminor version, such as "1.2.29"), or the "preview" version. If unspecified, it defaults to the latest version.
Software Configuration PropertiesThe properties to set on daemon config files.
Status DetailOutput only. Optional details of cluster's state.
Status StateOutput only. The cluster's state.
Status State Start TimeOutput only. Time when this state was entered.
Status SubstateOutput only. Additional state information that includes status reported by the agent.
Submitted JobsIndicates the number of jobs that have been submitted to a cluster.
Submitted OperationsIndicates the number of operations that have been submitted to a cluster.
Unhealthy HDFS Blocks By StatusIndicates the number of unhealthy blocks inside the cluster.
Worker Configuration Accelerators Accelerator CountThe number of the accelerator cards of this type exposed to this instance.
Worker Configuration Accelerators Accelerator Type UriFull URL, partial URI, or short name of the accelerator type resource to expose to this instance.
Worker Configuration Disk Configuration Boot Disk Size (Gigabytes)Size of the boot disk (default is 500GB).
Worker Configuration Disk Configuration Boot Disk TypeType of the boot disk (default is "pd-standard"). Valid values: "pd-ssd" (Persistent Disk Solid State Drive) or "pd-standard" (Persistent Disk Hard Disk Drive).
Worker Configuration Disk Configuration Local SSD CountNumber of attached SSDs, from 0 to 4 (default is 0). If SSDs are not attached, the boot disk is used to store runtime logs and HDFS data. If one or more SSDs are attached, this runtime bulk data is spread across them, and the boot disk contains only basic config and installed binaries.
Worker Configuration Image UriThe Compute Engine image resource used for cluster instances. It can be specified or may be inferred from SoftwareConfig.image_version.
Worker Configuration Instance CountThe number of VM instances in the instance group. For master instance groups, must be set to 1.
Worker Configuration Instance NamesOutput only. The list of instance names. Cloud Dataproc derives the names from clusterName, numInstances, and the instance group.
Worker Configuration Is PreemptibleSpecifies that this instance group contains preemptible instances.
Worker Configuration Machine Type UriThe Compute Engine machine type used for cluster instances.
Worker Configuration Managed Group Configuration Instance Group Manager NameOutput only. The name of the Instance Group Manager for this group.
Worker Configuration Managed Group Configuration Instance Template NameOutput only. The name of the Instance Template used for the Managed Instance Group.
YARN Active ApplicationsIndicates the number of active YARN applications.
YARN Allocated Memory (%)The ratio of YARN memory allocated.
YARN ContainersIndicates the number of YARN containers.
YARN Memory Size (Gigabytes)Indicates the YARN memory size in GB.
YARN NodeManagersIndicates the number of YARN NodeManagers running inside cluster.
YARN Pending Memory Size (Gigabytes)The current memory request that is pending to be fulfilled by the scheduler.
YARN Virtual CPU (Cores)The amount of virtual CPU in YARN.

Job

NameDescription
Driver Control Files URIOutput only. If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
Driver Output Resource URIOutput only. A URI pointing to the location of the stdout of the job's driver program.
LabelsThe labels to associate with this job. Label keys must contain 1 to 63 characters, and must conform to RFC 1035. Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035. No more than 32 labels can be associated with a job.
Placement Cluster NameThe name of the cluster where the job will be submitted.
Placement Cluster UUIDOutput only. A cluster UUID generated by the Cloud Dataproc service when the job is submitted.
Project IDThe project ID in which this resource was created.
Reference Job IDThe job ID, which must be unique within the project. The job ID is generated by the server upon job submission or provided by the user as a means to perform retries without creating duplicate jobs. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 100 characters.
RegionThe region in which this resource is located.
Scheduling Maximum Failure Rate (per Hour)Maximum number of times a driver may be restarted as a result of driver terminating with non-zero code before job is reported failed. A job may be reported as thrashing if driver exits with non-zero code 4 times within 10 minute window. Maximum value is 10.
Status DetailsOutput only. Optional job state details, such as an error description if the state is ERROR.
Status StateOutput only. A state message specifying the overall job state.
Status State Start TimeOutput only. The time when this state was entered.
Status SubstateOutput only. Additional state information, which includes status reported by the agent.

YARN Application

NameDescription
Project IDThe project ID in which this resource was created.
Reference Job IDThe identifer of the parent job.
YARN Applications NameThe application name.
YARN Applications Progress (%)The numerical progress of the application, from 1 to 100.
YARN Applications StateThe application state.
YARN Applications Tracking URLThe HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.