Microsoft Azure HD Insights

❗️

This source has been deprecated

observIQ is in the process of transitioning a subset of BindPlane's monitoring capabilities to the observIQ OpenTelemetry Collector. As a result, this Source is no longer publicly available in BindPlane. If you need access to this Source, please reach out to our support via chat or via [email protected].

Please refer to the Microsoft Azure Sources topic for additional information on how to configure the LPU, and general Azure Data Collection setup details.

Least Privileged User

Steps:

  1. Using the Azure CLI Client, find the Subscription ID and Tenant ID from your account list
  2. Create a custom RBAC role using the JSON provided. Include your Subscription ID and rename the file to azure.json
  3. Create an Active Directory Service Principal and assign the custom RBAC role t it.

Creating custom roles using the Azure CLI:

https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles

Assigning roles using the Azure CLI:

https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal

{
  "Name": "LPU HDInsight",
  "Description": "LPU for HDInsight",
  "Actions": [
    "Microsoft.HDInsight/clusters/*/read",
    "Microsoft.Insights/metrics/*/read",
    "Microsoft.Authorization/*/read"
  ],
  "AssignableScopes": [
    "/subscriptions/[Subscription ID]"
  ]
}

Connection Parameters

NameRequired?Description
Subscription IDRequiredGUID Subscription ID
Tenant IDRequiredGUID Tenant ID (also known as Directory ID)
Client IDRequiredGUID Client ID (also known as Application ID)
Client SecretRequiredThe Secret (also known as Key) corresponding to the Client ID.
Maximum HTTP Retry Time (seconds)The maximum amount of time in seconds to retry each API request when the API is throttling.
HTTP Request Timeout (seconds)The maximum amount of time in seconds before a single HTTP request will fail.

Metrics

API Usage

NameDescription
Average PagesThe average amount of pages needed for a paged resource type.
Average Request RetriesThe average number of retry requests per unique requests made.
Average Retry AttemptsThe average number of retry requests made per unique request that was retried.
Average Retry Wait (Milliseconds)The average amount of time retried requests spent waiting.
Client IDThe client ID used to make API calls.
Failed RequestsThe total number of requests that returned a failure response.
Maximum PagesThe most amount of pages needed for a paged resource type.
Maximum RetriesThe highest number of retries made for a single request.
Maximum Retry Wait (Milliseconds)The most amount of time a retried request spent waiting.
Minimum PagesThe least amount of pages needed for a paged resource type.
Minimum Retry Wait (Milliseconds)The least amount of time a retried request spent waiting.
Other Status ResponsesThe total number of successful requests that responded with some other accepted status.
Request TimeoutsThe total number of requests that timed out waiting for a response.
Requests RetriedThe number of unique requests that were retried.
Retry Status ResponsesThe total number of successful requests that responded with the status TOO MANY REQUESTS (429).
Retry TimeoutsThe total number of requests that needed to be retried, but the request retry time exceeded the maximum retry time.
Status OK ResponsesThe total number of successful requests that responded with the status OK (200).
Subscription IDThe subscription ID used to make API calls.
Successful RequestsThe total number of requests that returned a successful response.
Tenant IDThe tenant ID used to make API calls.
Total Monitor RequestsThe total number of requests made to get monitor metrics.
Total Paged RequestsThe total amount of resource types that required paging.
Total RequestsThe total number of requests made during collection.
Total RetriesThe total number of retry requests that were made.
Unique Monitor RequestsThe number of unique requests made to get monitor metrics.
Unique RequestsThe number of requests made with unique endpoints.

Application

NameDescription
IDUnique resource identifier.
NameSpecifies the name of the application. The name in the form 'clusterName/applicationName'.
Provisioning StateIndicates the current provisioning state.
StateIndicates the more detailed application state while provisioning is in progress.
TypeMicrosoft Azure resource type.

Cluster

NameDescription
BlueprintSpecifies the cluster blueprint url.
Categorized Gateway Requests (Requests)Number of gateway requests by categories (1xx/2xx/3xx/4xx/5xx).
CPU Used (Cores)Amount of CPU allocation used by cluster.
Created DateDatetime when the cluster create request was received.
Gateway Requests (Requests)Number of gateway requests.
IDSpecifies the resource identifier of the cluster.
KindSpecifies the cluster type. Valid values are hadoop, hbase, storm & spark.
LocationSpecifies the supported Azure location where the cluster is located.
NameSpecifies the name of the cluster.
OS TypeSpecifies the Operating system for the cluster. Valid values are Linux and Windows.
Provisioning StateIndicates the current provisioning state.
StateIndicates the more detailed HDInsight cluster state while provisioning is in progress.
TagsSpecifies the tags that are assigned to the cluster.
TierMicrosoft Azure tier type for the cluster.
TypeMicrosoft Azure resource type.
VersionSpecifies the cluster version.

Endpoint

NameDescription
LocationSpecifies the URL to connect.
NameFriendly name for the connectivity endpoint.
Parent IDThe Cluster ID that the endpoint belongs to.
PortSpecifies the port to connect.
ProtocolSpecifies the Protocol to use (example: HTTPS, SSH).

Role

NameDescription
Instance CountSpecifies the target instance count for the role.
NameSpecifies the role name.
Parent IDThe Cluster ID that the role belongs to.
Virtual Machine SizeSpecifies the size (tier) of the Virtual Machine.