Microsoft Azure HD Insights
This source has been deprecated
observIQ is in the process of transitioning a subset of BindPlane's monitoring capabilities to the observIQ OpenTelemetry Collector. As a result, this Source is no longer publicly available in BindPlane. If you need access to this Source, please reach out to our support via chat or via [email protected].
Please refer to the Microsoft Azure Sources topic for additional information on how to configure the LPU, and general Azure Data Collection setup details.
Least Privileged User
Steps:
- Using the Azure CLI Client, find the Subscription ID and Tenant ID from your account list
- Create a custom RBAC role using the JSON provided. Include your Subscription ID and rename the file to azure.json
- Create an Active Directory Service Principal and assign the custom RBAC role t it.
Creating custom roles using the Azure CLI:
https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles
Assigning roles using the Azure CLI:
https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal
{
"Name": "LPU HDInsight",
"Description": "LPU for HDInsight",
"Actions": [
"Microsoft.HDInsight/clusters/*/read",
"Microsoft.Insights/metrics/*/read",
"Microsoft.Authorization/*/read"
],
"AssignableScopes": [
"/subscriptions/[Subscription ID]"
]
}
Connection Parameters
Name | Required? | Description |
---|---|---|
Subscription ID | Required | GUID Subscription ID |
Tenant ID | Required | GUID Tenant ID (also known as Directory ID) |
Client ID | Required | GUID Client ID (also known as Application ID) |
Client Secret | Required | The Secret (also known as Key) corresponding to the Client ID. |
Maximum HTTP Retry Time (seconds) | The maximum amount of time in seconds to retry each API request when the API is throttling. | |
HTTP Request Timeout (seconds) | The maximum amount of time in seconds before a single HTTP request will fail. |
Metrics
API Usage
Name | Description |
---|---|
Average Pages | The average amount of pages needed for a paged resource type. |
Average Request Retries | The average number of retry requests per unique requests made. |
Average Retry Attempts | The average number of retry requests made per unique request that was retried. |
Average Retry Wait (Milliseconds) | The average amount of time retried requests spent waiting. |
Client ID | The client ID used to make API calls. |
Failed Requests | The total number of requests that returned a failure response. |
Maximum Pages | The most amount of pages needed for a paged resource type. |
Maximum Retries | The highest number of retries made for a single request. |
Maximum Retry Wait (Milliseconds) | The most amount of time a retried request spent waiting. |
Minimum Pages | The least amount of pages needed for a paged resource type. |
Minimum Retry Wait (Milliseconds) | The least amount of time a retried request spent waiting. |
Other Status Responses | The total number of successful requests that responded with some other accepted status. |
Request Timeouts | The total number of requests that timed out waiting for a response. |
Requests Retried | The number of unique requests that were retried. |
Retry Status Responses | The total number of successful requests that responded with the status TOO MANY REQUESTS (429). |
Retry Timeouts | The total number of requests that needed to be retried, but the request retry time exceeded the maximum retry time. |
Status OK Responses | The total number of successful requests that responded with the status OK (200). |
Subscription ID | The subscription ID used to make API calls. |
Successful Requests | The total number of requests that returned a successful response. |
Tenant ID | The tenant ID used to make API calls. |
Total Monitor Requests | The total number of requests made to get monitor metrics. |
Total Paged Requests | The total amount of resource types that required paging. |
Total Requests | The total number of requests made during collection. |
Total Retries | The total number of retry requests that were made. |
Unique Monitor Requests | The number of unique requests made to get monitor metrics. |
Unique Requests | The number of requests made with unique endpoints. |
Application
Name | Description |
---|---|
ID | Unique resource identifier. |
Name | Specifies the name of the application. The name in the form 'clusterName/applicationName'. |
Provisioning State | Indicates the current provisioning state. |
State | Indicates the more detailed application state while provisioning is in progress. |
Type | Microsoft Azure resource type. |
Cluster
Name | Description |
---|---|
Blueprint | Specifies the cluster blueprint url. |
Categorized Gateway Requests (Requests) | Number of gateway requests by categories (1xx/2xx/3xx/4xx/5xx). |
CPU Used (Cores) | Amount of CPU allocation used by cluster. |
Created Date | Datetime when the cluster create request was received. |
Gateway Requests (Requests) | Number of gateway requests. |
ID | Specifies the resource identifier of the cluster. |
Kind | Specifies the cluster type. Valid values are hadoop, hbase, storm & spark. |
Location | Specifies the supported Azure location where the cluster is located. |
Name | Specifies the name of the cluster. |
OS Type | Specifies the Operating system for the cluster. Valid values are Linux and Windows. |
Provisioning State | Indicates the current provisioning state. |
State | Indicates the more detailed HDInsight cluster state while provisioning is in progress. |
Tags | Specifies the tags that are assigned to the cluster. |
Tier | Microsoft Azure tier type for the cluster. |
Type | Microsoft Azure resource type. |
Version | Specifies the cluster version. |
Endpoint
Name | Description |
---|---|
Location | Specifies the URL to connect. |
Name | Friendly name for the connectivity endpoint. |
Parent ID | The Cluster ID that the endpoint belongs to. |
Port | Specifies the port to connect. |
Protocol | Specifies the Protocol to use (example: HTTPS, SSH). |
Role
Name | Description |
---|---|
Instance Count | Specifies the target instance count for the role. |
Name | Specifies the role name. |
Parent ID | The Cluster ID that the role belongs to. |
Virtual Machine Size | Specifies the size (tier) of the Virtual Machine. |
Updated almost 2 years ago