Cloud Foundry

Pivotal Cloud Foundry (PCF) Monitoring

❗️

This source has been deprecated

observIQ is in the process of transitioning a subset of BindPlane's monitoring capabilities to the observIQ OpenTelemetry Collector. As a result, this Source is no longer publicly available in BindPlane. If you need access to this Source, please reach out to our support via chat or via [email protected].

Data Collection Setup

Metrics are collected via 3 distinct APIs within the PCF Foundation: the Blue Medora Nozzle, Pivotal Cloud Controller, and Pivotal BOSH Director API.

Network Requirements

Nozzle API (Tile Deployment): 443 (TCP) Default

  • Must match your nozzle configuration. (e.g., BOSH install default port is 8081).

Cloud Controller API Port: 443 (TCP) Default

  • The source also accesses the UAA at the address specified in the authorization_endpoint in /info on the Cloud Controller API.

BOSH Director API Port: 25555 (TCP) Default

  • The source also accesses the UAA at the address specified in the user_authentication endpoint in /info on the BOSH Director API.

Obtaining Nozzle, Cloud Controller, and BOSH Director Credentials.

The Data Provider for PCF requires the following credential types:

  • Blue Medora Nozzle for PCF Credentials (UAA Client)
  • Cloud Controller API Credentials (UAA User)
  • BOSH Director Credentials (BOSH Director UAA Client)

Blue Medora Nozzle for PCF Credentials (UAA Client)

For the Blue Medora Nozzle for PCF, you will need a UAA Client and password

📘

The UAA Client used for your nozzle credentials must be the same UAA Client used when configuring your Nozzle.

To create a UAA Client for the nozzle:

  1. Complete steps 1-4 in Cloud Foundry's Create an Admin User topic. These steps walk you through installing the UAA CLI, targeting your UAA server, and obtaining an access token for the admin client from the UAA server.
  2. Next, execute the following command:
    Be sure to replace the {clientname} and {foundation} parameters with the name you want to assign your UAA client and your PCF foundation, respectively.
uaac client add {clientname} \
    --authorities "oauth.login,doppler.firehose" \
    --scope "openid,oauth.approvals,doppler.firehose" \
    --authorized_grant_types "authorization_code,client_credentials,refresh_token" \
    --access_token_validity 1209600 \
    --autoapprove true \
    --redirect_uri https://apps.{foundation}

Cloud Controller API Credentials (UAA User)

For the Cloud Controller API, you will need a UAA User and password in our data provider configuration.

Complete steps 1-4 in Cloud Foundry's Create an Admin Read-Only User topic, but adding the new user to the cloud_controller.admin group instead of cloud_controller.admin_read_only, as indicated in step 4. (Make sure to add the new user to the uaa.admin and scim.read groups as well.)

📘

User Privileges with the Cloud Controller API

NOTE: You may choose to assign the new user lesser privileges by adding it to the cloud_controller.admin_read_only group instead of cloud_controller.admin; however, CPU, Memory, and Disk Usage for Applications and Application Instances will not be returned with that lesser permission.

BOSH Director Credentials (BOSH Director UAA Client)

For BOSH Director, you will need a BOSH Director UAA Client and password

To create a BOSH Director UAA Client:

Complete steps 1-5 in Pivotal Cloud Foundry's Creating UAA Clients for BOSH Director topic, but adding the new user to the bosh.read group instead of bosh.admin, as indicated in step 5.

Setting up the Blue Medora Nozzle for PCF

The Blue Medora Nozzle for PCF is a Cloud Foundry component that is required by the management pack in order to connect to the Cloud Foundry Loggregator Firehose. It exposes PCF metrics via a RESTful API.

📘

PCF Nozzle Documentation

We recommend upgrading to the latest version of the nozzle for optimized performance. See our nozzle documentation on the Pivotal Partners site for details.

Prerequisites

You will need Blue Medora Nozzle for PCF credentials prior to deploying and configuring the Nozzle. See: Obtaining Nozzle, Cloud Controller, and BOSH Director Credentials above.

Recommended Deployment Method: Blue Medora Nozzle for PCF Tile

Download the Blue Medora Nozzle for PCF Tile from the Pivotal Network, then follow the instructions for installing, configuring, and using the nozzle as documented on our Pivotal Partners documentation site.

Least Privileged User

See:
Cloud Controller API Credentials (UAA User) section above

Supported Versions

Pivotal Cloud Foundry Central:
1.8+, 2.x

Connection Parameters

NameRequired?Description
BOSH Director Auth URL
BOSH Director API HostRequiredThe BOSH Director to connect to.
BOSH Director API PortThe port for communication to the BOSH Director.
BOSH Director API UsernameRequired
BOSH Director API PasswordRequired
SSL ConfigurationThe SSL mode to use when connecting to the target. Can be configured to not use SSL (No SSL), use SSL but do not verify the target's certificate (No Verify), and use SSL and verify the target's certificate (Verify).
Use Nozzle?Whether to expect Nozzle connection information. If set to false, events, resources, and metrics depending on the Loggregator Firehose will be missing.
Use Cloud Controller API?Whether to expect Cloud Controller connection information. If false, Applications, Services, Spaces, and Routes metrics and relationships may be missing.
Nozzle API HostRequiredThe Nozzle API to connect to.
Nozzle API PortThe port for communication to the Nozzle API.
Nozzle API UsernameRequired
Nozzle API PasswordRequired
Cloud Controller API HostRequiredThe Cloud Controller API to connect to.
Cloud Controller API PortThe port for communication to the Cloud Controller API.
Cloud Controller API UsernameRequired
Cloud Controller API PasswordRequired
Foundation NameSpecifies the name for the resource representing the Foundation, which is the top-level resource describing a Pivotal Cloud deployment. Enter a Foundation Name so you can distinguish data from other Pivotal Cloud deployments in your environment.
Connection Timeout (seconds)The number of seconds to allow for connecting to the target.
Collect EventsControls whether events are collected and returned.

Metrics

Access

NameDescription
File Server Active GoRoutinesInstantaneous number of active GoRoutines in the process
File Server CPUsNumber of CPUs on the machine
File Server Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
File Server Memory AllocationsLifetime number of memory allocations
File Server Memory DeallocationsLifetime number of memory deallocations
File Server Used Heap Memory (Bytes)Instantaneous count memory allocated on the main heap and still in use
File Server Used Memory (Bytes)Instantaneous count of memory allocated and still in use
File Server Used Stack Memory (Bytes)Instantaneous count of memory allocated on the main stack and still in use
SSH Proxy Active GoRoutinesInstantaneous number of active GoRoutines in the process
SSH Proxy CPUsNumber of CPUs on the machine
SSH Proxy Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
SSH Proxy Memory AllocationsLifetime number of memory allocations
SSH Proxy Memory DeallocationsLifetime number of memory deallocations
SSH Proxy Used Heap Memory (Bytes)Instantaneous count memory allocated on the main heap and still in use
SSH Proxy Used Memory (Bytes)Instantaneous count of memory allocated and still in use
SSH Proxy Used Stack Memory (Bytes)Instantaneous count of memory allocated on the main stack and still in use

Application

Name
Average Instance Disk Usage (Bytes)
Average Instance Memory Usage (Bytes)
Instance Count
Instance CPU Utilization (%)
Total Instance Disk Quota (Bytes)
Total Instance Disk Usage (Bytes)
Total Instance Memory Quota (Bytes)
Total Instance Memory Usage (Bytes)

Application Instance

Name
CPU Utilization (%)
Disk Quota (Bytes)
Disk Usage (Bytes)
File Descriptors Quota
Memory Quota (Bytes)
Memory Usage (Bytes)

BBS

NameDescription
Active GoRoutinesInstantaneous number of active GoRoutines in the Doppler process
Average Request Latency (Nanoseconds)Average time the BBS took to handle requests to its API endpoints in the last 15 minutes.
CF-Apps Domain Up-To-DateWhether the 'cf-apps' domain is Up-To-Date, so that CF apps from CC have been synchronized with DesiredLRPs for Diego to run.
CF-Tasks Domain Up-To-DateWhether the 'cf-tasks' domain is Up-To-Date, so that CF tasks from CC have been synchronized with tasks for Diego to run.
CPU CountNumber of CPUs on the machine
Cumulative Convergence Tasks KickedCumulative number of times the BBS has updated a Task during its Task convergence pass.
Cumulative Convergence Tasks PrunedCumulative number of times the BBS has deleted a malformed Task during its Task convergence pass.
Cumulative LRP Convergence Pass RunsCumulative number of times BBS has run its LRP convergence pass.
Cumulative Malformed ActualLRP Instances DeletedCumulative number of times the BBS has detected and deleted a malformed ActualLRP in its LRP convergence pass.
Cumulative Malformed DesiredLRP RunInfo DetectedCumulative number of times the BBS has detected a malformed DesiredLRP RunInfo in its LRP convergence pass.
Cumulative Malformed DesiredLRP SchedulingInfo DetectedCumulative number of times the BBS has detected a malformed DesiredLRP SchedulingInfo in its LRP convergence pass.
Cumulative Request CountCumulative number of requests the BBS has handled through its API.
Cumulative Task Convergence Pass RunsCumulative number of times the BBS has run its Task convergence pass.
etcd Leader Received Data Rate (Bytes per Second)Number of data per second this node is receiving (leader only). This value is undefined on single member clusters.
etcd Leader Received Request Rate (per Second)Number of requests per second this node is receiving (leader only). This value is undefined on single member clusters.
etcd Leader Send Data Rate (Bytes per Second)Number of data per second this node is sending (leader only). This value is undefined on single member clusters.
etcd Leader Send Request Rate (per Second)Number of requests per second this node is sending (leader only). This value is undefined on single member clusters.
etcd Raft TermRaft term of the etcd cluster.
etcd WatchersNumber of watches set against the etcd cluster.
Extra LRP Instance AverageAverage number of LRP instances that are no longer desired but still have a BBS record over the last 5 minutes.
Extra LRP InstancesTotal number of LRP instances that are no longer desired but still have a BBS record.
Is MasterSignals if BBS is on master
Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Lock Held Duration (Nanoseconds)Time the active BBS has held the BBS lock.
Lock Is HeldWhether a BBS holds the BBS lock.
LRP Convergence Pass Duration (Nanoseconds)Time the BBS took to run its LRP convergence pass.
LRP Convergence Pass Duration Maximum (Nanoseconds)Maximum Time the BBS took to run its LRP convergence pass in the last 15 minutes.
LRP Instances DesiredTotal number of LRP instances desired across all LRPs.
LRP Instances RunningTotal number of LRP instances that are running on cells.
Maximum Locket Active LocksThe maximum count of how many locks the system components are holding in the past 5 minutes.
Maximum Locket Active PresencesMaximum count of active presences in the past 15 minutes. Presences are defined as the registration records that the cells maintain to advertise themselves to the platform.
Memory AllocationsLifetime number of memory allocations
Memory DeallocationsLifetime number of memory deallocations
Migration Duration (Nanoseconds)Time the BBS took to run migrations against its persistence store.
Reporting Metrics Duration (Nanoseconds)Time the BBS took to emit metrics about etcd.
Request Latency (Nanoseconds)Time the BBS took to handle requests to its API endpoints.
Task Convergence Pass DurationTime the BBS took to run its Task convergence pass.
Unclaimed LRP InstancesTotal number of LRP instances that have not yet been claimed by a cell.
Used Heap Memory (Bytes)Instantaneous count of memory allocated on the main heap and still in use
Used Memory (Bytes)Instantaneous count of memory allocated and still in use
Used Stack Memory (Bytes)Instantaneous count of memory used by the stack allocator
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode

Bosh Job

Name
VM Active Memory Usage (Kibibytes)
VM Active Memory Utilization (%)
VM CPU Utilization (%)
VM Ephemeral Disk Utilization (%)
VM Load Average (15m)
VM Persistent Disk Utilization (%)
VM Swap Usage (Kibibytes)
VM Swap Utilization (%)
VM System Disk Utilization (%)
VM User CPU Utilization (%)

CC-Bridge

NameDescription
CC-Uploader Active GoRoutinesInstantaneous number of active GoRoutines in the process
CC-Uploader CPUsNumber of CPUs on the machine
CC-Uploader Heap Memory Usage (Bytes)Instantaneous count memory allocated on the main heap and still in use
CC-Uploader Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
CC-Uploader Memory AllocationsLifetime number of memory allocations
CC-Uploader Memory DeallocationsLifetime number of memory deallocations
CC-Uploader Memory Usage (Bytes)Instantaneous count of memory allocated and still in use
CC-Uploader Stack Memory Usage (Bytes)Instantaneous count of memory allocated on the main stack and still in use
Nsync Bulker Active GoRoutinesInstantaneous number of active GoRoutines in the process
Nsync Bulker Convergence Lock HeldWhether an Nsync Bulker holds the Nsync Bulker lock.
Nsync Bulker Convergence Lock Held Duration (Nanoseconds)Time the active Nsync Bulker has held the convergence lock.
Nsync Bulker Invalid DesiredLRP InstancesNumber of invalid DesiredLRPs found during Nsync Bulker periodic synchronization
Nsync Bulker Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Nsync Bulker LRP Instances DesiredCumulative number of LRPs desired through the Nsync API.
Nsync Bulker LRP Synchronization Duration (Nanoseconds)Time the Nsync Bulker took to synchronize CF apps and Diego DesiredLRPs.
Stager Staging Tasks FailedCumulative number of failed staging tasks handled by each stager.
Stager Staging Tasks Failed Duration (Nanoseconds)Time the failed staging task took to run.
Stager Staging Tasks RequestsCumulative number of requests to start a staging task.
Stager Staging Tasks SucceededCumulative number of successful staging tasks handled by each stager.
Stager Staging Tasks Succeeded Duration (Nanoseconds)Time the successful staging task took to run.
TPS Watcher Lock HeldWhether a tps-watcher holds the tps-watcher lock.
TPS Watcher Lock Held Duration (Nanoseconds)Time the active tps-watcher has held the convergence lock

Cloud Controller

NameDescription
All Log MessagesTotal number of log messages, sum of messages of all severity levels.
cc-generic Failed JobsNumber of failed jobs in the cc-generic queue.
cc-generic Delayed JobsNumber of delayed jobs in the cc-generic queue.
Completed RequestsNumber of requests that have been processed.
CPU CoresThe number of CPU cores of a host machine.
CPU Utilization (%)Amount of CPU spent in the Cloud Controller process.
Free Memory (Bytes)Total memory available according to the OS.
HTTP ResponsesNumber of HTTP responses.
Memory Usage (Bytes)Total memory used (active + wired) according to the OS.
Outstanding RequestsNumber of requests currently being processed.
Real Memory (Bytes)The RSS data (resident set size) or real memory of the Cloud Controller process.
Running TasksNumber of currently running tasks.
Running Tasks Memory Usage (Megabytes)Memory being consumed by all currently running tasks.
Thread ConnectionsNumber of open connections to event machine
Thread Result Scheduled TasksNumber of scheduled tasks in the result
Thread Result Unscheduled TasksNumber of unscheduled tasks in the result
Thread Scheduled TasksNumber of scheduled tasks in the threadqueue
Thread Unscheduled TasksNumber of unscheduled tasks in the threadqueue
ThreadsTotal number of threads that are either runnable or stopped
Uptime (Seconds)The uptime of the Cloud Controller process.
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode

Consul

NameDescription
Active GoRoutinesInstantaneous number of active GoRoutines in the process.
CIDVirtual Machine Cloud ID
Counter EventsNumber of CounterEvents aggregated in Metron.
CPUsNumber of CPUs on the machine.
DeploymentDeployment
Doppler Sent MessagesNumber of messages sent to Doppler regardless of protocol.
Heap Memory Usage (Bytes)Instantaneous count of memory allocated on the main heap and still in use.
IndexIndex
IPIP
JobJob
Job StateJob State
Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause.
Memory AllocationsLifetime number of memory allocations.
Memory DeallocationsLifetime number of memory deallocations.
Memory Usage (Bytes)Instantaneous count of memory allocated and still in use.
Processed Uncategorized EventsNumber of non-(CounterEvent HTTPStart HTTPStop) events processed by aggregator.
Stack Memory Usage (Bytes)Instantaneous count of memory used by the stack allocator.
TCP Send Errors (Errors)Number of errors if writing to Doppler over TCP fails
TCP Sent Data (Bytes)Lifetime data sent to Doppler over TCP
TCP Sent MessagesNumber of sent messages to Doppler over TCP
TLS Send Errors (Errors)Number of errors if writing to Doppler over TLS fails
TLS Sent Data (Bytes)Lifetime data sent to Doppler over TLS
TLS Sent MessagesNumber of sent messages to Doppler over TLS
UDP Send Errors (Errors)Number of errors if writing to Doppler over UDP fails
UDP Sent Data (Bytes)Lifetime data sent to Doppler over UDP
UDP Sent MessagesNumber of sent messages to Doppler over UDP
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode

Diego Brain

NameDescription
Auctioneer Active GoRoutinesInstantaneous number of active GoRoutines in the process
Auctioneer CPUsNumber of CPUs on the machine
Auctioneer Fetch State Duration (Nanoseconds)Time the auctioneer took to fetch state from all the cells when running its auction.
Auctioneer Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Auctioneer Lock Held Duration (Nanoseconds)Time the active auctioneer has held the auctioneer lock.
Auctioneer Lock Is HeldWhether an auctioneer holds the auctioneer lock.
Auctioneer Memory AllocationsLifetime number of memory allocations
Auctioneer Memory DeallocationsLifetime number of memory deallocations
Auctioneer RequestsCumulative number of requests the auctioneer has handled through its API.
Auctioneer Used Heap Memory (Bytes)Instantaneous count memory allocated on the main heap and still in use
Auctioneer Used Memory (Bytes)Instantaneous count of memory allocated and still in use
Auctioneer Used Stack Memory (Bytes)Instantaneous count of memory allocated on the main stack and still in use
CIDVirtual Machine Cloud ID
Convergence Tasks PrunedCumulative number of times the BBS has deleted a malformed Task during its Task convergence pass.
Converger Active GoRoutinesInstantaneous number of active GoRoutines in the process
Converger CPUsNumber of CPUs on the machine
Converger Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Converger Lock Held Duration (Nanoseconds)Time the active converger has held the converger lock.
Converger Lock Is HeldWhether an converger holds the auctioneer lock.
Converger Memory AllocationsLifetime number of memory allocations
Converger Memory DeallocationsLifetime number of memory deallocations
Converger Task Pass Duration (Nanoseconds)Time the BBS took to run its Task convergence pass.
Converger Task Pass RunsCumulative number of times the BBS has run its Task convergence pass.
Converger Tasks KickedCumulative number of times the BBS has updated a Task during its Task convergence pass.
Converger Used Heap Memory (Bytes)Instantaneous count memory allocated on the main heap and still in use
Converger Used Memory (Bytes)Instantaneous count of memory allocated and still in use
Converger Used Stack Memory (Bytes)Instantaneous count of memory allocated on the main stack and still in use
Cumulative Auctioneer LRP Instances FailedCumulative number of LRP instances that the auctioneer failed to place on Diego cells.
Cumulative Auctioneer LRP Instances StartedCumulative number of LRP instances that the auctioneer successfully placed on Diego cells.
Cumulative Auctioneer Task Auctions StartedCumulative number of Tasks that the auctioneer successfully placed on Diego cells.
Cumulative Auctioneer Task Placements FailedCumulative number of Tasks that the auctioneer failed to place on Diego cells.
Delta Average Cumulative Auctioneer LRP Instances FailedThe delta average over the past 5 minutes for each minute of the cumulative number of LRP instances that the auctioneer failed to place on Diego cells.
Delta Average Cumulative Auctioneer LRP Instances StartedThe delta average over the past 5 minutes for each minute of the cumulative number of LRP instances that the auctioneer successfully placed on Diego cells.
Delta Average Cumulative Auctioneer Task Placements FailedThe delta average over the past 5 minutes for each minute of the cumulative number of Tasks that the auctioneer failed to place on Diego cells.
DeploymentDeployment
IndexIndex
IPIP
JobJob
Job StateJob State
Maximum Auctioneer Fetch State Duration (Nanoseconds)The maximum duration over the past 5 minutes the auctioneer took to fetch state from all the cells when running its auction.
Maximum Auctioneer Locks HeldWhether an Auctioneer instance holds the expected Auctioneer lock (in Locket). 1 means the active Auctioneer holds the lock, and 0 means the lock was lost. The metric may occasionally be 0 when the Auctioneer instances are performing a leader transition, but a prolonged value of 0 indicates an issue with Auctioneer.
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode

Diego Cell

NameDescription
Active GoRoutinesInstantaneous number of active GoRoutines in the process
CIDVirtual Machine Cloud ID
Consul Down ModeHealth status of the consul associated with this route-rmitter.
Counter EventsNumber of Counter Events aggregated in Metron
CPUsNumber of CPUs on the machine
Current Buffer Count (Bytes)Number of Dropsonde messages read by UDP socket but not yet unmarshalled
DeploymentDeployment
Dropped MessagesNumber of intentionally dropped messages from Metron's batch writer buffer. Batch writing is performed over TCP/TLS only.
Garden Active GoRoutinesInstantaneous number of active GoRoutines in the process
Garden Backing StoresNumber of container backing store files
Garden CPUsNumber of CPUs on the machine
Garden Depot DirectoriesNumber of directories in the Garden depot
Garden Heap Memory Usage (Bytes)Instantaneous count memory allocated on the main heap and still in use
Garden Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Garden Loop DevicesNumber of attached loop devices
Garden Memory AllocationsLifetime number of memory allocations
Garden Memory DeallocationsLifetime number of memory deallocations
Garden Memory Usage (Bytes)Instantaneous count of memory allocated and still in use
Garden Metrics ReportingHow long it took to emit the BackingStores, DepotDirs, and LoopDevices metrics
Garden Stack Memory Usage (Bytes)Instantaneous count of memory allocated on the main stack and still in use
Heap Memory Usage (Bytes)Instantaneous count memory allocated on the main heap and still in use
HTTPStart CountNumber of HTTPStart aggregated in Metron
HTTPStartStop CountNumber of HTTPStartStop events emitted by Metron (created by combining HTTPStart and HTTPStop events)
HTTPStop CountNumber of HTTPStop aggregated in Metron
IndexIndex
IPIP
JobJob
Job StateJob State
Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Log MessagesCount of application log messages.
Marshalled ContainerMetric MessagesNumber of ContainerMetric messages marshalled
Marshalled CounterEvent MessagesNumber of CounterEvent messages marshalled
Marshalled Error MessagesNumber of Error messages marshalled
Marshalled Heartbeat MessagesNumber of Heartbeat messages marshalled
Marshalled HTTPStart MessagesNumber of HttpStart messages marshalled
Marshalled HTTPStartStop MessagesNumber of HttpStartStop messages marshalled
Marshalled HTTPStop MessagesNumber of HttpStop messages marshalled
Marshalled LogMessage MessagesNumber of LogMessage messages marshalled
Marshalled ValueMetric MessagesNumber of ValueMetric messages marshalled
Marshalling Errors (Errors)Number of errors when marshalling messages
Maximum Route Emitter Synchronization Pass Duration (Nanoseconds)The max time for the last 15 minutes the active route-emitter took to perform its synchronization pass.
Memory AllocationsLifetime number of memory allocations
Memory DeallocationsLifetime number of memory deallocations
Memory Usage (Bytes)Instantaneous count of memory allocated and still in use
Received Data (Bytes)Number of bytes of Dropsonde messages read by UDP socket
Received Legacy Data (Bytes)Number of bytes of Legacy messages read by UDP socket
Received Legacy MessagesNumber of Legacy messages read by UDP socket
Received MessagesNumber of Dropsonde messages read by UDP socket
REP Active GoRoutinesInstantaneous number of active GoRoutines in the process
REP ActualLRP Synchronization Duration (Nanoseconds)Time the cell rep took to synchronize the ActualLRPs it has claimed with its actual garden containers.
REP ActualLRP Synchronization Maximum Duration (Nanoseconds)The maximum time for the last 15 minutes the cell rep took to synchronize the ActualLRPs it has claimed with its actual garden containers.
REP ContainersNumber of containers hosted on the cell.
REP Containers Remaining SpaceRemaining number of containers this cell can host.
REP CPUsNumber of CPUs on the machine
REP Garden Creation Time (Nanoseconds)Time the rep Garden backend took to create a container.
REP Heap Memory Usage (Bytes)Instantaneous count memory allocated on the main heap and still in use
REP is UnhealthyWhether the cell has failed to pass its health check against the garden backend.
REP Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
REP Memory AllocationsLifetime number of memory allocations
REP Memory DeallocationsLifetime number of memory deallocations
REP Memory Usage (Bytes)Instantaneous count of memory allocated and still in use
REP Remaining Disk Available (Mebibytes)Remaining amount of disk available for this cell to allocate to containers.
REP Remaining Memory Available (Mebibytes)Remaining amount of memory available for this cell to allocate to containers.
REP Stack Memory Usage (Bytes)Instantaneous count of memory allocated on the main stack and still in use
REP Total Available ContainersTotal number of containers this cell can host.
REP Total Available Disk (Mebibytes)Total amount of disk available for this cell to allocate to containers.
REP Total Available Memory (Mebibytes)Total amount of memory available for this cell to allocate to containers.
Route Emitter Active GoroutinesInstantaneous number of active GoRoutines in the process
Route Emitter CPUsNumber of CPUs on the machine
Route Emitter Heap Memory Usage (Bytes)Instantaneous count of memory allocated on the main heap and still in use
Route Emitter Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Route Emitter Lock HeldWhether a Route Emitter holds the Route Emitter lock.
Route Emitter Lock Held Duration (Nanoseconds)Time the active Route Emitter has held the convergence lock.
Route Emitter Memory AllocationsLifetime number of memory allocations
Route Emitter Memory DeallocationsLifetime number of memory deallocations
Route Emitter Memory Usage (Bytes)Instantaneous count of memory allocated and still in use
Route Emitter Messages EmittedThe cumulative number of registration messages that this process has sent
Route Emitter Registered RoutesCumulative number of route registrations emitted from the route-emitter as it reacts to changes to LRPs
Route Emitter RoutesNumber of routes in the route-emitter's routing table.
Route Emitter Stack Memory Usage (Bytes)Instantaneous count of memory used by the stack allocator
Route Emitter Synchronization Pass Duration (Nanoseconds)Time the active route-emitter took to perform its synchronization pass.
Route Emitter Synchronized RoutesCumulative number of route registrations emitted from the route-emitter during its periodic route-table synchronization
Route Emitter Unregistered RoutesCumulative number of route unregistrations emitted from the route-emitter as it reacts to changes to LRPs
Sent MessagesNumber of messages sent to Doppler regardless of protocol
Stack Memory Usage (Bytes)Instantaneous count of memory allocated on the main stack and still in use
TCP Send Errors (Errors)Number of errors if writing to Doppler over TCP fails
TCP Sent Data (Bytes)Lifetime data sent to Doppler over TCP
TCP Sent MessagesNumber of sent messages to Doppler over TCP
TLS Send Errors (Errors)Number of errors if writing to Doppler over TLS fails
TLS Sent Data (Bytes)Lifetime data sent to Doppler over TLS
TLS Sent MessagesNumber of sent messages to Doppler over TLS
UDP Send Errors (Errors)Number of errors if writing to Doppler over UDP fails
UDP Sent Data (Bytes)Lifetime data sent to Doppler over UDP
UDP Sent MessagesNumber of sent messages to Doppler over UDP
Uncategorized EventsNumber of non-(CounterEvent HTTPStart HTTPStop) events processed by aggregator
Unmarshalled ContainerMetric MessagesNumber of ContainerMetric messages unmarshalled
Unmarshalled CounterEvent MessagesNumber of CounterEvent messages unmarshalled
Unmarshalled Error MessagesNumber of Error messages unmarshalled
Unmarshalled HTTPStart MessagesNumber of HttpStart messages unmarshalled
Unmarshalled HTTPStartStop MessagesNumber of HttpStartStop messages unmarshalled
Unmarshalled HTTPStop MessagesNumber of HttpStop messages unmarshalled
Unmarshalled LogMessage MessagesNumber of LogMessage messages unmarshalled
Unmarshalled ValueMetric MessagesNumber of ValueMetric messages unmarshalled
Unmarshalling Errors (Errors)Number of errors when unmarshalling messages
Unmatched HTTPStart CountNumber of HTTPStart events for which no HTTPStop was received
Unmatched HTTPStop CountNumber of HTTPStop events for which no HTTPStart was received
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode

Doppler Server

NameDescription
Active GoRoutinesInstantaneous number of active Goroutines in the Doppler process
CIDVirtual Machine Cloud ID
Container Metric SinksInstantaneous number of container metric sinks known to the SinkManager
CPUsNumber of CPUs on the machine
DeploymentDeployment
Dropped MessagesNumber of messages dropped inside Doppler for various reasons (downstream consumer can't keep up internal object wasn't ready for message, etc.)
Dump SinksInstantaneous number of dump sinks known to the SinkManager
Errors When Unmarshalling (Errors)Number of errors when unmarshalling messages
File HandlesNumber of file handles for the Doppler's process
Firehose Dropped MessagesNumber of messages dropped by the firehose
Firehose Dropped Messages Delta MaximumThe maximum delta between each minute over the past 5 minutes of the cumulative number of messages dropped by the firehose
Firehose Received Messages Delta MaximumThe maximum delta between each minute over the past 5 minutes of the cumulative number of messages received across all Doppler listeners (UDP, TCP, TLS)
Firehose Sent MessagesNumber of sent messages through the firehose per subscription id
Firehose SinksInstantaneous number of firehose sinks known to the SinkManager
Firehose Total Received MessagesTotal number of messages received across all of Doppler's listeners (UDP, TCP, TLS)
Heap Memory Usage (Bytes)Instantaneous count of memory allocated on the main heap and still in use
IndexIndex
Invalid Signature Errors (Errors)Number of messages received with an invalid signature
IPIP
JobJob
Job StateJob State
Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause in nanoseconds
Memory AllocationsLifetime number of memory allocations
Memory DeallocationsLifetime number of memory deallocations
Memory Usage (Bytes)Instantaneous count of memory allocated and still in use
Messages ReceivedNumber of messages received by Doppler's internal MessageRouter
Missing Signature Errors (Errors)Number of messages received that are too small to contain a signature
Stack Memory Usage (Bytes)Instantaneous count of memory used by the stack allocator
Syslog SinksInstantaneous number of syslog sinks known to the SinkManager
TCP Errors (Errors)Number of errors encountered by Doppler's TCP Listener while handshaking, decoding or reading from the connection
TCP Messages ReceivedNumber of messages received by Doppler's TCP Listener
TCP Received (Bytes)Lifetime data received by Doppler's TCP Listener
TLS Errors (Errors)Number of errors encountered by Doppler's TLS Listener while handshaking, decoding or reading from the connection
TLS Messages ReceivedNumber of messages received by Doppler's TLS Listener
TLS Received (Bytes)Lifetime data received by Doppler's TLS Listener
Truncating Buffer Dropped MessagesNumber of messages intentionally dropped by Doppler from the sink for the specific sink. This counter event will correspond with log messages "Log message output is too high."
UDP Errors (Errors)Number of errors encountered by Doppler's UDP Listener while reading from the connection
UDP Messages ReceivedNumber of messages received by Doppler's UDP Listener
UDP Received (Bytes)Lifetime data received by Doppler's UDP Listener
Unmarshalled ContainerMetric MessagesNumber of ContainerMetric messages unmarshalled
Unmarshalled CounterEvent MessagesNumber of CounterEvent messages unmarshalled
Unmarshalled Error MessagesNumber of Error messages unmarshalled
Unmarshalled HttpStart MessagesLifetime number of HttpStart messages unmarshalled
Unmarshalled HttpStartStop MessagesNumber of HttpStartStop messages unmarshalled
Unmarshalled HttpStop MessagesNumber of HttpStop messages unmarshalled
Unmarshalled LogMessage MessagesNumber of LogMessage messages unmarshalled
Unmarshalled ValueMetric MessagesNumber of ValueMetric messages unmarshalled
Uptime (Seconds)Uptime for the Doppler's process
Valid SignaturesNumber of messages received with valid signatures
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode
Websocket SinksInstantaneous number of websocket sinks known to the SinkManager

etcd Server

NameDescription
Appends ReceivedNumber of append requests this node has processed
Appends SentNumber of append requests this node has sent
CIDVirtual Machine Cloud ID
Compare and Delete Fail Operations (Operations)Compare And Delete Fail operation count
Compare and Delete Success Operations (Operations)Compare And DeleteSuccess operation count
Compare and Swap Fail Operations (Operations)Compare And SwapFail operation count
Compare and Swap Success Operations (Operations)Compare And SwapSuccess operation count
Create Fail Operations (Operations)Create Fail operation count
Create Success Operations (Operations)Create Success operation count
Delete Fail Operations (Operations)Delete Fail operation count
Delete Success Operations (Operations)Delete Success operation count
DeploymentDeployment
etcd FollowersNumber of etcd followers
Expire Count Operations (Operations)Expire Count operation count
Gets Fail Operations (Operations)Gets Fail operation count
Gets Success Operations (Operations)Gets Success operation count
IndexIndex
IPIP
Is LeaderSignals if etcd is leader
JobJob
Job StateJob State
Leader Latency (Seconds)Current latency to a specific follower
Raft TermX-Raft-Term value from the /stats/store endpoint
Received Data Rate (Bytes per Second)Number of data received
Received Request Rate (per Second)Number of requests received.
Send Data Rate (Bytes per Second)Number of data sent.
Send Request Rate (per Second)Number of requests sent.
Sets Fail Operations (Operations)Sets Fail operation count
Sets Success Operations (Operations)Sets Success operation count
Update Fail Operations (Operations)Update Fail operation count
Update Success Operations (Operations)Update Success operation count
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode
Watchers Operations (Operations)Watchers operation count

GoRouter

NameDescription
Active GoRoutinesInstantaneous number of active GoRoutines in the process.
Average Route-Emitter Received Messages Rate (per Second)Average number of router register messages received for Route-Emitter rate.
Average Total Requests (Requests per Second)Average rate of Requests.
Bad Gateway CountThe number of bad gateways associated with this GoRouter.
CIDVirtual Machine Cloud ID
CPUsNumber of CPUs on the machine.
Delta Average Total Routes RegisteredThe average change in the number of routes associated with this GoRouter.
DeploymentDeployment
Exhausted Connections Maximum Rate (per Minute)The maximum number of file descriptors currently used by the Gorouter job over the last 5 minutes
Heap Memory Usage (Bytes)Instantaneous count of memory allocated on the main heap and still in use.
IndexIndex
IPIP
JobJob
Job StateJob State
Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Maximum Bad Gateway CountThe maximum number of bad gateways associated with this GoRouter in the last 5 minutes.
Maximum File DescriptorsThe maximum number of file descriptors currently used by the Gorouter job over the last 5 minutes
Maximum Server Errors RateThe maximum number of server errors associated with this GoRouter in the last 5 minutes.
Maximum Time Since Last Registry Update (Milliseconds)Maximum number of milliseconds since the last registry update in the last 5 minutes.
Memory AllocationsLifetime number of memory allocations.
Memory DeallocationsLifetime number of memory deallocations.
Memory Usage (Bytes)Instantaneous count of memory allocated and still in use.
Router Handling Average Latency (Milliseconds)The router handling latency.
Router Handling Latency (Milliseconds)The router handling latency.
Server ErrorsThe number of server errors associated with this GoRouter.
Stack Memory Usage (Bytes)Instantaneous count of memory used by the stack allocator.
Time Since Last Registry Update (Milliseconds)Number of milliseconds since the last registry update.
Total Cloud Controller Received MessagesTotal number of router register messages received for Cloud Controller.
Total Cloud Controller Received RequestsTotal number of requests received for Cloud Controller.
Total HTTP ResponsesTotal number of HTTP responses.
Total Received MessagesTotal number of router register messages received.
total requestsTotal number of request received.
Total Route-Emitter Received MessagesTotal number of router register messages received for Route-Emitter.
Total Routes RegisteredThe number of routes associated with this GoRouter.
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode

Organization

NameDescription
Application CountApplication Count
Billing EnabledBilling Enabled
Creation DateCreation Date
GUIDGUID
Service CountService Count
Space CountSpace Count
StatusStatus

Service

NameDescription
Creation DateCreation Date
DescriptionDescription
GUIDGUID
Is ActiveIs Active
Is BindableIs Bindable

Service Plan

NameDescription
DescriptionDescription
GUIDGUID

Space

NameDescription
Application CountApplication Count
Creation DateCreation Date
GUIDGUID
Organization GUIDOrganization GUID
Service CountService Count
SSH AllowedSSH Allowed

Syslog Drain Binder

NameDescription
Active GoRoutinesInstantaneous number of active GoRoutines in the Doppler process
CIDVirtual Machine Cloud ID
Cloud Controller Polled CountNumber of times the syslog drain binder has polled the cloud controller for syslog drain bindings
CPUsNumber of CPUs on the machine
DeploymentDeployment
Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause
Heap Memory Usage (Bytes)Instantaneous count of memory allocated on the main heap and still in use
IndexIndex
IPIP
JobJob
Memory AllocationsLifetime number of memory allocations
Memory DeallocationsLifetime number of memory deallocations
Memory Usage (Bytes)Instantaneous count of memory allocated and still in use
Stack Memory Usage (Bytes)Instantaneous count of memory used by the stack allocator
Syslog Drains Returned by Cloud ControllerNumber of syslog drains returned by cloud controller

Traffic Controller

NameDescription
Active GoRoutinesInstantaneous number of active GoRoutines in the Doppler process
CIDVirtual Machine Cloud ID
CPUsNumber of CPUs on the machine
DeploymentDeployment
File HandlesNumber of file handles for the TrafficController's process
Heap Memory Usage (Bytes)Instantaneous count of memory allocated on the main heap and still in use
IndexIndex
IPIP
JobJob
Job StateJob State
Last Garbage Collector Pause (Nanoseconds)Duration of the last Garbage Collector pause in nanoseconds
Memory AllocationsLifetime number of memory allocations
Memory DeallocationsLifetime number of memory deallocations
Memory Usage (Bytes)Instantaneous count of memory allocated and still in use
Serving Container Metrics Duration (Milliseconds)Duration for serving container metrics via the containermetrics endpoint
Serving Recent Logs Duration (Milliseconds)Duration for serving recent logs via the recentLogs endpoint
Stack Memory Usage (Bytes)Instantaneous count of memory used by the stack allocator
Uptime (Seconds)Uptime for the Traffic Controller's process
VM Active Memory Usage (Kibibytes)VM Active Memory Usage
VM Active Memory Utilization (%)VM Active Memory Utilization
VM CPU Utilization (%)Combined amount of CPU spent in system and user processes
VM Ephemeral Disk Utilization (%)VM Ephemeral Disk Utilization
VM Load Average (15m)Unix-style load average for the past 15 minutes
VM Persistent Disk Utilization (%)VM Persistent Disk Utilization
VM Swap Usage (Kibibytes)VM Swap Usage
VM Swap Utilization (%)VM Swap Utilization
VM System Disk Utilization (%)VM System Disk Utilization
VM User CPU Utilization (%)Time CPU spent in user mode