Bindplane

The Bindplane Developer Hub

Welcome to the Bindplane developer hub. You'll find comprehensive guides and documentation to help you start working with Bindplane as quickly as possible, as well as support if you get stuck. Let's jump right in!

Apache Kafka

Monitor Apache Kafka Distributed Streaming Platform

Data Collection Setup

Monitoring metrics are collected via the JMX API on the Kafka Broker server.

Network Requirements

Port: 9999 (TCP) connection to the Apache Kafka Broker

Least Privilege User

If authentication is setup on the Kalfka Broker server, than use a Kafka user.
For more information see: Security in the Apache Kafka Documentation

Supported Versions

Apache Kafka: 2.0.0-2.0.10+

Connection Parameters

Name
Required?
Description

Brokers

Required

Comma-separated list of one or more brokers to connect to.

Port

The port for communication to the Apache Kafka platform.

Producers

Comma-separated list of one or more producers to connect to.

Consumers

Comma-separated list of one or more consumers to connect to.

Username

Password

SSL Configuration

The SSL mode to use when connecting to the target. Can be configured to not use SSL (No SSL), use SSL but do not verify the target's certificate (No Verify), and use SSL and verify the target's certificate (Verify).

Connection Timeout (seconds)

The number of seconds to allow for connecting to the Apache Kafka platform.

Request Timeout (seconds)

The number of seconds to allow for the API to return a response.

Metrics

Broker

Name
Description

Error Count (Errors)

Number of errors in responses.

Error Rate (per Second)

Rate of errors.

Free Memory Size (Bytes)

Free memory on the server.

Hostname

Hostname of the server.

Incoming Broker Data (Bytes)

Amount of data incoming from other brokers.

Incoming Broker Traffic Rate (Bytes per Second)

Rate of traffic incoming from other brokers.

Incoming Client Data (Bytes)

Amount of data incoming from clients.

Incoming Client Traffic Rate (Bytes per Second)

Rate of traffic incoming from clients.

Incoming Messages

Total number of incoming messages.

Incoming Message Rate (per Second)

Rate of incoming messages.

ISR Expansion Count

Number of in-sync replica expansions.

ISR Expansion Rate (per Second)

Rate of in-sync replica expansions.

ISR Shrink Count

Number of in-sync replica shrinks.

ISR Shrink Rate (per Second)

Rate of in-sync replica shrinks.

Leader Election Count

Number of leader elections.

Leader Election Rate (per Second)

Rate of leader elections.

Memory Utilization (%)

Memory utilization of the server.

Message Conversion Count

Number of records which required message format conversion.

Message Conversion Rate (per Second)

Rate of records which required message format conversion.

Message Conversion Time (Milliseconds)

Time spent on message format conversions.

Offline Partitions Count

Number of partitions that are offline.

OS

Operating system of the server.

Outgoing Broker Data (Bytes)

Amount of data outgoing to other brokers.

Outgoing Broker Traffic Rate (Bytes per Second)

Rate of traffic outgoing to other brokers.

Outgoing Client Data (Bytes)

Amount of data outgoing to clients.

Outgoing Client Traffic Rate (Bytes per Second)

Rate of traffic outgoing to clients.

Partitions Count

Number of partitions.

Process CPU Load (%)

CPU load from Kafka processes.

Purgatory Size

Number of requests waiting in purgatory.

Request Count (Requests)

Number of requests.

Request Rate (per Second)

Rate of requests.

System CPU Load (%)

CPU load of the server.

Temporary Memory Size (Bytes)

Temporary memory used for message format conversions and decompression.

Total Memory Size (Bytes)

Total memory of the server.

Under Replicated Partitions Count

Number of partitions that are under-replicated.

Cluster

Name
Description

Broker Count

Number of brokers.

Consumer Count

Number of consumers.

ID

ID of the Kafka cluster.

Producer Count

Number of producers.

Total Incoming Message Rate (per Second)

Total rate of incoming messages over all brokers in the cluster.

Total Incoming Traffic Rate (Bytes per Second)

Total rate of incoming traffic over all brokers in the cluster from other brokers.

Total Outgoing Traffic Rate (Bytes per Second)

Total rate of outgoing traffic over all brokers in the cluster to other brokers.

Total Request Rate (per Second)

Total rate of requests over all brokers in the cluster.

Version

Kafka version.

Consumer

Name
Description

Assigned Partition Count

Number of partitions currently assigned to this consumer.

Average Commit Latency (Nanoseconds)

Average time taken for a commit request.

Average IO Wait Time (Nanoseconds)

Average length of time the I/O thread spent waiting for a socket ready for reads or writes.

Average Join Time (Nanoseconds)

Average time taken for a group rejoin.

Average Sync Time (Nanoseconds)

Average time taken for a group sync.

Commit Count

Total number of commit calls.

Commit Rate (per Second)

Rate of commit calls.

Connection Close Count

Total number of connections closed.

Connection Close Rate (per Second)

Rate of connections closed.

Connection Count

Number of connections.

Connection Create Count

Total number of new connections established.

Connection Create Rate (per Second)

Rate of connections established.

Hostname

Hostname of the server.

Incoming Data (Bytes)

Total amount of data read off all sockets.

Incoming Traffic Rate (Bytes per Second)

Rate of data read off all sockets.

IO Time (Nanoseconds)

Total time the I/O thread spent doing I/O.

IO Time Ratio (%)

Fraction of time the I/O thread spent doing I/O.

IO Wait Time (Nanoseconds)

Total time the I/O thread spent waiting.

IO Wait Time Ratio (%)

Fraction of time the I/O thread spent waiting.

Join Count

Total number of group joins.

Join Rate (per Second)

Rate of group joins.

Maximum Commit Latency (Nanoseconds)

Maximum time taken for a commit request.

Maximum Join Time (Nanoseconds)

Maximum time taken for a group rejoin.

Maximum Lag (Records)

The maximum lag in terms of number of records for any partition in this window.

Maximum Sync Time (Nanoseconds)

Maximum time taken for a group sync.

OS

Operating system of the server.

Outgoing Data (Bytes)

Total amount of outgoing data sent to all servers.

Outgoing Traffic Rate (Bytes per Second)

Total rate of outgoing data sent to all servers.

Request Count (Requests)

Total number of requests sent.

Request Rate (per Second)

Rate of requests sent.

Response Count

Total number of responses received.

Response Rate (per Second)

Rate of responses received.

Sync Count

Total number of group syncs.

Sync Rate (per Second)

Rate of group syncs.

Producer

Name
Description

Available Buffer Memory (Bytes)

Total amount of buffer memory that is not being used (either unallocated or in the free list).

Average IO Wait Time (Nanoseconds)

Average length of time the I/O thread spent waiting for a socket ready for reads or writes.

Average Request Latency (Nanoseconds)

Average latency of requests.

Connection Close Count (Connections)

Total number of connections closed.

Connection Close Rate (Connections per Second)

Rate of connections closed.

Connection Count (Connections)

Number of connections.

Connection Create Count (Connections)

Total number of new connections established.

Connection Create Rate (Connections per Second)

Rate of connections established.

Hostname

Hostname of the server.

Incoming Data (Bytes)

Total amount of data read off all sockets.

Incoming Traffic Rate (Bytes per Second)

Rate of data read off all sockets.

IO Time (Nanoseconds)

Total time the I/O thread spent doing I/O.

IO Time Ratio (%)

Fraction of time the I/O thread spent doing I/O.

IO Wait Time (Nanoseconds)

Total time the I/O thread spent waiting.

IO Wait Time Ratio (%)

Fraction of time the I/O thread spent waiting.

Maximum Request Latency (Nanoseconds)

Maximum latency of requests.

OS

Operating system of the server.

Outgoing Data (Bytes)

Total amount of outgoing data sent to all servers.

Outgoing Traffic Rate (Bytes per Second)

Total rate of outgoing data sent to all servers.

Record Error Count (Errors)

Total number of record sends that resulted in errors.

Record Error Rate (Errors per Second)

Rate of record sends that resulted in errors.

Record Retry Count

Total number of retried record sends.

Record Retry Rate (per Second)

Rate of retried record sends.

Record Send Count (Records)

Total number of records sent.

Record Send Rate (Records per Second)

Rate of records sent.

Request Count (Requests)

Total number of requests sent.

Request Rate (per Second)

Rate of requests sent.

Response Count

Total number of responses received.

Response Rate (per Second)

Rate of responses received.

Total Buffer Memory (Bytes)

Maximum amount of buffer memory the client can use (whether or not it is currently used).

Waiting Thread Count (Threads)

Number of user threads blocked waiting for buffer memory to enqueue their records.