Gitiles
Code Review Sign In
gerrit.lfbroadband.org / voltha-lib-go / 222b61b4ab1b4a3676d1cbf1258e71adcc506ea3 / . / vendor / github.com / IBM / sarama / sarama.go
blob: 4d5f60a6664e022e30866587212d5b126446fa4e [file] [log] [blame]
Abhay Kumar40252eb2025-10-13 13:25:53 +0000[diff] [blame]1/*
2Package sarama is a pure Go client library for dealing with Apache Kafka (versions 0.8 and later). It includes a high-level
3API for easily producing and consuming messages, and a low-level API for controlling bytes on the wire when the high-level
4API is insufficient. Usage examples for the high-level APIs are provided inline with their full documentation.
5
6To produce messages, use either the AsyncProducer or the SyncProducer. The AsyncProducer accepts messages on a channel
7and produces them asynchronously in the background as efficiently as possible; it is preferred in most cases.
8The SyncProducer provides a method which will block until Kafka acknowledges the message as produced. This can be
9useful but comes with two caveats: it will generally be less efficient, and the actual durability guarantees
10depend on the configured value of `Producer.RequiredAcks`. There are configurations where a message acknowledged by the
11SyncProducer can still sometimes be lost.
12
13To consume messages, use Consumer or Consumer-Group API.
14
15For lower-level needs, the Broker and Request/Response objects permit precise control over each connection
16and message sent on the wire; the Client provides higher-level metadata management that is shared between
17the producers and the consumer. The Request/Response objects and properties are mostly undocumented, as they line up
18exactly with the protocol fields documented by Kafka at
19https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
20
21Metrics are exposed through https://github.com/rcrowley/go-metrics library in a local registry.
22
23Broker related metrics:
24
25 +---------------------------------------------------------+------------+---------------------------------------------------------------+
26 | Name | Type | Description |
27 +---------------------------------------------------------+------------+---------------------------------------------------------------+
28 | incoming-byte-rate | meter | Bytes/second read off all brokers |
29 | incoming-byte-rate-for-broker-<broker-id> | meter | Bytes/second read off a given broker |
30 | outgoing-byte-rate | meter | Bytes/second written off all brokers |
31 | outgoing-byte-rate-for-broker-<broker-id> | meter | Bytes/second written off a given broker |
32 | request-rate | meter | Requests/second sent to all brokers |
33 | request-rate-for-broker-<broker-id> | meter | Requests/second sent to a given broker |
34 | request-size | histogram | Distribution of the request size in bytes for all brokers |
35 | request-size-for-broker-<broker-id> | histogram | Distribution of the request size in bytes for a given broker |
36 | request-latency-in-ms | histogram | Distribution of the request latency in ms for all brokers |
37 | request-latency-in-ms-for-broker-<broker-id> | histogram | Distribution of the request latency in ms for a given broker |
38 | response-rate | meter | Responses/second received from all brokers |
39 | response-rate-for-broker-<broker-id> | meter | Responses/second received from a given broker |
40 | response-size | histogram | Distribution of the response size in bytes for all brokers |
41 | response-size-for-broker-<broker-id> | histogram | Distribution of the response size in bytes for a given broker |
42 | requests-in-flight | counter | The current number of in-flight requests awaiting a response |
43 | | | for all brokers |
44 | requests-in-flight-for-broker-<broker-id> | counter | The current number of in-flight requests awaiting a response |
45 | | | for a given broker |
46 | protocol-requests-rate-<api-key> | meter | Number of api requests sent to the brokers for all brokers |
47 | | | https://kafka.apache.org/protocol.html#protocol_api_keys | |
48 | protocol-requests-rate-<api-key>-for-broker-<broker-id> | meter | Number of packets sent to the brokers by api-key for a given |
49 | | | broker |
50 +---------------------------------------------------------+------------+---------------------------------------------------------------+
51
52Note that we do not gather specific metrics for seed brokers but they are part of the "all brokers" metrics.
53
54Producer related metrics:
55
56 +-------------------------------------------+------------+--------------------------------------------------------------------------------------+
57 | Name | Type | Description |
58 +-------------------------------------------+------------+--------------------------------------------------------------------------------------+
59 | batch-size | histogram | Distribution of the number of bytes sent per partition per request for all topics |
60 | batch-size-for-topic-<topic> | histogram | Distribution of the number of bytes sent per partition per request for a given topic |
61 | record-send-rate | meter | Records/second sent to all topics |
62 | record-send-rate-for-topic-<topic> | meter | Records/second sent to a given topic |
63 | records-per-request | histogram | Distribution of the number of records sent per request for all topics |
64 | records-per-request-for-topic-<topic> | histogram | Distribution of the number of records sent per request for a given topic |
65 | compression-ratio | histogram | Distribution of the compression ratio times 100 of record batches for all topics |
66 | compression-ratio-for-topic-<topic> | histogram | Distribution of the compression ratio times 100 of record batches for a given topic |
67 +-------------------------------------------+------------+--------------------------------------------------------------------------------------+
68
69Consumer related metrics:
70
71 +-------------------------------------------+------------+--------------------------------------------------------------------------------------+
72 | Name | Type | Description |
73 +-------------------------------------------+------------+--------------------------------------------------------------------------------------+
74 | consumer-batch-size | histogram | Distribution of the number of messages in a batch |
75 | consumer-fetch-rate | meter | Fetch requests/second sent to all brokers |
76 | consumer-fetch-rate-for-broker-<broker> | meter | Fetch requests/second sent to a given broker |
77 | consumer-fetch-rate-for-topic-<topic> | meter | Fetch requests/second sent for a given topic |
78 | consumer-fetch-response-size | histogram | Distribution of the fetch response size in bytes |
79 | consumer-group-join-total-<GroupID> | counter | Total count of consumer group join attempts |
80 | consumer-group-join-failed-<GroupID> | counter | Total count of consumer group join failures |
81 | consumer-group-sync-total-<GroupID> | counter | Total count of consumer group sync attempts |
82 | consumer-group-sync-failed-<GroupID> | counter | Total count of consumer group sync failures |
83 +-------------------------------------------+------------+--------------------------------------------------------------------------------------+
84*/
85package sarama
86
87import (
88 "io"
89 "log"
90)
91
92var (
93 // Logger is the instance of a StdLogger interface that Sarama writes connection
94 // management events to. By default it is set to discard all log messages via io.Discard,
95 // but you can set it to redirect wherever you want.
96 Logger StdLogger = log.New(io.Discard, "[Sarama] ", log.LstdFlags)
97
98 // PanicHandler is called for recovering from panics spawned internally to the library (and thus
99 // not recoverable by the caller's goroutine). Defaults to nil, which means panics are not recovered.
100 PanicHandler func(interface{})
101
102 // MaxRequestSize is the maximum size (in bytes) of any request that Sarama will attempt to send. Trying
103 // to send a request larger than this will result in an PacketEncodingError. The default of 100 MiB is aligned
104 // with Kafka's default `socket.request.max.bytes`, which is the largest request the broker will attempt
105 // to process.
106 MaxRequestSize int32 = 100 * 1024 * 1024
107
108 // MaxResponseSize is the maximum size (in bytes) of any response that Sarama will attempt to parse. If
109 // a broker returns a response message larger than this value, Sarama will return a PacketDecodingError to
110 // protect the client from running out of memory. Please note that brokers do not have any natural limit on
111 // the size of responses they send. In particular, they can send arbitrarily large fetch responses to consumers
112 // (see https://issues.apache.org/jira/browse/KAFKA-2063).
113 MaxResponseSize int32 = 100 * 1024 * 1024
114)
115
116// StdLogger is used to log error messages.
117type StdLogger interface {
118 Print(v ...interface{})
119 Printf(format string, v ...interface{})
120 Println(v ...interface{})
121}
122
123type debugLogger struct{}
124
125func (d *debugLogger) Print(v ...interface{}) {
126 Logger.Print(v...)
127}
128func (d *debugLogger) Printf(format string, v ...interface{}) {
129 Logger.Printf(format, v...)
130}
131func (d *debugLogger) Println(v ...interface{}) {
132 Logger.Println(v...)
133}
134
135// DebugLogger is the instance of a StdLogger that Sarama writes more verbose
136// debug information to. By default it is set to redirect all debug to the
137// default Logger above, but you can optionally set it to another StdLogger
138// instance to (e.g.,) discard debug information
139var DebugLogger StdLogger = &debugLogger{}