By default Kafka Connect sends its output to stdout, so you’ll see it on the console, Docker logs, or wherever. Sometimes you might want to route it to file, and you can do this by reconfiguring log4j. You can also change the configuration to get more (or less) detail in the logs by changing the log level.
Finding the log configuration file #
The configuration file is called connect-log4j.properties and usually found in etc/kafka/connect-log4j.properties.
To find out the config file location simply inspect the running Kafka Connect process, or the existing stdout, and look for the location of connect-log4j.properties in -Dlog4j.configuration:
[2019-01-15 10:34:22,755] INFO WorkerInfo values:
jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/var/folders/q9/2tg_lt9j6nx29rvr5r5jn_bw0000gp/T/confluent.tcrfo4zH/connect/logs, -Dlog4j.configuration=file:/Users/Robin/cp/confluent-5.1.0/bin/../etc/kafka/connect-log4j.propertiesBy default the file looks like this:
log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERRORChanging the log level #
In troubleshooting Kafka Connect it can be useful to increase the log level to DEBUG (or even TRACE). To do this simply change the config file’s rootlogger:
#log4j.rootLogger=INFO, stdout
log4j.rootLogger=DEBUG, stdoutSending logs to file #
To send the output to file instead, change it to this:
log4j.rootLogger=INFO, logFile
log4j.appender.logFile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logFile.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.logFile.File=/tmp/connect-worker.log
log4j.appender.logFile.layout=org.apache.log4j.PatternLayout
log4j.appender.logFile.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERRORThis will write it to /tmp/connect-worker.log.
Sending logs to both stdout and file #
You can get fancy and send TRACE to file whilst retaining INFO (and above) on stdout, using the Threshold option for the stdout appender:
log4j.rootLogger=TRACE, traceFile, stdout
log4j.appender.traceFile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.traceFile.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.traceFile.File=/tmp/connect-worker-trace.log
log4j.appender.traceFile.layout=org.apache.log4j.PatternLayout
log4j.appender.traceFile.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Threshold=INFO
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR