Ubuntu 18.10 Tomcat8 catalina.out file path

In previosu version, I saw that the catalina.out file was created at /var/log/tomcat8/catalina.out location.

As I was study for my current installation, Ubuntu 18.10 and last package version, I saw that this file is always empty.

I studyed the Tomcat8 configuration and saw that this output is managed with file /etc/tomcat8/logging.properties

There si something like : 1catalina.org.apache.juli.AsyncFileHandler.level = FINE 1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs 1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.

This means that a new file is created everyday in /var/log/tomcat8 with a file name like catalina.2018-11-30.

Also, the logrorate configuration is still there but configured on /var/log/tomcat8/catalina.out. So it has no impact.

Moreover, the logrotate configuration was able to change the group name for the file, with adm. Now, owners are set as tomcat8:tomcat8

Is it expected behaviour or something missing in the configuration set for the tomcat8 package ?

Regards

Etienne Jouvin

2 Answers

It seems as if catalina.out logging was broken in 18.10. tomcat8 logs to catalina.out through rsyslog, as you can see in the package's /etc/rsyslog.d/tomcat8.conf

$template TomcatFormat,"[%timegenerated:::date-year%-%timegenerated:::date-
month%-%timegenerated:::date-day% %timegenerated:::date-
hour%:%timegenerated:::date-minute%:%timegenerated:::date-second%]
[%syslogseverity-text%]%msg%\n"
:programname, startswith, "tomcat8" { /var/log/tomcat8/catalina.out;TomcatFormat stop
}

Unfortunately, it seems like the permissions on Ubuntu 18.10 for rsyslog are not correct to make this work. I found this in my syslog:

rsyslogd: file '/var/log/tomcat8/catalina.out': open error: Permission denied [v8.32.0 try ]

This causes rsyslog to send all tomcat8 logging to /var/log/syslog

As a side-effect, this causes it to seem like there is "pointless" logrotate on catalina.out. It is probably intended to still be working.

You should ensure that the group syslog has write permissions on the catalina.out file, then restart rsyslog. Logging should behave normally from this point on.

One (naive) way to ensure this permission:

sudo service tomcat8 stop
sudo touch /var/log/tomcat8/catalina.out
sudo chown tomcat8:syslog /var/log/tomcat8 -R
sudo chmod g+rwx /var/log/tomcat8
sudo chmod g+rw /var/log/tomcat8 -R
sudo chmod g+s /var/log/tomcat8 -R
sudo service rsyslog restart
sudo service tomcat8 start

(this is mostly comment - but a bit long to post in the comments section - and I hate answers which simply say "I did this and it worked")

The Tomcat9 logging is a similar mess on Ubuntu 18.10

I also have web access logs being written to /var/log/tomcat9.

Although I see the files described by @Christina there is no mention there of the localhost_access_log.YYYY-MM-DD.txt files.

Although the catalina*.out files are named with a date, that is the date the files are created, they are not being rotated.

However something is later compressing them. This turns out to be cron job (/etc/). This is aligned with the file naming used in the Tomcat logging configuration but does not force Tomcat to flush and close the logs.

The log files are owned by the tomcat user (implying that these are not maintained by rsyslog) and when I check with fuser it is tomcat that has open file handles, not rsyslog.

It looks to me that the errors in /var/log/syslog reporting permission denied to rsyslog are a red herring - there is a rsyslog config, but the default Tomcat configuration sends no logs to syslog. Indeed, since it is using java.util.logging it can't send syslog data to rsyslog directly.

The logrotate script is also broken / irrelevant to how tomcat9 is configured for logging. It uses the wrong file pattern and does nothing to tell tomcat to flush/reopen its log files.

So we have three different mechanisms for log management in the package, none of which provide a solution to the log rotation and which will actually break the integrity of your log files.

As I plan on implementing log aggregation, rather than fix the file mechanism I chose to go down the root of sending data to syslog. The default build of Tomcat on Ubuntu uses java.util.logging to provide the logging facility. Although this supports network socket based logging, it does not implement the syslog message protocol. However the tomcat process is managed by systemd, and the default tomcat configuration sends log messages to stdout as well as to files, hence I amended /lib/systemd/system/tomcat9.service to add these lines:

StandardOutput=syslog
StandardError=syslog

Note there is already an entry for SyslogIdentifier=tomcat.

That gets the data into rsyslog. The next step is to get it out of rsyslog into a file. I added a new file (/etc/rsyslog.d/10-symcbean.conf) containing:

$CreateDirs on
template(name="myfile" type="string" string="/var/log/apps/%programname%.log")
if ($programname startswith "tomcat") then { action(type="omfile" dynaFile="myfile") stop
}

(here the "startswith" string refers to the "SyslogIdentifier" in the systemd config).

I also created the directory /var/log/apps owned by tomcat:syslog (the tomcat user does not need access to this directory for the standard logs now handled by rsyslog, but I will be writing my garbage collection logs in there too).

Then it was simply a matter of restarting rsyslog and tomcat, and I saw the same log entries appearing in /var/log/apps/tomcat.log as were being written to /var/log/tomcat9 . The next step will be to retired the directly written logs in /var/log/tomcat9 by editing /var/lib/tomcat9/conf/logging.properties as follows:

# handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler
handlers = java.util.logging.ConsoleHandler
# .handlers = 1catalina.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler
.handlers = java.util.logging.ConsoleHandler

I don't need the web access logs as the HTTP traffic is mediated by a nginx proxy which handles the required logging. I commented out the entry in conf/server.xml for these.

The last part of the puzzle is to enable log rotation. This is a new file in /etc/logrotate.d containing:

/var/log/apps/tomcat*.log
{ rotate 20 su syslog syslog create 0644 syslog syslog daily dateext missingok notifempty compress delaycompress sharedscripts postrotate /usr/lib/rsyslog/rsyslog-rotate endscript
}

(the wildcard on the filename is to allow for different log streams/multiple log files).