Logrotate Configuration Setup in AlmaLinux 8.9

Logrotate helps manage log files by automatically rotating, compressing, and removing them when they become too large or outdated, preventing excessive disk space usage and ensuring system stability.

Table of Contents

• What is logrotate in a nutshell?
• Requirements
• Version Information for Logrotate Linux Tomcat and MySQL
• Scope of logs per logrotate configuration file
• Implementation of Tomcat Log Rotation
  • Prerequisite - Disable internal Tomcat log rotation process
    • Edit /opt/tomcat/conf/logging.properties configuration file
    • Edit /opt/tomcat/conf/server.xml configuration file
    • Delete Unnecessary Archived Logs in /opt/tomcat/logs/archived/ directory
  • Logrotate Configuration for catalina.out log files
  • Logrotate Configuration for Tomcat Miscellaneous Logs
• Logrotate Configuration for httpd log files
• Logrotate Configuration for unison.log file
  • Changing the SELinux Context for /root/unison.log
  • Custom cron job for unison log
• Logrotate Configuration for MySQL-related log files
  • Configuring MySQL Authentication for Log Rotation
• Logrotate Troubleshooting and Best Practices
  • Refer to Linux Manual Page
  • Maintain Root Ownership for Logrotate Configuration Files
  • Verify Ownership Permission and SELinux settings of log files
  • Understanding the ‘su’ Directive with Specific ‘create’ Settings
  • Run Manual Test
  • Debug Mode
  • Last Log Rotation Timestamps by Logrotate
  • Compression Issue
  • Verify Full File Paths
  • Check for Typos
  • Curly Braces and Comments

What is logrotate in a nutshell?

Logrotate is designed to ease the administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.

Source: https://linux.die.net/man/8/logrotate

Requirements

Implement custom logrotate configurations for Tomcat and MySQL DB virtual servers for disk space optimisation. To do this, we define specific directives in the logrotate configurations, including:

• Setting the occurrence of log rotation (e.g., daily) for timely rotation of log files.
• Specifying the retention policy for rotated log files (e.g., retaining the last 10 rotations).
• Leveraging the compress directive.
• Configuring the ownership and permissions for the rotated log files to ensure appropriate access and security.

Version Information for Logrotate Linux Tomcat and MySQL

The logrotate configuration detailed in this document has been developed and tested on the following system versions. Knowing the version compatibility across these components is crucial to ensure that everything functions correctly and performs optimally.

Scope of logs per logrotate configuration file

The table below outlines the scope of log files managed by each logrotate configuration. Complete scripts for each logrotate configuration file are provided in the subsequent sections of this document.

Implementation of Tomcat Log Rotation

Prerequisite - Disable internal Tomcat log rotation process

As a prerequisite before configuring logrotate for the Tomcat-related log files, it is necessary to disable Tomcat's internal log rotation process to avoid any conflicts with logrotate. To do this, update the following configuration files on the Tomcat virtual machines.

• File: /opt/tomcat/conf/logging.properties
• File: /opt/tomcat/conf/server.xml

Edit /opt/tomcat/conf/logging.properties configuration file

• Set maxDays to -1. This allows log entries to be continuously written to a single log file without a retention period as specified in the documentation.
• Set the rotatable parameter to false. This enables external tools such as logrotate to manage the log rotation cycle.
• Comment out the manager and host-manager logs as they are no longer needed.

Apply the changes in the logging.properties file accordingly as shown below:

 11catalina.org.apache.juli.AsyncFileHandler.level = FINE
 21catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
 31catalina.org.apache.juli.AsyncFileHandler.prefix = catalina.
 4# Retain log file (no deletion) on the server file system, and disable internal Tomcat log rotation,
 5# as log rotation and cleanup are managed externally by logrotate.
 61catalina.org.apache.juli.AsyncFileHandler.maxDays = -1
 71catalina.org.apache.juli.AsyncFileHandler.rotatable = false
 81catalina.org.apache.juli.AsyncFileHandler.encoding = UTF-8
 9
102localhost.org.apache.juli.AsyncFileHandler.level = FINE
112localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
122localhost.org.apache.juli.AsyncFileHandler.prefix = localhost.
13# Retain log file (no deletion) on the server file system, and disable internal Tomcat log rotation,
14# as log rotation and cleanup are managed externally by logrotate.
152localhost.org.apache.juli.AsyncFileHandler.maxDays = -1
162localhost.org.apache.juli.AsyncFileHandler.rotatable = false
172localhost.org.apache.juli.AsyncFileHandler.encoding = UTF-8
18
19# These logs are no longer required
20#3manager.org.apache.juli.AsyncFileHandler.level = FINE
21#3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
22#3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
23#3manager.org.apache.juli.AsyncFileHandler.maxDays = -1
24#3manager.org.apache.juli.AsyncFileHandler.rotatable = false
25#3manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8
26
27# These logs are no longer required
28#4host-manager.org.apache.juli.AsyncFileHandler.level = FINE
29#4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
30#4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.
31#4host-manager.org.apache.juli.AsyncFileHandler.maxDays = -1
32#4host-manager.org.apache.juli.AsyncFileHandler.rotatable = false
33#4host-manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8
34
35java.util.logging.ConsoleHandler.level = FINE
36java.util.logging.ConsoleHandler.formatter = org.apache.juli.OneLineFormatter
37java.util.logging.ConsoleHandler.encoding = UTF-8
38
39
40############################################################
41# Facility specific properties.
42# Provides extra control for each logger.
43############################################################
44
45org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
46org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.AsyncFileHandler
47
48# These logs are no longer required for our project
49#org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level = INFO
50#org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers = 3manager.org.apache.juli.AsyncFileHandler
51
52# These logs are no longer required for our project
53#org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level = INFO
54#org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers = 4host-manager.org.apache.juli.AsyncFileHandler

After making the changes, perform the following steps:

• Restart Tomcat service: systemctl restart tomcat
• Check Tomcat service status: sudo systemctl status tomcat
• Reboot the VM reboot command.

Delete Unnecessary Archived Logs in /opt/tomcat/logs/archived/ directory

Logs such as task-debug.YYYY-MM-DD.log and debug.YYYY-MM-DD.2.log are generated for debugging purposes and are not needed for this project. These can be safely removed to free up disk space. As a temporary solution, we are using a cron job to remove the contents of this directory. To prevent these logs from being created in the future, their generation and rotation must be disabled at the application level.

To set up the cron job:

• As the root user, open the crontab editor by running the following command: crontab -e
• Add the following line to the crontab file:

1# Daily at 12:15 AM remove all contents from '/opt/tomcat/logs/archived' as these logs are not needed.
2# Disable log generation and rotation at the application level to prevent future creation.
315 0 * * 0 rm -rf /opt/tomcat/logs/archived/*

Logrotate Configuration for catalina.out log files

1. Use SSH to access the Tomcat #01 server as the root user.
2. Navigate to the directory by entering cd /etc/logrotate.d in the terminal.
3. Copy the existing file tomcat.disabled and rename it to catalina_out using the command: cp -a tomcat.disabled catalina_out

Note: It is crucial to include the -a option in the cp command to ensure that the new file retains the ownership, permissions, and SELinux context of the original file.

 1/opt/tomcat/logs/catalina.out 
 2{
 3    su tomcat adminops
 4    copytruncate
 5    daily                        
 6    rotate 7                    
 7    compress
 8    missingok
 9    notifempty
10    dateext
11    dateformat -%Y-%m-%d-%H%M
12    create 770 tomcat adminops    
13}

4. For testing purposes, you can perform a dry run of the log rotation process to see what would happen without actually rotating the logs: logrotate -vd /etc/logrotate.d/catalina_out
5. Repeat the previous steps for each additional Tomcat VM to apply the same logrotate configuration.

Logrotate Configuration for Tomcat Miscellaneous Logs

1. Use SSH to access the Tomcat #01 server as the root user.
2. Navigate to the directory by entering cd /etc/logrotate.d in the terminal.
3. Copy the existing file tomcat.disabled and rename it to catalina_out using the command: cp -a tomcat.disabled tomcat_misc

Note: It is crucial to include the -a option in the cp command to ensure that the new file retains the ownership, permissions, and SELinux context of the original file.

4. Update the file with the following custom configuration

 1/opt/tomcat/logs/catalina.log
 2/opt/tomcat/logs/localhost.log
 3/opt/tomcat/logs/localhost_access_log.txt
 4/opt/tomcat/logs/debug.log
 5/opt/tomcat/logs/task-debug.log
 6/opt/tomcat/logs/task-generation.log
 7{
 8    copytruncate
 9    daily                        
10    rotate 7                    
11    compress
12    missingok
13    notifempty
14    su tomcat tomcat
15    create 640 tomcat tomcat
16    sharedscripts
17    postrotate
18        /bin/systemctl reload tomcat.service > /dev/null 2>/dev/null || true
19    endscript
20}

5. For testing purposes, you can perform a dry run of the log rotation process to see what would happen without actually rotating the logs: logrotate -vd /etc/logrotate.d/tomcat_misc
6. Repeat the previous steps for each additional Tomcat VM to apply the same logrotate configuration.

Logrotate Configuration for httpd log files

1. Use SSH to access the Tomcat #01 server as the root user.
2. Navigate to the directory by entering cd /etc/logrotate.d in the terminal.
3. Copy the existing file tomcat.disabled and rename it to catalina_out using the command: cp -a tomcat.disabled catalina_out

Note: It is crucial to include the -a option in the cp command to ensure that the new file retains the ownership, permissions, and SELinux context of the original file.

4. Update the file with the following custom configuration

 1/var/log/httpd/access_log 
 2/var/log/httpd/error_log 
 3{
 4    su root adminops
 5    daily
 6    rotate 7
 7    compress
 8    missingok
 9    notifempty
10    create 770 root adminops
11    sharedscripts
12    postrotate
13        /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
14    endscript
15}

5. For testing purposes, you can perform a dry run of the log rotation process to see what would happen without actually rotating the logs: logrotate -vd /etc/logrotate.d/httpd
6. Repeat the previous steps for each additional Tomcat VM to apply the same logrotate configuration.

Logrotate Configuration for unison log files

1. Use SSH to access the Tomcat #01 server as the root user.
2. Navigate to the directory by entering cd /etc/logrotate.d in the terminal.
3. Copy the existing file tomcat.disabled and rename it to unison using the command: cp -a tomcat.disabled unison

Note: It is crucial to include the -a option in the cp command to ensure that the new file retains the ownership, permissions, and SELinux context of the original file.

4. Update the file with the following custom configuration

 1# The 'daily' directive is intentionally omitted because it caused issues with the default logrotate daily cron job.
 2# Instead, logrotate is manually invoked via a custom cron job set to run daily at 12:45 AM.
 3# Cron job: 45 0 * * * /usr/sbin/logrotate /etc/logrotate.d/unison
 4# This ensures that unison logs are rotated daily without conflicts.
 5
 6/root/unison.log 
 7{
 8    rotate 7
 9    copytruncate
10    compress
11    dateext
12    dateformat -%Y-%m-%d-%H%M
13    missingok
14    notifempty
15    create 600 root root
16}

Changing the SELinux Context for /root/unison.log

The following steps outline a workaround for the issue where logrotate cannot rotate unison.log due to SELinux permissions, as the file is initially located under the /root directory. As the root user, change the SELinux context type for the /root/unison.log file from admin_home_t to var_log_t using the following commands:

• Check the current SELinux context: ls -Z /root/unison.log
• Set the new SELinux context: semanage fcontext -a -t var_log_t "/root/unison.log"
• Apply the context change: restorecon -v /root/unison.log
• Verify the new SELinux context has been applied: ls -Z /root/unison.log
• Output expected: system_u:object_r:var_log_t:s0 /root/unison.log

Custom cron job for unison.log

Due to an issue with the daily directive in logrotate not working as expected, we need to set up a custom cron job to manually run logrotate for Unison logs. Follow these steps to create the cron job:

• As the root user, open the crontab editor by running the following command: crontab -e
• Add the following line to the crontab file to manually run logrotate for Unison logs daily at 12:30 AM:

1# Manually run logrotate for unison logs daily at 12:45 AM as a workaround, since the 'daily' directive in logrotate was not working for some reason.
2
330 0 * * * /usr/sbin/logrotate /etc/logrotate.d/unison

• Save and close the crontab file.
• Repeat the previous steps for each additional Tomcat VM to apply the same logrotate configuration. However, schedule the daily crontab on the other Tomcat VMs for 12:45 AM. This timing ensures that the master Unison sync, which runs on the Tomcat 01 VM, completes first, allowing the other VMs to rotate their unison.log files with a slight delay.

Logrotate Configuration for MySQL-related log files

1. Use SSH to access the Tomcat #01 server as the root user.
2. Navigate to the directory by entering cd /etc/logrotate.d in the terminal.
3. Open the file named mysql for editing (e.g., vim or nano).

Important Note: We encountered an issue where rotated MySQL log files were not being compressed as expected. To resolve this, the MySQL logrotate configuration file was updated with additional directives. The key directives added are the following:

• delaycompress
• dateext
• dateformat -%Y-%m-%d-%H%M

Outcome: These updates have successfully addressed the compression issue with rotated log files.

4. Update the file with the following configuration:

 1# The log file name and location can be set in
 2# /etc/my.cnf by setting the "log-error" option
 3# in [mysqld]  section as follows:
 4#
 5# [mysqld]
 6# log-error=/var/log/mysqld.log
 7#
 8# For the mysqladmin commands below to work, root account
 9# password is required. Use mysql_config_editor(1) to store
10# authentication credentials in the encrypted login path file
11# ~/.mylogin.cnf
12#
13# Example usage:
14#
15#  mysql_config_editor set --login-path=client --user=root --host=localhost --password
16#
17# When these actions has been done, un-comment the following to
18# enable rotation of mysqld's log error.
19#
20
21/var/log/mysqld.log
22/var/log/mysql.log
23/var/lib/mysql/log_slow_query.log
24{
25        daily
26        rotate 5
27        copytruncate
28        compress
29        delaycompress
30        dateext
31        dateformat -%Y-%m-%d-%H%M
32        missingok
33        notifempty
34        create 640 mysql mysql
35    postrotate
36       # just if mysqld is really running
37       if test -x /usr/bin/mysqladmin && \
38          /usr/bin/mysqladmin --login-path=logrotate ping &>/dev/null
39       then
40          /usr/bin/mysqladmin --login-path=logrotate flush-logs
41       fi
42    endscript
43}

Configuring MySQL Authentication for Log Rotation

1. Set up a secure login path to store the root user credentials: mysql_config_editor set --login-path=logrotate --user=root --host=localhost --password
2. Enter the root password when prompted. This command stores the credentials securely and avoids using plain text passwords in scripts.
3. Verify the stored login paths: mysql_config_editor print --all
4. Check the current permissions of the ~/.mylogin.cnf file: ls -l ~/.mylogin.cnf
5. If the permissions are not set to -rw-------, update them to ensure that only the file owner can read and write to it: chmod 600 ~/.mylogin.cnf
6. Repeat the previous steps for each additional MySQL DB VM to apply the same log rotation configuration.

Logrotate Troubleshooting and Best Practices

Resources for Logrotate

• Linux Shell Terminal: man logrotate
• Online Ressource: https://linux.die.net/man/8/logrotate

Maintain Root Ownership for Logrotate Configuration Files

The logrotate configuration files should have permissions set to 644 and ownership set to root, which are the default settings. Additionally, the SELinux context type should be system_u:object_r:etc_t. When creating a new custom logrotate file, it is advisable to use cp -a <existing_logrotate_conf> <custom_new_logrotate_config> to ensure that the permissions, ownership, and SELinux settings are preserved.

Verify Ownership, Permission, and SELinux settings of log files

• Use the ls -l command to ensure that the ownership and permissions of the log files specified in the logrotate configuration file (e.g., etc/logrotate.d/<config_file>) match the settings provided in the logrotate configuration. For example: create 640 mysql mysql

• Use ls-Z to display the SELinux (Security-Enhanced Linux) security context of files and directories.

Understanding the ‘su’ Directive with Specific ‘create’ Settings

When logrotate is configured with create 770 root adminops, it sets new log files to 770 permissions, accessible only by the owner and group (root and adminops), and restricts access for others. Include the su directive (su root adminops) in the logrotate configuration to enforce this ownership.

Run Manual Test

Perform a manual test by executing the following command as root user: logrotate -vf /etc/logrotate.d/<config_file> Note: Ensure that the original log file contains log entries, as an empty log file will not be rotated.

Debug Mode

This mode is purely for verifying what logrotate would do under normal operational conditions. If you want to see logrotate performing the operations without making changes, run the following: logrotate --debug /etc/logrotate.d/<config_file>

Last Log Rotation Timestamps by Logrotate

cat /var/lib/logrotate/logrotate.status

Compression Issue

If the first rotated file is not being compressed, ensure that both the compress and copytruncate directives are declared in the logrotate configuration file.

Verify Full File Paths

Double-check that the full file paths of the log files are correctly declared in the logrotate configuration file. If uncertain, navigate to the directory containing the log files and run the command pwd to confirm.

Check for Typos

Review the logrotate configuration file for any typographical errors that may be causing issues.

Curly Braces and Comments

Review the logrotate configuration file for any typographical errors that may be causing issues.