rsyslog in Drupal

Drupal Logs with rsyslog

Managing Drupal logs with rsyslog in Containerized Workloads

In containerized environments, logging agents collect application logs and send them to appropriate databases. These logs are typically captured from the stdout of the containers’ PID 1 process. Therefore, it is essential for containerized services to correctly route their logs to relevant streams.

In Drupal, the default logging mechanism stores application logs in the database, which can be accessed via the Drupal Admin Panel. However, the Admin Panel offers limited functionality, making it cumbersome to efficiently view and manage logs. This is where rsyslog becomes useful. By integrating the rsyslog module with Drupal, logs can be redirected to files or even to standard streams like stdout and stderr, offering greater flexibility in log management.

In this article, I will demonstrate how to format Drupal logs in JSON using rsyslog, allowing logging agents like FluentBit and Promtail to efficiently collect and parse them.

1) Enable Syslog Module in Drupal

You can enable it by running drush en syslog command

Or you can navigate to /admin/modules of your website and enable

Or you can add it under module section in core.extension.yml file

2) Install rsyslog into Drupal Container

for Debian-based images:

RUN apt update && \
    apt install -y rsyslog \
    --no-install-recommends \
    && \
    apt clean autoclean && \
    apt autoremove --yes

for Alpine-based images:

RUN apk add --update --no-cache rsyslog

3) Configure rsyslog

First create a drupal.conf file. Then update rsyslog.conf accordingly. Here is an example of drupal.conf file. I just created a basic template called RSYSLOG_NEW_File_Format. And also edited the conditions not to get php-specific logs, I only need Drupal logs

template(name="RSYSLOG_NEW_File_Format" type="list") {
     property(name="msg" spifno1stsp="on")
     property(name="msg" droplastlf="on")
     constant(value="\n")
}

if $programname startswith 'drupal' then {
    
    if (($msg contains "php") or ($msg contains "#")) then {
      action(type="omfile" file="/var/log/php.log" template="RSYSLOG_NEW_File_Format")
    } else {
      action(type="omfile" file="/var/log/drupal.log" template="RSYSLOG_NEW_File_Format")
    }

    stop # Stops syslog posting to other files, such as /var/log/syslog

}

Here we comment out imklog to avoid getting the “activation of module imklog failed” error and ActionFileDefaultTemplate to disable using default Template because we will use RSYSLOG_NEW_File_Format described above in drupal.conf file. Finally, the most important part, creating a symbolic link to make sure sending all Drupal logs to the stdout of PID 1 process so that log collectors like Fluentbit and Promtail can collect them

ARG RSYSLOG_CONF_FILE=/etc/rsyslog.conf
ARG RSYSLOG_CONF_DIR=/etc/rsyslog.d

COPY drupal.conf ${RSYSLOG_CONF_DIR}/drupal.conf

RUN sed -i -e '/imklog/s/^/#/' -e '/ActionFileDefaultTemplate/s/^/#/' ${RSYSLOG_CONF_FILE} \
    && echo "local0.* /var/log/drupal.log" >> ${RSYSLOG_CONF_FILE} \
    && ln -sfT /proc/1/fd/1 /var/log/drupal.log


# for debian-based 
CMD ["sh", "-c", "service rsyslog start && ***other-commands***"]

# for alpine-based 
CMD ["sh", "-c", "rsyslogd && ***other-commands***"]

4) Configure syslog.settings.yml

identity: drupal
facility: 128
format: '{"level":"!severity","log_type":"!type","referer":"!referer","team":"my-team","source":"/var/log/drupal.log","service_name":"drupal-container","trace_id":"!uid","message":"!message","request_uri":"!request_uri","requester_ip":"!ip"}'

Example

Here you can see the full Dockerfile. I am using a Debian-based Drupal image that runs on an Apache2 server with a custom apache.conf file

FROM drupal:apache-bullseye

WORKDIR /opt/drupal

ARG RSYSLOG_CONF_FILE=/etc/rsyslog.conf
ARG RSYSLOG_CONF_DIR=/etc/rsyslog.d
ARG APACHE_SITES_ENABLED=/etc/apache2/sites-enabled
ENV APACHE_LOG_DIR=/var/log/apache2

# Install rsyslog
RUN apt update && \
    apt install -y \
    --no-install-recommends \
    rsyslog && \
    apt clean autoclean && \
    apt autoremove --yes

# Install and Implement Rsyslog
RUN sed -i -e '/imklog/s/^/#/' -e '/ActionFileDefaultTemplate/s/^/#/' ${RSYSLOG_CONF_FILE} \
    && echo "local0.* /var/log/drupal.log" >> ${RSYSLOG_CONF_FILE} \
    && ln -sfT /proc/1/fd/1 /var/log/drupal.log \
    && unlink ${APACHE_LOG_DIR}/access.log \
    && unlink ${APACHE_LOG_DIR}/other_vhosts_access.log

# Copy Config files
COPY composer.json .
COPY composer.lock .
COPY drupal.conf ${RSYSLOG_CONF_DIR}/drupal.conf
COPY apache.conf ${APACHE_SITES_ENABLED}/000-default.conf

# Copy Drupal folders
COPY web/themes web/themes
COPY web/profiles web/profiles
COPY web/sites web/sites
COPY web/modules web/modules

# Secure files and folders
RUN chmod 555 web/sites/default && \
    chmod 444 web/sites/default/settings.php && \
    chmod 444 web/sites/default/services.yml

# Install drupal packages
RUN composer install --no-progress --no-interaction --no-dev

ServerTokens Prod

ServerSignature Off

TraceEnable Off

ServerName ${MY_HOSTNAME}

<VirtualHost ${MY_HOSTNAME}:80>
 
   ServerName ${MY_HOSTNAME}
   ServerAdmin webmaster@localhost
   DocumentRoot /var/www/html


   <Directory "/var/www/html/">

      AllowOverride None
      AllowOverrideList None

   </Directory>

   ErrorLog /proc/1/fd/2
   CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

In conclusion, with the help of rsyslog we were able to get the drupal logs, format them in JSON, and then send them to specific files or standard streams.

RESOURCES

https://www.drupal.org/docs/8/core/modules/syslog/overview

Using a different log Format for all Files - rsyslog