How To Migrate the Ignition Loggers using Logback.XML

Introduction:

Ignition utilizes a third party library to log its gateway modules. Errors or warnings due to scripts that run in the gateway, device connections, database connections will all report to the gateway logs, which can be viewed from the Logs section of the gateway webpage. These gateway logs are stored in a location defined by Ignition in the ignition.conf file to store files named wrapper.log in the data/logs folder.

Ignition contains loggers that can provide events at regular intervals depending on the level of events. TRACE, DEBUG, INFO, WARN, and ERROR. TRACE and DEBUG are especially noisy loggers that can provide new logs every couple of seconds. The benefit of these TRACE and DEBUG levels is that they can let you observe the behavior of specific modules. This can be especially beneficial for the Support team when troubleshooting intermittent issues as these logs make it possible to observe the behavior of a module at the timestamp that an issue occurs. The problem is the frequency of the logs. You do not want to flood the gateway logs page with the same logs because that will prevent users from seeing any upcoming ERROR and WARN logs. Therefore, sometimes you may want to output the TRACE and DEBUG logs to a separate location.

You can control the output location of Ignition loggers. Ignition uses a third party logging library called Logback to output its logs. How logs are stored is determined through a file called Logback.XML which is located in the “data” folder of the Ignition install directory. By editing the Logback.XML, you can determine the frequency and location to store log information.

Instructions - Logback.XML

There is no need to remove any of the current code from the default Logback.XML. The following steps need to be completed in the Logback.XML to store files in separate location:

1. Set up a folder location to store the new log files.
2. Create an appender to determine which method to store the logs.
3. Set up a policy in the appender to determine how the logs will be stored.
4. Set-up a size limit for a log file before it creates a new file.
5. Set-up a limit to the number of log files that can exist before the oldest is pruned.
6. Specify the logger that you want to log to the files using which policy.
7. Optional - Set the time format for the logs
8. Optional - Disable the logger from appearing in the gateway webpage logs and the default log files.
9. Optional - Filter for specific logger levels

All additions should be within the opening and closing "configuration" tags but you should avoid adding any code within the default values in the logback.xml.

<configuration debug="true">
<appender name="SysoutAppender" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%.-1p [%-30c{1}] [%d{HH:mm:ss,SSS}]: %m %X%n</pattern>
</encoder>
</appender>

<appender name="DB" class="com.inductiveautomation.logging.SQLiteAppender">
<dir>logs</dir>
<!--
Maintenance Settings entryLimit: The maximum number of entries in the database. However, at any given time, there may be more than this number, due to how cleanup works.
maxEventsPerMaintenance: The number of event that can happen before a maintenance cycle occurs.
minTimeBetweenMaintenance: The minimum time (max frequency) between maintenance events. Takes precedent over max events.
vacuumFrequency: The number of maintenance cycles before a "vacuum" is performed, to recover disk space.

On disk, most log events are between 600-800 bytes.
<entryLimit>50000</entryLimit>
<maxEventsPerMaintenance>5000</maxEventsPerMaintenance>
<minTimeBetweenMaintenance>60000</minTimeBetweenMaintenance>
<vacuumFrequency>3</vacuumFrequency>
-->
</appender>
<appender name="SysoutAsync" class="ch.qos.logback.classic.AsyncAppender" queueSize="1000" discardingThreshold="0">
<appender-ref ref="SysoutAppender" />
</appender>
<appender name="DBAsync" class="ch.qos.logback.classic.AsyncAppender" queueSize="100000" discardingThreshold="0">
<appender-ref ref="DB" />
</appender>
<root level="INFO">
<appender-ref ref="SysoutAsync"/>
<appender-ref ref="DBAsync"/>
</root>

<!--Add additional code here -->

</configuration>

1. Set up a folder location to store the new log files.

This is the folder location where your new log files will be generated. We will use a property tag, so we can reference this value in other areas of the .XML document.

Place the following code below within the <configuration> tags. 

<property name="LOG_HOME" value="TagHistoryLogs" /> <!--Location for the logs folder. -->

If you don't specify folder path with the drive and a forward slash(E.g. C:/), then the folder will be created in the root of the Ignition install directory.

2. Create an appender to determine which method to store the logs.

Logback delegates the task of writing a logging event to components called appenders. Different appenders store logs differently, and it is possible to have multiple appenders in one logback.xml. In our example, we will use the RollingFileAppender class because it will continually create new log files while following certain conditions. RollingFileAppender can log to a file, and once a certain condition is met (e.g. reached max size of a log file), will continue logging to another file.

Place the following code within the <configuration> tags.

<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <!-- Creates separate log file -->    
</appender>

3. Set up a policy in the appender to determine how the logs will be stored.

RollingPolicy is responsible for the rollover procedure which involves file creation and pruning. We will use the SizeAndTimeBasedRollingPolicy. It defines a rollover policy based on datetime, for example by a specific day or month. We can provide a specific pattern to the name of log files using the fileNamePattern tags provided for SizeAndTimeBasedRollingPolicy. The naming pattern is a string that uses a date conversion string designated by date conversion token, %d, and an incremental conversion token, %i. The date conversion syntax follows the format accepted by java.text.SimpleDateFormat. The conversion token is used to increment each log file that is generated. Both the %i and %d tokens are mandatory for SizeAndTimeBasedRollingPolicy.

The appender requires that you specify the filepath for the .log file where the logs will be stored. We can reference the LOG_HOME property that we created in Step 1 by referencing the property name in curly braces and use it for the fileNamePattern. 

Place the following code inside the <appender> tags from Step 2.

<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- Sets policy for generating and deleting log files -->
  <fileNamePattern>${LOG_HOME}\\TagHistoryQuery.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
  <maxFileSize>10MB</maxFileSize> <!-- Max file size -->
  <maxHistory>10</maxHistory> <!-- Max 10 Days of History (Days is inferred from the value of fileNamePattern) -->
  <totalSizeCap>1GB</totalSizeCap> <!--Maximum sum of the log files' size -->
</rollingPolicy>

Steps 4 and 5 will explain a few of the values that you can adjust in the code above.

4. Set-up a size limit for a log file before it creates a new file.

Once a file reaches 10MB, then another log file will be created.

<maxFileSize>10MB</maxFileSize> <!-- Max file size -->

5. Set-up a limit to the number of log files that can exist before the oldest is pruned.

These lines limit the number of log files for TimeBasedRollingPolicy. The maxHistory value limits the number of historical days, months, years the stored logs can range and whether that unit is days, months, or years is inferred by the structure of the fileNamePattern tag. The totalSizeCap value limits the sum of all the log files to 100MB before the oldest file is pruned.

<maxHistory>10</maxHistory> <!-- Max 10 Days -->
<totalSizeCap>100MB</totalSizeCap> <!--Maximum sum of the log files' size -->

6. Specify the logger that you want to log to the files using which policy.

To specify a logger that you want to log, you should have an individual open and closing <logger> tag for each logger. If you have three loggers you want to log, then you should have three individual open and closing <logger> tags. In the appender-ref tag, you will provide the reference to the appender created in Step 2.

Place the following code outside of the <appender> tags but inside the <configuration> tags for each logger. 

<logger name="Tags.History.Query" level="TRACE">
  <appender-ref ref="FILE"/>
</logger>

Note: The Logback.xml is case sensitive to logger names. Some loggers names changed from lower case to upper case between Ignition versions. If the logger name provided does not have the correct case, then it will not be migrated to a separate file. 

7. Set the date and time format for the logs

To determine the date and time format that is utilized in the logs, you would add the line below inside the appender tags but outside the rollingPolicy tags and adjust the values in the %d{pattern, timezone} section, where pattern is a date pattern in the java.text.SimpleDateFormat format (e.g. MM:dd:YYYY HH:mm:ss) and the time zone is a name listed from the tz database. 

By default, the timezone of the logs is in the UTC+0 timezone. If you wanted the logs to be in a different timezone (e.g. to match your gateway's timezone), then you will need specify that in the logback.xml with the Timezone name. For Pacific Standard Time, you would use America/Los_Angeles.

Place the following code within the <appender> tags but outside the <rollingPolicy> tags.

<encoder>
  <pattern>%.-1p [%-30c{1}] [%d{MM:dd:YYYY HH:mm:ss, America/Los_Angeles}]: %m %X%n</pattern> <!-- Format of the logs -->
</encoder>

8. Optional - Disable the logger from appearing in the gateway webpage logs and the default log files.

By setting the additivity of a logger to false, you will prevent that logger from storing in other log files as well as the Ignition gateway webpage logs.

Place the additivity parameter to any of your logger tags and set it equal to "false". 

<logger name="Tags.History.Query" level="TRACE" additivity="false">
  <appender-ref ref="FILE"/>
</logger>

9. Optional - Filter for specific logger levels

In some cases, you may want to primarily store or omit the logs for a specific logger level. The following lines are commented out in the final result. 

Place the following code within the <configuration> tags but outside of the <rollingPolicy> and <encoder> tags.

<!-- Accepts DEBUG level -->
  <filter class="ch.qos.logback.classic.filter.LevelFilter">
    <level>DEBUG</level>
    <onMatch>ACCEPT</onMatch>
    <onMismatch>DENY</onMismatch>
  </filter>

Final Result

<property name="LOG_HOME" value="TagHistoryLogs" />

<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <!-- Creates separate log file -->
<!-- Filter logger level. -->
  <!-- <filter class="ch.qos.logback.classic.filter.LevelFilter">
    <level>DEBUG</level>
    <onMatch>ACCEPT</onMatch>
    <onMismatch>DENY</onMismatch>
  </filter> -->
  <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- Sets policy for generating and deleting log files -->
    <fileNamePattern>${LOG_HOME}\\TagHistoryQuery.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
    <maxFileSize>10MB</maxFileSize> <!-- Max file size -->
    <maxHistory>10</maxHistory> <!-- Max 10 Days of History (Days is inferred from the value of fileNamePattern) -->
    <totalSizeCap>1GB</totalSizeCap> <!--Maximum sum of the log files' size -->
  </rollingPolicy>
  <encoder>
    <pattern>%.-1p [%-30c{1}] [%d{MM:dd:YYYY HH:mm:ss, America/Los_Angeles}]: %m %X%n</pattern> <!-- Format of the logs -->
  </encoder>
</appender>

<logger name="Tags.History.Query" level="TRACE" additivity="false">
  <appender-ref ref="FILE"/>
</logger>

After Editing the Logback.xml

Once you have finished making change to the Logback.xml, the Ignition gateway needs to be restarted to load the new logback.xml configuration for its loggers.

Your new logback.xml will remain configured on your gateway until you decide to edit it again. Note that the logback.xml is not carried over with a .gwbk.

Additional Resources

Logback Manual: http://logback.qos.ch/manual/configuration.html

SizeAndTimeBasedRollingPolicy: https://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy

DateTimeFormat pattern: https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html

List of Timezones: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

Using patterns in Logback: http://logback.qos.ch/manual/layouts.html#conversionWord