800-92 - Guide to Computer Security Log Management

A log is a record of the events occurring within an organization’s systems and networks. Logs are composed of log entries; each entry contains information related to a specific event that has occurred within a system or network.

Because of the widespread deployment of networked servers, workstations, and other computing devices, and the ever-increasing number of threats against networks and systems, the number, volume, and variety of computer security logs has increased greatly. This has created the need for computer security log management, which is the process for generating, transmitting, storing, analyzing, and disposing of computer security log data.

Security software is a major source of computer security log data. Common types of network-based and host-based security software include the following:

• Antimalware Software. The most common form of antimalware software is antivirus software, which typically records all instances of detected malware, file and system disinfection attempts, and file quarantines.

• Intrusion detection and intrusion prevention systems record detailed information on suspicious behavior and detected attacks, as well as any actions intrusion prevention systems performed to stop malicious activity in progress. Some intrusion detection systems, such as file integrity checking software, run periodically instead of continuously, so they generate log entries in batches instead of on an ongoing basis.

• VPN systems typically log successful and failed login attempts, as well as the dates and times each user connected and disconnected, and the amount of data sent and received in each user session.

• Web proxies can be used to restrict Web access and to add a layer of protection between Web clients and Web servers. Web proxies often keep a record of all URLs accessed through them.

• Vulnerability management software, which includes patch management software and vulnerability assessment software, typically logs the patch installation history and vulnerability status of each host. Vulnerability management software may also record additional information about hosts’ configurations.

• Authentication servers, including directory servers and single sign-on servers, typically log each authentication attempt, including its origin, username, success or failure, and date and time.

• Routers that block traffic are usually configured to log only the most basic characteristics of blocked activity.

• Firewalls can track the state of network traffic and perform content inspection. Firewalls tend to have more complex policies and generate more detailed logs of activity than routers.

• Network Quarantine Servers. Some organizations check each remote host’s security posture before allowing it to join the network. Network quarantine servers log information about the status of checks, including which hosts were quarantined and for what reasons.

The most common types of security-related OS data are as follows:

• System events are operational actions performed by OS components, such as shutting down the system or starting a service. Typically, failed events and the most significant successful events are logged, but many OSs permit administrators to specify which types of events will be logged.

• Audit records contain security event information such as successful and failed authentication attempts, file accesses, security policy changes, account changes and use of privileges.

OS logs are most beneficial for identifying or investigating suspicious activity involving a particular host. After suspicious activity is identified by security software, OS logs are often consulted to get more information on the activity.

Some applications generate their own log files, while others use the logging of the OS on which they are installed. The following lists some of the most commonly logged types of information and the potential benefits of each:

• Client requests and server responses, which can be very helpful in reconstructing sequences of events and determining their apparent outcome.

• Account information such as successful and failed authentication attempts.

• Usage information such as the number of transactions occurring in a certain period (e.g., minute, hour) and the size of transactions (e.g., e-mail message size, file transfer size).

• Significant operational actions such as application startup and shutdown, application failures, and major application configuration changes.

In a typical organization, many hosts’ OSs, security software, and other applications generate and store logs. This complicates log management in the following ways:

• Many Log Sources. Logs are located on many hosts throughout the organization, necessitating log management to be performed throughout the organization.

• Inconsistent Log Content. Each log source records certain pieces of information in its log entries, such as host IP addresses and usernames. For efficiency, log sources often record only the pieces of information that they consider most important. This can make it difficult to link events recorded by different log sources because they may not have any common values recorded.

• Inconsistent Timestamps. Each host that generates logs typically references its internal clock when setting a timestamp for each log entry. If a host’s clock is inaccurate, the timestamps in its logs will also be inaccurate. This can make analysis of logs more difficult, particularly when logs from multiple hosts are being analyzed.

Inconsistent Log Formats. Many of the log source types use different formats for their logs, such as comma-separated or tab-separated text files, databases, syslog, Simple Network Management Protocol (SNMP), Extensible Markup Language (XML), and binary files. Some logs are designed for humans to read, while others are not; some logs use standard formats, while others use proprietary formats.

To facilitate analysis of logs, organizations often need to implement automated methods of converting logs with different content and formats to a single standard format with consistent data field representations.

Many logs have a maximum size. When the size limit is reached, the log might overwrite old data with new data or stop logging altogether, both of which would cause a loss of log data availability.

To meet data retention requirements, organizations might need to keep copies of log files for a longer period of time than the original log sources can support, which necessitates establishing log archival processes. Because of the volume of logs, it might be appropriate in some cases to reduce the logs by filtering out log entries that do not need to be archived.

Despite the many challenges an organization faces in log management, there are a few key practices an organization can follow to avoid and even solve many of these obstacles it confronts. The following four measures give a brief explanation of these solutions:

• An organization should define its requirements and goals for performing logging and monitoring logs. The organization can then prioritize its goals based on balancing the organization’s reduction of risk with the time and resources needed to perform log management functions.

• Establish policies and procedures for log management. Policies and procedures are beneficial because they ensure a consistent approach throughout the organization. Periodic audits are one way to confirm that logging standards and guidelines are being followed throughout the organization.

• Create and maintain a secure log management infrastructure. It is critical to create an infrastructure robust enough to handle not only expected volumes of log data, but also peak volumes during extreme situations (e.g., widespread malware incident, penetration testing, vulnerability scans).

• Provide adequate support for all staff with log management responsibilities. organizations should ensure that they provide the necessary training to relevant staff regarding their log management responsibilities as well as skill instruction for the needed resources to support log management.

A log management infrastructure consists of the hardware, software, networks, and media used to generate, transmit, store, analyze, and dispose of log data.

A log management infrastructure typically comprises the following three tiers:

• Log Generation. The first tier contains the hosts that generate the log data. Some hosts run logging client applications or services that make their log data available through networks to log servers in the second tier. Other hosts make their logs available through other means, such as allowing the servers to authenticate to them and retrieve copies of the log files.

• Log Analysis and Storage. The second tier is composed of one or more log servers that receive log data or copies of log data from the hosts in the first tier. The data is transferred to the servers either in a real-time or near-real-time manner, or in occasional batches based on a schedule or the amount of log data waiting to be transferred. Servers that receive log data from multiple log generators are sometimes called collectors or aggregators. Log data may be stored on the log servers themselves or on separate database servers.

• Log Monitoring. The third tier contains consoles that may be used to monitor and review log data and the results of automated analysis. Log monitoring consoles can also be used to generate reports. In some log management infrastructures, consoles can also be used to provide management for the log servers and clients.

A motivation for using a separate logging network is protecting log data on the organization’s regular networks from eavesdropping. If a separate logging network is not used, logging communications on the regular network could be protected using additional security controls, such as data encryption. Another benefit of having a separate logging network is protecting those components that are only used for log management from attacks.

The following items describe common log management infrastructure functions:

• Log parsing is extracting data from a log so that the parsed values can be used as input for another logging process.

• Event filtering is the suppression of log entries from analysis, reporting, or long-term storage because their characteristics indicate that they are unlikely to contain information of interest.

• In event aggregation, similar entries are consolidated into a single entry containing a count of the number of occurrences of the event.

• Log rotation is closing a log file and opening a new log file when the first file is considered to be complete. Log rotation is typically performed according to a schedule (e.g., hourly, daily, weekly) or when a log file reaches a certain size. The primary benefits of log rotation are preserving log entries and keeping the size of log files manageable.

• Log archival is retaining logs for an extended period of time, typically on removable media, a storage area network (SAN), or a specialized log archival appliance or server. Logs often need to be preserved to meet legal or regulatory requirements.

• Log compression is storing a log file in a way that reduces the amount of storage space needed for the file without altering the meaning of its contents.

• Log reduction is removing unneeded entries from a log to create a new log that is smaller. A similar process is event reduction, which removes unneeded data fields from all log entries.

• Log conversion is parsing a log in one format and storing its entries in a second format. For example, conversion could take data from a log stored in a database and save it in an XML format in a text file.

• In log normalization, each log data field is converted to a particular data representation and categorized consistently. One of the most common uses of normalization is storing dates and times in a single format. Normalizing the data makes analysis and reporting much easier when multiple log formats are in use.

• Log file integrity checking involves calculating a message digest for each file and storing the message digest securely to ensure that changes to archived logs are detected.

• Event correlation is finding relationships between two or more log entries. The most common form of event correlation is rule-based correlation, which matches multiple log entries from a single source or multiple sources based on logged values, such as timestamps, IP addresses, and event types.

• Log viewing is displaying log entries in a human-readable format.

• Log reporting is displaying the results of log analysis. Log reporting is often performed to summarize significant activity over a particular period of time or to record detailed information related to a particular event or series of events.

• Log clearing is removing all entries from a log that precede a certain date and time.

Many log sources either use syslog as their native logging format or offer features that allow their log formats to be converted to syslog format.

Syslog assigns a priority to each message based on the importance of the following two attributes:

• Message type, known as a facility. Examples of facilities include kernel messages, mail system messages, authorization messages, printer messages, and audit messages.

• Severity. Each log message has a severity value assigned, from 0 (emergency) to 7 (debug).

Syslog can be configured to handle log entries differently based on each message’s facility and severity. For example, it could forward severity 0 kernel messages to a centralized server for further review, and simply record all severity 7 messages without forwarding them.

Syslog is intended to be very simple, and each syslog message has only three parts. The first part specifies the facility and severity as numerical values. The second part of the message contains a timestamp and the hostname or IP address of the source of the log. The third part is the actual log message content.

Some current syslog implementations offer more robust filtering capabilities, such as handling messages differently based on the host or program that generated a message, or a regular expression matching content in the body of a message.

Some syslog implementations can initiate actions when certain events are detected. Examples of actions include sending SNMP traps, alerting administrators through pages or e-mails, and launching a separate program or script.

SIEM products have one or more log servers that perform log analysis, and one or more database servers that store the logs. Most SIEM products support two ways of collecting logs from log generators:

• Agentless. The SIEM server receives data from the individual log generating hosts without needing to have any special software installed on those hosts. Some servers pull logs from the hosts. In other cases, the hosts push their logs to the server. The server then performs event filtering and aggregation and log normalization and analysis on the collected logs.

• Agent-Based. An agent program is installed on the log generating host to perform event filtering and aggregation and log normalization for a particular type of log, then transmit the normalized log data to an SIEM server.

The primary advantage of the agentless approach is that agents do not need to be installed, configured, and maintained on each logging host. The primary disadvantage is the lack of filtering and aggregation at the individual host level, which can cause significantly larger amounts of data to be transferred over networks and increase the amount of time it takes to filter and analyze the logs.

SIEM products usually include support for several dozen types of log sources. This significantly improves the normalization, analysis, and correlation of log data over that performed by software with a less granular understanding of specific log sources and formats.

SIEM products usually include several features to help log monitoring staff, such as the following:

• Graphical user interfaces (GUI) that are specifically designed to assist analysts in identifying potential problems and reviewing all available data related to each problem.

• A security knowledge base, with information on known vulnerabilities, the likely meaning of certain log messages, and other technical data.

• Incident tracking and reporting capabilities.

• Asset information storage and correlation (e.g., giving higher priority to an attack that targets a vulnerable OS or a more important host).

The major operational processes for log management, which are as follows:

• Configure the log sources, including log generation, storage, and security • Perform analysis of log data • Initiate appropriate responses to identified events • Manage the longterm storage of log data.

System-level administrators need to configure log sources so that they capture the necessary information in the desired format and locations, as well as retain the information for the appropriate period of time.

Assuming that a log source offers configuration options, it is generally prudent to be conservative when selecting initial logging settings. A single setting could cause an enormous number of log entries to be recorded, or far too much information to be logged for each event.

Software vendors and other parties may also have information available on the logging capabilities and typical effects of various logging settings, which can be very helpful in determining an initial configuration.

The storage options for log entries are as follows:

• Not stored. Entries that are determined to be of little or no value to the organization, such as debugging messages that can only be understood by the software vendor, or error messages that do not log any details of the activity, generally do not need to be stored.

• System level only. Entries that might be of some value or interest to the system-level administrator, but are not sufficiently important to be sent to the log management infrastructure, should be stored on the system.

• Both system level and infrastructure level. Entries deemed to be of particular interest should be retained on the system and also transmitted to the log management infrastructure.

• Infrastructure level only. If logs are stored on the infrastructure servers, generally it is preferable to also store them at the system level. However, this is not always possible, such as systems with little capacity for logs or log sources not capable of storing logs locally.

Security considerations for securing logs on systems, in storage, and in transit include the following:

• Limit access to log files. Users should not be able to rename, delete, or perform other filelevel operations on log files.

• Avoid recording unneeded sensitive data. When feasible, logging should be configured not to record information that is not required and would present a substantial risk if accessed by unauthorized parties.

• Protect archived log files. This could include creating and securing message digests for the files, encrypting log files, and providing adequate physical protection for archival media.

• Secure the processes that generate the log entries. Unauthorized parties should not be able to manipulate log source processes, executable files, configuration files.

The key to performing log analysis is understanding the typical activity associated with each system. Although some log entries are very easy to understand, many are not. The primary reasons for this are as follows:

• Need for Context. The meaning of an entry often depends upon the context surrounding it. Administrators need to determine how this context is defined, such as through additional log entries in one or more logs, or through non-log sources (e.g., configuration management records). Context is needed to validate unreliable log entries, such as security software that often generates false positives when looking for malicious activity. Infrastructure administrators should reach out to system-level administrators as needed to help provide context for entries.

• Unclear Messages. A log entry might contain a cryptic message or code that is meaningful to the software vendor but not to the administrator reviewing the entry. Such entries might necessitate discussions with the software vendor to determine their meanings. Using SIEM software to analyze logs typically reduces the number of unclear messages because the SIEM software often has detailed knowledge of software vendors’ logging practices. However, even SIEM software cannot understand every message, such as new message types associated with a just-released update to a product.

The most effective way to gain a solid understanding of log data is to review and analyze portions of it regularly (e.g., every day). The goal is to eventually gain an understanding of the baseline of typical log entries, likely encompassing the vast majority of log entries on the system.

Daily log reviews should include those entries that have been deemed most likely to be important, as well as some of the entries that are not yet fully understood.

Another motivation for understanding the log entries is so that the analysis process can be automated as much as possible. By determining which types of log entries are of interest and which are not, administrators can configure automated filtering of the log entries.

The filtering should be configured so that it presents administrators with a reasonable number of entries for manual analysis. One effective technique is to have two reports: one for the entries thought to be most important, and another for entries that are not yet fully understood.

Organizations should consider assigning their own priorities to log entries based on a combination of factors, including the following:

• Entry type (message class CRITICAL)

• Newness of the entry type (i.e., has this type of entry appeared in the logs before?) • Log source

• Source address on a blacklist, destination address of a critical system

• Time of day or day of the week

• Frequency of the entry (e.g., x times in y seconds).

To perform effective reviews and analysis, system-level and infrastructure administrators should have solid understanding of each of the following from training or hands-on experience:

• The organization’s policies regarding acceptable use, so that administrators can recognize violations of the policies.

• The security software used by their hosts, including the types of security-related events that each program can detect and the general detection profile of each program (e.g., known false positives).

• The operating systems and major applications (e.g., e-mail, Web) used by their hosts, particularly each OS’s and major application’s security and logging and characteristics.

• The characteristics of common attack techniques, especially how the use of these techniques might be recorded on each system.

• The software needed to perform analysis, such as log viewers, log reduction scripts, and database query tools.

Infrastructure and system-level administrators need to provide additional types of support for logging operations. They should perform the following actions regularly:

• Monitor the logging status of all log sources to ensure that each source is enabled, configured properly, and functioning as expected.

• Monitor log rotation and archival processes to ensure that logs are archived and cleared correctly and that old logs are destroyed once they are no longer needed.

• Check for upgrades and patches for logging software; acquire, test, and deploy the updates.

• Ensure that each system’s clock is synched to a common time source so that its timestamps will match those generated by other systems.

• Reconfigure logging as needed based on factors such as policy changes, audit findings, technology changes, and new security needs.

• Document anomalies detected in log settings, configurations, and processes. Such anomalies might indicate malicious activity, deviations from policy and procedures, and flaws in logging mechanisms.