(Notes prepared by Multazim A. R. Deshmukh)

(Any corrections & suggestions are welcome)

Part I - Syslog

1.1) Facility Levels

1.2) Severity levels

1.3) Common Linux log files name and usage

1.4) syslog variants

Part II - syslog-ng

2.1) Features:

2.1.1) Client Side Features:

2.1.2) Server Side Features:

2.2) Installation:

2.3) Overview

2.4) Modes of Operation:

2.4.1) Client mode

2.4.2) Relay mode

2.4.3) Server mode

2.5) Example configuration file

2.6) the configuration syntax in detail

2.6.1) Global options

2.6.2) Syslog-ng modules

2.6.2.1) Loading modules

2.6.3) Managing complex syslog-ng configurations

2.6.3.1) including configuration files:

2.6.4) Sources

2.6.4.1) Collecting process accounting logs on Linux:

2.6.4.2) Receiving messages from external applications

2.6.5) Destinations

2.6.6) Creating Filters for Messages

2.6.7) Log Paths

2.6.8) Managing incoming and outgoing messages with flow-control

2.6.9) Have syslog-ng reload the configuration file

2.6.10) log to another file

2.7) Configuring as a loghost

2.8) Client side configurations

2.9) Improve Performance

2.9.1) Write every so often

2.9.2) Avoid redundant processing and disk space

2.10) PostgreSQL Destination

2.11) ISO 8601 timestamps

2.12) Log Levels

2.13) Macros and Variables

2.14) Troubleshooting syslog-ng:

2.15) Possible causes of losing log messages:

2.16) Procedure – Creating syslog-ng core files

2.17) Collecting debugging information with strace, truss, or tusc

2.18) Apache and MySQL Logging with Syslog-ng

2.19) build a Secure Logging Server with syslog-ng using SSL/TLS

2.20) syslog-ng + logrotate

2.21) Reference

Part III - rsyslog

3.1) Features:

3.2) Installation

3.3) Configuration Structure

3.3.1) Modules

3.3.2) Configuration Directives

3.3.3) Rule line

3.3.4) A Sample Configuration

3.3.5) Templates

3.4) Central Logging Server

3.5) Encrypting Syslog Traffic with TLS (SSL)

3.6) Unreliability of plain tcp syslog

3.7) TLS secured syslog via RELP

3.8) References:

Part IV. WEB GUIS to MANAGE Logging in Linux

Part I - Syslog

Syslog is a standard for computer data logging. It separates the software that generates messages from the system that stores them and the software that reports and analyzes them. There will be a syslog-compatible service running on your server that implements the standard. Some applications, such as the apache web server, do not use the syslog service and instead manage their own logging. If applications implement their own logging their configuration files will say where to find the log files.

Syslog can be used for computer system management and security auditing as well as generalized informational, analysis, and debugging messages. It is supported by a wide variety of devices (like printers and routers) and receivers across multiple platforms. Because of this, syslog can be used to integrate log data from many different types of systems into a central repository.

Messages are labeled with a facility code (auth, authpriv, daemon, cron, ftp, lpr, kern, mail, news, syslog, user, uucp, local0 to local7) indicating the type of software that generated the messages, and are assigned a severity (Emergency, Alert, Critical, Error, Warning, Notice, Info, Debug).

Implementations are available for many operating systems. Specific configuration may permit directing messages to various devices (console), files (/var/log/) or remote syslog servers. Most implementations also provide a command line utility, often called logger, that can send messages to the syslog. Some implementations permit the filtering and display of syslog messages.

Syslog is now standardized within the Syslog working group of the IETF.

Syslog was developed in the 1980s by Eric Allman as part of the Sendmail project, and was initially used solely for Sendmail. It proved so valuable that other applications began using it as well. Syslog has since become the standard logging solution on Unix and Unix-like systems; there have also been a variety of syslog implementations on other operating systems and is commonly found in network devices such as routers.

1.1) Facility Levels

A facility level is used to specify what type of program is logging the message. This lets the configuration file specify that messages from different facilities will be handled differently.^[4] The list of facilities available: (defined by RFC 3164)

Facility Number	Keyword	Facility Description
0	kern	kernel messages
1	user	user-level messages
2	mail	mail system
3	daemon	system daemons
4	auth	security/authorization messages
5	syslog	messages generated internally by syslogd
6	lpr	line printer subsystem
7	news	network news subsystem
8	uucp	UUCP subsystem
9		clock daemon
10	authpriv	security/authorization messages
11	ftp	FTP daemon
12	-	NTP subsystem
13	-	log audit
14	-	log alert
15	cron	clock daemon
16	local0	local use 0 (local0)
17	local1	local use 1 (local1)
18	local2	local use 2 (local2)
19	local3	local use 3 (local3)
20	local4	local use 4 (local4)
21	local5	local use 5 (local5)
22	local6	local use 6 (local6)
23	local7	local use 7 (local7)

1.2) Severity levels

RFC 5424 defines eight severity levels:

Code	Severity	Keyword	Description	General Description
0	Emergency	emerg (panic)	System is unusable.	A "panic" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call.
1	Alert	alert	Action must be taken immediately.	Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a primary ISP connection.
2	Critical	crit	Critical conditions.	Should be corrected immediately, but indicates failure in a secondary system, an example is a loss of a backup ISP connection.
3	Error	err (error)	Error conditions.	Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time.
4	Warning	warning (warn)	Warning conditions.	Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full - each item must be resolved within a given time.
5	Notice	notice	Normal but significant condition.	Events that are unusual but not error conditions - might be summarized in an email to developers or admins to spot potential problems - no immediate action required.
6	Informational	info	Informational messages.	Normal operational messages - may be harvested for reporting, measuring throughput, etc. - no action required.
7	Debug	debug	Debug-level messages.	Info useful to developers for debugging the application, not useful during operations.

A common mnemonic used to remember the syslog levels from bottom to top is: "Do I Notice When Evenings Come around Early".

In Linux, Almost all logfiles are located under /var/log directory (and subdirectory). You can change to this directory using cd command but you need to be the root user. You can use less, more, cat or tail command to see the logs.

1.3) Common Linux log files name and usage

/var/log/messages: General message and system related stuff
/var/log/auth.log: Authenication logs
/var/log/kern.log: Kernel logs
/var/log/btmp : Invalid Login Attempts
/var/log/cron.log: Crond logs (cron job)
/var/log/dmesg : Kernel boot messages
/var/log/maillog: Mail server logs
/var/log/qmail/ : Qmail log directory (more files inside this directory)
/var/log/httpd/: Apache access and error logs directory
/var/log/lastlog: user last login information
/var/log/boot.log : System boot log
/var/log/mysqld.log: MySQL database server log file
/var/log/secure: Authentication log
/var/log/utmp or /var/log/wtmp : Login records file
/var/log/yum.log: Yum log files

For more info refer: http://www.thegeekstuff.com/2011/08/linux-var-log-files/
In short /var/log is the location where you should find all Linux logs file. However some applications such as httpd have a directory within /var/log/ for their own log files. You can rotate log file using logrotate software and monitor logs files using logwatch software.

1.4) Syslog variants

There are several implementations of the syslog standard, including syslog-ng, sysklogd and rsyslogd. These applications are examples of the message logging software used by Linux, and their configuration files can be found in the /etc directory.
Many systems use sysklogd to provide syslog functionality. Its configuration file can be found at /etc/syslog.conf.
Some other distributions use rsyslogd. Depending on the version, its configuration files will be in either /etc/rsyslog.conf or /etc/rsyslog.d/ (the default configuration file in that directory is /etc/rsyslog.d/50-default.conf).
Still other distributions might use syslog-ng, which stores its config file at /etc/syslog-ng/syslog-ng.conf. Once you find your syslog config file you can search for the facility entry to see where those entries are logged.

Part II - syslog-ng

syslog-ng is an open source implementation of the Syslog protocol for Unix and Unix-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport. As of today syslog-ng is developed by Balabit IT Security Ltd. It has two editions with a common codebase. The first is called syslog-ng Open Source Edition (OSE) with the license LGPL. The second is called Premium Edition (PE) and has additional plugins (modules) under proprietary license.

2.1) Features:

Syslog-ng provides a number of features other than transporting syslog messages and storing them to plain text log files:

2.1.1) Client Side Features:

Secure logging using SSL/TLS

Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng uses the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows the mutual authentication of the host and the server using X.509 certificates.

Reliable log transfer

The syslog-ng application enables you to send the log messages of your hosts to remote servers. The logs of different servers can be collected and stored centrally on dedicated log servers. Transferring log messages using the TCP protocol ensures that no messages are lost.

Support for standard syslog protocols

The syslog-ng application can send and receive traditional syslog messages (as described in RFC3164) and also messages conforming to the latest IETF-syslog protocol standard (as described in RFC5424) using the UDP, TCP, or TLS protocol over IPv4 and IPv6 networks.

Collecting local messages

The syslog-ng application can collect messages of local applications and processes using UNIX domain sockets, named pipes, and plain text files. Messages can be received also from the standard output (stdout).

2.1.2) Server Side Features:

Flexible message filtering and sorting

The syslog-ng application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. Directories, files, and database tables can be created dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.

Message parsing and rewriting

The syslog-ng application can segment log messages to named fields or columns. These fields can be used like macros in filenames and database tables. It is also possible to modify the values of these fields, for example, to add missing values.

Message classification

The syslog-ng application can compare the contents of the log messages to a database of predefined message patterns. By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.

Handle extreme load

The syslog-ng application is optimized for performance, and can handle enormous amount of messages. Depending on its exact configuration, it has been known to process over 500,000 messages per second real-time, and over 24 GB raw logs per hour on average server hardware.

Direct database access

Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The syslog-ng application supports the following databases: MySQL, Microsoft SQL (MSSQL), Oracle, PostgreSQL, and SQLite.

IPv4 and IPv6 support

The syslog-ng application can operate in both IPv4 and IPv6 network environments; it can receive messages from and send messages to both types of networks.

MongoDB support

Using MongoDB instead of SQL is faster and allows better representation of log data.

AMQP destination

The destination allows one to publish log messages to an AMQP-capable messaging server.

2.2) Installation:

See list of Supported Platforms in below link:

http://www.balabit.com/network-security/syslog-ng/central-syslog-server/specifications/

a) Compiling / installing from source code:

To compile syslog-ng Open Source Edition (OSE) from the source code, complete the following steps.

1. Download the latest version of syslog-ng OSE from https://www.balabit.com/downloads/files?path=/syslog-ng/sources/. The source code is available as a tar.gz archive file.

2. Download the latest version of the EventLog library available at https://www.balabit.com/downloads/files/eventlog/0.2/.

3. Install the following packages that are required to compile syslog-ng. These packages are available for most UNIX/Linux systems. Alternatively, you can also download the sources and compile them.

o the gcc C compiler (at least version 2.7.2),

o the GNU flex lexical analyser generator, available at http://flex.sourceforge.net/;

o the bison parser generator, available at http://ftp.gnu.org/gnu/bison/;

o and the development files of the glib library, available at http://freshmeat.net/projects/glib/.

4. If you want to use the spoof-source function of syslog-ng, install the development files of the libnet library, available at http://libnet.sourceforge.net.

5. If you want to use the /etc/hosts.deny and /etc/hosts.allow for TCP access, install the development files of the libwrap (also called TCP-wrappers) library, available at ftp://ftp.porcupine.org/pub/security/index.html.

6. Uncompress the eventlog archive using the

7.              $ tar xvfz eventlog-x.x.x.x.tar.gz

Or the

$ gunzip -c eventlog-x.x.x.x.tar.gz | tar xvf -

Command. A new directory containing the source code of eventlog will be created.

8. By default, eventlog creates a file used by the syslog-ng configure script in the /usr/local/lib/pkgconfig directory. Issue the following command to add this directory to your PKG_CONFIG_PATH:

9.              PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH

10. Enter the new directory and issue the following commands:

11.           $ ./configure

12.           $ make

13.           $ make install

14. Uncompress the syslog-ng archive using the

15.           tar xvfz syslog-ng-x.xx.tar.gz

or the

unzip -c syslog-ng-x.xx.tar.gz | tar xvf -

command. A new directory containing the source code of syslog-ng will be created.

16. Enter the new directory and issue the following commands:

17.           $ ./configure

18.           $ make

19.           $ make install

These commands will build syslog-ng using its default options.

20. If needed, use the following options to change how syslog-ng is compiled using the following command syntax:

21.           $ ./configure --compile-time-option-name

For an up-to-date list of options run:

$ ./configure --help

b) Installing from Binaries: For list of 3rd party binaries of syslog-ng, refer below link

http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/downloads/3rd-party

For CentOS or RHEL, you have to install epel repo and then do ‘yum install sysog-ng’. That’s it.

2.3) Overview

Syslog-ng takes incoming log messages from defined 'sources' and forwards them to the appropriate destinations, based on powerful filter directives. The syslog-ng application can receive messages from files, remote hosts, and other sources.
Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.
Sources and destinations are independent objects; log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations; messages arriving from a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.
Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.
Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.
In a typical simple set-up, syslog-ng will read messages from three sources:

1. the default /dev/log device, where most logs are sent

2. syslog-ng "internal" log messages

3. /proc/kmsg kernel messages

Sources are defined using the "source" directive. These incoming messages are then filtered according to defined filters ("filter" keyword), i.e. according to originating program or log level, and sent to the appropriate "destination". Destinations include log files (e.g. /var/log/messages.log), printing messages on a console and remote servers. The pivotal function is log. This function defines which filters should be applied to a certain source, and where the resulting messages should be sent to.

2.4) Modes of Operation:

The syslog-ng Open Source Edition application has three typical operation scenarios: Client, Server, and Relay.

2.4.1) Client mode

Figure 2.4.1. Client-mode operation

In client mode, syslog-ng collects the local logs generated by the host and forwards them through a network connection to the central syslog-ng server or to a relay. Clients often also log the messages locally into files.

2.4.2) Relay mode

Figure 2.4.2. Relay-mode operation

In relay mode, syslog-ng receives logs through the network from syslog-ng clients and forwards them to the central syslog-ng server using a network connection. Relays also log the messages from the relay host into a local file, or forward these messages to the central syslog-ng server.

2.4.3) Server mode

Figure 2.4.3. Server-mode operation

In server mode, syslog-ng acts as a central log-collecting server. It receives messages from syslog-ng clients and relays over the network, and stores them locally in files, or passes them to other applications, for example log analyzers.

2.5) Example configuration file

For a quick start, here there is a classic configuration file. This example includes logging to a remote server destination.
******************************************************************************************

# Global Options

#@version:3.2

options {

# sync (0);

flush_lines (0);

time_reopen (10);

dns_cache_hosts(/etc/hosts);

dns_cache_expire(87600);

dns_cache_hosts(/etc/hosts);

# chain_hostnames(0);

# long_hostnames (off);

use_dns (persist_only);

use_fqdn (no);

create_dirs(yes);

owner(root);

group(root);

perm(0640);

dir_owner(root);

dir_group(root);

dir_perm(0740);

keep_hostname (yes);

#stats (3600);

stats_freq(0);

time_reap(360);

ts_format(iso);

};

#Define network log sources

#source s_net { syslog(ip(127.0.0.1) port(1000) transport(tcp)); };

#source s_net { syslog(ip(127.0.0.1) port(514) transport(udp)); };

#source s_net { udp(ip(192.168.2.9) port(514));};

source s_net { tcp(ip(192.168.2.9) port(1514)); };

#Define Local log sources

source s_sys {

file ("/proc/kmsg" log_prefix("kernel: "));

unix-stream ("/dev/log");

internal();

# udp(ip(0.0.0.0) port(514));

};

# source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); };

# source s_pipe { pipe("/dev/pipe" pad_size(2048)); };

# Define destinations for local logs

destination d_cons { file("/dev/console"); };

destination d_mesg { file("/var/log/messages"); };

destination d_auth { file("/var/log/secure"); };

destination d_mail { file("/var/log/maillog" flush_lines(10)); };

destination d_spol { file("/var/log/spooler"); };

destination d_boot { file("/var/log/boot.log"); };

destination d_cron { file("/var/log/cron"); };

destination d_kern { file("/var/log/kern"); };

destination d_mlal { usertty("*"); };

# Define destinations for remote logs

destination d_remote_alllogs { file ( "/var/log/syslog/remote/$FULLHOST/$YEAR/$MONTH/$DAY/$FULLHOST-$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) template("$DATE $FULLHOST $PROGRAM $TAG [$FACILITY.$LEVEL] $MESSAGE\n")); };

#destination d_net_everything { file("/var/log/syslog/remote/$YEAR.$MONTH.$DAY/everything.log" template("$HOUR:$MIN:$SEC [$LEVEL] [$FACILITY] [$PROGRAM] $MSG\n") template_escape(no)); };

destination d_net_auth { file("/var/log/syslog/remote/$HOST/auth.log"); };

destination d_net_cron { file("/var/log/syslog/remote/$HOST/cron.log"); };

destination d_net_daemon { file("/var/log/syslog/remote/$HOST/daemon.log"); };

destination d_net_kern { file("/var/log/syslog/remote/$HOST/kern.log"); };

destination d_net_lpr { file("/var/log/syslog/remote/$HOST/lpr.log"); };

destination d_net_mail { file("/var/log/syslog/remote/$HOST/mail.log"); };

destination d_net_syslog { file("/var/log/syslog/remote/$HOST/syslog.log"); };

destination d_net_user { file("/var/log/syslog/remote/$HOST/user.log"); };

destination d_net_uucp { file("/var/log/syslog/remote/$HOST/uucp.log"); };

destination d_net_debug { file("/var/log/syslog/remote/$HOST/debug"); };

destination d_net_error { file("/var/log/syslog/remote/$HOST/error"); };

destination d_net_messages { file("/var/log/syslog/remote/$HOST/messages"); };

destination d_net_mailinfo { file("/var/log/syslog/remote/$HOST/mail/mail.info"); };

destination d_net_mailwarn { file("/var/log/syslog/remote/$HOST/mail/mail.warn"); };

destination d_net_mailerr { file("/var/log/syslog/remote/$HOST/mail/mail.err"); };

destination d_net_error { file("/var/log/syslog/remote/$HOST/error"); };

destination d_net_ppp { file("/var/log/syslog/remote/$HOST/ppp.log"); };

destination d_newscrit { file("/var/log/syslog/remote/$HOST/news/news.crit"); };

destination d_newserr { file("/var/log/syslog/remote/$HOST/news/news.err"); };

destination d_newsnotice { file("/var/log/syslog/remote/$HOST/news/news.notice"); };

destination d_console_all { file("/dev/tty12"); };

#Define Filters

#filter f_dbg { level(debug); };

filter f_info{ level(info); };

filter f_notice{ level(notice); };

filter f_warn { level(warn); };

filter f_err { level(err); };

filter f_crit { level(crit..emerg); };

filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };

filter f_error { level(err .. emerg) ; };

#filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news);};

filter f_auth { facility(auth, authpriv) and not filter(f_debug); };

filter f_cron { facility(cron) and not filter(f_debug); };

filter f_daemon { facility(daemon) and not filter(f_debug); };

filter f_kern { facility(kern) and not filter(f_debug); };

filter f_lpr { facility(lpr) and not filter(f_debug);};

#filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); };

filter f_mail { facility(mail) and not filter(f_debug); };

filter f_news { facility(news) and not filter(f_debug); };

filter f_syslog { program(syslog-ng); };

filter f_user { facility(user) and not filter(f_debug); };

filter f_uucp { facility(uucp) and not filter(f_debug); };

filter f_ppp { facility(local2) and not filter(f_debug); };

filter f_console { level(warn..emerg); };

filter f_default { level(info..emerg) and not (facility(mail)or facility(authpriv) or facility(cron)); };

filter f_emergency { level(emerg); };

#filter f_news { facility(uucp) or(facility(news) and level(crit..emerg)); };

filter f_boot { facility(local7); };

# Define logging for local logs

#log { source(s_sys); filter(f_kernel); destination(d_cons); };

log { source(s_sys); filter(f_kern); destination(d_kern); };

log { source(s_sys); filter(f_default); destination(d_mesg); };

log { source(s_sys); filter(f_auth); destination(d_auth); };

log { source(s_sys); filter(f_mail); destination(d_mail); };

log { source(s_sys); filter(f_emergency); destination(d_mlal); };

log { source(s_sys); filter(f_news); destination(d_spol); };

log { source(s_sys); filter(f_boot); destination(d_boot); };

log { source(s_sys); filter(f_cron); destination(d_cron); };

#Define Logging for Network logs

log { source(s_net); destination(d_remote_alllogs) ; };

#log { source(s_net); destination(d_net_everything); };

log { source(s_net); filter(f_auth); destination(d_net_auth); };

log { source(s_net); filter(f_user); destination(d_net_user); };

log { source(s_net); filter(f_auth); destination(d_net_auth); };

log { source(s_net); filter(f_user); destination(d_net_user); };

log { source(s_net); filter(f_cron); destination(d_net_cron); };

log { source(s_net); filter(f_daemon); destination(d_net_daemon); };

log { source(s_net); filter(f_kern); destination(d_net_kern); };

log { source(s_net); filter(f_lpr); destination(d_net_lpr); };

log { source(s_net); filter(f_mail); destination(d_net_mail); };

log { source(s_net); filter(f_uucp); destination(d_net_uucp); };

log { source(s_net); filter(f_mail); filter(f_info); destination(d_net_mailinfo); };

log { source(s_net); filter(f_mail); filter(f_warn); destination(d_net_mailwarn); };

log { source(s_net); filter(f_mail); filter(f_err); destination(d_net_mailerr); };

log { source(s_net); filter(f_debug); destination(d_net_debug); };

log { source(s_net); filter(f_news); filter(f_crit); destination(d_newscrit); };

log { source(s_net); filter(f_news); filter(f_err); destination(d_newserr); };

log { source(s_net); filter(f_news); filter(f_notice); destination(d_newsnotice); };

#log { source(s_net); filter(f_messages); destination(d_net_messages); };

log { source(s_net); destination (d_net_messages); };

log { source(s_net); filter(f_ppp); destination(d_net_ppp); };

log { source(s_net); filter(f_error); destination(d_net_error); };

log { source(s_net); filter(f_syslog); destination(d_net_syslog); };

log { source(s_net); destination(d_console_all); };

# vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:

******************************************************************************************

2.6) the configuration syntax in detail

Every syslog-ng configuration file must begin with a line containing the version information of syslog-ng. For syslog-ng version 3.3, this line looks like:

@version: 3.3

Versioning the configuration file was introduced in syslog-ng 3.0. If the configuration file does not contain the version information, syslog-ng assumes that the file is for syslog-ng version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later will correctly operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.
Some options are global options, or can be set globally, for example, whether syslog-ng OSE should use DNS resolution to resolve IP addresses.

options { use_dns(no); };

2.6.1) Global options

Global options are detailed at below link:

http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html-single/index.html#chapter-global-options.

Note:

· All identifiers, attributes, and any other strings used in the syslog-ng configuration file are case sensitive.

· To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

Tip: Before activating a new configuration, check that your configuration file is syntactically correct using the “syslog-ng --syntax-only” command.

To activate the configuration, reload the configuration of syslog-ng using “/etc/init.d/syslog-ng reload “command or you can run # systemctl reload syslog-ng

Note: For RHEL 6.x, while reloading syslog-ng , you might get below error.

Plugin module not found in 'module-path'; module-path='/lib64/syslog-ng', module='afsql'

Just do “yum install syslog-ng-libdbi” and then reload syslog-ng

2.6.2) Syslog-ng modules

Starting with syslog-ng Open Source Edition version 3.3, syslog-ng OSE became modular to increase its flexibility and also to simplify the development of additional modules. Most of the functionality of syslog-ng OSE has been moved to separate modules. That way it becomes also possible to finetune the resource requirements of syslog-ng OSE for example, by loading only the modules that are actually used in the configuration, or simply omitting modules that are not used but require large amount of memory.
Each module contains one or more plugins, which add some functionality to syslog-ng OSE, for example, a destination or a source driver.

§ To display the list of available modules, execute the syslog-ng --version command.

§ To the description of the available modules, execute the syslog-ng --module-registry command.

§ To customize which modules are loaded automatically when syslog-ng OSE is started, use the --default-modules command-line option of syslog-ng OSE.

§ To request loading a module from the syslog-ng OSE configuration file, see below section.

2.6.2.1) Loading modules

The syslog-ng Open Source Edition application loads every available module during startup.
To load a module that is not loaded automatically, include the following statement in the syslog-ng OSE configuration file:

@module <module-name>

Note the following points about the @module statement:

§ The @module statement is a top-level statement, that is, it cannot be nested into any other statement. Usually it is used immediately after the @version statement.

§ Every @module statement loads a single module: loading multiple modules requires a separate @module statement for every module.

§ In the configuration file, the @module statement of a module must be earlier than the module is used.

2.6.3) Managing complex syslog-ng configurations

The following sections describe some methods that can be useful to simplify the management of large-scale syslog-ng installations.

2.6.3.1) including configuration files:

The syslog-ng application supports including external files in its configuration file, so parts of its configuration can be managed separately. To include the contents of a file in the syslog-ng configuration, use the following syntax

include "<filename>";

Where <filename> can be file name, path or even a directory (where all files are included, except files beginning with ~ (tilde) or . (dot)). Including a directory is not recursive.
This imports the entire file into the configuration of syslog-ng, at the location of the include statement. If you specify a directory, syslog-ng will try to include every file in alphabetic order. When including configuration files, consider the following points:

§ If an object is defined twice (for example the original syslog-ng configuration file and the file imported into this configuration file both define the same option, source, or other object), then the object that is defined later in the configuration file will be effective. For example, if you set a global option at the beginning of the configuration file, and later include a file that defines the same option with a different value, then the option defined in the imported file will be used.

§ Files can be embedded into each other: the included files can contain include statements as well, up to a maximum depth of 15 levels.

§ You cannot include complete configuration files into each other, only configuration snippets can be included. This means that the included file cannot have a @version statement.

§ Include statements can only be used at top level of the configuration file. For example, the following is correct:

@version: 3.3

include "example.conf";

But the following is not:

source s_example {

    include "example.conf"

};

Warning : The syslog-ng application will not start if it cannot find a file that is to be included in its configuration. Always double-check the filenames, paths, and access rights when including configuration files, and use the --syntax-only command-line option to check your configuration.

2.6.4) Sources:

syslog-ng receives log messages from a source. To define a source you should follow the following syntax:

source <identifier> { source-driver(params); source-driver(params); ... };

You can look at the identifiers and source-drivers in the official manuals. This guide will follow the manual to explain the configuration file above. The unix-stream() source-driver opens the given AF_UNIX socket and starts listening on it for messages. The internal() source-driver gets messages generated by syslog-ng.
Therefore, the following means: src gets messages from the /dev/log socket and syslog-ng.

source src { unix-stream("/dev/log"); internal(); };

The kernel sends log messages to /proc/kmsg and the file() driver reads log messages from files. Therefore, the following means: kernsrc gets messages from file /proc/kmsg

source kernsrc { file("/proc/kmsg"); };

In the default configuration file after emerging syslog-ng, the source is defined as:

source src { unix-stream("/dev/log"); internal(); pipe("/proc/kmsg"); };

Reading messages by pipe("/proc/kmsg") gives a better performance but because it opens its argument in read-write mode can be a security hazard as the syslog-ng admin guide states in section 3.3.3:
"Pipe is very similar to the file() driver, but there are a few differences, for example pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg
To open a port to read data from a remote server a source must be defined with this syntax:

for UDP ->  source s_net { udp(); };

source s_net { tcp(); };

to receive log messages via TCP. Both listen on port 514.
To listen on user defined ports or specific IPs of logging server, define source as below:

source s_net { udp(ip(192.168.2.9) port(514));};

# Here syslog-ng will read on IP 192.168.2.9 of log server on udp port 514 for incoming logs.

source s_net { tcp(ip(192.168.2.8) port(1514)); };

# Here syslog-ng will read on IP 192.168.2.8 of log server on tcp port 1514 for incoming logs.

2.6.4.1) Collecting process accounting logs on Linux:

Starting with version 3.2, syslog-ng OSE can collect process accounting logs on Linux systems.Process accounting is the method of recording and summarizing commands executed on Linux, for example, the commands being run, the user who executed the command, CPU time used by the process, exit code, and so on. When process accounting (also called pacct) is enabled on a system, the kernel writes accounting records to the /var/log/account/pacct file (might vary between different Linux distributions).
To use the pacct() driver, the following conditions must be met:

§ The syslog-ng OSE application must be compiled with the --enable-pacct option. Execute the syslog-ng -V command to check if your binary supports process accounting.

§ The pacctformat plugin must be loaded. By default, syslog-ng OSE automatically loads the available modules.

§ The scl.conf file must be included in your syslog-ng configuration:

@include "scl.conf"

§ Process accounting must be running on the host. You can enable it with the accton command.

The pacct() driver parses the fields of the accounting logs and transforms them into name-value pairs. The fields are defined in the manual page of the accounting log file (man acct), syslog-ng OSE prepends every field with the .pacct. prefix. For example, the ac_uid field that contains the id of the user who started the process will be available under the $.pacct.ac_uid name. These can be used as macros in templates, in filters to select specific messages, and so on.
To use the pacct() driver, use the following syntax.

@version: @techversion;

@include "scl.conf"

source s_pacct { pacct(); };

...

log { source(s_pacct); destination(...); };

Also see pacct() options In manual

2.6.4.2) Receiving messages from external applications

The program driver starts an external application and reads messages from the standard output (stdout) of the application. It is mainly useful to receive log messages from daemons that accept incoming messages and convert them to log messages.
The program driver has a single required parameter, specifying the name of the application to start.

Declaration:

program(filename);

Example Using the program() driver

source s_program { program("/etc/init.d/mydaemon"); };

2.6.5) Destinations

In syslog-ng, log messages are sent to files. The syntax is very similar to sources:

destination <identifier> {destination-driver(params); destination-driver(params); ... };

You will be normally logging to a file, but you could log to a different destination-driver: pipe, Unix socket, TCP-UDP ports, terminals or to specific programs. Therefore, this means sending authlog messages to /var/log/auth.log:

destination authlog { file("/var/log/auth.log"); };

If the user is logged in, usertty() sends messages to the terminal of the specified user. If you want to send console messages to root's terminal if it is logged in:

destination console { usertty("root"); };

Messages can be sent to a pipe with pipe(). The following sends xconsole messages to the pipe /dev/xconsole.

destination xconsole { pipe("/dev/xconsole"); };

To send messages on the network, use udp(). The following will send your log data out to another server.

destination remote_server { udp("10.0.0.2" port(514)); };

Similarly you can use TCP for transport.

2.6.6) Creating Filters for Messages

The syntax for the filter statement is:

filter <identifier> { expression; };

Functions can be used in the expression, such as the function facility() which selects messages based on the facility codes. The Linux kernel has a few facilities you can use for logging. Each facility has a log-level; where debug is the most verbose, and panic only shows serious errors. You can find the facilities, log levels and priority names in /usr/include/sys/syslog.h. e.g. To filter those messages coming from authorization, use the following:

filter f_auth { facility(auth); };

The facility expression can use the boolean operators and, or, and not, so the following filter selects those messages not coming from authorization, network news or mail:

filter f_debug { not facility(auth, authpriv, news, mail); };

The function level() selects messages based on its priority level, so if you want to select informational levels:

filter f_info { level(info); };

Functions and boolean operators can be combined in more complex expressions. The following line filters messages with a priority level from informational to warning not coming from auth, authpriv, mail and news facilities:

filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); };

Messages can also be selected by matching a regular expression in the message with the function

match("regex"
value("keyword"))

. For example:

filter f_failed { match("regex" value("failed")); };

To filter messages received from a particular remote host, the host() function must be used:

filter f_host { host( "192.168.1.1" ); };

2.6.7) Log Paths

syslog-ng connects sources, filters and destinations with log statements. The syntax is:

log {source(s1); source(s2); ...

filter(f1); filter(f2); ...

destination(d1); destination(d2); ...

flags(flag1[, flag2...]); };

The following for example sends messages from src source to mailinfo destination filtered by f_info filter.

log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); };

2.6.8) Managing incoming and outgoing messages with flow-control

This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message losses. To use flow-control, the flow-control flag must be enabled for the particular log path.
The syslog-ng application monitors (polls) the sources defined in its configuration file, periodically checking each source for messages. When a log message is found in one of the sources, syslog-ng polls every source and reads the available messages. These messages are processed and put into the output buffer of syslog-ng (also called fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.
In large-traffic environments many messages can arrive during a single poll loop, therefore syslog-ng reads only a fixed number of messages from each source. The log_fetch_limit() option specifies the number of messages read during a poll loop from a single source.

Figure 2.6.8.1. Managing log messages in syslog-ng

`Note: The log_fetch_limit() parameter can be set as a global option, or for every source individually.`

Every destination has its own output buffer. The output buffer is needed because the destination might not be able to accept all messages immediately. The log_fifo_size() parameter sets the size of the output buffer. The output buffer must be larger than the log_fetch_limit() of the sources, to ensure that every message read during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output buffer must be large enough to store the incoming messages of every source.
TCP and unix-stream sources can receive the logs from several incoming connections (for example many different clients or applications). For such sources, syslog-ng reads messages from every connection, thus the log_fetch_limit() parameter applies individually to every connection of the source.

Figure 2.6.8.2 Managing log messages of TCP sources in syslog-ng

The flow-control of syslog-ng introduces a control window to the source that tracks how many messages can syslog-ng accept from the source. Every message that syslog-ng reads from the source lowers the window size by one; every message that syslog-ng successfully sends from the output buffer increases the window size by one. If the window is full (that is, its size decreases to zero), syslog-ng stops reading messages from the source. The initial size of the control window is by default 1000: the log_fifo_size() must be larger than this value in order for flow-control to have any effect. If a source accepts messages from multiple connections, all messages use the same control window.

`Note: Starting with syslog-ng OSE version 3.3, if the source can handle multiple connections (for example, tcp()), the size of the control window is divided by the value of the max_connections() parameter and this smaller control window is applied to each connection of the source.`

When flow-control is used, every source has its own control window. As a worst-case situation, the output buffer of the destination must be set to accommodate all messages of every control window, that is, the log_fifo_size() of the destination must be greater than number_of_sources*log_iw_size(). This applies to every source that sends logs to the particular destination. Thus if two sources having several connections and heavy traffic send logs to the same destination, the control window of both sources must fit into the output buffer of the destination. Otherwise, syslog-ng does not activate the flow-control, and messages may be lost.
The syslog-ng application handles outgoing messages the following way:

Figure 2.6.8.3 Handling outgoing messages in syslog-ng OSE

§ Output queue: Messages from the output queue are sent to the target syslog-ng server. The syslog-ng application puts the outgoing messages directly into the output queue, unless the output queue is full. The output queue can hold 64 messages, this is a fixed value and cannot be modified.

§ Overflow queue: If the output queue is full, syslog-ng puts the outgoing messages into the overflow queue of the destination. (The overflow queue is identical to the output buffer used by other destinations.) The log_fifo_size() parameter specifies the number of messages stored in the overflow queue. For details on sizing the log_fifo_size() parameter,

2.6.9) Log to another file

In order to move some log from /var/log/messages to another file:

#sshd configuration

destination ssh { file("/var/log/ssh.log"); };

filter f_ssh { program("sshd"); };

log { source(src); filter(f_ssh); destination(ssh); };

2.7) Configuring as a loghost

Configuring your system to be a loghost is quite simple. Drop the following into your configuration, and create the needed directory. With this simple configuration, log filenames will be based on the FQDN of the remote host, and located in /var/log/remote/. After creating the remote directory, reload your syslog-ng configuration.

source net { udp(); };

destination remote { file("/var/log/remote/${FULLHOST}-log"); };

log { source(net); destination(remote); };

2.8) Client Side Configurations:

For rsyslog clients add the following line in /etc/rsyslog.d/rsyslog.conf:

For TCP based transfer: *.* @@ipaddress:1514

For UDP based transfer: *.* @ipaddress:514

For syslog-ng clients, configuration will be as follows:

source s_sys {

file ("/proc/kmsg" log_prefix("kernel: "));

unix-stream ("/dev/log");

internal();

# udp(ip(0.0.0.0) port(514));

};

destination d_net {tcp(192.168.2.9 port(1514)); };

#destination d_net{udp(192.168.2.9);};

log { source(s_sys); destination(d_net); };

Also you will have to allow above ports on firewall,

e.g.

iptables -A INPUT -m state --state NEW -p udp --dport 514 -j ACCEPT

iptables -A INPUT -m state --state NEW -p tcp --dport 1514 -j ACCEPT

2.9) Improve Performance

Syslog-ng’s performance can be improved in different ways:

2.9.1) Write every so often

It seems that the old sync(X) option is called flush_lines(X) now, where the writing to the file is buffered for X lines. Default is 0 (no buffering).

2.9.2) Avoid redundant processing and disk space

A single log message can be sent to different log files several times. For example, in the initial configuration file, we have the following definitions:

destination cron { file("/var/log/cron.log"); };

destination messages { file("/var/log/messages"); };

filter f_cron { facility(cron); };

filter f_messages { level(info..warn)

       and not facility(auth, authpriv, mail, news); };

log { source(src); filter(f_cron); destination(cron); };

log { source(src); filter(f_messages); destination(messages); };

The same message from the cron facility will end up in both the cron.log and messages files. To change this behavior we can use the final flag, ending up further processing with the message. Therefore, in this example, if we want messages from the cron facility not ending up in the messages file, we should change the cron's log sentence by:

 log { source(src); filter(f_cron); destination(cron); flags(final); };

another way is to exclude the cron facility from f_messages filter:

 filter f_messages { level(info..warn) and not facility(cron, auth, authpriv, mail, news); };

2.10) PostgreSQL Destination

This section will use two roles: syslog and logwriter. syslog will be the administrator of the database syslog and logwriter will only be able to add records to the logs table.
No longer needed to create table for logs. syslog-ng will create automatically.

psql -U postgres

postgres=# CREATE ROLE syslog WITH LOGIN;

postgres=# \password syslog    # Using the \password function is secure because

postgres=# \password logwriter # the password isn't saved in history.

postgres=# CREATE DATABASE syslog OWNER syslog;

postgres=# \q # You're done here for the moment

Edit pg_hba.conf to allow syslog and logwriter to establish a connection to PostgreSQL.

/var/lib/postgresql/8.4/data/pg_hba.conf

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

host    syslog      logwriter   192.168.0.1/24        md5

host    syslog      syslog      192.168.0.10/32       md5

Tell PostgreSQL to reload the configuration files:

/etc/rc.d/postgresql-8.4 reload

Edit /etc/syslog-ng.conf so that it knows where and how to write to PostgreSQL. syslog-ng will utilize the logwriter role.

# SQL logging support

destination d_pgsql {

  sql(type(pgsql)

  host("127.0.0.1") username("logwriter") password("password")

  database("syslog")

  table("logs_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") #or whatever you want, example ${HOST}" for hosts, ${LEVEL}" for levels.. etc

  columns("datetime timestamp with time zone", "host varchar(32)", "program varchar(16)", "pid varchar(16)", "message varchar(200)")

  values("$R_ISODATE", "$HOST", "$PROGRAM", "$PID", "$MSG")

  indexes("datetime", "host", "program", "pid", "message"));

};

log { source(src); destination(d_pgsql); };

Finally, restart syslog-ng.

/etc/rc.d/syslog-ng restart

And check to see if things are being logged.

psql -U logwriter -d syslog

syslog=> SELECT * FROM <your table name> ORDER BY datetime DESC LIMIT 10;

2.11) ISO 8601 timestamps

Before:

Note: Logger makes entries in the system log.  It provides a shell command interface to the syslog system log module.

Run below command

#logger These timestamps are not optimal.

#tail -n 1 /var/log/messages.log

Feb 18 14:25:01 hostname logger: These timestamps are not optimal.

Hence Add ts_format(iso); to /etc/syslog-ng/syslog-ng.conf in the options section.

Then:

# /etc/rc.d/syslog-ng reload

After:

#logger Now THAT is a timestamp!

#tail -n 2 /var/log/messages.log

Feb 18 14:25:01 hostname logger: These timestamps are not optimal.

2010-02-18T20:23:58-05:00 electron logger: Now THAT is a timestamp!

2.12) Log Levels

Log levels are defined separately for each logged facility in syslog-ng config. Available log levels are listed in /usr/include/sys/syslog.h :

define LOG_EMERG       0       /* system is unusable */

define LOG_ALERT       1       /* action must be taken immediately */

define LOG_CRIT        2       /* critical conditions */

define LOG_ERR         3       /* error conditions */

define LOG_WARNING     4       /* warning conditions */

define LOG_NOTICE      5       /* normal but significant condition */

define LOG_INFO        6       /* informational */

define LOG_DEBUG       7       /* debug-level messages */

2.13) Macros and Variables

Macros can be used in both templates, and in destination file names. For More information, Refer Macros of syslog-ng OSE from the manual.
The following code will write the log lines to /var/log/ syslog/remote in the format of hierarchy separating logs year wise, then month wise for each year, then day wise for each month. Log files will have information as per template defined later in below destination file name.

destination d_alllogs { file ( "/var/log/syslog/remote/$FULLHOST/$YEAR/$MONTH/$DAY/$FULLHOST-$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) template("$DATE $FULLHOST $PROGRAM $TAG [$FACILITY.$LEVEL] $MESSAGE\n")); };

log { source(s_sys); destination(d_alllogs);  };

Note: hostname must be set in /etc/sysconfig/network for all clients; also you need to have entries in /etc/hosts on both client and server then only logs will be separated hostname wise.

If above settings are not done then different folders will be created for each client and folder names will be same as IP of clients.

2.14) Troubleshooting syslog-ng:

As a general rule, first try to get logging the messages to a local file. Once this is working, you know that syslog-ng is running correctly and receiving messages, and you can proceed to forwarding the messages to the server.

If the syslog-ng server does not receive the messages, use tcpdump or a similar packet sniffer tool on the client to verify that the messages are sent correctly, and on the server to verify that it receives the messages.

e.g. create logs on client using logger command and run “tcpdump –vn port <port no>” on log server to see if it is receiving logs or not.

If syslog-ng is closing the connections for no apparent reason, be sure to check the log messages of syslog-ng. You might also want to run syslog-ng with the --verbose or --debug command-line options for more-detailed log messages. Starting from syslog-ng OSE version 3.1, you can enable these messages without restarting syslog-ng using the syslog-ng-ctl verbose --set=on command. For details, see the syslog-ng-ctl man page at syslog-ng-ctl(1).

To start syslog-ng in debug mode, use command: /sbin/syslog-ng –Fedv

Similarly, build up encrypted connections step-by-step: first create a working unencrypted (for example TCP) connection, then add TLS encryption, and finally client authentication if needed.

2.15) Possible causes of losing log messages:

During the course of a message from the sending application to the final destination of the message, there are a number of locations where a message may be lost, even though syslog-ng does its best to avoid message loss. Usually losing messages can be avoided with careful planning and proper configuration of syslog-ng and the hosts running syslog-ng. The following list shows the possible locations where messages may be lost, and provides methods to minimize the risk of losing messages.

§ Between the application and the syslog-ng client: Make sure to use an appropriate source to receive the logs from the application (for example from /dev/log). For example, use unix-stream instead of unix-dgram whenever possible.

§ When syslog-ng is sending messages: If syslog-ng cannot send messages to the destination and the output buffer gets full, syslog-ng will drop messages. The number of dropped messages is displayed per destination in the log message statistics of syslog-ng.

§ On the network: When transferring messages using the UDP protocol, messages may be lost without any notice or feedback — such is the nature of the UDP protocol. Always use the TCP protocol to transfer messages over the network whenever possible.

§ In the socket receive buffer: When transferring messages using the UDP protocol, the UDP datagram (that is, the message) that reaches the receiving host placed in a memory area called the socket receive buffer. If the host receives more messages than it can process, this area overflows, and the kernel drops messages without letting syslog-ng know about it. Using TCP instead of UDP prevents this issue. If you must use the UDP protocol, increase the size of the receive buffer using the so_rcvbuf() option.

§ When syslog-ng is receiving messages: The receiving syslog-ng (for example the syslog-ng server or relay) may drop messages if the fifo of the destination file gets full. The number of dropped messages is displayed per destination in the log message statistics of syslog-ng.

§ When the destination cannot handle large load: When syslog-ng is sending messages at a high rate into an SQL database, a file, or another destination, it is possible that the destination cannot handle the load, and processes the messages slowly. As a result, the buffers of syslog-ng fill up, syslog-ng cannot process the incoming messages, and starts to loose messages. For details, see the previous entry. Use the throttle parameter to avoid this problem.

§ As a result of an unclean shutdown of the syslog-ng server: If the host running the syslog-ng server experiences an unclean shutdown, it takes time until the clients realize that the connection to the syslog-ng server is down. Messages that are put into the output TCP buffer of the clients during this period are not sent to the server.

2.16) Procedure – Creating syslog-ng core files

Purpose: When syslog-ng crashes for some reason, it can create a core file that contains important troubleshooting information. To enable core files, complete the following procedure:
Steps:

1. Core files are produced only if the

maximum
core file size

ulimit is set to a high value in the init script of syslog-ng. Add the following line to the init script of syslog-ng:

ulimit -c unlimited

2. Verify that syslog-ng has permissions to write the directory it is started from, for example /opt/syslog-ng/sbin/.

3. If syslog-ng crashes, it will create a core file in the directory syslog-ng was started from.

4. To test that syslog-ng can create a core file, you can create a crash manually. For this, determine the PID of syslog-ng (for example using the ps -All|grep syslog-ng command), then issue the following command: kill -ABRT <syslog-ng pid>

This should create a core file in the current working directory.

2.17) Collecting debugging information with strace, truss, or tusc

To properly troubleshoot certain situations, it can be useful to trace which system calls syslog-ng OSE performs. How this is performed depends on the platform running syslog-ng OSE. In general, note the following points:

§ When syslog-ng OSE is started, a supervisor process might stay in the foreground, while the actual syslog-ng daemon goes to the background. Always trace the background process.

§ Apart from the system calls, the time between two system calls can be important as well. Make sure that your tracing tool records the time information as well. For details on how to do that, refer to the manual page of your specific tool (for example, strace on Linux, or truss on Solaris and BSD).

§ Run your tracing tool in verbose mode, and if possible, set it to print long output strings, so the messages are not truncated.

§ When using strace, also record the output of lsof to see which files are accessed.

The following are examples for tracing system calls of syslog-ng on some platforms. The output is saved into the /tmp/syslog-ng-trace.txt file, sufficed with the PID of the related syslog-ng process.The path of the syslog-ng binary assumes that you have installed syslog-ng OSE from the official syslog-ng OSE binaries available at the BalaBit website — native distribution-specific packages may use different paths.

§ Linux: strace -o /tmp/trace.txt -s256 -ff -ttT /opt/syslog-ng/sbin/syslog-ng -f /opt/syslog-ng/etc/syslog-ng.conf -Fdv

§ HP-UX: tusc -f -o /tmp/syslog-ng-trace.txt -T /opt/syslog-ng/sbin/syslog-ng

§ IBM AIX and Solaris: truss -f -o /tmp/syslog-ng-trace.txt -r all -w all -u libc:: /opt/syslog-ng/sbin/syslog-ng -d -d -d

Tip: To execute these commands on an already running syslog-ng OSE process, use the -p <pid_of_syslog-ng> parameter.

loggen — Generate syslog messages at a specified rate

The loggen application is tool to test and stress-test your syslog server and the connection to the server. It can send syslog messages to the server at a specified rate, using a number of connection types and protocols, including TCP, UDP, and unix domain sockets. The messages can be generated automatically (repeating the PADDstring over and over), or read from a file or the standard input.
When loggen finishes sending the messages, it displays the following statistics:

§ average rate: Average rate the messages were sent in messages/second.

§ count: The total number of messages sent.

§ time: The time required to send the messages in seconds.

§ average message size: The average size of the sent messages in bytes.

§ bandwidth: The average bandwidth used for sending the messages in kilobytes/second.

For options see manual page of loggen.

Examples:

1) The following command generates 100 messages per second for ten minutes, and sends them to port 2010 of the localhost via TCP. Each message is 300 bytes long.

loggen --size 300 --rate 100 --interval 600 127.0.0.1 2010

2) The following command is similar to the one above, but uses the UDP protocol.

loggen --inet --dgram --size 300 --rate 100 --interval 600 127.0.0.1 2010

3) Send a single message on TCP6 to the ::1 IPv6 address, port 1061:

loggen --ipv6 --number 1 ::1 1061

4) Send a single message on UDP6 to the ::1 IPv6 address, port 1061:

loggen --ipv6 --dgram --number 1 ::1 1061

5) Send a single message using a unix domain-socket: loggen --unix --stream --number 1 </path/to/socket>
6) Read messages from the standard input (stdio) and send them to the localhost:

loggen 127.0.0.1 1061 --read-file –

2.18) Apache and MySQL Logging with Syslog-ng

http://blog.monitis.com/index.php/2011/09/05/apache-and-mysql-logging-with-syslog-ng/

http://www.linuxgeek.net/apache-with-syslog-ng/

2.19) build a Secure Logging Server with syslog-ng using SSL/TLS

http://www.enterprisenetworkingplanet.com/netsysm/article.php/3598146/Build-a-Secure-Logging-Server-with-syslogng-Part-2.htm

http://www.nineproductions.com/syslog-ng-with-tls-setup-for-centralized-logging/

http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-4.0-guides/en/syslog-ng-pe-v4.0-guide-admin-en/html/procedure-configuring-mutual-tls-server.html

http://www.syslog.org/forum/syslog-ng/syslog-ng-using-tls-require-help/

http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html/procedure-configuring-tls-server.html

https://www.icts.uiowa.edu/confluence/display/ICTSit/Add+TLS+Functionality+to+your+syslog-ng+setup

2.20) syslog-ng + logrotate

http://www.mpking.com/2013/03/logrotate-with-syslog-ng.html

http://mewbies.com/how_to_log_users_using_syslog-ng_tutorial.htm

http://www.thegeekstuff.com/2010/07/logrotate-examples/

https://wiki.archlinux.org/index.php/Logrotate

2.21) running syslog-ng in chrooted environment and collecting logs from chrooted applications

http://eagain.net/articles/syslog-ng-chroot/

http://www.campin.net/syslog-ng/chroot-jail.html

http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html/example_chroot.html

http://www.balabit.com/wiki/syslog-ng-faq-chroot

2.22) References:

The syslog-ng Open Source Edition 3.3 Administrator Guide (contains every bit of information for syslog-ng)

http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html-single/index.html

See also man syslog-ng, man syslog-ng.conf, man syslog-ng-ctl

List of syslog-ng OSE parameters

http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/html/syslog-ng-parameter-index.html

Macros of syslog-ng OSE

http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/html/reference-macros.html

Books about syslog-ng

http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/overview/books

Performane tuning syslog-ng

http://codeascraft.com/2012/08/13/performance-tuning-syslog-ng/

^{How to Log to PostgreSQL
with Syslog-ng}

http://blog.monitis.com/index.php/2011/09/08/how-to-log-to-postgresql-with-syslog-ng/

Remote Logging with SSH and Syslog-NG

http://www.deer-run.com/~hal/sysadmin/SSH-SyslogNG.html

Chapter 3 - Rsyslog

Rsyslog is the default logging program on several Linux distributions including Debian and Red Hat based systems. Apart from implementing the syslog protocol, rsyslog adds additional features such as content-based filtering. This also uses TCP for transporting, and provides lot of configuration options.
Rsyslog uses the quasi-standard BSD syslog protocol, specified in RFC 3164. As the text of RFC 3164 is just a vague informational description and not a standard, various incompatible extensions of it emerged. Rsyslog supports many of these extensions. The format of relayed messages can be customized.

3.1) Features:

The most important extensions of the original protocol supported by rsyslog are:

ISO 8601 timestamp with millisecond granularity and timezone information
the addition of the name of relays in the host fields to make it possible to track the path a given message has traversed
reliable transport using TCP
support GSS-API and TLS
logging directly into various database engines.
support for RFC 5424, RFC 5425, RFC 5426
support for RELP
support for buffered operation modes where messages are buffered locally if the receiver is not ready

For more Features of rsyslog: see http://www.rsyslog.com/doc/features.html https://wiki.archlinux.org/index.php/Rsyslog
Following setup will help you to analyze the log files of all the servers in your infrastructure from a central log server.

3.2) Installation

Rsyslog comes as the default logging program in Debian Distribution and Red Hat based systems. If you system doesn’t have rsyslog, install it as shown below depending on your distro.
1) From RPM: Refer Link - http://www.rsyslog.com/installing-rsyslog-from-rpm/

apt-get install rsyslog rsyslog-doc

(or)

yum install rsyslog rsyslog-doc

see also : http://www.rsyslog.com/doc/install.html

Rsyslog configurations are stored in /etc/ryslog.conf file and the files under /etc/rsyslog.d/ directory.

3.3) Configuration Structure

Before understanding how to setup the central logging sever, it is good to understand the configuration structure of rsyslog.
Rsyslog configuration files are structured in the following manner

1. Modules

2. Configuration Directives

3. Rule line

3.3.1) Modules

Rsyslog has a modular architecture. It enables functionalities to be added dynamically through these modules. The modules are categorized as:

Input Modules – Used to gather messages from various sources
Output Modules – Used to write the messages to various places ( file, socket etc.. )
Parser Modules – Used to parse the message content

Please note that there are also other categories of modules available. This is to give an overview of what modules can do.
For more Info see http://www.rsyslog.com/doc/modules.html http://www.rsyslog.com/doc/rsyslog_conf_modules.html

3.3.2) Configuration Directives

All configuration directives must be specified one per line and must start with dollar sign ($). It affects the rules.
For Complete List see: http://www.rsyslog.com/doc/rsyslog_conf_global.html

3.3.3) Rule line

Every rule line consists of two fields, a ‘selector field’ and an ‘action field’. The selector field is divided into two, ‘facilities & priorities’. Action specifies what action must be taken for the matched rule.
See also: http://www.rsyslog.com/doc/multi_ruleset.html

3.3.4) A Sample Configuration

######################

                    MODULES

######################

$ModLoad imuxsock

$ModLoad imklog

######################

                    Directives

######################

# Set the default permissions for all log files.

$FileOwner root

$FileGroup adm

$FileCreateMode 0640

$DirCreateMode 0755

######################

                    RULES

######################

mail.info      /var/log/mail.info

mail.warn     /var/log/mail.warn

mail.err        /var/log/mail.err

daemon.*     /var/log/daemon.log

3.3.5) Templates

Templates are a very important features provided by rsyslog. It allows the user to log the messages in their desirable format. It can also be used to create dynamic file names to log the messages. In case of database logging, the templates are used to convert the message into a proper SQL statement.
A sample template will look like:

$template mytemplate “Text-Before %msg% Text-After\n”

The above template will log the message “This is hello from rsyslog” as:

Text-Before this is hello from rsyslog Text-After

We will see how to use the template for generate the log files dynamically.
See http://www.rsyslog.com/doc/rsyslog_conf_templates.html
http://www.rsyslog.com/tag/use-a-template/
http://www.rsyslog.com/doc/rsyslog_conf_examples.html

3.4) Central Logging Server

The above sections should have given an overview about rsyslog and how to configure it. Now we will move on to setup a central logging system.
For our discussion we will have server IP as “192.168.1.1” for the central log server, where all the log messages from client should be forwarded.
Add the following lines to the rsyslog.conf of the central log server servers (In this example, the following line was added on the log server with ip-address 192.168.1.1):

# provides support for local system logging

$ModLoad imuxsock

# provides kernel logging support (previously done by rklogd)

$ModLoad imklog

# provides UDP syslog reception. For TCP, load imtcp.

$ModLoad imudp

# For TCP, InputServerRun 514

$UDPServerRun 514

# This one is the template to generate the log filename dynamically, depending on the client's IP address.

$template FILENAME,"/var/log/%fromhost-ip%/syslog.log"

# Log all messages to the dynamically formed file. Now each clients log (192.168.1.2, 192.168.1.3,etc...), will be under a separate directory which is formed by the template FILENAME.

*.* ?FILENAME

After adding the above lines to the rsyslog.conf, restart the rsyslog process. Now the rsyslog server will be ready to accept messages.

# service rsyslog restart

Shutting down system logger:                               [  OK  ]

Starting system logger:                                    [  OK  ]

Add the following lines to the rsyslog.conf on the individual client machines that should send their log messages to the central server.

$ModLoad imuxsock

$ModLoad imklog

# Provides UDP forwarding. The IP is the server's IP address

*.* @192.168.1.1:514

# Provides TCP forwarding. But the current server runs on UDP

# *.* @@192.168.1.1:514

Restart the rsyslog process on the clients. Now the rsyslog central server (In this example, 192.168.1.1) will receive all the log messages from the configured clients and each client’s log will be placed under a separate directory.
See Also: http://www.howtoforge.com/centralized-rsyslog-server-monitoring (very important link)

3.5) Encrypting Syslog Traffic with TLS (SSL)

Refer Link: http://www.rsyslog.com/doc/rsyslog_tls.html

3.6) Unreliability of plain tcp syslog à http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html

3.7) TLS secured syslog via RELP

Refer Link: http://www.rsyslog.com/tls-secured-syslog-via-relp/

Sunday, 21 December 2014

Logging in Linux