Interesting Facts about Linuz

Exactly 15 years ago, on March 1994, Linux kernel version 1.0.0 was humbly released for the world to tinker with. To celebrate the historic moment, I have collected some really interesting facts about the Linux kernel. So if you are a Linux fan, grab a beer and read on.

1. A 21 year-old Finnish college student created the Linux kernel as a hobby. (Do you know him?)

2. An asteroid was named after the creator of the Linux kernel.

3. Thousands of developers/programmers scattered all around the world are continuously contributing to the development of the Linux kernel.

4. The Linux kernel's official mascot is a penguin named Tux.

5. According to a study funded by the European Union, the estimated cost to redevelop the most recent kernel versions would be at $1.14 billion USD.

6. As of today, only 2% of the Linux kernel has been written by Linus Torvalds.

7. The Linux kernel is written in the version of the C programming language.

8. Linux is now one of the most widely ported operating system kernels, running on a diverse range of systems from handheld computers to mainframe servers.

9. Linux kernel 1.0.0 was released with 176,250 lines of code. The latest Linux kernel has over 10 million lines of code.

10. Microsoft Windows and the Linux kernel can run simultaneously in parallel on the same machine using a software called Cooperative Linux (coLinux).

11. At first, Torvalds wanted to call the kernel he developed Freax (a combination of "free", "freak", and the letter X to indicate that it is a Unix-like system), but his friend Ari Lemmke, who administered the FTP server where the kernel was first hosted for downloading, named Torvalds' directory linux.

12. A guy name William Della Croce, Jr. trademarked the name Linux and eventually demanded royalties for its use. He later agreed to assign the trademark to Torvalds.

13. The Linux kernel can be found on more than 87% of systems on the world's Top 500 supercomputers.

14. A "vanilla kernel" is not an ice cream flavor but an unmodified version of the Linux kernel.

15. The Linux Kernel is not in any way related to the army rank called ‘Colonel’. (hehe)

Cheers!

SendMail

Sendmail

http://www.sendmail.org/~ca/email/sm-X/design-2005-05-05/main/node1.html

SendMail Architecture

---

Sendmail X: Architecture

This chapter describes the architecture of sendmail X. It presents some possible design choices for various parts of sendmail X and explains why a particular choice has been made. Notice: several decisions haven't been made yet, there are currently a lot of open questions.

General Architecture of sendmail X

sendmail X consists of several communicating modules. A strict separation of functionality allows for a flexible, maintainable, and scalable program. It also enhances security by running only those parts with special privileges (e.g., root, which will be used as a synonym for the required privileges in this text) that really require it.

Ref URL : http://www.sendmail.org/~ca/email/sm-X/design-2005-05-05/main/node3.html  
 
   

Configuring the DHCP Server

Introduction

Normally if you have a cable modem or DSL, you get your home PC's IP address dynamically assigned from your service provider. If you install a home cable/DSL router between your modem and home network, your PC will most likely get its IP address at boot time from the home router instead. You can choose to disable the DHCP server feature on your home router and set up a Linux box as the DHCP server.

This chapter covers only the configuration of a DHCP server that provides IP addresses. The configuration of a Linux DHCP client that gets its IP address from a DHCP server on Linux Networking.

Download and Install the DHCP Package

Most RedHat and Fedora Linux software products are available in the RPM format. Downloading and installing RPMs aren't hard.

When searching for the file, remember that the DHCP server RPM's filename usually starts with the word dhcp followed by a version number like this: dhcp-3.0.1rc14-1.i386.rpm.

Debian Note: With Debian / Ubuntu the package name may include a version number. Use the dpkg --list | grep dhcp command to get a list of all your dhcp packages and use the output to infer what the DHCP server package name would be. In this case we can guess that the package name should be dhcp3-server.

root@u-bigboy:/tmp# dpkg --list | grep dhcp
ii  dhcp3-client   3.0.3-6ubuntu7  DHCP Client
ii  dhcp3-common   3.0.3-6ubuntu7  Files used by all the dhcp3* packages
root@u-bigboy:/tmp#

The /etc/dhcpd.conf File

When DHCP starts, it reads the file /etc/dhcpd.conf. It uses the commands here to configure your network. The standard DHCP RPM package doesn't automatically install a /etc/dhcpd.conf file, but you can find a sample copy of dhcpd.conf in the following directory which you can always use as a guide.

/usr/share/doc/dhcp-/dhcpd.conf.sample

You have to copy the sample dhcpd.conf file to the /etc directory and then you'll have to edit it. Here is the command to do the copying for the version 3.0p11 RPM file:

[root@bigboy tmp]# cp /usr/share/doc/dhcp-3.0pl1/dhcpd.conf.sample /etc/dhcpd.conf

Debian Note: With Debian / Ubuntu the configuration file name is /etc/dhcp*/dhcpd.conf and has the same syntax as that used by Redhat / Fedora.

Here is a quick explanation of the dhcpd.conf file: Most importantly, there must be a subnet section for each interface on your Linux box.

ddns-update-style interim
ignore client-updates

subnet 192.168.1.0 netmask 255.255.255.0 {

  # The range of IP addresses the server
  # will issue to DHCP enabled PC clients
  # booting up on the network

  range 192.168.1.201 192.168.1.220;

  # Set the amount of time in seconds that
  # a client may keep the IP address

 default-lease-time 86400;
 max-lease-time 86400;

  # Set the default gateway to be used by
  # the PC clients

  option routers 192.168.1.1;
  # Don't forward DHCP requests from this
  # NIC interface to any other NIC
  # interfaces

  option ip-forwarding off;

  # Set the broadcast address and subnet mask
  # to be used by the DHCP clients

 option broadcast-address 192.168.1.255;
 option subnet-mask 255.255.255.0;

  # Set the DNS server to be used by the
  # DHCP clients

 option domain-name-servers 192.168.1.100;

  # Set the NTP server to be used by the
  # DHCP clients

  option nntp-server 192.168.1.100;

  # If you specify a WINS server for your Windows clients,
  # you need to include the following option in the dhcpd.conf file:

 option netbios-name-servers 192.168.1.100;

  # You can also assign specific IP addresses based on the clients'
  # ethernet MAC address as follows (Host's name is "laser-printer":

 host laser-printer {
     hardware ethernet 08:00:2b:4c:59:23;
    fixed-address 192.168.1.222;
  }
}
#
# List an unused interface here
#
subnet 192.168.2.0 netmask 255.255.255.0 {
}

There are many more options statements you can use to configure DHCP. These include telling the DHCP clients where to go for services such as finger and IRC. Check the dhcp-options man page after you do your install:

[root@bigboy tmp]# man dhcp-options

Note: The host statement seen in the sample dhcpd.conf file can be very useful. Some devices such as network printers default to getting their IP addresses using DHCP, but users need to access them by a fixed IP address to print their documents. This statement can be used to always provide specific IP address to DHCP queries from a predefined a NIC MAC address. This can help to reduce systems administration overhead.

How to Get DHCP Started

To get DHCP started:

1) Some older Fedora/RedHat versions of the DHCP server will fail unless there is an existing dhcpd.leases file. Use the command touch /var/lib/dhcp/dhcpd.leases to create the file if it does not exist.

[root@bigboy tmp]# touch /var/lib/dhcp/dhcpd.leases

2) Use the chkconfig command to get DHCP configured to start at boot:

[root@bigboy tmp]# chkconfig dhcpd on

With Debian / Ubuntu the equivalent command for the dhcp3-server package would be:

root@u-bigboy:/tmp# sysv-rc-conf dhcp3-server on

3) Use the service command to instruct the /etc/init.d/dhcpd script to start/stop/restart DHCP after booting

[root@bigboy tmp]# service dhcpd start
[root@bigboy tmp]# service dhcpd stop
[root@bigboy tmp]# service dhcpd restart

With Debian / Ubuntu the equivalent commands would be:

root@u-bigboy:/tmp# /etc/init.d/dhcp*-server start
root@u-bigboy:/tmp# /etc/init.d/dhcp*-server stop
root@u-bigboy:/tmp# /etc/init.d/dhcp*-server restart

4) Remember to restart the DHCP process every time you make a change to the conf file for the changes to take effect on the running process. You also can test whether the DHCP process is running with the following command; you should get a response of plain old process ID numbers:

[root@bigboy tmp]# pgrep dhcpd

5) Finally, always remember to set your PC to get its IP address via DHCP.

DHCP Servers with Multiple NICs

When a DHCP configured PC boots, it requests its IP address from the DHCP server. It does this by sending a standardized DHCP broadcast request packet to the DHCP server with a source IP address of 255.255.255.255.

If your DHCP server has more than one interface, you have to add a route for this 255.255.255.255 address so that it knows the interface on which to send the reply; if not, it sends it to the default gateway. (In both of the next two examples, we assume that DHCP requests will be coming in on interface eth0).

Note: You can't run your DHCP sever on multiple interfaces because you can only have one route to network 255.255.255.255. If you try to do it, you'll discover that DHCP serving working on only one interface.

Temporary Solution

You can temporarily add a route to 255.255.255.255 using the route add command as seen below.

[root@bigboy tmp]# route add -host 255.255.255.255 dev eth0

If you want this routing state to be maintained after a reboot, then use the permanent solution that's discussed next.

Permanent Solution

The new Fedora Linux method of adding static routes doesn't seem to support sending traffic out an interface that's not destined for a specific gateway IP address. The DHCP packet destined for address 255.255.255.255 isn't intended to be relayed to a gateway, but it should be sent using the MAC address of the DHCP client in the Ethernet frame.

You have one of two choices. Add the route add command to your /etc/rc.local script, or add an entry like this to your /etc/sysconfig/static-routes file.

#
# File /etc/sysconfig/static-routes
#
eth0 host 255.255.255.255

Note: The /etc/sysconfig/static-routes file is a deprecated feature and Fedora support for it will eventually be removed.

Now that you have configured your server, it's time to take a look at the DHCP clients.

Configuring Linux Clients to Use DHCP

A Linux NIC interface can be configured to obtain its IP address using DHCP . Please refer to this chapter if you need a quick refresher on how to configure a Linux DHCP client.

Configuring Windows Clients to Use DHCP

Fortunately Windows defaults to using DHCP for all its NIC cards so you don't have to worry about doing any reconfiguration.

Using a Single DHCP Server to Serve Multiple Networks

As stated before, DHCP clients send their requests for IP addresses to a broadcast address which is limited to the local LAN. This would imply that a DHCP server is required on each subnet. Not so. It is possible to configure routers to forward DHCP requests to a DHCP server many hops away. This is done by inserting the IP address of the router's interface on the DHCP client's network into the forwarded packet. To the DHCP server, the non-blank router IP address field takes precedence over the broadcast address and it uses this value to provide a DHCP address that is meaningful to the client. The DHCP server replies with a broadcast packet, and the router, which has kept track of the initial forwarded request, forwards it back towards the client. You can configure this feature on Cisco devices by using the ip helper-address command on all the interfaces on which DHCP clients reside. Here is a configuration sample that points to a DHCP server with the IP address 192.168.36.25:

interface FastEthernet 2/1
 ip address 192.168.1.30 255.255.255.0
 ip helper-address 192.168.36.25

Simple DHCP Troubleshooting

The most common problems with DHCP usually aren't related to the server; after the server is configured correctly there is no need to change any settings and it therefore runs reliably. The problems usually occur at the DHCP client's end for a variety of reasons. The following sections present simple troubleshooting steps that you can go through to ensure that DHCP is working correctly on your network.

DHCP Clients Obtaining 169.254.0.0 Addresses

Whenever Microsoft DHCP clients are unable to contact their DHCP server they default to selecting their own IP address from the 169.254.0.0 network until the DHCP server becomes available again. This is frequently referred to as Automatic Private IP Addressing (APIPA). Here are some steps you can go through to resolve the problem:

• Ensure that your DHCP server is configured correctly and use the pgrep command discussed earlier to make sure the DHCP process is running. Pay special attention to your 255.255.255.255 route, especially if your DHCP server has multiple interfaces.
• Give your DHCP client a static IP address from the same range that the DHCP server is supposed to provide. See whether you can ping the DHCP server. If you cannot, double-check your cabling and your NIC cards.
• DHCP uses the BOOTP protocol for its communication between the client and server. Make sure there are no firewalls blocking this traffic. DHCP servers expect requests on UDP port 67 and the DHCP clients expect responses on UDP port 68. Use tcpdump on the server's NIC to verify the correct traffic flows.

Other DHCP Failures

If the DHCP server fails to start then use your regular troubleshooting techniques, to help rectify your problems. Most problems with an initial setup are often due to:

• Incorrect settings in the /etc/dhcpd.conf file such as not defining the networks for which the DHCP server is responsible;
• Firewall rules that block the DHCP bootp protocol on UDP ports 67 and 68;
• Routers failing to forward the bootp packets to the DHCP server when the clients reside on a separate network.

Always check your /var/logs/messages file for dhcpd errors and remember that mandatory keywords in your configuration file may change when you upgrade your operating system. Always read the release notes to be sure.

Conclusion

In most home-based networks, a DHCP server isn't necessary because the DSL router / firewall usually has DHCP capabilities, but it is an interesting project to try. Just remember to make sure that the range of IP addresses issued by all DHCP servers on a network doesn't overlap because it could possibly cause unexpected errors. You might want to disable the router/firewall's DHCP server capabilities to experiment with your new Linux server.

A DHCP server may be invaluable in an office environment where the time and cost of getting a network engineer to get the work done may make it simpler for Linux systems administrators to do it by themselves.

Creating a Linux DHCP server is straightforward and touches all the major themes in the previous chapters. Now it's time to try something harder, but before we do, we'll do a quick refresher on how to create the Linux users who'll be using many of the applications outlined in the rest of the book.

The Linux Boot Sequence

You might remember when you installed Linux that the installation process prompted you for a list of partitions and the sizes of each in which your filesystems would be placed.

When allocating disk space for the partitions, the first sector, or data unit, for each partition is always reserved for programmable code used in booting. The very first sector of the hard disk is reserved for the same purpose and is called the master boot record (MBR).

When booting from a hard disk, the PC system BIOS loads and executes the boot loader code in the MBR. The MBR then needs to know which partitions on the disk have boot loader code specific to their operating systems in their boot sectors and then attempts to boot one of them.

Fedora Linux is supplied with the GRUB boot loader which is fairly sophisticated and therefore cannot entirely fit in the 512 bytes of the MBR. The GRUB MBR boot loader merely searches for a special boot partition and loads a second stage boot loader. This then reads the data in the /boot/grub/grub.conf configuration file, which lists all the available operating systems and their booting parameters. When this is complete, the second stage boot loader then displays the familiar Fedora branded splash screen that lists all the configured operating system kernels for your choice.

Note: In some operating systems, such as Debian / Ubuntu, the /boot/grub/grub.conf file may also be referred to by the name /boot/grub/menu.lst.

Figure 7-1 shows a typical grub.conf file for a system that can boot both Fedora Linux and Windows 2000. This structure of this file is discussed in Chapter 33, "Modifying the Kernel to Improve Performance".

Figure 7-1 Sample grub.conf file

default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
title Fedora Core (2.6.8-1.521)
      root (hd0,0)
      kernel /vmlinuz-2.6.8-1.521 ro root=LABEL=/
      initrd /initrd-2.6.8-1.521.img
title Windows 2000
    rootnoverify (hd0,1)
    chainloader +1

When Linux begins to boot with its kernel, it first runs the /sbin/init program, which does some system checks, such as verifying the integrity of the file systems, and starts vital programs needed for the operating system to function properly. It then inspects the /etc/inittab file to determine Linux's overall mode of operation or runlevel. A listing of valid runlevels can be seen in Table 7-1.

Table 7-1 Linux Runlevels

Mode	Directory	Run Level Description
0	/etc/rc.d/rc0.d	Halt
1	/etc/rc.d/rc1.d	Single-user mode
2	/etc/rc.d/rc2.d	Not used (user-definable)
3	/etc/rc.d/rc3.d	Full multi-user mode (no GUI interface)
4	/etc/rc.d/rc4.d	Not used (user-definable)
5	/etc/rc.d/rc5.d	Full multiuser mode (with GUI interface)
6	/etc/rc.d/rc6.d	Reboot

Based on the selected runlevel, the init process then executes startup scripts located in subdirectories of the /etc/rc.d directory. Scripts used for runlevels 0 to 6 are located in subdirectories /etc/rc.d/rc0.d through /etc/rc.d/rc6.d, respectively.

Here is a directory listing of the scripts in the /etc/rc.d/rc3.d directory:

[root@bigboy tmp]# ls /etc/rc.d/rc3.d
...    ...    K75netfs      K96pcmcia    ...    ...
...    ...    K86nfslock    S05kudzu     ...    ...
...    ...    K87portmap    S09wlan      ...    ...
...    ...    K91isdn       S10network   ...    ...
...    ...    K92iptables   S12syslog    ...    ...
...    ...    K95firstboot  S17keytable  ...    ...
[root@bigboy tmp]#

As you can see, each filename in these directories either starts with an "S" which signifies the script should be run at startup, or a K, which means the script should be run when the system is shutting down. If a script isn't there, it won't be run.

Most Linux packages place their startup script in the /etc/init.d directory and place symbolic links (pointers) to this script in the appropriate subdirectory of /etc/rc.d. This makes file management a lot easier. The deletion of a link doesn't delete the file, which can then be used for another day.

The number that follows the K or S specifies the position in which the scripts should be run in ascending order. In our example, kudzu with a value 05 will be started before wlan with a value of 09. Fortunately you don't have to be a scripting/symbolic linking guru to make sure everything works right because Fedora comes with a nifty utility called chkconfig while Debian / Ubuntu uses the update-rc.d command to do it all for you. This is explained later.

Determining the Default Boot runlevel

The default boot runlevel is set in the file /etc/inittab with the initdefault variable. When set to 3, the system boots up with the text interface on the VGA console; when set to 5, you get the GUI. Here is a snippet of the file (delete the initdefault line you don't need):

# Default runlevel. The runlevels used by RHS are:
# 0 - halt (Do NOT set initdefault to this)
# 1 - Single user mode
# 2 - Multiuser, without NFS (The same as 3, if you do not have networking)
# 3 - Full multiuser mode
# 4 - unused
# 5 - X11
# 6 - reboot (Do NOT set initdefault to this)
#
id:3:initdefault:                         # Console Text Mode
id:5:initdefault:                         # Console GUI Mode

Note the following:

• Most home users boot up with a Windows like GUI (runlevel 5)
• Most techies will tend to boot up with a plain text-based command-line-type interface (runlevel 3)
• Changing initdefault from 3 to 5, or vice-versa, has an effect upon your next reboot. See the following section on how to get a GUI login all the time until the next reboot.
• Of course, don't set the initdefault value to 6 or your system will constantly reboot. Setting it to 0 will never allow it to start!

Getting a GUI Console

Manual Method: You can start the X terminal GUI application each time you need it by running the startx command at the VGA console. Remember that when you log out you will get the regular text-based console again.

[root@bigboy tmp]# startx

Automatic Method: You can have Linux automatically start the X terminal GUI console for every login attempt until your next reboot by using the init command. You will need to edit your initdefault variable in your /etc/inittab file, as mentioned in the preceding section to keep this functionality even after you reboot.

[root@bigboy tmp]# init 5

When the CPU capacity or available memory on your server is low or you want to maximize all system resources, you might want to operate in text mode runlevel 3 most of the time, using the GUI only as necessary with the startx command.

Servers that double as personal workstations, or servers that might have to be operated for an extended period of time by relatively nontechnical staff, may need to be run at runlevel 5 all the time through the init 5 command. Remember you can make runlevel 5 permanent even after a reboot by editing the /etc/inittab file.

Get a Basic Text Terminal Without Exiting the GUI

There are a number of ways for you to get a command prompt when running a Linux GUI. This can be important if you need quick access to commands or you are not familiar with the GUI menu option layout.

Using a GUI Terminal Window

You can open a GUI-based window with a command prompt inside by doing the following:

• Click on the Fedora logo button in the bottom left hand corner of the screen.
• Click on Systems Tools and then Terminal

Using Virtual Consoles

By default, Linux runs six virtual console or TTY sessions running on the VGA console. These are defined by the mingetty statements in the /etc/inittab file. The X terminal GUI console creates its own virtual console using the first available TTY that is not controlled by mingetty. This makes the GUI run as number 7:

• You can step through each virtual console session by using the Ctl-Alt-F1 through F6 key sequence. You'll get a new login prompt for each attempt.
• You can get the GUI login with the sequence Ctl-Alt-F7 only in run level 5, or if the GUI is running after launching startx.

System Shutdown and Rebooting

It is usually not a good idea to immediately power off your system when you are finished using it. This can cause files that are being updated to become corrupted, or worse, you could corrupt the filesystem directory structure. Linux has a number of ways to gracefully shut down and reboot your system which will be outlined in this section.

Halt/Shut Down The System

The init command will allow you to change the current runlevel, and for a shutdown, that value is 0. Here is an example:

[root@bigboy tmp]# init 0

Fedora also has a shutdown command which can also be used to the same effect. It often prompts you as to whether you are sure you want to execute the command, which can be avoided with the -y switch. The -h switch forces the system to halt, and the first argument tells it how long to wait before starting the procedure, in this case 0 minutes. You can also specify shutting down at a specific time of the day; please refer to the man pages for details. Another advantage of the shutdown command is that it warns people that the shutdown is going to occur.

[root@bigboy tmp]# shutdown -hy 0

Broadcast message from root (pts/0) (Sat Nov  6 13:15:27 2004):

The system is going down for system halt NOW!
[root@bigboy tmp]#

Reboot The System

You can also use the init command to reboot the system immediately by entering runlevel 6.

[root@bigboy tmp]# init 6

The "reboot" command has the same effect, but it also sends a warning message to all users.

[root@bigboy tmp]# reboot

Broadcast message from root (pts/0) (Sat Nov  6 12:39:31 2004):

The system is going down for reboot NOW!
[root@bigboy tmp]#

More graceful reboots can be done with the shutdown command using the -r switch and specifying a delay, which in this case is 10 minutes.

[root@bigboy root]# shutdown -ry 10

Broadcast message from root (pts/0) (Sat Nov  6 13:26:39 2004):

The system is going DOWN for reboot in 10 minutes!

Broadcast message from root (pts/0) (Sat Nov  6 13:27:39 2004):

The system is going DOWN for reboot in 9 minutes!
...
...
...
Broadcast message from root (pts/0) (Sat Nov  6 13:36:39 2004):

The system is going down for reboot NOW!

Entering Single-user Mode

Some activities require you to force the system to log off all users, third-party applications and networking so that only the systems administrator has access to the system from the VGA console. A typical scenario is the addition of a new hard disk, as mentioned in Chapter 27, "Expanding Disk Capacity", or the troubleshooting of a failed boot process.

Another reason is the recovery of your root password.

Switching to Single-user Mode

When the system is running normally, this can be done by using the init command to enter runlevel 1. It is best to do this from the console, because if you do it from a remote terminal session you'll be logged out.

[root@bigboy root]# init 1
...
...
bash-2.05b#

Unfortunately, this gives no prior warning to users, and the shutdown command doesn't have a single-user mode option. This can be overcome by running the shutdown command with a delay in minutes as the only argument.

[root@bigboy tmp]# shutdown 1

Broadcast message from root (pts/0) (Sat Nov  6 13:44:59 2004):

The system is going DOWN to maintenance mode in 1 minute!

Broadcast message from root (pts/0) (Sat Nov  6 13:45:59 2004):

The system is going down to maintenance mode NOW!

...
...
bash-2.05b#

Entering Single-user Mode At The Grub Splash Screen

You can enter single user mode directly after turning on the power to your system. The steps to do this are listed below.

1. Power on your system. Wait for the "Grub loading" message to appear and, depending on your Linux distribution, get ready to hit either any key or the ESC key to enter the grub boot menu.

Grub loading, please wait ...
Press ESC to enter the menu

or

Grub loading, please wait ...
Press any key to enter the menu

2. You will then get grub's main menu which will display a list of available kernels. Use the arrow keys to scroll to your desired version of the kernel and then press e for "edit".

Fedora Core (2.6.18-1.2239.fc5smp)
Fedora Core (2.6.18-1.2200.fc5smp)

3. The kernel's boot menu will appear. Use the arrow keys to scroll to the "kernel" line and then press e for "edit".

root (hd0,0)
kernel /vmlinuz-2.6.18-1.2239.fc5smp ro root=LABEL=/
initrd /initrd-2.6.18-1.2239.fc5smp.img

4. A grub edit prompt will appear. Use the arrow keys to move to the end of the line and add the word "single" to the end, separated by a space. Change

grub edit> kernel /vmlinuz-2.6.18-1.2239.fc5smp ro root=LABEL=/

to

grub edit> kernel /vmlinuz-2.6.18-1.2239.fc5smp ro root=LABEL=/ single

5. Press enter to save your changes, and then b for "boot". 6. The system will continue to boot, but will go straight to the root # prompt without first asking for a username and password.

Reverting To Your Default runlevel From Single User Mode

The exit command forces the system to exit runlevel 1 and revert to the default runlevel for the system. You can also use the init command (for example init 3 and init 5) to alter this default behavior:

bash-2.05b# exit
INIT: Entering runlevel: 3
...
...
...
Fedora Core release 2 (Tettnang)
Kernel 2.6.8-1.521 on an i686
bigboy login:

Root Password Recovery

Sometimes you might forget the root password, or the previous systems administrator may move on to a new job without giving it to you. To do this, follow these steps:

1. Go to the VGA console and press Ctrl-Alt-Del. The system will then shut down in an orderly fashion.
2. Reboot the system and enter single-user mode.
3. Once at the command prompt, change your password. Single user mode assumes the person at the console is the systems administrator root, so you don't have to specify a root username.
4. Return to your default runlevel by using the exit command.

Starting and Stopping Daemons

A simple definition of a daemon is a programs that runs unattended even when nobody is logged into your system. Common examples of daemons are the syslog daemon which receives system error messages and writes them to a log file; the apache or httpd daemon that serves web pages to Internet web browsers and the sendmail daemon that places email it receives into your inbox.

The startup scripts I have been mentioning in the /etc/init.d directory govern the activation of daemons that were installed with some of your Linux packages. The commands to start and stop them are universal.

Starting a Daemon

If a startup script exists in the /etc/init.d directory, then its daemon can be started by specifying its filename followed by the keyword "start" as seen here:

root@u-bigboy:~# /etc/init.d/apache start
* Starting apache 1.3 web server...
 ...done.
root@u-bigboy:~#

Stopping a Daemon

Daemons can be stopped by specifying its script filename followed by the keyword "stop":

root@u-bigboy:~# /etc/init.d/apache stop
* Stopping apache 1.3 web server...
 ...done.
root@u-bigboy:~#

Restarting a Daemon

Daemons usually only read their configuration files when they are started, therefore if you edit the file, you have to restart the daemon for the new settings to become active. This can be done with the keyword "restart":

root@u-bigboy:~# /etc/init.d/apache restart
* Restarting apache 1.3 web server...
 ...done.
root@u-bigboy:~#

Don’t worry about configuring your daemons. Later we'll be covering some commonly used daemons and will discuss them with ample examples.

The service command

Some operating systems such as Fedora and Redhat also come with the shortcut service command which allows you to control daemons with the "start", "stop" and "restart" keywords, but with less typing. Here are some quick, intuitive examples of doing this:

[root@bigboy ~]# service httpd start
[root@bigboy ~]# service httpd stop
[root@bigboy ~]# service httpd restart

The service command also has the "status" keyword which will provide a brief report on what the daemon is doing.

[root@bigboy ~]# service httpd status
httpd (pid 6135 6133 6132 6131 6130 6129 6128 6127 1561) is running...
[root@bigboy ~]#

Using chkconfig to Start Daemons at Each runlevel

As stated earlier, the chkconfig command can be used to adjust which applications start at each runlevel. You can use this command with the --list switch to get a full listing of packages listed in /etc/init.d and the runlevels at which they will be on or off:

[root@bigboy tmp]# chkconfig --list
keytable 0:off 1:on  2:on  3:on 4:on  5:on 6:off
atd      0:off 1:off 2:off 3:on 4:on  5:on 6:off
syslog   0:off 1:off 2:on  3:on 4:on  5:on 6:off
gpm      0:off 1:off 2:on  3:on 4:on  5:on 6:off
kudzu    0:off 1:off 2:off 3:on 4:on  5:on 6:off
wlan     0:off 1:off 2:on  3:on 4:on  5:on 6:off
sendmail 0:off 1:off 2:off 3:on 4:off 5:on 6:off
netfs    0:off 1:off 2:off 3:on 4:on  5:on 6:off
network  0:off 1:off 2:on  3:on 4:on  5:on 6:off
random   0:off 1:off 2:on  3:on 4:on  5:on 6:off
...
...

chkconfig Examples

You can use chkconfig to change runlevels for particular packages. Here we see sendmail will start with a regular startup at runlevel 3 or 5. Let's change it so that sendmail doesn't startup at boot.

Use Chkconfig to Get a Listing of sendmail's Current Startup Options

The chkconfig command can be used with grep to determine the run levels in which sendmail will run. Here we see it will run at levels 3 and 5.

[root@bigboy tmp]# chkconfig --list | grep mail
sendmail 0:off 1:off 2:off 3:on 4:off 5:on 6:off
[root@bigboy tmp]#

Switch Off sendmail Starting Up in Levels 3 and 5

The chkconfig command with the --level switch indicates that some action needs to be done at the runlevels entered as its values. The first argument in the command is the package you want to affect and the second defines whether you want it on or off. In this case we want sendmail not to be started when entering runlevels 3 and 5:

[root@bigboy tmp]# chkconfig --level 35 sendmail off
[root@bigboy tmp]#

By not specifying the runlevels with the --level switch, chckconfig will make the changes for runlevels 3 and 5 automatically:

[root@bigboy tmp]# chkconfig sendmail off

Because the intention is to permanently shutdown sendmail permanently, we might also have to stop it from running now.

[root@bigboy tmp]# service sendmail stop
Shutting down sendmail: [  OK  ]
Shutting down sm-client: [  OK  ]
[root@bigboy tmp]#

Double-check that sendmail Will Not Start Up

We can then use chkconfig to double-check our work.

[root@bigboy tmp]# chkconfig --list | grep mail
sendmail 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@bigboy tmp]#

Turn On sendmail Again

To reactivate sendmail, we can use chkconfig once more, but with the on argument. Start sendmail again to get it running immediately, not just after the next reboot.

[root@bigboy tmp]# chkconfig sendmail on
[root@bigboy tmp]# chkconfig --list | grep mail
sendmail 0:off 1:off 2:off 3:on 4:off 5:on 6:off
[root@bigboy tmp]# service sendmail start
Starting sendmail: [  OK  ]
Starting sm-client: [  OK  ]
[root@bigboy tmp]#

Using chkconfig to Improve Security

A default Fedora installation automatically starts a number of daemons that you may not necessarily need for a Web server. This usually results in your system listening on a variety of unexpected TCP/IP ports that could be used as doors into your system by hackers.

The screen output of the netstat -an command below shows a typical case. Some ports are relatively easy to recognize. TCP ports 25 and 22 are for mail and SSH, respectively, but some others are less obvious. Should you use the chkconfig command and the scripts in the /etc/init.d directory to shut these down permanently?

[root@bigboy tmp]# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address
State
tcp    0    0 0.0.0.0:32768    0.0.0.0:*     LISTEN
tcp    0    0 127.0.0.1:32769  0.0.0.0:*     LISTEN
tcp    0    0 0.0.0.0:111      0.0.0.0:*     LISTEN
tcp    0    0 127.0.0.1:631    0.0.0.0:*     LISTEN
tcp    0    0 127.0.0.1:25     0.0.0.0:*     LISTEN
tcp    0    0 :::22            :::*          LISTEN
udp    0    0 0.0.0.0:32768    0.0.0.0:*
udp    0    0 0.0.0.0:930      0.0.0.0:*
udp    0    0 0.0.0.0:68       0.0.0.0:*
udp    0    0 0.0.0.0:111      0.0.0.0:*
udp    0    0 0.0.0.0:631      0.0.0.0:*
...
...
[root@bigboy tmp]#

For example, how do you know which startup script is responsible for TCP port 111? The answer is to use the lsof command which lists all open, or actively used, files and can be given additional options to extend its scope to include the TCP/IP protocol stack.

In the next examples we see that TCP ports 111 and 32769, and UDP port 123 are being used by the portmap, xinetd and ntp daemons respectively. The portmap daemon is required for the operation of NFS and NIS, topics that are covered in Chapters 29, "Remote Disk Access with NFS", and 30, "Configuring NIS". portmap also has many known security flaws that makes it advisable to be run on a secured network. If you don't need any of these three applications, it's best to shut down portmap permanently. NTP, which is covered in Chapter 24, "The NTP Server", is required for synchronizing your time with a reliable time source, and may be necessary. A number of network applications are reliant on xinetd, as explained in Chapter 16, "Telnet, TFTP, and xinetd", and it might be required for their operation:

[root@ bigboy tmp]# lsof -i tcp:111
COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
portmap 1165  rpc    4u  IPv4   2979       TCP *:sunrpc (LISTEN)
[root@ bigboy tmp #

[root@bigboy tmp]# lsof -i tcp:32769
COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
xinetd  1522 root    5u  IPv4   2764       TCP probe-001:32769 (LISTEN)
[root@bigboy tmp]#

[root@bigboy root]# lsof -i udp:123
COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
ntpd    1321  ntp    4u  IPv4   3390       UDP *:ntp
...
...
[root@bigboy root]#

In some cases it's tricky to determine the application based on the results of the lsof command. In the example below, we've discovered that TCP port 32768 is being used by rpc.statd, but there is no rpc.statd file in the /etc/init.d directory. The simple solution is to use the grep command to search all the files for the string rpc.statd to determine which one is responsible for its operation. We soon discover that the nfslock daemon uses it. If you don't need nfslock, then shut it down permanently.

[root@bigboy tmp]# lsof -i tcp:32768
COMMAND    PID    USER   FD   TYPE DEVICE SIZE NODE NAME
rpc.statd 1178 rpcuser    6u  IPv4   2400       TCP *:32768 (LISTEN)
[root@bigboy tmp]# ls /etc/init.d/rpc.statd
ls: /etc/init.d/rpc.statd: No such file or directory
[root@bigboy tmp]# grep -i statd /etc/init.d/*
/etc/init.d/nfslock:[ -x /sbin/rpc.statd ] || exit 0
...
...
[root@bigboy tmp]#

As a rule of thumb, applications listening only on the loopback interface (IP address 127.0.0.1) are usually the least susceptible to network attack and probably don't need to be stopped for network security reasons. Those listening on all interfaces, depicted as IP address 0.0.0.0, are naturally more vulnerable and their continued operation should be dependent on your server's needs. I usually shutdown nfs, nfslock, netfs, portmap, and cups printing as standard practice on Internet servers. I keep sendmail running as it is always needed to send and receive mail (see Chapter 21, "Configuring Linux Mail Servers", for details). Your needs may be different.

Remember to thoroughly research your options thoroughly before choosing to shut down an application. Use the Linux man pages, reference books and the Internet for information. Unpredictable results are always undesirable.

Shutting down applications is only a part of server security. Firewalls, physical access restrictions, password policies, and patch updates need to be considered. Full coverage of server and network security is beyond the scope of this book, but you should always have a security reference guide on hand to guide your final decisions.

Final Tips on chkconfig

Remember the following:

• In most cases, you want to modify runlevels 3 and 5 simultaneously and with the same values.
• Don't add/remove anything to other runlevels unless you absolutely know what you are doing. Don't experiment, unless in a test environment.
• chkconfig doesn't start the programs in the /etc/init.d directory, it just configures them to be started or ignored when the system boots up. The commands for starting and stopping the programs covered in this book are covered in each respective chapter.

Using sysv-rc-conf to Start Daemons at Each runlevel

With Debian / Ubuntu Linux, the update-rc.d command replaces chkconfig as the default package for modifying /etc/init.d script links. Unfortunately, the utility was written to facilitate link modification when packages are installed or removed, but is less friendly when you need to alter links for existing packages you want to keep.

Fortunately there is hope for the harried systems administrator in the form of the sysv-rc-conf package which uses an almost identical syntax to chkconfig, and has a GUI mode if you run it from the command line without any arguments. This section will show you some important tips in using sysv-rc-conf.

Installing sysv-rc-conf

The sysv-rc-conf package can be installed easily using apt-get. Here is an example.

root@u-bigboy:~# apt-get install sysv-rc-conf

Listing the runlevels for Daemons

This can be done with the --list option. In this example below we get a listing for only the apache daemon.

root@u-bigboy:~#  sysv-rc-conf  --list apache
apache       0:off      1:off   2:on    3:on    4:on    5:on    6:off
root@u-bigboy:~#

Here we get a listing for everything.

root@u-bigboy:~#  sysv-rc-conf  --list
acpi-support 0:off      1:off   2:on    3:on    4:on    5:on    6:off
acpid        0:off      1:off   2:on    3:on    4:on    5:on    6:off
alsa-utils   0:off      6:off
…
…
…
vbesave      2:on       3:on    4:on    5:on
x11-common   S:on
root@u-bigboy:~#

Setting the runlevels for Daemons

The sysv-rc-conf program has further similarities to the chkconfig syntax. Here we set the apache daemon to start automatically at levels 2 through 5.

root@u-bigboy:~# sysv-rc-conf  apache on

Similarly, we can set the apache daemon not to start at levels 2 through 5 with this command:

root@u-bigboy:~# sysv-rc-conf  apache off

Finally we can use set the apache daemon to start only at levels 3 and 5.

root@u-bigboy:~# sysv-rc-conf  --level 35 apache on

Conclusion

The topics discussed in this chapter might seem simple, but like syslog, which was covered in Chapter 5, "Troubleshooting Linux with syslog", they are an essential part of Linux administration that gets frequently overlooked especially when new software is installed.

Whenever possible, always try to reboot your system to make sure all the newly installed applications start up correctly. Sometimes they start but give errors listed only in the /var/log directory. Taking the time to configure and test your startup scripts could prevent you from being awakened in the middle of the night while you are on vacation! It is really important.

Troubleshooting Linux with syslog

Introduction

There are hundreds of Linux applications on the market, each with their own configuration files and help pages. This variety makes Linux vibrant, but it also makes Linux system administration daunting. Fortunately, in most cases, Linux applications use the syslog utility to export all their errors and status messages to files located in the /var/log directory.

This can be invaluable in correlating the timing and causes of related events on your system. It is also important to know that applications frequently don't display errors on the screen, but will usually log them somewhere. Knowing the precise message that accompanies an error can be vital in researching malfunctions in product manuals, online documentation, and Web searches.

syslog, and the logrotate utility that cleans up log files, are both relatively easy to configure but they frequently don't get their fair share of coverage in most texts. I've included syslog here as a dedicated chapter to both emphasize its importance to your Linux knowledge and prepare you with a valuable skill that will help you troubleshoot all the Linux various applications that will be presented throughout the book

syslog

syslog is a utility for tracking and logging all manner of system messages from the merely informational to the extremely critical. Each system message sent to the syslog server has two descriptive labels associated with it that makes the message easier to handle.

• The first describes the function (facility) of the application that generated it. For example, applications such as mail and cron generate messages with easily identifiable facilities named mail and cron.

• The second describes the degree of severity of the message. There are eight in all and they are listed in Table 5-1:

You can configure syslog's /etc/syslog.conf configuration file to place messages of differing severities and facilities in different files. This procedure will be covered next.

Table 5-1 Syslog Facilities

Severity Level	Keyword	Description
0	emergencies	System unusable
1	alerts	Immediate action required
2	critical	Critical condition
3	errors	Error conditions
4	warnings	Warning conditions
5	notifications	Normal but significant conditions
6	informational	Informational messages
7	debugging	Debugging messages

The /etc/syslog.conf File

The files to which syslog writes each type of message received is set in the /etc/syslog.conf configuration file. This file consists of two columns. The first lists the facilities and severities of messages to expect and the second lists the files to which they should be logged. By default, RedHat/Fedora's /etc/syslog.conf file is configured to put most of the messages in the file /var/log/messages. Here is a sample:

*.info;mail.none;authpriv.none;cron.none           /var/log/messages

In this case, all messages of severity "info" and above are logged, but none from the mail, cron or authentication facilities/subsystems. You can make this logging even more sensitive by replacing the line above with one that captures all messages from debug severity and above in the /var/log/messages file. This example may be more suitable for troubleshooting.

*.debug                                          /var/log/messages

In this example, all debug severity messages; except auth, authpriv, news and mail; are logged to the /var/log/debug file in caching mode. Notice how you can spread the configuration syntax across several lines using the slash (\) symbol at the end of each line.

 *.=debug;\
     auth,authpriv.none;\
     news.none;mail.none     -/var/log/debug

Here we see the /var/log/messages file configured in caching mode to receive only info, notice and warning messages except for the auth, authpriv, news and mail facilities.

*.=info;*.=notice;*.=warn;\
     auth,authpriv.none;\
     cron,daemon.none;\
     mail,news.none          -/var/log/messages

You can even have certain types of messages sent to the screen of all logged in users. In this example messages of severity emergency and above triggers this type of notification. The file definition is simply replaced by an asterisk to make this occur.

*.emerg                         *

Certain applications will additionally log to their own application specific log files and directories independent of the syslog.conf file. Here are some common examples:

Files:

/var/log/maillog             : Mail
/var/log/httpd/access_log    : Apache web server page access logs

Directories:

/var/log
/var/log/samba                      : Samba messages
/var/log/mrtg                       : MRTG messages
/var/log/httpd                      : Apache webserver messages

Note: In some older versions of Linux the /etc/syslog.conf file was very sensitive to spaces and would recognize only tabs. The use of spaces in the file would cause unpredictable results. Check the formatting of your /etc/syslog.conf file to be safe.

Activating Changes to the syslog Configuration File

Changes to /etc/syslog.conf will not take effect until you restart syslog. Issue this command to do so:

[root@bigboy tmp]# service syslog restart

In Ubuntu / Debian systems the command would be:

root@u-bigboy:~# /etc/init.d/sysklogd restart

How to View New Log Entries as They Happen

If you want to get new log entries to scroll on the screen as they occur, then you can use this command:

[root@bigboy tmp]# tail -f /var/log/messages

Similar commands can be applied to all log files. This is probably one of the best troubleshooting tools available in Linux. Another good command to use apart from tail is grep. grep will help you search for all occurrences of a string in a log file; you can pipe it through the more command so that you only get one screen at a time. Here is an example:

[root@bigboy tmp]# grep string /var/log/messages | more

You can also just use the plain old more command to see one screen at a time of the entire log file without filtering with grep. Here is an example:

[root@bigboy tmp]# more /var/log/messages

Logging syslog Messages to a Remote Linux Server

Logging your system messages to a remote server is a good security practice. With all servers logging to a central syslog server, it becomes easier to correlate events across your company. It also makes covering up mistakes or malicious activities harder because the purposeful deletion of log files on a server cannot simultaneously occur on your logging server, especially if you restrict the user access to the logging server.

Configuring the Linux Syslog Server

By default syslog doesn't expect to receive messages from remote clients. Here's how to configure your Linux server to start listening for these messages.

As we saw previously, syslog checks its /etc/syslog.conf file to determine the expected names and locations of the log files it should create. It also checks the file /etc/sysconfig/syslog to determine the various modes in which it should operate. Syslog will not listen for remote messages unless the SYSLOGD_OPTIONS variable in this file has a -r included in it as shown below.

# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages received with -r
# See syslogd(8) for more details

SYSLOGD_OPTIONS="-m 0 -r"

# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
#    once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details

KLOGD_OPTIONS="-2"

Note: In Debian / Ubuntu systems you have to edit the syslog startup script /etc/init.d/sysklogd directly and make the SYSLOGD variable definition become "-r".

# Options for start/restart the daemons
#   For remote UDP logging use SYSLOGD="-r"
#
#SYSLOGD="-u syslog"
SYSLOGD="-r"

You will have to restart syslog on the server for the changes to take effect. The server will now start to listen on UDP port 514, which you can verify using either one of the following netstat command variations.

[root@bigboy tmp]# netstat -a | grep syslog
udp        0      0 *:syslog                *:*
[root@bigboy tmp]# netstat -an | grep 514
udp        0      0 0.0.0.0:514             0.0.0.0:*
[root@bigboy tmp]#

Configuring the Linux Client

The syslog server is now expecting to receive syslog messages. You have to configure your remote Linux client to send messages to it. This is done by editing the /etc/hosts file on the Linux client named smallfry. Here are the steps:

1) Determine the IP address and fully qualified hostname of your remote logging host.

2) Add an entry in the /etc/hosts file in the format:

IP-address    fully-qualified-domain-name    hostname    "loghost"

Example:

192.168.1.100    bigboy.my-site.com    bigboy     loghost

Now your /etc/hosts file has a nickname of "loghost" for server bigboy.

3) The next thing you need to do is edit your /etc/syslog.conf file to make the syslog messages get sent to your new loghost nickname.

*.debug                                       @loghost
*.debug                                       /var/log/messages

You have now configured all debug messages and higher to be logged to both server bigboy ("loghost") and the local file /var/log/messages. Remember to restart syslog to get the remote logging started.

You can now test to make sure that the syslog server is receiving the messages with a simple test such as restarting the lpd printer daemon and making sure the remote server sees the messages.

Linux Client

[root@smallfry tmp]# service lpd restart
Stopping lpd: [  OK  ]
Starting lpd: [  OK  ]
[root@smallfry tmp]#

Linux Server

[root@bigboy tmp]# tail /var/log/messages
...
...
Apr 11 22:09:35 smallfry lpd: lpd shutdown succeeded
Apr 11 22:09:39 smallfry lpd: lpd startup succeeded
...
...
[root@bigboy tmp]#

Syslog Configuration and Cisco Network Devices

syslog reserves facilities "local0" through "local7" for log messages received from remote servers and network devices. Routers, switches, firewalls and load balancers each logging with a different facility can each have their own log files for easy troubleshooting. Appendix 4 has examples of how to configure syslog to do this with Cisco devices using separate log files for the routers, switches, PIX firewalls, CSS load balancers and LocalDirectors.

Logrotate

The Linux utility logrotate renames and reuses system error log files on a periodic basis so that they don't occupy excessive disk space.

The /etc/logrotate.conf File

This is logrotate's general configuration file in which you can specify the frequency with which the files are reused.

• You can specify either a weekly or daily rotation parameter. In the case below the weekly option is commented out with a #, allowing for daily updates.
• The rotate parameter specifies the number of copies of log files logrotate will maintain. In the case below the 4 copy option is commented out with a #, while allowing 7 copies.
• The create parameter creates a new log file after each rotation

Therefore, our sample configuration file will create daily archives of all the logfiles and store them for seven days. The files will have the following names with, logfile being current active version:

logfile
logfile.0
logfile.1
logfile.2
logfile.3
logfile.4
logfile.5
logfile.6

Sample Contents of /etc/logrotate.conf

# rotate log files weekly
#weekly

# rotate log files daily
daily

# keep 4 weeks worth of backlogs
#rotate 4

# keep 7 days worth of backlogs
rotate 7

# create new (empty) log files after rotating old ones
create

The /etc/logrotate.d Directory

Most Linux applications that use syslog will put an additional configuration file in this directory to specify the names of the log files to be rotated. It is a good practice to verify that all new applications that you want to use the syslog log have configuration files in this directory. Here are some sample files that define the specific files to be rotated for each application.

Here is an example of a custom file located in this directory that rotates files with the .tgz extension which are located in the /data/backups directory. The parameters in this file will override the global defaults in the /etc/logrotate.conf file. In this case, the rotated files won't be compressed, they'll be held for 30 days only if they are not empty, and they will be given file permissions of 600 for user root.

/data/backups/*.tgz {

 daily
 rotate 30
 nocompress
 missingok
 notifempty
 create 0600 root root
}

Note: In Debian / Ubuntu systems the /etc/cron.daily/sysklogd script reads the /etc/syslog.conf file and rotates any log files it finds configured there. This eliminates the need to create log rotation configuration files for the common system log files in the /etc/logrotate.d directory. As the script resides in the /etc/cron.daily directory it automatically runs every 24 hours. In Fedora / Redhat systems this script is replaced by the /etc/cron.daily/logrotate daily script which does not use the contents of the syslog configuration file, relying mostly on the contents of the /etc/logrotate.d directory.

Activating logrotate

The above logrotate settings in the previous section will not take effect until you issue the following command:

[root@bigboy tmp]# logrotate -f

If you want logrotate to reload only a specific configuration file, and not all of them, then issue the logrotate command with just that filename as the argument like this:

[root@bigboy tmp]# logrotate -f /etc/logrotate.d/syslog

Compressing Your Log Files

On busy Web sites the size of your log files can become quite large. Compression can be activated by editing the logrotate.conf file and adding the compress option.

#
# File: /etc/logrotate.conf
#

# Activate log compression

compress

The log files will then start to become archived with the gzip utility, each file having a .gz extension.

[root@bigboy tmp]# ls /var/log/messages*
/var/log/messages      /var/log/messages.1.gz /var/log/messages.2.gz
/var/log/messages.3.gz /var/log/messages.4.gz /var/log/messages.5.gz
/var/log/messages.6.gz /var/log/messages.7.gz
[root@bigboy tmp]#

Viewing the contents of the files still remains easy because the zcat command can quickly output their contents to the screen. Use the command with the compressed file's name as the argument as seen below.

[root@bigboy tmp]# zcat /var/log/messages.1.gz
...
...
Nov 15 04:08:02 bigboy httpd: httpd shutdown succeeded
Nov 15 04:08:04 bigboy httpd: httpd startup succeeded
Nov 15 04:08:05 bigboy sendmail[6003]: iACFMLHZ023165: to=,  delay=2+20:45:44, xdelay=00:00:02, mailer=esmtp, pri=6388168, relay=www.clematis4spiders.info. [222.134.66.34], dsn=4.0.0, stat=Deferred: Connection refused by www.clematis4spiders.info.
[root@bigboy tmp]#

syslog-ng

The more recent syslog-ng application combines the features of logrotate and syslog to create a much more customizable and feature rich product. This can be easily seen in the discussion of its configuration file that follows.

The /etc/syslog-ng/syslog-ng.conf file

The main configuration file for syslog-ng is the /etc/syslog-ng/sylog-ng.conf file but only rudimentary help on its keywords can be found using the Linux man pages.

[root@zippy tmp]# man syslog-ng.conf

Figure 5-1 has a sample syslog-ng.conf file and outlines some key features. The options section that covers global characteristics is fully commented, but it is the source, destination and log sections that define the true strength of the customizability of syslog-ng.

Figure 5-1 A Sample syslog-ng.conf File

options {

        # Number of syslog lines stored in memory before being written to files
        sync (0);

        # Syslog-ng uses queues
        log_fifo_size (1000);

        # Create log directories as needed
        create_dirs (yes);

        # Make the group "logs" own the log files and directories
        group (logs);
        dir_group (logs);

        # Set the file and directory permissions
        perm (0640);
        dir_perm (0750);

        # Check client hostnames for valid DNS characters
        check_hostname (yes);

        # Specify whether to trust hostname in the log message.
        # If "yes", then it is left unchanged, if "no" the server replaces
        # it with client's DNS lookup value.
        keep_hostname (yes);

        # Use DNS fully qualified domain names (FQDN)
        # for the names of log file folders
        use_fqdn (yes);
        use_dns (yes);

        # Cache DNS entries for up to 1000 hosts for 12 hours
        dns_cache (yes);
        dns_cache_size (1000);
        dns_cache_expire (43200);

      };


# Define all the sources of localhost generated syslog
# messages and label it "d_localhost"
source s_localhost {
        pipe ("/proc/kmsg" log_prefix("kernel: "));
        unix-stream ("/dev/log");
        internal();
};

# Define all the sources of network generated syslog
# messages and label it "d_network"
source s_network {
        tcp(max-connections(5000));
        udp();
};

# Define the destination "d_localhost" log directory
destination d_localhost {
         file ("/var/log/syslog-ng/$YEAR.$MONTH.$DAY/localhost/$FACILITY.log");
};

# Define the destination "d_network" log directory
destination d_network {
        file ("/var/log/syslog-ng/$YEAR.$MONTH.$DAY/$HOST/$FACILITY.log");
};

# Any logs that match the "s_localhost" source should be logged
# in the "d_localhost" directory

log { source(s_localhost);
    destination(d_localhost);
};

# Any logs that match the "s_network" source should be logged
# in the "d_network" directory

log { source(s_network);
     destination(d_network);
};

In our example, the first set of sources is labeled s_localhost. It includes all system messages sent to the Linux /dev/log device, which is one of syslog's data sources, all messages that syslog-ng views as being of an internal nature and additionally inserts the prefix "kernel" to all messages it intercepts on their way to the /proc/kmsg kernel message file.

Unlike a regular syslog server which listens for client messages on UDP port 514, syslog-ng also listens on TCP port 514. The second set of sources is labeled s_network and includes all syslog messages obtained from UDP sources and limits TCP syslog connections to 5000. Limiting the number of connections to help regulate system load is a good practice in the event that some syslog client begins to inundate your server with messages.

Our example also has two destinations for syslog messages, one named d_localhost, the other, d_network. These examples show the flexibility of syslog-ng in using variables. The $YEAR, $MONTH and $DAY variables map to the current year, month and day in YYYY, MM and DD format respectively. Therefore the example:

/var/log/syslog-ng/$YEAR.$MONTH.$DAY/$HOST/$FACILITY.log

refers to a directory called /var/log/syslog-ng/2005.07.09 when messages arrive on July 9, 2005. The $HOST variable refers to the hostname of the syslog client and will map to the client's IP address if DNS services are deactivated in the options section of the syslog-ng.conf file. Similarly the $FACILITY variable refers to the facility of the syslog messages that arrive from that host.

Installing syslog-ng

The most recent syslog-ng and its companion eventlog tar files can be downloaded from the www.balabit.com website. The installation procedure is straightforward, but you will need to have the Linux gcc C programming language compiler preinstalled to be successful. Here are the steps.

1. Download the tar files from the BalaBit website. In this case we have browsed the website beforehand and know the exact URLs to use with the wget command.

[root@zippy tmp]# wget wget http://www.balabit.com/downloads/syslog-ng/2.0/src/eventlog-0.2.5.tar.gz
--12:34:17--  wget http://www.balabit.com/downloads/syslog-ng/2.0/src/eventlog-0.2.5.tar.gz
         => `eventlog-0.2.5.tar.gz'
...
...
...

12:34:19 (162.01 KB/s) - `eventlog-0.2.5.tar.gz' saved [345231]

[root@zippy tmp]# wget http://www.balabit.com/downloads/syslog-ng/2.0/src/syslog-ng-2.0.0.tar.gz
--12:24:21--  wget http://www.balabit.com/downloads/syslog-ng/2.0/src/syslog-ng-2.0.0.tar.gz
         => ` syslog-ng-2.0.0.tar.gz'
...
...
...

12:24:24 (156.15 KB/s) - ` syslog-ng-2.0.0.tar.gz' saved [383589]

[root@zippy tmp]#

2. Install the prerequisite glib libraries.

[root@zippy tmp]# yum -y install glib

3. Using the tar command we extract the files in the pre-requisite eventlog archive and then use the configure; make and make install commands to install them correctly. Pay special attention to the output of the configure command to make sure that all the pre-installation tests are passed. If not, install the packages the error messages request and then start again.

[root@zippy tmp]# tar -xzf eventlog-0.2.5.tar.gz
[root@zippy tmp]# cd eventlog-0.2.5
[root@zippy eventlog-0.2.5]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
...
...
...
[root@zippy eventlog-0.2.5]# make
Making all in utils
make[1]: Entering directory `/tmp/eventlog-0.2.5/utils'
sed -e "s,_SCSH_,/usr/bin/scsh," make_class.in >make_class
...
...
...
[root@zippy eventlog-0.2.5]# make install
Making install in utils
make[1]: Entering directory `/tmp/eventlog-0.2.5/utils'
make[2]: Entering directory `/tmp/eventlog-0.2.5/utils'
...
...
...
make[2]: Leaving directory `/tmp/eventlog-0.2.5'
make[1]: Leaving directory `/tmp/eventlog-0.2.5'
[root@zippy eventlog-0.2.5]#

4. The next step is to install the prerequisite glib package on your system.

[root@zippy eventlog-0.2.5]# yum -y install glib

5. Some environmental variables also need to be set prior to the installation of the syslog-ng files.

[root@zippy eventlog-0.2.5]# PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
[root@zippy eventlog-0.2.5]# export PKG_CONFIG_PATH

6. Using the tar command we extract the files in the pre-requisite syslog-ng archive and then use the configure, make clean, make and make install commands to install them correctly. In this case we the --sysconfdir directive with the configure command to make sure syslog-ng searches for its configuration file in the /etc directory. Once again, pay close attention to the pre-installation tests that the configure command executes.

[root@zippy eventlog-0.2.5]# cd /tmp
[root@zippy tmp]# tar -xzf syslog-ng-2.0.0.tar.gz
[root@zippy tmp]# cd syslog-ng-2.0.0
[root@zippy syslog-ng-2.0.0]# make clean
[root@zippy syslog-ng-2.0.0]# ./configure --sysconfdir=/etc
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
...
...
...
[root@zippy syslog-ng-2.0.0]# make; make install
Making all in src
make[1]: Entering directory `/tmp/ syslog-ng-2.0.0/src'
...
...
...
[root@zippy syslog-ng-2.0.0]#

7. The installation has template init.d/syslog-ng scripts and syslog-ng.conf files in the contribs/ directory.

[root@zippy syslog-ng-2.0.0]# ls contrib/
fedora-packaging  init.d.RedHat-7.3      init.d.SuSE
Makefile.in       rhel-packaging         syslog-ng.conf.HP-UX
syslog-ng.vim     init.d.HP-UX           init.d.solaris   
Makefile          README                 syslog2ng         
init.d.RedHat     syslog-ng.conf.RedHat  init.d.SunOS
Makefile.am       relogger.pl            syslog-ng.conf.doc
syslog-ng.conf.SunOS
[root@zippy syslog-ng-2.0.0]#

8. Copy the versions for your operating system to the /etc/init.d and /etc , /etc/logrotate.d , /etc/sysconfig directories. The /etc/syslog-ng/ directory needs to be created beforehand. Redhat and Fedora installations have their own subdirectories contrib/.

[root@zippy syslog-ng-2.0.0]# mkdir /etc/syslog-ng/
[root@zippy syslog-ng-2.0.0]# cp contrib/fedora-packaging/syslog-ng.init \
 /etc/init.d/syslog-ng
[root@zippy syslog-ng-2.0.0]# cp contrib/fedora-packaging/syslog-ng.conf \
 /etc
[root@zippy syslog-ng-2.0.0]# cp contrib/fedora-packaging/syslog-ng.sysconfig \
 /etc/sysconfig/syslog-ng
[root@zippy syslog-ng-2.0.0]# cp contrib/fedora-packaging/syslog-ng.logrotate \
 /etc/logrotate.d/syslog-ng

Remember that you may want to customize your syslog-ng.conf file.

9. Change the permissions on your new /etc/inid.d/syslog-ng file.

[root@zippy syslog-ng-2.0.0]# chmod 755 /etc/init.d/syslog-ng

10. You need to be careful. The init.d script may refer to a syslog-ng binary file that's in an incorrect location. Find its true location and edit the script.

[root@zippy syslog-ng-2.0.0]# updatedb
[root@zippy syslog-ng-2.0.0]# locate syslog-ng | grep bin
/usr/local/sbin/syslog-ng
[root@zippy syslog-ng-2.0.0]# vi /etc/init.d/syslog-ng
...
#exec="/sbin/syslog-ng"
exec="/usr/local/sbin/syslog-ng"
...
:wq
[root@zippy syslog-ng-2.0.0]#

11. Next create the /etc/syslog-ng directory for the configuration files and the /var/log/syslog-ng directory for the log files.

[root@zippy syslog-ng-2.0.0]# chkconfig syslog off
[root@zippy syslog-ng-2.0.0]# chkconfig syslog-ng on
[root@zippy syslog-ng-2.0.0]# service syslog stop
Shutting down kernel logger: [  OK  ]
Shutting down system logger: [  OK  ]
[root@zippy syslog-ng-2.0.0]# service syslog-ng start
syslog-ng: unrecognized service
[root@zippy syslog-ng-2.0.0]#

12. The sample syslog-ng.conf file in Figure 5-1 was configured to have all directories owned by the group logs. This user group needs to be created and any users that need access to the directories need to added to this group using the usermod command. In this case the user peter is added to the group and the groups command is used to verify success.

[root@zippy tmp]# groupadd logs
[root@zippy tmp]# usermod -G logs peter
[root@zippy tmp]# groups peter
peter: users logs
[root@zippy tmp]# usermod -G logs peter

13. You can now configure syslog-ng to start on the next reboot with the chkconfig command and then use the service command to start it immediately. Remember to stop the old syslog process beforehand.

[root@zippy tmp]# service syslog stop
Shutting down kernel logger: [  OK  ]
Shutting down system logger: [  OK  ]
[root@zippy tmp]# chkconfig syslog off
[root@zippy tmp]# chkconfig syslog-ng on
[root@zippy tmp]# service syslog-ng start
Starting system logger: [  OK  ]
Starting kernel logger: [  OK  ]
[root@zippy tmp]#

14. Now, your remote hosts should log begin logging to the /var/log/syslog-ng directory. According to our preliminary configuration file, there should be sub-directories categorized by date inside it. Each of these sub-directories in turn will have directories beneath them named after the IP address and/or hostname of the various remote syslog clients and will contain files categorized by syslog facility. In this example we see that the 2005.07.09 directory as received messages from three hosts, 192.168.1.1, 192.168.1.100 and localhost.

[root@zippy tmp]# ls /var/log/syslog-ng/
2005.07.09
[root@zippy tmp]# ll /var/log/syslog-ng/2005.07.09/
drwxr-x---  2 root  logs  4096 Jul  9 17:01 192-168-1-1.my-web-site.org
drwxr-x---  2 root  logs  4096 Jul  9 16:45 192-168-1-99.my-web-site.org
drwxr-x---  2 root  logs  4096 Jul  9 23:24 LOGGER
[root@zippy tmp]# ls /var/log/syslog-ng/2005.07.09/localhost/
cron.log  kern.log  local7.log  syslog.log
[root@zippy tmp]#

Using syslog-ng your system can now be used as a much more customizable tool to help troubleshoot devices attached to your network. Each day syslog-ng will automatically create new sub-directories to match the current date and at the end of each calendar quarter the files will be moved to a special archive directory containing all the data for the previous three months. This archived data can then be periodically deleted as needed. For very large deployments, or for better searching and correlation capabilities, it is possible to send the output of syslog-ng to a SQL type database. This is beyond the scope of this book, but it is a worthwhile feature to keep in mind.

Configuring syslog-ng Clients

Clients logging to the syslog-ng server don't need to have syslog-ng installed on them, a regular syslog client configuration will suffice.

Simple syslog Security

One of the shortcomings of a syslog server is that it doesn't filter out messages from undesirable sources. It is therefore wise to implement the use of TCP wrappers or a firewall to limit the acceptable sources of messages when your server isn't located on a secure network. This will help to limit the effectiveness of syslog based denial of service attacks aimed at filling up your server's hard disk or taxing other system resources that could eventually cause the server to crash.

Remember that regular syslog servers listen on UDP port 514 and syslog-ng servers rely on port 514 for both UDP and TCP. Please refer to Chapter 14, "Linux Firewalls Using iptables", on Linux firewalls for details on how to configure the Linux iptables firewall application and Appendix I, "Miscellaneous Linux Topics", for further information on configuring TCP wrappers.

Conclusion

In the next chapter we cover the installation of Linux applications, and the use of syslog will become increasingly important especially in the troubleshooting of Linux-based firewalls which can be configured to ignore and then log all undesirable packets; the Apache Web server which logs all application programming errors generated by some of the popular scripting languages such as PERL and PHP; and finally, Linux mail whose configuration files are probably the most frequently edited system documents of all and which correspondingly suffer from the most mistakes.