Debian stable Web Server step-by-step

This is a series of specific steps for setting up Debian stable as a nice PHP web server. You are welcome to correct any mistakes or errors, improve the content, or add anything that seems to be missing.

Installation onto a local server

Run memtest overnight

First run Memtest86+ on the machine overnight, to test that the CPU and memory are error-free. Also run a CPU stress test for 5 to 10 minutes to check CPU is reliable and adequately cooled. The easiest way to download and run these memory and CPU testing tools is with the Ultimate Boot CD.

Also use System Rescue CD to check bad blocks on the hard drive, before trusting it with any data: badblocks -v -s -w -c 4096 /dev/sda

If the machine does not pass memtest and CPU stress and HDD tests without any errors or warnings at all, then do NOT proceed any further until this is rectified.

Install Debian Stable

• Boot from latest stable Debian Netinst CD
• Enter (default installation)
• Lang: English [default]
• Country: Australia
• Keymap: American English [default]
• Primary network interface: Eth0 [default]
• "Go back" after network autodetection
• Configure network manually
• 192.168.0.100 as IP address, and then take the defaults for netmask, gateway, and DNS.
• Hostname: netserver
• Domain name: YourHostName.com
• Partitioning method: Guided - yes entire disk [default]
• Disk to partition: "SCSI3 (1,0,0) (sda) - 18.2 Gb MegaRAID LD 0 RAID1 17G" [default, only choice - only applies if you have an HP netserver 1000R with a NetRAID card]
• Partitioning scheme: All files in one partition [default]
• Finish partitioning, and write changes to disk [default]
• Yes, write changes to disk. [default]
• City: Sydney [default]
• Root password: standard
• Extra Username: nickj
• Extra Username Password: standard
• Use a network mirror: Yes [default]
• Australian mirror [default]
• use: ftp.au.debian.org [default]
• Proxy: leave blank [default]
• Participate in package user survey: No [default]
• Choose software to install: Web server + File server + Mail server + SQL database + Standard system
• Continue installing libc-client without maildir support: Yes [default]
• Workgroup name: WORKGROUP
• Modify smb.conf to use WINS settings for DHCP: Yes
• Install the GRUB boot loader to the master boot record: Yes [default]
• remove CD
• Continue [default]
• Machine reboots
• Check boots okay, login as root, check networking okay, check SSH running.
• SSH into the box, and can now do remaining stuff below remotely via SSH.

Update /etc/apt/sources.list

"editor /etc/apt/sources.list", and comment out the "deb cdrom:" line or lines.

This makes all data come off the Internet, which is what we want.

Alternative installation for VPS

VPS machines tend to already come with a very very minimal installation of Debian stable pre-installed, so they don't need the OS installed. However, in order to get some of the basic system configuration that you'd normally do whilst installing locally, some additional steps may be required, namely:

Set the locale

aptitude install locales
dpkg-reconfigure locales

• enable all of the EN_US, EN_AU, and EN_GB locals.
• Set the en_US.UTF-8 local as the default.

Go from singe DHCP IP to multiple static IP addresses

If you have purchased multiple IP addresses, you can activate these, as well as change from DHCP to static IPs, by doing "editor /etc/network/interfaces", and entering the network configuration there, such as:

# The loopback device, always include this.
auto lo
iface lo inet loopback

# The first IP address:
auto eth0
allow-hotplug eth0
iface eth0 inet static
address 173.213.150.12
netmask 255.255.255.0
gateway 173.213.150.1

# The second IP address:
auto eth0:1
iface eth0:1 inet static
address 173.213.150.83
netmask 255.255.255.0
gateway 173.213.150.1

Set the timezone

dpkg-reconfigure tzdata

Set the hostname

echo "node" > /etc/hostname
hostname -F /etc/hostname
editor /etc/hosts

... and add a line like this, replacing the IP with your real IP, and the names with real names of this machine.

173.213.150.12 name.FQDN.domainname.com name othername

Run unixbench

If you want a rough-and-ready benchmark of a VPS to check it's suitable for your requirements, one test is to run UnixBench. Unfortunately there is no Debian package for Unixbench, so here's how to install & run it. Source of instructions.

# Install the requirements
aptitude install libx11-dev libgl1-mesa-dev libxext-dev perl perl-modules make
# Download and extract the test.
cd
mkdir unixbench
cd unixbench
wget http://byte-unixbench.googlecode.com/files/unixbench-5.1.2.tar.gz
tar zxvf unixbench-5.1.2.tar.gz
cd unixbench-5.1.2
# Run the test - it takes about 30 minutes, so run it in a separate terminal.
./Run
# Once the test is done, you'll get a number to measure performance.
# The prerequisites can then be removed (omitting removal of perl since it's useful to me) :
aptitude remove libx11-dev libgl1-mesa-dev libxext-dev
# And the test can be deleted.
cd
rm -rf unixbench

Get the latest and greatest packages

Get the latest and greatest packages (currently there are no updates, but later there probably will be).

aptitude update
aptitude upgrade
aptitude dist-upgrade

Remove unneeded or unwanted packages

Remove unneeded or unwanted packages for a headless webserver, most especially those that talk to the network.

Note: if you are using the system as a Linux desktop, and especially if you are running Ubuntu, then either skip this step, or take care to make sure that the following doesn't remove packages needed for the desktop to work. For example, on Ubuntu, do NOT remove anything where it will also remove the "ubuntu-desktop" package (use 'q' to quit if aptitude says it is going to remove this).

# remove NFS
aptitude --purge remove nfs-common nfs-kernel-server 

# Remove appletalk compat:
aptitude --purge remove netatalk

# Remove PHP4 (we'll install PHP5 later)
aptitude --purge remove php4-common libapache2-mod-php4

# RPC daemon
aptitude --purge remove portmap

# Remove postgres (we'll install MySQL later)
aptitude --purge remove postgresql postgresql-client postgresql-common \
 postgresql-doc postgresql-contrib

# remove email related stuff :
aptitude --purge remove qpopper sa-exim uw-imapd

# remove ident:
aptitude --purge remove pidentd

# remove analog
aptitude --purge remove analog

# Remove unneeded packages:
aptitude --purge remove spamassassin spamc python-newt mutt libapache2-mod-perl2 \
                        libapache2-mod-python

These are steps that make sense on a webserver, but you should probably skip these on a laptop or Linux Desktop:

# DHCP and related functionality
aptitude --purge remove winbind dhcp3-client dhcp3-common 

# remove samba:
aptitude --purge remove samba samba-common samba-doc smbfs

Then purge the old RC files from uninstalled packages:

dpkg --purge $(COLUMNS=132 dpkg -l | grep ^rc | awk '{ print $2; }')

Need to close down open ports. Check what ports are open with these 3 commands, should be stripped back now:

nmap localhost
lsof -i
netstat -l

Install LAMP

# Install LAMP:
aptitude install php5 php5-gd php5-cgi php5-mysql mysql-server mysql-client php5-curl lynx php-pear \
ca-certificates xml-core apache2 libapache2-mod-php5

Install extra useful software

Installing extra useful software:

aptitude install nmap

# This will install the Exim V4 mail server - when prompted, select the "internet site; mail is sent and received directly using SMTP" configuration option :
aptitude install logcheck

aptitude install diffstat
aptitude install zip unzip
aptitude install subversion subversion-tools patch

# Tidying web pages.
aptitude install tidy

# package info:
aptitude install grep-dctrl

# for debugging apache crashes:
aptitude install gdb

# useful utility
aptitude install curl

# For building PHP modules:
aptitude install php5-dev

# For converting HTML to text in a console-only environment
aptitude install html2text

# To be able to compile PHP from snaps.php.net
aptitude install flex libxml2-dev

# Get the "dig" command:
aptitude install dnsutils

# Get the "cruft" command:
aptitude install cruft

# Get the "sloccount" command:
aptitude install sloccount

# Get the p7zip command:
aptitude install p7zip

# Install mailer for sending attachments
aptitude install mutt

# Other useful packages:
aptitude install bzip2 deborphan p7zip-full fakeroot indent fdutils xsltproc

# Get the "uprecords" command:
aptitude install uptimed

# S.M.A.R.T. monitoring tools (skip if running inside a virtual machine/box/VPS - cannot monitor disks from inside a guest - needs to be done by the host O/S).
aptitude install smartmontools
# Then enable smartd:
editor /etc/default/smartmontools
# ... and uncomment the start smartd line.

# Get pinfo, a better GUI for viewing info and manual pages.
aptitude install pinfo

# Get ddrescue, useful for recovering data from failing hard disks (skip if running inside a virtual machine/box/VPS).
aptitude install ddrescue

# Get the "lsb_release -a" command (not installed by default).
aptitude install lsb-release

# The at, atd, and batch commands:
aptitude install at

# Find out if any setuid files change. Will automatically add a daily cron job on installation:
aptitude install checksecurity 

# Rsync:
aptitude install rsync

# Calculates PI. One way of benchmarking.
aptitude install pi

# Small maths tool, used some command line utils (e.g. the mysql tuning script).
aptitude install bc

# Visual top tool:
aptitude install htop

# Safe rm command, to prevent accidents from trashing the filesystem:
aptitude install safe-rm

Disable IPv6

Disabling IPv6 on physical machine or VirtualBox

editor /etc/modprobe.d/aliases

and change this line:

- #alias net-pf-10 ipv6
+ alias net-pf-10 off

Then have to reboot for this change to take effect:

reboot

Disabling IPv6 on a VPS

VPS boxes sometimes don't have modprobe files, so another approach is needed. You can disable ipv6 with a sysctl call:

sysctl -w net.ipv6.conf.all.disable_ipv6=1

You can now check that ifconfig doesn't show ipv6 addresses anymore. In order for this to persist across reboots, create a file, editor /etc/sysctl.d/60-noipv6.conf which contains:

net.ipv6.conf.all.disable_ipv6 = 1

Then you also have to explicitly disable IPv6 for some daemons, like SSH. So do editor /etc/ssh/sshd_config and add this line:

AddressFamily inet # IPv4 only

Then restart SSH:

/etc/init.d/ssh restart

Testing if IPv6 is disabled

Can check has been successfully disabled with:

ip a | grep inet6

(should give no output when IPv6 disabled)

... and:

/usr/bin/lsof -i | grep LISTEN

Should only list "IPv4" entries.

... and:

netstat -l

Should not show "tcp6" and "udp6" after the reboot if it is working.

Stop console screen from blanking

Stop console screen from blanking, due to power-saving:

editor /etc/console-tools/config

Change this line:

BLANK_TIME=30

To:

BLANK_TIME=0

Exim 4 configuration

Exim 4 configuration:

Need to configure exim rather than remove it because the "at" package depends on a mail-transport-agent package.

dpkg-reconfigure exim4-config

Then choose these options, for a server in a data centre:

General type of mail configuration:  internet site; mail is sent and received directly using SMTP
System mail name: YourHostName.com   <Note: this should be a valid domain to reply to, so use this instead of the machine's name>
IP-addresses to listen on for incoming SMTP connections: 127.0.0.1
Other destinations for which mail is accepted:  <blank>
Domains to relay mail for: YourOtherHostName.com
machines to relay for:  <blank>
Keep number of DNS-queries minimal (Dial-on-Demand)?  No
Delivery method for local mail:  mbox format in /var/mail/
Split configuration into small files? No [default]

[this allows mail to be sent locally, but not by other hosts connecting to this one]

For a development or test server, on a broadband connection, have to use smarthosts, as most ISPs block port 25 for outgoing mail. For example, the Exim4 config for a dev machine is the same as above, except for the following:

General type of mail configuration:  mail sent by smarthost; no local mail
IP address or host name of the outgoing smarthost: "mail.internode.on.net" (internode) or "mail-hub.bigpond.net.au" (telstra)
Visible domain name for local users? YourHostName.com

Then try sending a test message with the mail command.

If sending mail is not working:
To see the queue, and try to clear it out, do this:

exim4 -v -v -v -qff

To view the log messages, do this:

tail -f /var/log/exim4/mainlog

Try sending some mail (e.g. to root) from the console, and test if it gets received.

You can check the queue with :

exim -bp

If you need to delete everything from the queue, do this until the queue is empty:

exim -bp | exiqgrep -i | xargs exim -Mrm

Refer to the exim cheatsheet for more commands if Exim troubleshooting is needed.

Customizing Apache configuration

Customizing Apache:

editor /etc/apache2/conf.d/security 

Make these changes:

- ServerTokens Full
+ ServerTokens Prod

- ServerSignature On
+ ServerSignature Off

- TraceEnable On
+ TraceEnable Off

Save, and test config:

apache2ctl -t

Restart the web server:

/etc/init.d/apache2 force-reload

Then to test from the command line that these changes have taken effect:

telnet localhost 80
GET / HTTP/1.0
(press enter twice).

... and the response header should contain "Server: Apache", without the version number.

Then general configuration:

editor /etc/apache2/mods-enabled/alias.conf

Comment this out:

#    Alias /icons/ "/usr/share/apache2/icons/"
#
#    <Directory "/usr/share/apache2/icons">
#        Options Indexes MultiViews
#        AllowOverride None
#        Order allow,deny
#        Allow from all
#    </Directory>

Removing some more unwanted things:

editor /etc/apache2/sites-available/default

Then change these lines, to specify the admin's email address, prevent showing indexes, and allow HTTPauth prompting for a password when accessing directories with .htaccess and .htpasswd files ("AllowOverride AuthConfig"), as well as other overrides ("AllowOverride All").

- ServerAdmin webmaster@localhost
+ ServerAdmin some@valid.email.com

  <Directory /var/www/>
- Options Indexes FollowSymLinks MultiViews
+ Options -Indexes FollowSymLinks MultiViews
- AllowOverride None
+ AllowOverride All

... and delete this section (directory we don't use):

-       <Directory />
-               Options FollowSymLinks
-               AllowOverride None
-       </Directory>

... and delete this section (special directory we don't use):

-       ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
-       <Directory "/usr/lib/cgi-bin">
-               AllowOverride None
-               Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
-               Order allow,deny
-               Allow from all
-       </Directory>

... and delete this section (another special directory we don't use):

-    Alias /doc/ "/usr/share/doc/"
-    <Directory "/usr/share/doc/">
-        Options Indexes MultiViews FollowSymLinks
-        AllowOverride None
-        Order deny,allow
-        Deny from all
-        Allow from 127.0.0.0/255.0.0.0 ::1/128
-    </Directory>

Test config:

apache2ctl -t

Then reload apache so that it gets these changes:

/etc/init.d/apache2 reload

Note: if HTTPS has already been setup, then need to repeat the above steps for the SSL site configuration files too.

Then remove mod_autoindex, not needed:

a2dismod autoindex

Then reload apache so that it gets this change:

/etc/init.d/apache2 force-reload

To see a list of which Apache2 modules are enabled, do:

ls /etc/apache2/mods-enabled/

Can then remove unwanted modules like so:

a2dismod perl
a2dismod mod_python

(although these modules won't be enabled if following these instructions).

Reload apache:

/etc/init.d/apache2 force-reload 

Enable the expires module:

a2enmod expires

Enable the rewrite module, which is useful if ever need to redirect web requests:

a2enmod rewrite

Reload apache:

/etc/init.d/apache2 force-reload

Few more apache tweaks - allow index.php3 as index file - "editor /etc/apache2/mods-enabled/dir.conf", and change:

- DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm
+ DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm index.php3

Then "editor /etc/apache2/mods-enabled/php5.conf", and change to prevent PHP .inc file source code from being visible (e.g.: http://IP-address/lib-functions.inc )

- AddType application/x-httpd-php .php .phtml .php3
+ AddType application/x-httpd-php .php .phtml .php3 .inc

Then check above syntax is OK:

apache2ctl -t

Then restart apache:

/etc/init.d/apache2 force-reload

Note: if HTTPS has already been setup, then need to repeat the above steps for the SSL site configuration files too. ( editor /etc/apache2/sites-available/ssl )

Set up Apache virtual hosting

Setting up Apache virtual hosting:

Load the mass virtual hosting module:

a2enmod vhost_alias

Then "editor /etc/apache2/apache2.conf", and add this to the end of the file to enable mass virtual hosting:

# ---------------------------------------------------
# ---------------- Virtual Hosting ------------------
<IfModule mod_vhost_alias.c>

# Get the server name from the Host: header
UseCanonicalName Off

# The virtual document root is under /var/www/__name_of_whatever_was_requested__
VirtualDocumentRoot /var/www/%0

# Note: Regrettably, cannot set up separate SSL certs for each virtual host here (need different IPs or ports)
# see: http://www.mail-archive.com/modssl-users@modssl.org/msg15648.html

</IfModule>
# ---------------------------------------------------

Then check above syntax is OK:

apache2ctl -t

Then restart apache:

/etc/init.d/apache2 force-reload

Then try to access a page by the hostname (e.g. "http://IP-address"). Should get an error like "The requested URL / was not found on this server." Also a "tail -f /var/log/apache2/error.log" should show an entry like: "File does not exist: /var/www/IP-address". This indicates that virtual hosting is working.

Apache content compression

Setting up Apache content compression:

Source.

Load the headers module:

a2enmod headers

Then "editor /etc/apache2/sites-available/default", and add this to the end of the file (but inside the <VirtualHost> block) to enable mod_deflate, and turn on logging for it.

# ---------------------------------------------------
# -------------- Enabling Mod_Deflate ---------------
<IfModule mod_deflate.c>
  # Compress all content, manually excluding specified file types
  # place filter 'DEFLATE' on all outgoing content
  SetOutputFilter DEFLATE
  # exclude uncompressible content via file type
  SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|rar|zip)$ no-gzip

  # properly handle requests coming from behind proxies
  <IfModule mod_headers.c>
      Header append Vary User-Agent env=!dont-vary
  </IfModule>

  # Keep a log of compression ratio on each request
  DeflateFilterNote Input instream
  DeflateFilterNote Output outstream
  DeflateFilterNote Ratio ratio
  LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' deflate
  CustomLog /var/log/apache2/deflate.log deflate

  # Properly handle old browsers that do not support compression
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
</IfModule>
# ---------------------------------------------------

Then check above syntax is OK:

apache2ctl -t

Then restart apache:

/etc/init.d/apache2 force-reload

Can then check it is working by accessing some pages, and doing:

tail -f /var/log/apache2/deflate.log

Note: Have to paste this into /etc/apache2/sites-available/ssl too (if have created it), or otherwise just do HTTPS / SSL setup at the end, which makes things easier.

Configure php.ini

Configuring PHP.ini (note: for Ubuntu rather than debian, the path is probably /etc/php5/cgi rather than /etc/php5/apache2)

cp /etc/php5/apache2/php.ini /etc/php5/apache2/orig-php.ini
editor /etc/php5/apache2/php.ini

... and make these changes:

- ;date.timezone =
+ date.timezone = Australia/Sydney

# hide PHP in the headers
- expose_php = On
+ expose_php = Off

- max_execution_time = 30
+ max_execution_time = 300 ; Equals 5 mins
 
- upload_max_filesize = 2M
+ upload_max_filesize = 10M

- allow_url_fopen = On
+ allow_url_fopen = 0

Then need to restart apache for changes to take effect:

/etc/init.d/apache2 force-reload

Then soft-link the command line php.ini to the apache php.ini, so that they can have a shared configuration:

# diff -u --ignore-all-space /etc/php5/cli/php.ini /etc/php5/apache2/php.ini
mv /etc/php5/cli/php.ini /etc/php5/cli/php.ini-orig
ln -s /etc/php5/apache2/php.ini /etc/php5/cli/php.ini
ls -al /etc/php5/cli/php.ini

Enable SSL in Apache 2

• Enabling SSL should come after the default sites-available config, in the previous sections, to avoid having to repeat the steps for both the non-SSL and for the SSL configurations.
• "editor /etc/apache2/sites-available/default-ssl", and update as per the "default" file (delete doc, icons, update email address, AllowOverride, Options -Indexes, and add the mod_deflate block).
• Then use the following steps to enable the SSL module, and configure OpenSSL's defaults:

a2enmod ssl
editor /etc/ssl/openssl.cnf

• Then change default_bits from 1024 to 2048 (or 4096), due to mandated increases in SSL key length.
• Also change default_days from 365 to 790 (that's 2 years and 2 months), since most SSL certs can be bought for more than 1 year.
• Then under the "[ req_distinguished_name ]" section, it's useful to set the defaults (using the "_default" entries) for your SSL cert, as this will make regeneration easier when you need to repeat it in a few years.
• Then generate the server.key and server.csr files with:

t="/etc/apache2/ssl"
mkdir $t
export RANDFILE=/dev/random
openssl req -nodes -new -keyout $t/server.key -out $t/server.csr

When prompts, will ask a series of questions. Some fictional answers (the defaults will use the entries you supplied above, so if you have done this you can just press enter repeatedly) :

Country: AU
State: NSW
City: Sydney
Organisation Name: FooBar Pty Ltd
Dept: YourHostName
YOUR name: www.YourHostName.com   [must be the host's fully qualified name, or will get a msg that the certificate doesn't match the site name]
Email address: webmaster@YourHostName.com

Then

chmod 400 /etc/apache2/ssl/server.*

• Then usually want to buy a cert from one of the SSL providers, who will want the server.csr file, and who will email you a server.crt certificate file. Assume you have already bought and received a signed certificate from a CA (with the .CRT server certificate file from the CA, and a .KEY private key file that you generated), then "editor /etc/apache2/sites-available/default-ssl" and modify this :

   #   Server Certificate:
   SSLCertificateFile /etc/apache2/ssl/server.crt

   #   Server Private Key:
   #   If the key is not combined with the certificate, use this
   #   directive to point at the key file.
   SSLCertificateKeyFile /etc/apache2/ssl/server.key

Then enable this site:

a2ensite default-ssl

Then reload apache with:

/etc/init.d/apache2 force-reload

Can check for errors or warnings with:

tail -20 /var/log/apache2/error.log

If you get errors like this in apache's error log: "RSA server certificate CommonName (CN) `www.YourHostName.com' does NOT match server name!?" , then to make Apache happy, the first fully qualified domain name in the /etc/hosts for our IP address should match the name on the certificate. For example, if your SSL cert is for the "www.YourHostName.com" site, and your IP address is 182.32.121.108, then your /etc/hosts should start something like this:

box:~# cat /etc/hosts
127.0.0.1 localhost
182.32.121.108 www.YourHostName.com box

Change the date

Check if the time and date are correct:

date

To change the time and date, if it's wrong:

date --set 0:42:05
hwclock --systohc --utc

See the NTP step as the long-term solution to keeping time accurate.

Tweak MySQL configuration

Edit configuration file:

editor /etc/mysql/my.cnf

... and add these settings to increase performance a bit:

# change from listed default of 16 to 128 (assuming you have more than 1GB of RAM) :
- key_buffer              = 16M
+ key_buffer              = 128M
# uncomment, and reduce max_connections to save RAM:
- #max_connections        = 100
+ max_connections         = 65
# uncomment, and change from default of 64 to 256:
- # table_cache             = 64
+ table_cache             = 256
# Add this on the line after the above
+ sort_buffer             = 4M
+ join_buffer_size        = 1M
+ max_heap_table_size     = 64M 
+ tmp_table_size          = 64M

Also increase query cache a bit:

query_cache_limit       = 8M
query_cache_size        = 32M

Note: On a box with very restricted RAM, set max_connections to 20, leave key_buffer at 16M, and set query_cache_size to 16M.

Uncomment these lines to enable logging of slow queries (in this case, queries that take more than 5 seconds), and to enable logging of non indexed joins:

log_slow_queries        = /var/log/mysql/mysql-slow.log
long_query_time = 3
log-queries-not-using-indexes

Enable binary logging (which enables point in time recovery) by uncommenting this line:

log_bin                 = /var/log/mysql/mysql-bin.log

And if you're not using innodb, then uncomment this line:

skip-innodb

Then to make changes take effect:

/etc/init.d/mysql restart

After the host has been running with this configuration for a week or so, you try running the tuning primer script, which offers suggestions on MySQL tuning:

wget http://www.day32.com/MySQL/tuning-primer.sh
chmod +x ./tuning-primer.sh
./tuning-primer.sh

Also:

wget http://mysqltuner.pl/mysqltuner.pl
perl mysqltuner.pl

Installing and configuring portsentry

installing and configuring portsentry:

aptitude install portsentry

Then:

editor /etc/portsentry/portsentry.conf

... and change to add some ignored ports:

- ADVANCED_EXCLUDE_TCP="113,139"
+ ADVANCED_EXCLUDE_TCP="113,139,25,445,135"

# Enable blocking:
- BLOCK_UDP="0"
- BLOCK_TCP="0"
+ BLOCK_UDP="1"
+ BLOCK_TCP="1"

# Make a little less likely to react:
- SCAN_TRIGGER="0"
+ SCAN_TRIGGER="2"

Then:

editor /etc/portsentry/portsentry.ignore.static

... and add:

# Put hosts in here you never want blocked. This includes the IP addresses
# of all local interfaces on the protected host (i.e virtual host, mult-home)
# Keep 127.0.0.1 and 0.0.0.0 to keep people from playing games.
127.0.0.1
0.0.0.0
# Add the local IP address, the gateway address, DNS addresses,
# addresses of hosts you know you will be connecting from, etc etc.
192.168.0.0/24

Then:

editor /etc/default/portsentry

... (will be an empty or non-existent file), and add / change the two lines to ATCP and AUDP modes (these are the inverse modes) :

TCP_MODE="atcp"
UDP_MODE="audp"

Can then reload portsentry by doing:

/etc/init.d/portsentry restart

Can then test with:

nmap 127.0.0.1

Note: when not using atcp and audp, portsentry will open lots of ports, which will show up with the above command.

Locking down the IP functionality

Locking down the IP functionality to make the system behave sensibly: [from http://www.debian.org/doc/manuals/securing-debian-howto/ch4.en.html , section 4.17.3 ]:

editor /etc/sysctl.conf

... and add the following to the end of that file:

# Additional settings - adapted from the script contributed
# by Dariusz Puchala (see below)
# Ignore ICMP broadcasts
net/ipv4/icmp_echo_ignore_broadcasts = 1
#
# Ignore bogus ICMP errors
net/ipv4/icmp_ignore_bogus_error_responses = 1
# 
# Do not accept ICMP redirects (prevent MITM attacks)
net/ipv4/conf/all/accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net/ipv4/conf/all/secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
net/ipv4/conf/all/send_redirects = 0
#
# Do not forward IP packets (we are not a router)
# Note: Make sure that /etc/network/options has 'ip_forward=no'
net/ipv4/conf/all/forwarding = 0
#
# Enable TCP Syn Cookies
# Note: Make sure that /etc/network/options has 'syncookies=yes'
net/ipv4/tcp_syncookies = 1
#
# Log Martian Packets
# Commented out as this can create heavy load on server flooded with information:
# net/ipv4/conf/all/log_martians = 1
#
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
# Note: Make sure that /etc/network/options has 'spoofprotect=yes'
net/ipv4/conf/all/rp_filter = 1
#
# Do not accept IP source route packets (we are not a router)
net/ipv4/conf/all/accept_source_route = 0

... then do:

cat /proc/sys/net/ipv4/tcp_syncookies

... should say "0". Then do:

reboot

... then do again:

cat /proc/sys/net/ipv4/tcp_syncookies

... and if it worked, it should say "1".

Set limits.conf

editor /etc/security/limits.conf

... and add these two lines:

*                hard    core            0
*                hard    nproc           150

This makes DOS attacks marginally harder - sets the core size to 0 K, and the maximum number of processes to 150.

Install bastille

aptitude install bastille

On Debian 5 there are two files that need to be modified after installing the bastille package, so that it can run (otherwise get an error about not being a supported OS) -

• /usr/lib/Bastille/API.pm
• /usr/lib/Bastille/IOLoader.pm

Search for DB4.0 and you will see it grouped with the OS compatibility listings. Just add DB5.0 right after the DB4.0 and you're set. At least, it worked fine for me.

Then run:

InteractiveBastille

... to configure it. Some possible answers to questions:

accept terms
restrictive permissions on admin utils: no  [default]
disable SUID status for mount/unmount: yes [default]
disable SUID status for ping: yes [default]
disable SUID status for at: no   [differs from default, needed for my setup]
Should Bastille disable clear-text r-protocols that use IP-based authentication? [Y]  [default]
enforce password aging:  no [differs from default, don't want to have to do this]
restrict use to cron to admin accounts:  Yes. [default]
set default umask:  Yes. [default]
Umask: 002 - 
disallow root login on all ttys: n [default]
password-protect the GRUB prompt? [N] [default]
disable CTRL-ALT-DELETE rebooting? [N] [default]
password protect single user mode [Y] [default]
default-deny on TCP Wrappers and xinetd? [N]    [default]
ensure the telnet service does not run on this system? [y]  [default]
ensure inetd's FTP service does not run on this system? [y] [default]
display "Authorized Use" messages at log-in time? [Y]   [default]
disable the gcc compiler? [N] [default]
put limits on system resource usage? [N]  [default]   [would say yes to this next time, since did this manually above].
restrict console access to a small group of user accounts? [N] [default]
add additional logging? [Y]     [default]
remote logging host? [N]   [default]
install TMPDIR/TMP scripts? [N]    [default]
run the packet filtering script? [N]    [default] [asks all sorts of over-detailed Qs]
make changes? [y]        [default]

Install Sun's JDK

First we have to enable the non-free and contrib repositories, by adding these 3 sources. "editor /etc/apt/sources.list"

deb http://ftp.us.debian.org/debian/ lenny contrib non-free
deb http://security.debian.org/ lenny/updates contrib
deb-src http://security.debian.org/ lenny/updates contrib

Installing the SUN JDK:

aptitude install sun-java6-jdk sun-java6-jre

... and say "yes, accept" when prompted about the terms and conditions.

To show the available JVMs & JDKs :

update-alternatives --display java

Check current default java version with:

java -version

If you have multiple versions of Java installed, then you have the option of changing the system default from GCJ to sun's JDK:

update-java-alternatives --verbose --set java-6-sun

Check version has been updated with:

java -version

If ever need to restore to GCJ (if using this), can do so with this:update-java-alternatives --verbose --set java-gcj

Install some useful java libraries:

aptitude install libmysql-java junit junit-doc

Remove the avahi daemon, if it gets installed (can be pulled in by a recommends line as part of the sun-java6-bin installation) :

aptitude remove avahi-daemon

Install the NTP daemon to keep the time current

Install NTP daemon to keep the time current:

aptitude install ntp ntp-doc

Then

editor /etc/ntp.conf

... and set up as follows:

# /etc/ntp.conf, configuration for ntpd

driftfile /var/lib/ntp/ntp.drift
# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

# deny-by-default policy
restrict default ignore

# You do need to talk to an NTP server or two (or three).
server clock.psu.edu       iburst
server ntp0.cornell.edu    iburst
server ntp0.nl.net         iburst
server ntp2a.mcc.ac.uk     iburst
server salmon.maths.tcd.ie iburst

# Have to grant access to the above servers - however the servers are not allowed
# to modify the run-time configuration or query the NTP server.
restrict clock.psu.edu       nomodify nopeer notrap noquery
restrict ntp0.cornell.edu    nomodify nopeer notrap noquery
restrict ntp0.nl.net         nomodify nopeer notrap noquery
restrict ntp2a.mcc.ac.uk     nomodify nopeer notrap noquery
restrict salmon.maths.tcd.ie nomodify nopeer notrap noquery

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1           nomodify nopeer notrap

Then restart with:

/etc/init.d/ntp restart

Check that there were no errors in the system log with:

tail -f /var/log/syslog

Should get output like this:

Jul  7 17:17:31 ludo ntpd[3473]: ntpd 4.2.2p4@1.1585-o Sun Mar  4 13:21:35 UTC 2007 (1)
Jul  7 17:17:31 ludo ntpd[3474]: precision = 2.000 usec
Jul  7 17:17:31 ludo modprobe: WARNING: Not loading blacklisted module ipv6
Jul  7 17:17:31 ludo ntpd[3474]: Listening on interface wildcard, 0.0.0.0#123 Disabled
Jul  7 17:17:31 ludo ntpd[3474]: Listening on interface lo, 127.0.0.1#123 Enabled
Jul  7 17:17:31 ludo ntpd[3474]: Listening on interface eth0, 192.168.0.5#123 Enabled
Jul  7 17:17:31 ludo ntpd[3474]: kernel time sync status 0040
Jul  7 17:17:31 ludo ntpd[3474]: frequency initialized -70.588 PPM from /var/lib/ntp/ntp.drift

Can then check if it is working with:

ntpq -p

Should get output like this:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*otc2.psu.edu    128.118.25.12    2 u    1   64    1  256.700   -0.417   1.318
 cudns.cit.corne 192.5.41.209     2 u    2   64    1  263.631   -4.051   0.580
 ntp0.nl.uu.net  .GPS.            1 u    1   64    1  350.644    0.044   1.723
 maverick.mcc.ac 193.62.22.98     2 u    2   64    1  325.238   -3.433   0.123
 salmon.maths.tc 134.226.1.114    3 u    1   64    1  335.144   -0.182   0.198

Can then check for open UDP connections, like so:

netstat -l

Should get output which includes something like this:

udp        0      0 ludo.yourhostname.c:ntp *:*
udp        0      0 localhost:ntp           *:*
udp        0      0 *:ntp                   *:*

(There does not seem to be any way to turn these udp connections off, which is why we use the deny-by-default policy)

Automatic daily downloading and email notification of pending security updates

Automatic daily downloading and email notification of pending security updates. source. Note: this step is probably only appropriate for a headless web server, and a laptop or desktop-system has synaptic, which provides a nicer interface to package updating. Install:

aptitude install cron-apt

Configure:

editor /etc/cron-apt/config

Change this line:

MAILON="changes"

To test, can run with:

/usr/sbin/cron-apt

This will then automatically download any security updates each night, and send a daily email to root about what updates need to be applied. Then install apt-listchanges for a printout and emailed list of package changes when actually installing:

aptitude install apt-listchanges

Configure apt-listchanges to show information about what an update contains, before we actually install it:

dpkg-reconfigure apt-listchanges
Method for changes display: text
skip changes that have already been seen? No.
E-mail Address(es) which will receive changes: root
Changes displayed with apt: both
Prompt for confirmation after displaying changes? Yes

Then check the above configuration applied successfully:

cat /etc/apt/listchanges.conf

When ready to apply package updates, can then manually install the updates with:

aptitude update
aptitude upgrade

This won't apply any kernel updates, however. For these, need to do:

aptitude dist-upgrade

... and then "reboot". Note: a log of package upgrades, downloads, and installations is also kept locally, and can be viewed with:

less /var/log/aptitude

Test if all CPUs are being used

Q: On an SMP kernel, how to test if both CPUs are being used? How to see the load on each CPU?

aptitude install sysstat

Then to see the load on each CPU, do:

mpstat -P ALL

Output will be like so for 2 CPUs, or more entries if more are being detected and used:

www:~# mpstat -P ALL
Linux 2.6.18-4-686 (www)        04/07/07

21:06:12     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
21:06:12     all    0.40    7.09    0.24    0.12    0.00    0.00    0.00   92.15    253.26
21:06:12       0    0.40    6.99    0.23    0.15    0.00    0.00    0.00   92.23    252.79
21:06:12       1    0.39    7.20    0.24    0.10    0.00    0.00    0.00   92.07      0.48
www:~#

Move Cronjobs forward a few hours

Start the overnight cron jobs earlier - seem to be ending around 7:00 am, and this is too late. By default, crontab runs some jobs at 6:52 AM - this is way too late, want things to run from 1:55 AM instead, so that they are well and truly finished by the early morning, so as to not impact early-morning users.

editor /etc/crontab

Change as follows:

- 25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
- 47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
- 52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
+ 55 1    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
+ 01 4    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
+ 02 5    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

Alias the poweroff command

Sooner or later, probably at 5 PM on a Friday afternoon as you're rushing to get out of the office and down to the pub, you're going to stuff up and do something silly. For me, that silly stuff-up was mistaking a shell on an important production server for a shell on a testing machine, and powering it off, requiring an embarrassed call to a data centre tech to get it turned back on again ASAP. Because of this, I now like to alias the poweroff command, because on a production server, it's a command that you will hopefully only ever want to use once - and that's when it's about to be removed from the data centre. To do this:

editor ~/.bashrc

Uncomment the lines for a coloured ls, and then add these 2 lines:

# Warning on "poweroff" command.
alias poweroff='echo This is a production machine! If you REALLY want to poweroff, do /sbin/poweroff'

Then reload your bashrc with:

source ~/.bashrc

Permission changes

A few permission changes recommended by Tiger (a security auditing program) -

Fix the permissions in one manually created directory:

chmod g-w /usr/local/lib
chmod g-w /usr/local

Some other changes:

chmod 660 /var/log/btmp
chmod 600 /boot/grub/menu.lst

Logcheck configuration

"editor /etc/logcheck/logcheck.conf"

- REPORTLEVEL="server"
+ REPORTLEVEL="paranoid"
 
- SENDMAILTO="logcheck"
+ SENDMAILTO="your@full.email.com"

If you are using atd, then do "editor /etc/logcheck/ignore.d.paranoid/atd" (which will be a new file) and add this:

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ atd\[[0-9]+\]: \(pam_unix\) session opened for user INSERT_USERNAME by \(uid\=1\)$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ atd\[[0-9]+\]: \(pam_unix\) session closed for user INSERT_USERNAME

To stop alerts about time synchronisation, do "editor /etc/logcheck/ignore.d.paranoid/ntp" (which will be a new file), and add this:

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: synchronized to ([0-9.]{7,15}|[0-9a-fA-F:.]{4,39}), stratum [0-9]+$

Specify atd load parameter

The atd load parameter in Debian used to be explicit, but in 5.0, it's not, but you can make it explicit by doing the following:

"editor /etc/init.d/atd", and change as follows:

     start)
         log_daemon_msg "Starting deferred execution scheduler" "atd"
-        start_daemon $DAEMON
+        start_daemon $DAEMON -l 1.5 -b 22
         log_end_msg $?

Then restart atd:

/etc/init.d/atd restart

Install APC as the PHP opcode cache

Installing APC as the opcode cache:

Benchmark before installing, on some PHP page on your site:

ab -kc 10 -t 30 http://yourhostname.com/some_php_page.php

Install required dependencies:

aptitude install apache2-dev
aptitude install build-essential

Install APC (press enter when prompted on any questions to take the defaults) :

pecl install apc

Then:

editor /etc/php5/apache2/php.ini

... and add these 3 lines to the end:

; APC opcode cache:
extension=apc.so
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.ttl=7200
apc.shm_size=128

Then reload apache:

/etc/init.d/apache2 restart

Check that there are no errors in the error log:

tail -20 /var/log/apache2/error.log

Benchmark after installing:

ab -kc 10 -t 30 http://yourhostname.com/some_php_page.php

• For me this showed an increase from 895 completed requests to 3728 completed requests on a dev box with Debian 4.0 (approx 4.2x increase)
• For me this showed an increase from 2889 completed requests to 12023 completed requests on a server with Debian 4.0 (approx 4.2x increase)
• For me this showed an increase from 3494 completed requests to 43932 completed requests on a VPS with Debian 5.0 (approx 12x increase - no idea why this increase was so large)
• For me this showed an increase from 5675 completed requests to 24500 completed requests on a VirtualBox with Debian 5.0 (approx 4.3x increase)
• App1 from 521 -> 1698 requests on a Debian 6.0 VirtualBox ( approx 3.2x increase )
• App2 from 1659 -> 2966 requests on a Debian 6.0 VirtualBox ( approx 1.8x increase )

So overall a not bad increase in speed, typically around 4.2 times faster for serving PHP pages, definitely worth having.