NFS Configuration

9.3. NFS Client Configuration Files
Any NFS share made available by a server can be mounted using various methods. The share can be manually mounted, using the mount command. However, this requires that the root user type the mount command every time the system restarts. Two methods of configuring NFS shares to be mounted automatically at boot time include modifying the /etc/fstab or using the autofs service.
9.3.1. /etc/fstab
Placing a properly formatted line in the /etc/fstab file has the same effect as manually mounting the exported file system. The /etc/fstab file is read by the /etc/rc.d/init.d/netfs script at system startup and any NFS shares listed there will be mounted.
A sample /etc/fstab line to mount an NFS export looks like the following:
: nfs 0 0
The corresponds to hostname, IP address, or fully qualified domain name of the server exporting the file system.
The is the path to the exported directory.
The specifies where on the local file system to mount the exported directory. This mount point must exist before /etc/fstab is read or the mount will fail.
The nfs option specifies the type of file system being mounted.
The area specifies mount options for the file system. For example, if the options area states rw,suid, the exported file system will be mounted read-write and the user and groupid set by the server will be used. Note that parentheses are not to be used here. For more mount options, see Section 9.3.3 Common NFS Mount Options.
9.3.2. autofs
One drawback to using /etc/fstab is that, regardless of how infrequently a user may access the NFS mounted file system, the system must dedicate resources to keep that mount in place. This is not a problem with one or two mounts, but when the system is maintaining mounts to a dozen systems at one time, overall system performance can suffer. An alternative to /etc/fstab is to use the kernel-based automount utility, which will mount and unmount NFS file systems automatically, saving resources.
The autofs script, located in the /etc/rc.d/init.d/ directory, is used to control automount through the /etc/auto.master primary configuration file. While automount can be specified on the command line, it is more convenient to specify the mount points, hostname, exported directory, and options in a set of files rather than typing them all by hand. By running autofs as a service that starts and stops in designated runlevels, the mount configurations in the various files can be automatically implemented.
The autofs configuration files are arranged in a parent-child relationship. A main configuration file (/etc/auto.master) refers mount points on your system that are linked to a particular map type, which take the form of other configuration files, programs, NIS maps, and other less common mount methods. The auto.master file contains lines referring to each of these mount points, organized like this:

The element of this line indicates the location of the mount on the local file system. The relates to the way in which the mount point will be mounted. The most common method for auto mounting NFS exports is to use a file as the map type for the particular mount point. The map file, usually named auto., where is the mount point designated in auto.master, contains lines that look like this:
:
The refers to the directory within the mount point where the exported file system should be mounted. Much like a standard mount command, the host exporting the file system, as well as the file system being exported, are required in the : section. To specify particular options to be used when mounting the exported file system, place them in the section, separated by commas. For NFS mounts that use autofs, place -fstype=nfs in the section.
While autofs configuration files can be used for a variety of mounts to many types of devices and file systems, they are particularly useful in creating NFS mounts. For example, some organizations store a user's /home/ directory on a central server via an NFS share. Then, they configure the auto.master file on each of the workstations to point to an auto.home file containing the specifics for how to mount the /home/ directory via NFS. This allows the user to access personal data and configuration files in their /home/ directory by logging in anywhere on the internal network. The auto.master file in this situation would look similar to this:
/home /etc/auto.home
This sets up the /home/ mount point on the local system to be configured by the /etc/auto.home file, which may look similar to this:
* -fstype=nfs,soft,intr,rsize=8192,wsize=8192,nosuid server.example.com:/home
This line states that any directory a user tries to access under the local /home/ directory (due to the asterisk character) should result in an NFS mount on the server.example.com system within its exported /home/ file system. The mount options specify that each /home/ directory NFS mounts should use a particular collection of settings. For more information on mount options, including the ones used in this example, see Section 9.3.3 Common NFS Mount Options.
9.3.3. Common NFS Mount Options
Beyond mounting a file system via NFS on a remote host, a number of different options may be specified at the time of the mount that can make it easier to use. These options can be used with manual mount commands, /etc/fstab settings, and autofs, and other mounting methods.
The following options are the most popular for NFS mounts:
hard or soft — specifies whether the program using a file via an NFS connection should stop and wait (hard) for the server to come back online if the host serving the exported file system is unavailable, or if it should report an error (soft).
If hard is specified, the user cannot terminate the process waiting for the NFS communication to resume unless intr option is also specified.
If soft, is specified, the user can set an additional timeo= option, where specifies the number of seconds to pass before the error is reported.
intr — allows NFS requests to be interrupted if the server goes down or cannot be reached.
nolock — is occasionally required when connecting to older NFS server. To require locking, use the lock option.
noexec — does not permit the execution of binaries on the mounted file system. This is useful if the system is mounting a non-Linux file system via NFS that contains incompatible binaries.
nosuid — does not allow set-user-identifier or set-group-identifier bits to take effect.
rsize=8192 and wsize=8192 — may speed up NFS communication for reads (rsize) and writes (wsize) by setting a larger data block size, in bytes, to be transferred at one time. Be careful when changing these values; some older Linux kernels and network cards may not work well with larger block sizes.
nfsvers=2 or nfsvers=3 — specify which version of the NFS protocol to use.
Many more options are listed on the mount man page, including options for mounting non-NFS file systems.


1. Introduction
The Network File System is certainly one of the most widely used network services. Network file system (NFS) is based on the Remote procedure call. It allows the client to automount and therefore, transparently access the remote file systems on the network.
2. Scenario
In this scenario we are going to export the file system from the linuxconfig.org (IP address 10.1.1.200) host and mount it on linuxconfig.local(IP address 10.1.1.100).
3. Prerequisites
At this point, we assume that the NFS service daemon is already installed on your system, including portmap daemon on which NFS setupt depends. Moreover, your system needs to support the NFS file system.
$ cat /proc/filesystems

NFS daemon should be listening on both standard ports 2049 and portmap on port 111.

Another way to check if NFS is functioning, is to use the rpcinfo command.
# rpcinfo -p
You should get a response/output similar to one below:

4. Server export file
All NFS server exports need to be defined in /etc/exports file.
4.1. Most common exports options
Here are the most common export techniques and options:
/home/nfs/ 10.1.1.100(rw,sync) export /home/nfs directory for host with IP 10.1.1.100 with read, write permissions, and synchronized mode
/home/nfs/ 10.1.1.0/24(ro,sync) export /home/nfs directory for network 10.1.1.0 netmask 255.255.255.0 with read only permissions and synchronized mode
/home/nfs/ 10.1.1.100(rw,sync) 10.1.1.10(ro,sync) export /home/nfs directory for host with IP 10.1.1.100 with read, write permissions, synchronized mode, and also export /home/nfs directory for hosts with IP 10.1.1.10 with read only permissions and synchronized mode
/home/nfs/ 10.1.1.100(rw,sync,no_root_squash) export /home/nfs directory for host with IP 10.1.1.100 with read, write permissions, synchronized mode and the remote root user will not be treated as a root but as a default nfs user.
/home/nfs/ *(ro,sync) export /home/nfs directory for any host with a read only permission and synchronized mode
/home/nfs/ *.linuxconfig.org(ro,sync) export /home/nfs directory for any host within linuxconfig.org domain with a read only permission and synchronized mode
/home/nfs/ foobar(rw,sync) export /home/nfs directory for hostname foobar with read, write permissions and synchronized mode
4.2. Edit exports file
Open up your favorite text editor, for example, vim and edit /etc/exports file and add line /home/nfs/ *(ro,sync) to export /home/nfs directory for any host with read only permissions.
Be sure that the directory you export by NFS exists. You can also create a file inside the /home/nfs directory which will help you troubleshoot once you mount this file system remotely.
# touch /home/nfs/test_file
4.3. Restart NFS daemon
Once you edit /etc/exports file you need to restart NFS daemon to apply changes in the /etc/exports file. Depending on your Linux distribution, the restarting of NFS may differ. Debian users:
# /etc/init.d/nfs-kernel-server restart
Redhat users
# /etc/init.d/nfs restart
If you later decide to add more NFS exports to the /etc/exports file, you will need to either restart NFS daemon or run command exportfs:
# exportfs -ra
5. Mount remote file system on client
First we need to create a mount point:
# mkdir /home/nfs_local
If you are sure that the NFS client and mount point are ready, you can run the mount command to mount exported NFS remote file system:
# mount 10.1.1.200:/home/nfs /home/nfs_local
In case that you need to specify a type of the filesystem you can do this by:
# mount -t nfs 10.1.1.200:/home/nfs /home/nfs_local
You may get error message
mount: mount to NFS server failed: timed out (retrying).
This may mean that your server supports higher versions of nfs and therefore you need to pass one extra argument to your nfs client. In this example we use nfs version 3:
# mount -t nfs -o nfsvers=3 10.1.1.200:/home/nfs /home/nfs_local

Now you should be able to see that the file system is mounted. Notice that the mount command reports that the filesystem is mounted as "read and write", although you can see that it provides a "read only" permission.
6. Configure automount
To make this completely transparent to end users, you can automount the NFS file system every time a user boots a PC, or you can also use PAM modules to mount once a user logs in with a proper username and password. In this situation just edit /etc/fstab to mount system automatically during a system boot. You can use your favorite editor and create new line like this:
10.1.1.200:/home/nfs /home/nfs_local/ nfs defaults 0 0
in /etc/fstab or
# echo "10.1.1.200:/home/nfs /home/nfs_local/ nfs defaults 0 0" >> /etc/fstab

7. Conclusion
The Network File System comes with tons of export options. What has been shown here, just barely scratches the surface of NFS. Please visit Linux NFS-HOWTO hosted by linux documentation project or NFS homepage for more details.
8. Appendix A
Following section of this NFS tutorial is going to be devoted to RedHat like Linux systems which by default block all incoming traffic to a NFS server by engaging firewall using iptables rules. For this reason when the firewall is running on your NFS server, you might get this error when mounting NFS filesytem: mount.nfs: mount to NFS server '10.1.1.13' failed: System Error: No route to host. This error message has nothing to do with your NFS configuration, all what needs to be done is either turn off the firewall or add iptables rules to allow traffic on portmap port 111, nfs port 2049 and random ports for other nfs services.

There are two solutions to this problem: easy solution is to turn off the firewall completely and the right solution to add appropriate iptables rules.
8.1. Turn off firewall on Redhat like systems:
The easiest solution is to just turn off the firewall. This will automatically grant access to the nfs daemon to anyone. I would suggest this solution only for testing purposes of your NFS configuration. Enter the following command to stop firewall and clean up all iptables rules:
# service iptables stop
Now when your NFS settings are correct you should be able to mount nfs filesystem from you client machine.
8.2. Add iptables rules to allow NFS communication
This is a more complex but right solution to the given problem. First we need to set static port for nfs services such as rquotad, mountd, statd, and lockd by editing /etc/sysconfig/nfs file. Add or uncomment following lines in your /etc/sysconfig/nfs file:
LOCKD_TCPPORT=32803
LOCKD_UDPPORT=32769
MOUNTD_PORT=892
STATD_PORT=662

Restart you NFSD daemon with following commands:
# /etc/init.d/nfs restart
# /etc/init.d/nfslock restart
Use rpcinfo command to confirm a validity of your new ports settings:
# rpcinfo -p localhost
The output should be similar to the one below:
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100011 1 udp 999 rquotad
100011 2 udp 999 rquotad
100011 1 tcp 1002 rquotad
100011 2 tcp 1002 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 32769 nlockmgr
100021 3 udp 32769 nlockmgr
100021 4 udp 32769 nlockmgr
100021 1 tcp 32803 nlockmgr
100021 3 tcp 32803 nlockmgr
100021 4 tcp 32803 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100005 1 udp 892 mountd
100005 1 tcp 892 mountd
100005 2 udp 892 mountd
100005 2 tcp 892 mountd
100005 3 udp 892 mountd
100005 3 tcp 892 mountd
100024 1 udp 662 status
100024 1 tcp 662 status
Save your current iptables rules into iptables-rules-orig.txt :
# iptables-save > iptables-rules-orig.txt
Create file called iptables-nfs-rules.txt with the following content:
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2:200]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -d 224.0.0.251 -p udp -m udp --dport 5353 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 32769 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 32769 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 32803 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 32803 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 662 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 662 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 892 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 892 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
Apply new rules with iptables-restore, where the single argument will be iptables-nfs-rules.txt file:
NOTE: this will create a new set of iptables rules. If you have already defined some iptables rules previously, you may want to edit iptables-rules-orig.txt and use it with iptables-restore command instead.
# iptables-restore iptables-nfs-rules.txt
Save these new rules, so you do not have to apply new rules for nfs daemon next time you restart your server:
# service iptables save
Now your server is ready to accept client nfs requests. Optionally, you may restart iptables rules / firewall with the following command:
# service iptables restart

RMAN Backup Recovery

WHAT IS RMAN ?

Recovery Manager is a tool that: manages the process of creating backups and also manages the process of restoring and recovering from them.

WHY USE RMAN ?

* No extra costs …Its available free
* RMAN introduced in Oracle 8 it has become simpler with newer versions and easier than user managed backups
* Proper security
* You are 100% sure your database has been backed up.
* Its contains detail of the backups taken etc in its central repository
* Facility for testing validity of backups also commands like crosscheck to Check the status of backup.
* Faster backups and restores compared to backups without RMAN
* RMAN is the only backup tool which supports incremental backups.
* Oracle 10g has got further optimized incremental backup which has resulted in improvement of performance during backup and recovery time
* Parallel operations are supported
* Better querying facility for knowing different details of backup
* No extra redo generated when backup is taken..compared to online backup without RMAN which results in saving of space in hard disk
* RMAN an intelligent tool
* Maintains repository of backup metadata
* Remembers backup set location
* Knows what need to backed up
* Knows what is required for recovery
* Knows what backup are redundant

UNDERSTANDING THE RMAN ARCHITECTURE

* An oracle RMAN comprises of RMAN EXECUTABLE This could be present and fired even through client side TARGET DATABASE This is the database which needs to be backed up
* RECOVERY CATALOG Recovery catalog is optional otherwise backup details are stored in target database controlfile
* It is a repository of information queried and updated by Recovery Manager
* It is a schema or user stored in Oracle database
* One schema can support many databases
* It contains information about physical schema of target database datafile and archive log, backup sets and pieces

Recovery catalog is a must in following scenarios

* In order to store scripts
* For tablespace point in time recovery

Media Management Software

Media Management software is a must if you are using RMAN for storing backup in tape drive directly.

Backups in RMAN

Oracle backups in RMAN are of the following type

RMAN complete backup OR RMAN incremental backup

These backups are of RMAN proprietary nature

IMAGE COPY

Its again a kind of backup. The advantage of uing Image copy is its not in RMAN proprietary format.

Backup Format

RMAN backup is not in oracle format but in RMAN format. Oracle backup comprises of backup sets and it consists of backup pieces. Backup sets are logical entity. In oracle 9i it gets stored in a default location.

There are two type of backup sets

1. Datafile backup sets,
2. Archivelog backup sets

One more important point of data file backup sets is it do not include empty blocks. A backup set would contain many backup pieces. A single backup piece consists of physical files which are in RMAN proprietary format.

You can go to RMAN prompt by just typing rman. RMAN executable is present in ORACLE_HOME/bin location.

bash-2.05$ rman

Recovery Manager: Release 10.2.0.1.0 - Production on Mon Jul 23 02:16:55 2007

Copyright (c) 1982, 2005, Oracle. All rights reserved.

RMAN>

You can use target connect to connect to database. The database it will connect to depends on the environment variable ORACLE_HOME.

RMAN> connect target

connected to target database: ORCL (DBID=1156435946)

Alternatively you can use “rman TARGET SYS/oracle@test NOCATALOG” to connect to the RMAN of “test” instance.

Here we will be using target database control file to store all the information required for RMAN, like backupsets and backup image information etc.

Backup Database:

RMAN> shutdown immediate

using target database control file instead of recovery catalog
database closed
database dismounted
Oracle instance shut down

RMAN> startup mount

connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area 1258291200 bytes

Fixed Size 1978336 bytes
Variable Size 318771232 bytes
Database Buffers 922746880 bytes
Redo Buffers 14794752 bytes

RMAN> backup database
2> ;

Starting backup at 23-JUL-07
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=155 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00001 name=/dy/oracle/product/oradata/orcl/system01.dbf
input datafile fno=00003 name=/dy/oracle/product/oradata/orcl/sysaux01.dbf
input datafile fno=00005 name=/dy/oracle/product/oradata/orcl/example01.dbf
input datafile fno=00002 name=/dy/oracle/product/oradata/orcl/undotbs01.dbf
input datafile fno=00004 name=/dy/oracle/product/oradata/orcl/users01.dbf
channel ORA_DISK_1: starting piece 1 at 23-JUL-07
channel ORA_DISK_1: finished piece 1 at 23-JUL-07
piece handle=/dy/oracle/product/flash_recovery_area/ORCL/backupset/2007_07_23/

o1_mf_nnndf_TAG20070723T031355_3b8zv57d_.bkp tag=TAG20070723T031355 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:36
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including current control file in backupset
including current SPFILE in backupset
channel ORA_DISK_1: starting piece 1 at 23-JUL-07
channel ORA_DISK_1: finished piece 1 at 23-JUL-07
piece handle=/dy/oracle/product/flash_recovery_area/ORCL/backupset/2007_07_23/

o1_mf_ncsnf_TAG20070723T031355_3b8zy7xr_.bkp tag=TAG20070723T031355 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:06
Finished backup at 23-JUL-07
Creating RMAN Catalog

You can also create a repository for RMAN to store all this information. Repository will be just another small database which can store the catalog information. Creating a catalog is a 3 step process

1) Create database which will hold the catalog. Else you can use the existing database also. All you need is to create a seperate tablespace for holding the information about RMAN catalog.

2) Create RMAN tablespace and RMAN user

SQL> create tablespace rman_tbs datafile ‘/dy/oracle/product/db10g/dbf/rman01.dbf’ size 500M EXTENT MANAGEMENT LOCAL segment SPACE MANAGEMENT AUTO ;

Tablespace created.

SQL> create user rman identified by rman
2 default tablespace rman_tbs
3 TEMPORARY TABLESPACE TEMPTS1
4 QUOTA UNLIMITED ON rman_tbs account unlock;

User created.

After creating user, you need to grant RECOVERY_CATALOG_OWNER role to that user.

3) Create RMAN catalog.

bash-2.05$ rman catalog rman/rman@test

Recovery Manager: Release 10.2.0.1.0 - Production on Mon Jul 23 04:37:10 2007

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to recovery catalog database

RMAN> create catalog

recovery catalog created

For registering the database, you need to get connected to database as well as catalog at the same time. Here is how you can do.

RMAN> connect target

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-06004: ORACLE error from recovery catalog database: RMAN-20001: target database not found in recovery catalog

The above error is because the database we connected to is not found in the catalog database. We can register the database in catalog.

RMAN> register database;

database registered in recovery catalog
starting full resync of recovery catalog
full resync complete

Registering database will also resynch the information present in the target database control file and catalog database. Since we have taken 1 backup early, it will synchup that information with RMAN catalog. We can check the same using LIST BACKUP command at RMAN prompt.

RMAN> list backup
2> ;

List of Backup Sets
===================

BS Key Type LV Size Device Type Elapsed Time Completion Time
——- —- — ———- ———– ———— —————
18 Full 573.48M DISK 00:01:26 23-JUL-07
BP Key: 20 Status: AVAILABLE Compressed: NO Tag: TAG20070723T031355
Piece Name: /dy/oracle/product/flash_recovery_area/ORCL/backupset/2007_07_23/

o1_mf_nnndf_TAG20070723T031355_3b8zv57d_.bkp
List of Datafiles in backup set 18
File LV Type Ckp SCN Ckp Time Name
—- — —- ———- ——— —-
1 Full 562487 23-JUL-07 /dy/oracle/product/oradata/orcl/system01.dbf
2 Full 562487 23-JUL-07 /dy/oracle/product/oradata/orcl/undotbs01.dbf
3 Full 562487 23-JUL-07 /dy/oracle/product/oradata/orcl/sysaux01.dbf
4 Full 562487 23-JUL-07 /dy/oracle/product/oradata/orcl/users01.dbf
5 Full 562487 23-JUL-07 /dy/oracle/product/oradata/orcl/example01.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time
——- —- — ———- ———– ———— —————
19 Full 6.80M DISK 00:00:05 23-JUL-07
BP Key: 21 Status: AVAILABLE Compressed: NO Tag: TAG20070723T031355
Piece Name: /dy/oracle/product/flash_recovery_area/ORCL/backupset/2007_07_23/

o1_mf_ncsnf_TAG20070723T031355_3b8zy7xr_.bkp
Control File Included: Ckp SCN: 562487 Ckp time: 23-JUL-07
SPFILE Included: Modification time: 23-JUL-07

RMAN Command line

Backing up the Controlfile and Spfile

The control file can be automatically backed up after each RMAN backup and database structure change as a way to protect the RMAN repository (when we are not using a seperate catalog for RMAN).
RMAN> configure controlfile autobackup on;

Backing up control file

RMAN> backup current controlfile;

Creating Image copy of all datafiles in database

These are the image copies and are stored in ORACLE format and not in RMAN format. Backupsets and backuppieces are stored in internal RMAN format. Hence these image copies can be used for manual restore and recovery as well.

RMAN> backup as copy database;

Creating backupsets of all datafiles in database

If you specify BACKUP AS BACKUPSET, then RMAN stores its backups in backup sets. A backup set, consisting of one or more backup pieces, contains the physical file data being backed up. This backupset is written in a format that only RMAN can access. Only RMAN can create and restore backup sets. Backup sets can be written to disk or tape, and they are the only type of backup which RMAN can use to write backups to tape.

RMAN> backup as backupset database;

Backup individual tablespace as backupsets

RMAN> backup tablespace system, HTMLDB;

Backup individual tablespace as image copies

RMAN> backup as copy tablespace system;

Backup individual files as image copies

RMAN> backup as copy datafile ‘/dy/oracle/product/db10g/dbf/system01.dbf’;

Backup individual files as Backupsets

RMAN> backup datafile ‘/dy/oracle/product/db10g/dbf/system01.dbf’;

Backup archivelogs

We can backup the archive logs according to the output of some search condition. Example we want to backup only those archivelogs which starts with “ARCH_616814159_”.

RMAN> backup acrchivelog like ‘%ARCH_616814159_%’;

Copy archivelogs from some time stamp.

Suppose we want to copy the archivelogs of last 2 days, then we can use the following commands.

RMAN> BACKUP ARCHIVELOG from time ’sysdate-2?;

Backup all archivelog files

RMAN> Backup archivelog all;

Backup archivelog between some time.

RMAN> BACKUP ARCHIVELOG FROM TIME ‘SYSDATE-30? UNTIL TIME ‘SYSDATE-7?;

Specifying copies while backing up.

RMAN> backup copies 2 datafile ‘/dy/oracle/product/db10g/dbf/cs_tbs01.dbf’;

Remember that copies option cannot be used with image copies. It can be used only with backupsets.

Giving tags to backups

RMAN> BACKUP TAG ‘weekly_full_db_bkup’ DATABASE MAXSETSIZE 100M;

Backing up backupsets
RMAN> BACKUP BACKUPSET ALL;

Backup imagecopies

RMAN> Backup as copy backupset all;

List Imagecopies

RMAN> list copy;

List Backupsets

RMAN> list backup;

Restoring and Recovering the database

Use the RESTORE and RECOVER commands for RMAN restore and recovery of physical database files.

RMAN> STARTUP FORCE MOUNT;
RESTORE DATABASE;
RECOVER DATABASE;
ALTER DATABASE OPEN;

Recovering Current Tablespaces

RMAN> SQL ‘ALTER TABLESPACE users OFFLINE’;
RESTORE TABLESPACE users;
RECOVER TABLESPACE users;
SQL ‘ALTER TABLESPACE users ONLINE;

Recovering Current Datafiles

RMAN> SQL ‘ALTER DATABASE DATAFILE 7 OFFLINE’;
RESTORE DATAFILE 7;
RECOVER DATAFILE 7;
SQL ‘ALTER DATABASE DATAFILE 7 ONLINE’;

Recovering Individual Data Blocks

RMAN can recover individual corrupted datafile blocks. When RMAN performs a complete scan of a file for a backup, any corrupted blocks are listed in V$DATABASE_BLOCK_CORRUPTION. Corruption is usually reported in alert logs, trace files or results of SQL queries. Use BLOCKRECOVER to repair all corrupted blocks:

RMAN> BLOCKRECOVER CORRUPTION LIST;

You can also recover individual blocks, as shown in this example:
RMAN> BLOCKRECOVER DATAFILE 7 BLOCK 233, 235 DATAFILE 4 BLOCK 101;

Managing RMAN repository

As you know that if you do not use a recovery catalog, then control file of the target database is used as RMAN repository and eventually after some time the control file records for RMAN information will get overwritten.
Set this initialization parameter in the parameter file of the target database to determine how long records are kept:

CONTROL_FILE_RECORD_KEEP_TIME =

You can configure a retention policy to be used by RMAN to determine which backups are considered obsolete. This allows you to remove files from the repository that are no longer needed to meet your retention requirements. This policy can be based on a recovery window (the maximum number of days into the past for which you can recover) or redundancy (how many copies of each backed-up file to keep).

Two Parameters are used to set retention policies.

You can specify the days days between the current time and the earliest point of recoverability, this is called RECOVERY WINDOW. RMAN does not consider any full or level 0 incremental backup as obsolete if it falls within the recovery window.
Alternatively the REDUNDANCY parameter will instruct to store the number of copies of backup in RMAN repository. If the number of backups for a specific datafile or control file exceeds the REDUNDANCY setting considers the extra backups as obsolete.

RMAN> CONFIGURE RETENTION POLICY TO REDUNDANCY 3;

This will make the REDUNDANCY setting to 3. Meaning that it will at max store 3 copies of backups and ikmages of datafile. Any more images or backups are consider obsolete.

When we run the below command

RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;

it will configure new retention policy based on the RECOVERY WINDOW of 7 days. This means that all the backups which falls outside this window will be considered obsolute. So in this case you need to have backup scheduled every week to have atleast 1 valid backup.

Remember that at any point of time, only one policy can be active. It can be either REDUNDANCY or RECOVERY WINDOW.

When you change the retention policy to another one, it will suspend the previous policy as shown below.

RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;

old RMAN configuration parameters:
CONFIGURE RETENTION POLICY TO REDUNDANCY 3;
new RMAN configuration parameters:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete

Cross checking the available backups

RMAN> CROSSCHECK BACKUP;

Crosscheck is needed when an archivelog file or backup is manually removed, i.e., not deleted by RMAN. This command ensures that data about backups in the recovery catalog or control file is synchronized with corresponding data on disk or in the media management catalog. The CROSSCHECK command operates only on files that are recorded in the recovery catalog or the control file.

The CROSSCHECK command does not delete any files that it is unable to find, but updates their repository records to EXPIRED. Then, you can run DELETE EXPIRED to remove the repository records for all expired files as well as any existing physical files whose records show the status EXPIRED.

If some backup pieces or copies were erroneously marked as EXPIRED, for example, because the media manager was misconfigured, then after ensuring that the files really do exist in the media manager, run the CROSSCHECK BACKUP command again to restore those files to AVAILABLE status.

RMAN> CROSSCHECK COPY;

This will validate the image copies.

Deleting the backups

The DELETE command removes RMAN backups and copies from DISK marks the records in control file as DELETED or removes the records from the recovery catalog (if you use a catalog).

RMAN> DELETE BACKUPSET 101, 102, 103;

RMAN> delete controlfilecopy ‘%ctl’; # Pattern search is allowed.

RMAN> delete archivelog until sequence 20;

This will delete all the archives from the oldest one till the sequence we have specified.

RMAN> delete backup of tablespace system;

RMAN> DELETE ARCHIVELOG ALL BACKED UP 2 TIMES TO DEVICE TYPE DISK;

List commands

RMAN> list backup;

RMAN> list copy;

RMAN> list incarnation of database;

RMAN> list expired backupset;

RMAN> list expired copy;

RMAN> list backup of tablespace sysaux;

RMAN> list copy of datafile 3;

Reporting in RMAN

RMAN> report need backup;

Reports which database files need to be backed up to meet a configured or specified retention policy

RMAN> report unrecoverable;

Reports which database files require backup because they have been affected by some NOLOGGING operation such as a direct-path insert

RMAN> report need backup recovery window of 7 days;

Displays objects requiring backup to satisfy a recovery window-based retention policy.

Suppose in the above command we want to skip a perticular tablespace like perfstat then we can use the below command.

RMAN> report need backup recovery window of 2 days database skip tablespace perfstat;

RMAN> report need backup redundancy 3;

Displays objects requiring backup to satisfy a redundancy-based retention policy.

RMAN> report need backup days 7;

Displays files that require more than n days’ worth of archived redo log files for recovery.

RMAN> report need backup incremental 7;

Displays files that require application of more than n incremental backups for recovery.

RMAN> report need backup redundancy 2 datafile 3;

Gives report of files with less than 2 redundant backups.

RMAN> report need backup tablespace htmldb;

Report of files that must be backed up to satisfy current retention policy for this tablespace (htmldb)

RMAN> report need backup incremental 2;

Report of files that need more than 2 incrementals during recovery

RMAN> report need backup device type disk;

Report of files that must be backed up to satisfy current retention policy for database

RMAN> report obsolete;
Shows backups that are obsolete according to the current retention policy.

You can add the options RECOVERY WINDOW and REDUNDANCY with this command as given below.

RMAN> report obsolete recovery window of 3 days;

RMAN> report obsolete redundancy 2;

RMAN> report schema;

This command lists and displays information about the database files.

RMAN> report schema at time ’sysdate - 14?;

This command gives report on schema 14 days ago.

RMAN> report schema at SCN 10000;

This gives report on schema at scn 10000.

RMAN> report schema at sequence 55 thread 1;

Gives report of schema at sequence 55.

RMAN configuration

RMAN> show all;

RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO ‘%F’; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM ‘AES128?; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO ‘/dy/oracle/product/db10g/dbs/snapcf_test.f’; # default

We can change each of these parameters as per our requirements.

You can return any setting to its default value by using CONFIGURE… CLEAR

RMAN> CONFIGURE BACKUP OPTIMIZATION CLEAR;

RMAN> CONFIGURE RETENTION POLICY CLEAR;

By default, RMAN sends all backups to an operating system specific directory on disk. So default setting for DEVICE TYPE is DISK. You can configure to make backups by default on tape or any other device as given below.

RMAN> CONFIGURE DEFAULT DEVICE TYPE TO sbt;

You can configure backup sets or image copies as the default for a perticular device type, using either of the following commands:

RMAN> CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COPY; # Default becomes image copies
RMAN> CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO BACKUPSET; # Default becomes uncompressed

You can configure RMAN to use compressed backupsets by default on a particular device type, by using the CONFIGURE DEVICE TYPE command with the BACKUP TYPE TO COMPRESSED BACKUPSET option, as shown in the following examples.

RMAN> CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET;

RMAN> CONFIGURE DEVICE TYPE sbt BACKUP TYPE TO COMPRESSED BACKUPSET;

To disable compression you can use below command

RMAN> CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO BACKUPSET;

RMAN> CONFIGURE DEVICE TYPE sbt BACKUP TYPE TO BACKUPSET;

You can configure the parallelism for a device type.

RMAM> CONFIGURE DEVICE TYPE sbt PARALLELISM 2;

You can turn on/off controlfile autobackup using

RMAN> configure controlfile autobackup off;

RMAN> configure controlfile autobackup on;

We can configure RMAN channel to write backups and images in a specific format.

The following command configures RMAN to write disk backups to the /backup directory

CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT ‘/backup/ora_df%t_s%s_s%p’;

%t is replaced with a four byte time stamp,
%s with the backup set number, and
%p with the backup piece number.

You can also configure format for controlfile autobackup as well.
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE deviceSpecifier TO ’string’;

For example, you can run the following command:

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO ‘?/oradata/cf_%F’;

Here %F will be replaced by following

c-IIIIIIIIII-YYYYMMDD-QQ, where:

IIIIIIIIII stands for the DBID.
YYYYMMDD is a time stamp of the day the backup is generated
QQ is the hex sequence that starts with 00 and has a maximum of FF

You can clear the format using following command.
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK CLEAR;

References:

Metalink Note ID: 360416.1

Log file location in R12

Log files are useful in troubleshooting issues in Oracle Applications. Here is the list of Log file location in Oracle Applications for Startup/Shutdown, Cloning, Patching, DB & Apps Listener and various components in Apps R12/12i:

A. Startup/Shutdown Log files for Application Tier in R12

Instance Top is new TOP added in R12 (to read more click here)

–Startup/Shutdown error message text files like adapcctl.txt, adcmctl.txt…
$INST_TOP/apps/$CONTEXT_NAME/logs/appl/admin/log

–Startup/Shutdown error message related to tech stack (10.1.2, 10.1.3 forms/reports/web)
$INST_TOP/apps/$CONTEXT_NAME/logs/ora/ (10.1.2 & 10.1.3)
$INST_TOP/apps/$CONTEXT_NAME/logs/ora/10.1.3/Apache/error_log[timestamp]
$INST_TOP/apps/$CONTEXT_NAME/logs/ora/10.1.3/opmn/ (OC4J~…, oa*, opmn.log)$INST_TOP/apps/$CONTEXT_NAME/logs/ora/10.1.2/network/ (listener log)
$INST_TOP/apps/$CONTEXT_NAME/logs/appl/conc/log (CM log files)

B. Log files related to cloning in R12

Preclone log files in source instance
i) Database Tier - /$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/(StageDBTier_MMDDHHMM.log)

ii) Application Tier - $INST_TOP/apps/$CONTEXT_NAME/admin/log/ (StageAppsTier_MMDDHHMM.log)

Clone log files in target instance

Database Tier - $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_.log
Apps Tier - $INST_TOP/apps/$CONTEXT_NAME/admin/log/ApplyAppsTier_.log

—–
If your clone on DB Tier fails while running txkConfigDBOcm.pl (Check metalink note - 415020.1)
During clone step on DB Tier it prompts for “Target System base directory for source homes” and during this you have to give like /base_install_dir like ../../r12 and not oracle home like ../../r12/db/tech_st_10.2.0
—–

C. Patching related log files in R12

i) Application Tier adpatch log - $APPL_TOP/admin/$SID/log/
ii) Developer (Developer/Forms & Reports 10.1.2) Patch - $ORACLE_HOME/.patch_storage
iii) Web Server (Apache) patch - $IAS_ORACLE_HOME/.patch_storage
iv) Database Tier opatch log - $ORACLE_HOME/.patch_storage

D. Autoconfig related log files in R12
i) Database Tier Autoconfig log :
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/MMDDHHMM/adconfig.log
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/MMDDHHMM/NetServiceHandler.log

ii) Application Tier Autoconfig log - $INST_TOP/apps/$CONTEXT_NAME/admin/log/$MMDDHHMM/adconfig.log

Autoconfig context file location in R12 - $INST_TOP/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml

E. Other log files in R12
1) Database Tier
1.1) Relink Log files :
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME /MMDDHHMM/ make_$MMDDHHMM.log

1.2) Alert Log Files :
$ORACLE_HOME/admin/$CONTEXT_NAME/bdump/alert_$SID.log

1.3) Network Logs :
$ORACLE_HOME/network/admin/$SID.log

1.4) OUI Logs :
OUI Inventory Logs :
$ORACLE_HOME/admin/oui/$CONTEXT_NAME/oraInventory/logs

2) Application Tier
$ORACLE_HOME/j2ee/DevSuite/log
$ORACLE_HOME/opmn/logs
$ORACLE_HOME/network/logs

Tech Stack Patch 10.1.3 (Web/HTTP Server)
$IAS_ORACLE_HOME/j2ee/forms/logs
$IAS_ORACLE_HOME/j2ee/oafm/logs
$IAS_ORACLE_HOME/j2ee/oacore/logs
$IAS_ORACLE_HOME/opmn/logs
$IAS_ORACLE_HOME/network/log
$INST_TOP/logs/ora/10.1.2
$INST_TOP/logs/ora/10.1.3
$INST_TOP/logs/appl/conc/log
$INST_TOP/logs/appl/admin/log


Database Tier Scripts in R12
————————————–
For Database tier you need to start database and database listener. Scripts are located in Database_Install_Dir/db/tech_st/10.2.0/appsutil/scripts/$CONTEXT_NAME- For Database
Use script addbctl.sh

- For Database Listener
Use script addlnctl.sh

or alternatively you can use
lsnrctl startstop listener_name (For Database Listener)
sqlplus “/as sysdba”
SQL> startup shutdown immediate

Middle/Application Tier Scripts in R12
————————————————-

Scripts for Application Tier services in R12 are located in “Install_base/inst/apps/$CONTEXT_NAME/admin/scripts“
where CONTEXT_NAME is of format SID_HOSTNAME

i) adstrtal.sh
Master script to start all components/services of middle tier or application tier. This script will use Service Control API to start all services which are enabled after checking them in context file (SID_HOSTNAME.xml or CONTEXT_NAME.xml)

ii) adstpall.sh
Master script to stop all components/services of middle tier or application tier.

iii) adalnctl.sh
Script to start / stop apps listener (FNDFS and FNDFS). This listener will file will be in 10.1.2 ORACLE_HOME (i.e. Forms & Reports Home)
listener.ora file will be in $INST_TOP/apps/$CONTEXT_NAME/ora/10.1.2/network/admin directory
(Mostly similar to one in 11i with only change in ORACLE_HOME i.e. from 8.0.6 to 10.1.2 )

iv) adapcctl.sh
Script to start/stop Web Server or Oracle HTTP Server. This script uses opmn (Oracle Process Manager and Notification Server) with syntax similar to opmnctl [startstop]proc ohs
like opmnctl stopproc ohs .

(In 11i this script directly used to call apachectl executable but now calls opmnctl which in turn calls apachectl. In 11i web server oracle home was 1.0.2.2.2 but in R12 its 10.1.3)

v) adcmctl.sh
Script to start / stop concurrent manager, Similar to one in 11i. (This script in turn calls startmgr.sh )

vi) adformsctl.sh
Script to start / stop Forms OC4J from 10.1.3 Oracle_Home. This script will also use opmnctl to start/stop Forms OC4J like
opmnctl stopproc type=oc4j instancename=forms

vii) adformsrvctl.sh
This script is used only if you wish to start forms in socket mode. Default forms connect method in R12 is servlet.
If started this will start frmsrv executable from 10.1.2 Oracle_Home in Apps R12

viii) adoacorectl.sh
This script will start/stop oacore OC4J in 10.1.3 Oracle_Home. This scripts will also use opmnctl (similar to adapcctl & adformsctl) to start oacore instance of OC4J like
opmnctl startproc type=oc4j instancename=oacore

ix) adoafmctl.sh
This script will start/stop oafm OC4J in 10.1.3 Oracle_Home. This scripts will also use opmnctl (similar to above) to start oacore instance of OC4J like
opmnctl startproc type=oc4j instancename=oafm

x) adopmnctl.sh
This script will start/stop opmn service in 10.1.3 Oracle_Home. opmn will control all services in 10.1.3 Oracle_Home like web server or various oc4j instances. If any services are stopped abnormally opmn will/should start them automatically.

xi) jtffmctl.sh
This script will be used to start/stop one to one fulfilment server.

xii) mwactl.sh
To start / stop mwa telnet server where mwa is mobile application.

Squid Proxy Configuration

Webmin Install

[root@proxy backup]# rpm -ivh webmin-1.330-1.noarch.rpm
warning: webmin-1.330-1.noarch.rpm: V3 DSA signature: NOKEY, key ID 11f63c51
Preparing... ########################################### [100%]
Operating system is Redhat Enterprise Linux
1:webmin ########################################### [100%]
Webmin install complete. You can now login to http://proxy.chainsys.com:10000/
as root with your root password.

Install Squid Proxy Server on CentOS / Redhat enterprise Linux 5
Squid Basic Configuration
Squid configuration file located at /etc/squid/squid.conf. Open file using a text editor:
# vi /etc/squid/squid.conf
At least you need to define ACL (access control list) to work with squid. The defaults port is TCP 3128. Following example ACL allowing access from your local networks 192.168.1.0/24 and 192.168.2.0/24. Make sure you adapt to list your internal IP networks from where browsing should be allowed:
acl our_networks src 192.168.1.0/24 192.168.2.0/24
http_access allow our_networks
Save and close the file. Start squid proxy server:
# chkconfig squid on
# /etc/init.d/squid start
Output:
init_cache_dir /var/spool/squid... Starting squid: . [ OK ]
Verify port 3128 is open:
# netstat -tulpn | grep 3128
Output:
tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN 20653/(squid)
Open TCP port 3128
Finally make sure iptables is allowing to access squid proxy server. Just open /etc/sysconfig/iptables file:
# vi /etc/sysconfig/iptables
Append configuration:
-A RH-Firewall-1-INPUT -m state --state NEW,ESTABLISHED,RELATED -m tcp -p tcp --dport 3128 -j ACCEPT
Restart iptables based firewall:
# /etc/init.d/iptables restart
Output:
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
Applying iptables firewall rules: [ OK ]
Loading additional iptables modules: ip_conntrack_netbios_n[ OK ]
Client configuration
Open a webbrowser > Tools > Internet option > Network settings > and setup Squid server IP address and port # 3128.



Start/Stop squid :

/etc/init.d/squid restart
===============================================================
Blocking Sites :

ex : 1
acl orkut dstdomain .orkut.com
http_access deny orkut

acl youtube dstdomain .youtube.com
http_access deny youtube

ex 2 :

To deploy the web-site blocking mechanism in Squid, add the following entries to your Squid configuration file (in my system, it’s called squid.conf and it’s located in the /etc/squid directory):

acl bad url_regex "/etc/squid/squid-block.acl"
http_access deny bad

The file /etc/squid/squid-block.acl contains web sites or words you want to block. You can name the file whatever you like. If a site has the URL or word listed in squid-block.acl file, it won’t be accesible to your users. The entries below are found in squid-block.acl file used by my clients:

.oracle.com
.playboy.com.br
sex
...

With the squid-block.acl file in action, internet users cannot access the following sites:

* Sites that have addresses ending with .oracle.com
* Sites that have addresses ending with .playboy.com.br
* Sites containing the word “sex” in its pages

You should beware that by blocking sites containing the word “sex”, you will also block sites such as Middlesex University, Sussex University, etc. To resolve this problem, you can put those sites in a special file called squid-noblock.acl:

^http://www.middlesex.ac.uk
^http://www.sussex.ac.uk
EX 3 :
Squid content filtering: Block / download of music MP3, mpg, mpeg, exec files

Q. For security and to save bandwidth I would like to configure Squid proxy server such way that I do not want my users to download all of the following files:
MP3
MPEG
MPG
AVG
AVI
EXE
How do I configure squid content filtering?
A. You can use squid ACL (access control list) to block all these files easily.
How do I block music files using squid content filtering ACL?
First open squid.conf file /etc/squid/squid.conf:
# vi /etc/squid/squid.conf
Now add following lines to your squid ACL section:
acl blockfiles urlpath_regex "/etc/squid/blocks.files.acl"
You want display custom error message when a file is blocked:
# Deny all blocked extension
deny_info ERR_BLOCKED_FILES blockfiles
http_access deny blockfiles
Save and close the file.
Create custom error message HTML file called ERR_BLOCKED_FILES in /etc/squid/error/ directory or /usr/share/squid/errors/English directory.
# vi ERR_BLOCKED_FILES
Append following content:

File is blocked due to new IT policy

Please contact helpdesk for more information:

Phone: 555-12435 (ext 44)

Email: helpdesk@yourcorp.com

Caution: Do not include HTML close tags as it will be closed by squid.
Now create /etc/squid/blocks.files.acl file:
# vi /etc/squid/blocks.files.acl
Append following text:
\.[Ee][Xx][Ee]$
\.[Aa][Vv][Ii]$
\.[Mm][Pp][Gg]$
\.[Mm][Pp][Ee][Gg]$
\.[Mm][Pp]3$
Save and close the file. Restart Squid:
# /etc/init.d/squid restart

Squid Proxy Server Limit the number of simultaneous Web connections from a client with maxconn ACL

So how do you limit the number of simultaneous web connections from a client browser system using the open source Squid proxy server?
You need to use squid ACCESS CONTROLS feature called maxconn. It puts a limit on the maximum number of connections from a single client IP address. It is an ACL that will be true if the user has more than maxconn connections open. It is used in http_access to allow/deny the request just like all the other acl types.
Step # 1: Edit squid conf file
Open /etc/squid/squid.conf file:
# vi /etc/squid/squid.conf
Step # 2: Setup maxconn ACL
Locate your ACL section and append config directive as follows:
acl ACCOUNTSDEPT 192.168.5.0/24
acl limitusercon maxconn 3
http_access deny ACCOUNTSDEPT limitusercon
Where,
acl ACCOUNTSDEPT 192.168.3.0/24 : Our accounts department IP range
acl limitusercon maxconn 3 : Set 3 simultaneous web access from the same client IP
http_access deny ACCOUNTSDEPT limitusercon : Apply ACL
Save and close the file.
Restart squid
Restart the squid server, enter:
# /etc/init.d/squid restart

Squid deny users accessing a website

Q. How do I deny a user accessing particular website? For example block foo.com.
A. Squid cache is a popular open source web proxy server and web cache software.
It has a wide variety of uses, from speeding up a web server by caching repeated requests, to caching web, DNS and other network lookups for a group of people sharing network resources, to aiding security by filtering traffic.
Squid has powerful ACL (access control list). The primary use of the acl system is to implement simple access control.
How to deny a user from accessing particular site?
To block site called foo.com you need to add following two lines to your squid configuration file.
# vi /etc/squid/squid.conf
Search for `Access Controls' and append following two lines:
acl blocksites dstdomain .foo.com
http_access deny blocksites
Save and close the file. Restart Squid:
# /etc/init.d/squid restart
Let us say you would like to deny access for anyone who browses to a URL with the word "bar" in it. Append following ACL:
acl blockregexurl url_regex -i bar
http_access deny blockregexurl
Save and close the file.
Protection from malware using Squid proxy server

Malware (for "malicious software") is any program or file that is harmful to a computer user. Thus, malware includes computer viruses, worms, Trojan horses, and also spyware, programming that gathers information about a computer user without permission.

Ignoring the threat of malware is one of the most reckless things you can do in today's increasingly hostile computing environment. Malware is malicious code planted on your computer, and it can give the attacker a truly alarming degree of control over your system, network, and data - without your knowledge

Now, you can get complete protection from such malware by using/configuring the Malware Block List with squid so that you, your network and your users are protected.

Visit : http://www.malware.com.br for more information of the Malware Block List

To use the Malware Block List on a Squid proxy to block user access to URLs that contain Malware you need to perform the following simple steps:

* Download the block list:
wget -O - http://malware.hiperlinks.com.br/cgi/submit?action=list_squid > malware_block_list.txt

* Create an ACL in the main configuration file (squid.conf) pointing to a file which will have the list URLs:
acl malware_block_list url_regex -i "/etc/squid/malware_block_list.txt"

* Enable the ACL created previously:
http_access deny malware_block_list
deny_info http://malware.hiperlinks.com.br/denied.shtml malware_block_list

* Force Squid reconfiguration:
squid -k reconfigure

To have an up-to-date block list, create a cron job to run every 4 hours, pointing to a script like this:
#!/bin/sh
wget -O - http://malware.hiperlinks.com.br/cgi/submit?action=list_squid > /etc/squid /malware_block_list.txt
squid -k reconfigure

Use the chkconfig configure Squid to start at boot::
[root@bigboy tmp]# chkconfig squid on
Use the service command to start, stop, and restart Squid after booting:
[root@bigboy tmp]# service squid start
[root@bigboy tmp]# service squid stop
[root@bigboy tmp]# service squid restart
You can test whether the Squid process is running with the pgrep command:
[root@bigboy tmp]# pgrep squid
You should get a response of plain old process ID numbers.
The /etc/squid/squid.conf File
The main Squid configuration file is squid.conf, and, like most Linux applications, Squid needs to be restarted for changes to the configuration file can take effect.
The Visible Host Name
Squid will fail to start if you don't give your server a hostname. You can set this with the visible_hostname parameter. Here, the hostname is set to the real name of the server bigboy.
visible_hostname bigboy
Access Control Lists
You can limit users' ability to browse the Internet with access control lists (ACLs). Each ACL line defines a particular type of activity, such as an access time or source network, they are then linked to an http_access statement that tells Squid whether or not to deny or allow traffic that matches the ACL.
Squid matches each Web access request it receives by checking the http_access list from top to bottom. If it finds a match, it enforces the allow or deny statement and stops reading further. You have to be careful not to place a deny statement in the list that blocks a similar allow statement below it. The final http_access statement denies everything, so it is best to place new http_access statements above it
Note: The very last http_access statement in the squid.conf file denies all access. You therefore have to add your specific permit statements above this line. In the chapter's examples, I've suggested that you place your statements at the top of the http_access list for the sake of manageability, but you can put them anywhere in the section above that last line.
Squid has a minimum required set of ACL statements in the ACCESS_CONTROL section of the squid.conf file. It is best to put new customized entries right after this list to make the file easier to read.
Restricting Web Access By Time
You can create access control lists with time parameters. For example, you can allow only business hour access from the home network, while always restricting access to host 192.168.1.23.
#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/24
acl business_hours time M T W H F 9:00-17:00
acl RestrictedHost src 192.168.1.23

#
# Add this at the top of the http_access section of squid.conf
#
http_access deny RestrictedHost
http_access allow home_network business_hours
Or, you can allow morning access only:
#
# Add this to the bottom of the ACL section of squid.conf
#
acl mornings time 08:00-12:00

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow mornings
Restricting Access to specific Web sites
Squid is also capable of reading files containing lists of web sites and/or domains for use in ACLs. In this example we create to lists in files named /usr/local/etc/allowed-sites.squid and /usr/local/etc/restricted-sites.squid.
# File: /usr/local/etc/allowed-sites.squid
www.openfree.org
linuxhomenetworking.com

# File: /usr/local/etc/restricted-sites.squid
www.porn.com
illegal.com
These can then be used to always block the restricted sites and permit the allowed sites during working hours. This can be illustrated by expanding our previous example slightly.
#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/24
acl business_hours time M T W H F 9:00-17:00
acl GoodSites dstdomain "/usr/local/etc/allowed-sites.squid"
acl BadSites dstdomain "/usr/local/etc/restricted-sites.squid"

#
# Add this at the top of the http_access section of squid.conf
#
http_access deny BadSites
http_access allow home_network business_hours GoodSites
Restricting Web Access By IP Address
You can create an access control list that restricts Web access to users on certain networks. In this case, it's an ACL that defines a home network of 192.168.1.0.
#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/255.255.255.0
You also have to add a corresponding http_access statement that allows traffic that matches the ACL:
#
# Add this at the top of the http_access section of squid.conf
#
http_access allow home_network
Password Authentication Using NCSA
You can configure Squid to prompt users for a username and password. Squid comes with a program called ncsa_auth that reads any NCSA-compliant encrypted password file. You can use the htpasswd program that comes installed with Apache to create your passwords. Here is how it's done:
1) Create the password file. The name of the password file should be /etc/squid/squid_passwd, and you need to make sure that it's universally readable.
[root@bigboy tmp]# touch /etc/squid/squid_passwd
[root@bigboy tmp]# chmod o+r /etc/squid/squid_passwd
2) Use the htpasswd program to add users to the password file. You can add users at anytime without having to restart Squid. In this case, you add a username called www:
[root@bigboy tmp]# htpasswd /etc/squid/squid_passwd www
New password:
Re-type new password:
Adding password for user www
[root@bigboy tmp]#
3) Find your ncsa_auth file using the locate command.
[root@bigboy tmp]# locate ncsa_auth
/usr/lib/squid/ncsa_auth
[root@bigboy tmp]#
4) Edit squid.conf; specifically, you need to define the authentication program in squid.conf, which is in this case ncsa_auth. Next, create an ACL named ncsa_users with the REQUIRED keyword that forces Squid to use the NCSA auth_param method you defined previously. Finally, create an http_access entry that allows traffic that matches the ncsa_users ACL entry. Here's a simple user authentication example; the order of the statements is important:
#
# Add this to the auth_param section of squid.conf
#
auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd

#
# Add this to the bottom of the ACL section of squid.conf
#
acl ncsa_users proxy_auth REQUIRED

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow ncsa_users
5) This requires password authentication and allows access only during business hours. Once again, the order of the statements is important:
#
# Add this to the auth_param section of squid.conf
#
auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd

#
# Add this to the bottom of the ACL section of squid.conf
#
acl ncsa_users proxy_auth REQUIRED
acl business_hours time M T W H F 9:00-17:00

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow ncsa_users business_hours
Remember to restart Squid for the changes to take effect.
Groups in squid
acl groupA src 192.168.0.10 192.168.0.11
acl groupB src 192.168.0.40 192.168.0.41
acl listA dstdomain .ebay.co.uk
acl listB dstdomain .google.com

http_access allow GroupA listA
http_access allow GroupB listB
http_access deny all

acl groupA src 192.168.0.10 192.168.0.11
acl groupB src 192.168.0.40 192.168.0.41

acl GoodSitesA dstdomain "/usr/local/etc/allowed-sitesA.squid"
acl BadSitesA dstdomain "/usr/local/etc/restricted-sitesA.squid"

acl GoodSitesB dstdomain "/usr/local/etc/allowed-sitesB.squid"
acl BadSitesB dstdomain "/usr/local/etc/restricted-sitesB.squid"

http_access allow GroupA GoodSitesA
http_access deny GroupA BadSitesA

http_access allow GroupB GoodSitesA
http_access deny GroupB BadSitesB

Forcing Users To Use Your Squid Server
If you are using access controls on Squid, you may also want to configure your firewall to allow only HTTP Internet access to only the Squid server. This forces your users to browse the Web through the Squid proxy.
Making Your Squid Server Transparent To Users
It is possible to limit HTTP Internet access to only the Squid server without having to modify the browser settings on your client PCs. This called a transparent proxy configuration. It is usually achieved by configuring a firewall between the client PCs and the Internet to redirect all HTTP (TCP port 80) traffic to the Squid server on TCP port 3128, which is the Squid server's default TCP port.
Squid Transparent Proxy Configuration
Your first step will be to modify your squid.conf to create a transparent proxy. The procedure is different depending on your version of Squid.
Prior to version 2.6: In older versions of Squid, transparent proxy was achieved through the use of the httpd_accel options which were originally developed for http acceleration. In these cases, the configuration syntax would be as follows:
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
Version 2.6 and Beyond: Newer versions of Squid simply require you to add the word "transparent" to the default "http_port 3128" statement. In this example, Squid not only listens on TCP port 3128 for proxy connections, but will also do so in transparent mode.
http_port 3128 transparent
Configuring iptables to Support the Squid Transparent Proxy
The examples below are based on the discussion of Linux iptables in Chapter 14, "Linux Firewalls Using iptables". Additional commands may be necessary for you particular network topology.
In both cases below, the firewall is connected to the Internet on interface eth0 and to the home network on interface eth1. The firewall is also the default gateway for the home network and handles network address translation on all the network's traffic to the Internet.
Only the Squid server has access to the Internet on port 80 (HTTP), because all HTTP traffic, except that coming from the Squid server, is redirected.
If the Squid server and firewall are the same server, all HTTP traffic from the home network is redirected to the firewall itself on the Squid port of 3128 and then only the firewall itself is allowed to access the Internet on port 80.
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 \
-j REDIRECT --to-port 3128
iptables -A INPUT -j ACCEPT -m state \
--state NEW,ESTABLISHED,RELATED -i eth1 -p tcp \
--dport 3128
iptables -A OUTPUT -j ACCEPT -m state \
--state NEW,ESTABLISHED,RELATED -o eth0 -p tcp \
--dport 80
iptables -A INPUT -j ACCEPT -m state \
--state ESTABLISHED,RELATED -i eth0 -p tcp \
--sport 80
iptables -A OUTPUT -j ACCEPT -m state \
--state ESTABLISHED,RELATED -o eth1 -p tcp \
--sport 80
Note: This example is specific to HTTP traffic. You won't be able to adapt this example to support HTTPS web browsing on TCP port 443, as that protocol specifically doesn't allow the insertion of a "man in the middle" server for security purposes. One solution is to add IP masquerading statements for port 443, or any other important traffic, immediately after the code snippet. This will allow non HTTP traffic to access the Internet without being cached by Squid.
If the Squid server and firewall are different servers, the statements are different. You need to set up iptables so that all connections to the Web, not originating from the Squid server, are actually converted into three connections; one from the Web browser client to the firewall and another from the firewall to the Squid server, which triggers the Squid server to make its own connection to the Web to service the request. The Squid server then gets the data and replies to the firewall which then relays this information to the Web browser client. The iptables program does all this using these NAT statements:
iptables -t nat -A PREROUTING -i eth1 -s ! 192.168.1.100 \
-p tcp --dport 80 -j DNAT --to 192.168.1.100:3128
iptables -t nat -A POSTROUTING -o eth1 -s 192.168.1.0/24 \
-d 192.168.1.100 -j SNAT --to 192.168.1.1
iptables -A FORWARD -s 192.168.1.0/24 -d 192.168.1.100 \
-i eth1 -o eth1 -m state
--state NEW,ESTABLISHED,RELATED \
-p tcp --dport 3128 -j ACCEPT
iptables -A FORWARD -d 192.168.1.0/24 -s 192.168.1.100 \
-i eth1 -o eth1 -m state --state ESTABLISHED,RELATED \
-p tcp --sport 3128 -j ACCEPT
In the first statement all HTTP traffic from the home network except from the Squid server at IP address 192.168.1.100 is redirected to the Squid server on port 3128 using destination NAT. The second statement makes this redirected traffic also undergo source NAT to make it appear as if it is coming from the firewall itself. The FORWARD statements are used to ensure the traffic is allowed to flow to the Squid server after the NAT process is complete. The unusual feature is that the NAT all takes place on one interface; that of the home network (eth1).
You will additionally have to make sure your firewall has rules to allow your Squid server to access the Internet on HTTP TCP port 80 as covered in Chapter 14, "Linux Firewalls Using iptables".
Manually Configuring Web Browsers To Use Your Squid Server
If you don't have a firewall that supports redirection, then you need to configure your firewall to only accept HTTP Internet access from the Squid server, as well as configure your PC browser's proxy server settings manually to use the Squid server. The method you use depends on your browser.
For example, to make these changes using Internet Explorer
Click on the "Tools" item on the menu bar of the browser.
Click on "Internet Options"
Click on "Connections"
Click on "LAN Settings"
Configure with the address and TCP port (3128 default) used by your Squid server.
Here's how to make the same changes using Mozilla or Firefox.
Click on the "Edit" item on the browser's menu bar.
Click on "Preferences"
Click on "Advanced"
Click on "Proxies"
Configure with the address and TCP port (3128 default) used by your Squid server under "Manual Proxy Configuration"
Squid Disk Usage
Squid uses the /var/spool/squid directory to store its cache files. High usage squid servers need a large amount of disk space in the /var partition to get optimum performance.
Every webpage and image accessed via the Squid server is logged in the /var/log/squid/access.log file. This can get quite large on high usage servers. Fortunately, the logrotate program automatically purges this file.
Troubleshooting Squid
Squid logs both informational and error messages to files in the /var/log/squid/ directory. It is best to review these files first whenever you have difficulties.The squid.out file can be especially useful as it contains Squids' system errors.
Another source of errors could be unintended statements in the squid.conf file that cause no errors; mistakes in the configuration of hours of access and permitted networks that were forgotten to be added are just two possibilities.
By default, Squid operates on port 3128, so if you are having connectivity problems, you'll need to follow the troubleshooting steps in Chapter 4, "Simple Network Troubleshooting", to help rectify them.
Note: Some of Squid's capabilities go beyond the scope of this book, but you should be aware of them. For example, for performance reasons, you can configure child Squid servers on which certain types of content are exclusively cached. Also, you can restrict the amount of disk space and bandwidth Squid uses.


Sample squid.conf files :
ex 1 :

# The port on which squid will listen for requests
http_port 8080
# If 'cgi-bin' or '?' is in query, squid should not check with neighbours'/parents' cache
# and should go to target web-server.
hierarchy_stoplist cgi-bin ?
# If url contains 'cgi-bin' or '?', then it must not be cached
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
# Absolute path to squid access log.
access_log /var/log/squid/access.log squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
# Access control list to control every IP address
acl all src 0.0.0.0/0.0.0.0
# Access control list for source machine in LAN
acl lan_src src 192.168.0.0/16
# Access control list for destination machine in LAN
acl lan_dst dst 192.168.0.0/16
# Access control list to manage squid cache
acl manager proto cache_object
# Access control list to define IP address allowed for source localhost
acl localhost src 127.0.0.1/255.255.255.255
# Access control list to define IP addresses allowed for localhost as destination
acl to_localhost dst 127.0.0.0/8
# Access control list to define Safe ports that should be allowed by default
acl SSL_ports port 443 563 1863 5190 5222 5050 6667
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
# Allow cache management only from localhost
http_access allow manager localhost
# Deny cache management from remote hosts
http_access deny manager
# Deny http access via all the ports which are not listed as safe
http_access deny !Safe_ports
# Deny all connections via all ports which are not listed as safe
http_access deny CONNECT !SSL_ports
# Allow http access from localhost
http_access allow localhost
# Allow http access from machines on LAN
http_access allow lan_src
http_access deny all
http_reply_access allow all
icp_access allow all
# Deny caching for everyone so that there is not caching at all
cache deny all
coredump_dir /var/spool/squid
# Never allow direct connection to machines on the internet
prefer_direct off
never_direct allow all
# Allow direct connetion if the destination machine is on LAN
always_direct allow lan_dst
# Delete this line if you don't have /etc/hosts file
hosts_file /etc/hosts
# Allow AIM connections
# Delete the following 9 lines if you don't want people to connect to AIM
acl AIM_ports port 5190 9898 6667
acl AIM_domains dstdomain .oscar.aol.com .blue.aol.com .freenode.net
acl AIM_domains dstdomain .messaging.aol.com .aim.com
acl AIM_hosts dstdomain login.oscar.aol.com login.glogin.messaging.aol.com toc.oscar.aol.com irc.freenode.net
acl AIM_nets dst 64.12.0.0/255.255.0.0
acl AIM_methods method CONNECT
http_access allow AIM_methods AIM_ports AIM_nets
http_access allow AIM_methods AIM_ports AIM_hosts
http_access allow AIM_methods AIM_ports AIM_domains
# Allow connections to Yahoo Messenger
# Delete the following 6 lines if you don't want people to connect to Yahoo Messenger
acl YIM_ports port 5050
acl YIM_domains dstdomain .yahoo.com .yahoo.co.jp
acl YIM_hosts dstdomain scs.msg.yahoo.com cs.yahoo.co.jp
acl YIM_methods method CONNECT
http_access allow YIM_methods YIM_ports YIM_hosts
http_access allow YIM_methods YIM_ports YIM_domains
# Allow connections to Google Talk
# Delete the following 6 lines if you don't want people to connect to Google Talk
acl GTALK_ports port 5222 5050
acl GTALK_domains dstdomain .google.com
acl GTALK_hosts dstdomain talk.google.com
acl GTALK_methods method CONNECT
http_access allow GTALK_methods GTALK_ports GTALK_hosts
http_access allow GTALK_methods GTALK_ports GTALK_domains
# Allow connections to MSN
# Delete the following 6 lines if you don't want people to connect to Google Talk
acl MSN_ports port 1863 443 1503 begin_of_the_skype_highlighting              1863 443 1503      end_of_the_skype_highlighting begin_of_the_skype_highlighting              1863 443 1503      end_of_the_skype_highlighting begin_of_the_skype_highlighting              1863 443 1503      end_of_the_skype_highlighting
acl MSN_domains dstdomain .microsoft.com .hotmail.com .live.com .msft.net .msn.com .passport.com
acl MSN_hosts dstdomain messenger.hotmail.com
acl MSN_nets dst 207.46.111.0/255.255.255.0
acl MSN_methods method CONNECT
http_access allow MSN_methods MSN_ports MSN_hosts


==============
acl BlockExt url_regex -i \.mp3$ \.asx$ \.wma$ \.wmv$ \.avi$ \.mpeg$ \.mpg$ \.qt$ \.ram$ \.rm$ \.iso$ \.wav$ \.exe$

acl webRadioReq1 req_mime_type -i ^video/x-ms-asf$
acl webRadioReq2 req_mime_type -i ^application/vnd.ms.wms-hdr.asfv1$
acl webRadioReq3 req_mime_type -i ^application/x-mms-framed$
acl webRadioRep1 rep_mime_type -i ^video/x-ms-asf$
acl webRadioRep2 rep_mime_type -i ^application/vnd.ms.wms-hdr.asfv1$
acl webRadioRep3 rep_mime_type -i ^application/x-mms-framed$

acl WMP browser Windows-Media-Player/*

http_access deny BlockExt !UtentiGold
http_access deny WMP all
http_access deny webRadioReq1 all
http_access deny webRadioReq2 all
http_access deny webRadioReq3 all

http_reply_access deny webRadioRep1 all
http_reply_access deny webRadioRep2 all
http_reply_access deny webRadioRep3 all


(1) To allow http_access for only one machine with MAC Address 00:08:c7:9f:34:41
To use MAC address in ACL rules. Configure with option -enable-arp-acl.
acl all src 0.0.0.0/0.0.0.0
acl pl800_arp arp 00:08:c7:9f:34:41
http_access allow pl800_arp
http_access deny all
(2) To restrict access to work hours (9am - 5pm, Monday to Friday) from IP 192.168.2/24
acl ip_acl src 192.168.2.0/24
acl time_acl time M T W H F 9:00-17:00
http_access allow ip_acl time_acl
http_access deny all
(3) Can i use multitime access control list for different users for different timing.
AclDefnitions
acl abc src 172.161.163.85
acl xyz src 172.161.163.86
acl asd src 172.161.163.87
acl morning time 06:00-11:00
acl lunch time 14:00-14:30
acl evening time 16:25-23:59

Access Controls
http_access allow abc morning
http_access allow xyz morning lunch
http_access allow asd lunch
This is wrong. The description follows:
Here access line "http_access allow xyz morning lunch" will not work. So ACLs are interpreted like this ...
http_access RULE statement1 AND statement2 AND statement3 OR
http_access ACTION statement1 AND statement2 AND statement3 OR
........
So, the ACL "http_access allow xyz morning lunch" will never work, as pointed, because at any given time, morning AND lunch will ALWAYS be false, because both morning and lunch will NEVER be true at the same time. As one of them is false, and acl uses AND logical statement, 0/1 AND 0 will always be 0 (false).
That's because this line is in two. If now read:
http_access allow xyz AND morning OR
http_access allow xyz lunch
If request comes from xyz, and we're in one of the allowed time, one of the rules will match TRUE. The other will obviously match FALSE. TRUE OR FALSE will be TRUE, and access will be permitted.
Finally Access Control looks...
http_access allow abc morning
http_access allow xyz morning
http_access allow xyz lunch
http_access allow asd lunch
http_access deny all
(4) Rules are read from top to bottom. The first rule matched will be used. Other rules won't be applied.
Example:
http_access allow xyz morning
http_access deny xyz
http_access allow xyz lunch
If xyz tries to access something in the morning, access will be granted. But if he tries to access something at lunchtime, access will be denied. It will be denied by the deny xyz rule, that was matched before the 'xyz lunch' rule.
5.2. How do I make wget work with Squid?
It's simple. Create a file called .wgetrc and put it in your home directory. Insert the following lines in it and that's it!
HTTP_PROXY=192.168.1.1:8080
FTP_PROXY=192.168.1.1:8080
You can make it work globally for all users, type man wget to learn how.
For SMTP /POP
$ipt -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
$iptables -A FORWARD -m state --state NEW -p tcp --dport 25 -j ACCEPT
$iptables -A FORWARD -m state --state NEW -p tcp --dport 110 -j ACCEPT

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 110 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT


Open TCP port 3128
Finally make sure iptables is allowing to access squid proxy server. Just open /etc/sysconfig/iptables file:
# vi /etc/sysconfig/iptables
Append configuration:
-A RH-Firewall-1-INPUT -m state --state NEW,ESTABLISHED,RELATED -m tcp -p tcp --dport 3128 -j ACCEPT
Restart iptables based firewall:
# /etc/init.d/iptables restart
IPtables For Outlook Express Mails
A Squid proxy can share a Internet Service. But it will not download the mails in Outlook Express.
In this case IPtables can be used to retrieve mails.

A model Network setup:

Eth0 : Public N/W

Eth1 : Private N/W
IP Range : 192.168.10.0/24

So hence the Iptables rules for forwarding the packets ,

# iptables --table nat --append POSTROUTING -s 192.168.10.0/24 --out-interface eth1 -j MASQUERADE

# iptables --append FORWARD -s 192.168.10.0/24 --in-interface eth0 -j ACCEPT

# iptables -A INPUT -s 192.168.10.0/24 -j ACCEPT

Insert the above rules,
Save and restart the service

#service iptables save
# service iptables restart
Now you can download your mails in the Outlook express.

Client Side Configuration:
1. Set the Gateway address as Eth1 IP.
2. Open Outlook Express – Tools – Accounts – mail – add new
3. Configure your mail account.
iptables -A OUTPUT -p tcp -s 209.200.83.221 --sport 25 -d 0/0 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT


iptables -A OUTPUT -p tcp -s 209.200.83.221 --sport 1024:65535 -d 0/0 --dport 25 -m state --state NEW,ESTABLISHED -j ACCEPT

iptables -A INPUT -p tcp -s 0/0 --sport 25 -d 209.200.83.221 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

209.200.83.221
============================

ex :
# IPTABLES Configuration
#IPTABLES="/etc/sysconfig/iptables"
# Module Loading
/sbin/depmod -a
/sbin/modprobe ip_tables
/sbin/modprobe ip_conntrack
/sbin/modprobe iptable_filter
/sbin/modprobe iptable_mangle
/sbin/modprobe iptable_nat
/sbin/modprobe ipt_LOG
/sbin/modprobe ipt_limit
/sbin/modprobe ipt_state
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

# Creating chain for bad_tcp_packet chain
iptables -N bad_tcp_packets

# Creating Separate chains for ICMP,TCP & UDP
iptables -N allowed
iptables -N tcp_packets
iptables -N udp_packets
iptables -N icmp_packets

# bad_tcp_packets chain
iptables -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "New not Syn:"
iptables -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j DROP

# ALLOWED Chain
iptables -A allowed -p TCP --syn -j ACCEPT
iptables -A allowed -p TCP -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A allowed -p TCP -j ACCEPT

# TCP RUles
iptables -A tcp_packets -p TCP -s 0/0 --dport 21 -j allowed
iptables -A tcp_packets -p TCP -s 0/0 --dport 22 -j allowed
iptables -A tcp_packets -p TCP -s 0/0 --dport 23 -j allowed
iptables -A tcp_packets -p TCP -s 0/0 --dport 23 -j allowed
iptables -A tcp_packets -p TCP -s 0/0 --dport 80 -j allowed
iptables -A tcp_packets -p TCP -s 0/0 --dport 25 -j allowed
iptables -A tcp_packets -p TCP -s 0/0 --dport 110 -j allowed
iptables -A tcp_packets -p TCP -s 0/0 --dport 8080 -j allowed
#iptables -A tcp_packets -p TCP -s 0/0 --dport 3128 -j allowed
# Dropping MULTICAST
iptables -A udp_packets -p UDP -i $INET_IFACE -d $INET_BROADCAST --destination-port 135:139 -j DROP


# UDP Ports
iptables -A udp_packets -p UDP -s 0/0 --destination-port 53 -j ACCEPT
iptables -A udp_packets -p UDP -s 0/0 --destination-port 123 -j ACCEPT
iptables -A udp_packets -p UDP -s 0/0 --destination-port 2074 -j ACCEPT
iptables -A udp_packets -p UDP -s 0/0 --destination-port 4000 -j ACCEPT

# Dropping DHCP request from outside Network
iptables -A udp_packets -p UDP -i $INET_IFACE -d 255.255.255.255 --destination-port 67:68 -j DROP
#ICMP Rules
iptables -A icmp_packets -p ICMP -s 0/0 --icmp-type 8 -j ACCEPT
iptables -A icmp_packets -p ICMP -s 0/0 --icmp-type 11 -j ACCEPT

# INPUT CHain
# Bad TCP PACKETS
iptables -A INPUT -p TCP -j bad_tcp_packets

# Rules for Special Networks not part of INternet
iptables -A INPUT -p ALL -i $LAN_IFACE -s $LAN_IP_RANGE -j ACCEPT
iptables -A INPUT -p ALL -i $LO_IFACE -s $LO_IP -j ACCEPT
iptables -A INPUT -p ALL -i $LO_IFACE -s $LAN_IP -j ACCEPT
iptables -A INPUT -p ALL -i $LO_IFACE -s $INET_IP -j ACCEPT
iptables -A INPUT -p ALL -i $LAN_IFACE -d $LAN_BROADCAST_ADDRESS -j ACCEPT

# RULES for INCOMING Packets from the INternet
iptables -A INPUT -p ALL -d $INET_IP -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p TCP -i $INET_IFACE -j tcp_packets
iptables -A INPUT -p UDP -i $INET_IFACE -j udp_packets
iptables -A INPUT -p ICMP -i $INET_IFACE -j icmp_packets

# Dropping Multicast Packets
iptables -A INPUT -i $INET_IFACE -d 224.0.0.0/8 -j DROP

# LOG weird packets that don't match the abvoe
iptables -A INPUT -m limit --limit 3/minute --limit-burst 3 -j LOG --log-level DEBUG --log-prefix "IPT INPUT packet died"

# FORWARD Chain Rules
# BAD TCP Packets
iptables -A FORWARD -p tcp -j bad_tcp_packets
# Accept packets we wwant to forward
iptables -A FORWARD -i $LAN_IFACE -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# LOG weird packets that don't match the abvoe
iptables -A FORWARD -m limit --limit 3/minute --limit-burst 3 -j LOG --log-level DEBUG --log-prefix "IPT FORWARD packet died"

# OUTPUT CHAIN
# BAD TCP PACKETS we dont'want
iptables -A OUTPUT -p tcp -j bad_tcp_packets

# Special OUTPUT Rules to decide which IP's to allow
iptables -A OUTPUT -p ALL -s $LO_IP -j ACCEPT
iptables -A OUTPUT -p ALL -s $LAN_IP -j ACCEPT
iptables -A OUTPUT -p ALL -s $INET_IP -j ACCEPT

# LOG weird packets that don't match the abvoe
iptables -A OUTPUT -m limit --limit 3/minute --limit-burst 3 -j LOG --log-level DEBUG --log-prefix "IPT OUTPUT packet died"

# IP Forwarding and NAT
iptables -t nat -A POSTROUTING -o $INET_IFACE -j SNAT --to-source $INET_IP
iptables -t nat -A POSTROUTING -p TCP -s 0/0 -o $INET_IFACE --dport 25 -j SNAT --to-source $INET_IP
iptables -t nat -A POSTROUTING -p TCP -s 0/0 -o $INET_IFACE --dport 110 -j SNAT --to-source $INET_IP
iptables -t nat -A PREROUTING -p TCP -i $LAN_IFACE --dport 80 -j REDIRECT --to-port 8080

Opatch

Introduction:

This post has been written on request of one of regular visitor of my blog (Altaaf). This post is all about Oracle Patching Utility (OPATCH) and various options that can be used along with Opatch.

We will begin the discussion by knowing what exactly is Opatch?

- OPatch is a java based utility that allow the application and rolling back of interim patches to an Oracle product. The program has sub-commands that may take arguments. The program requires java(JVM) to be present on the current system.OPatch expects users to have commands such as fuser, jar, ar and make on Unix platforms and jar on Windows platforms to be available in their PATH.

Opatch Details

How to check the Opatch version?

-bash-3.00$ ./opatch version
Invoking OPatch 10.2.0.3.0

OPatch Version: 10.2.0.3.0

OPatch succeeded.

How to know which version is the correct version of Opatch for your RDBMS home?

You can verify whether the Opatch version for your RDBMS release is correct or not using metalink note ID 357221.1.

This note ID gives the copatibilities between OPatch version and RDBMS version.

How to get the latest version of OPatch?

You can download the latest version of OPatch from metalink using following URL.
http://updates.oracle.com/download/6880880.html

In the release dropdown you can select the OPatch release that you want to download based on your RDBMS version. Please read the README.txt before applying the patch.

What is Oracle Database Inventory and where it is located?

Oracle Inventory is the location or the place where all the information about an Oracle Home is stored and maintained. When ever we install an RDBMS Oracle Home, a new inventory gets created. Applying any new patch to Oracle Home will update the invnrtory for that Oracle Home and record the details for the patch applied. This inventory is in the form of XML files.

The location of inventory is defined in a file called oraInst.loc. The path for this file is provided while installing Oracle Home. If we dont supply any path, the is file will be present at central location /etc/oraInst.loc. Also we can have a central inventory if its not used by any other oracle installation. If the central inventory is used by previous Oracle installation we can create local inventory. The content of oraInst.loc file is as shown below.

-bash-3.00$ cd $ORACLE_HOME
-bash-3.00$ cat oraInst.loc
inventory_loc=/slot/ems2029/appmgr/apps/tech_st/10.1.2/oraInventory
inst_group=ems2029
-bash-3.00$

Here “inventory_loc” gives the location of Inventory directory where as inst_group gives the name of the group which is the owner of this inventory.

How to create Local Inventory?

You can create local inventory at any location while installing Oracle Home. You need to use -invPtrLoc variable along with runInstaller command and give the location of oraInst.loc. Otherwise the default path assumed for oraInst.loc is /etc/oraInst.loc.

./runInstaller -invPtrLoc /slot/ems2029/appmgr/apps/tech_st/10.1.2/oraInst.loc

After the installation starts it will ask for location of oraInventory directory and group which should own it. It will make entry of these into oraInst.loc file.

Having know the above information about opatch, now we will move to details about using opatch and various options available.

Opatch help

You can get all the options using opatch -help

-bash-3.00$ ./opatch -help
Invoking OPatch 11.1.0.6.0

Oracle Interim Patch Installer version 11.1.0.6.0
Copyright (c) 2007, Oracle Corporation. All rights reserved.

Usage: opatch [ -help ] [ -r[eport] ] [ command ]

command := apply
lsinventory
prereq
query
rollback
util
version

:= -help Displays the help message for the command.
-report Print the actions without executing.

example:
‘opatch -help’
‘opatch apply -help’
‘opatch lsinventory -help’
‘opatch prereq -help’
‘opatch rollback -help’
‘opatch util -help’

OPatch succeeded.

You can get specific help for any command of opatch using opatch -help. Example opatch apply -help.

Applying single patch using opatch

1. You MUST read the Readme.txt file included in opatch file, look for any prereq. steps/ post installation steps or and DB related changes. Also, make sure that you have the correct opatch version required by this patch.
2.Make sure you have a good backup of database.
3. Make a note of all Invalid objects in the database prior to the patch.
4. Shutdown All the Oracle Processes running from that Oracle Home , including the Listener and Database instance, Management agent etc.
5. You MUST Backup your oracle Home and Inventory

tar cvf - $ORACLE_HOME $ORACLE_HOME/oraInventory | gzip > Backup_Software_Version.tar.gz

6. Unzip the patch in $ORACLE_HOME/patches

If you are applying a single intrim patch to oracle home, then change the location to the directory of that patch and then invoke following command.

Example if I need to apply patch 6972343 to RDBMS Oracle Home

-bash-3.00$ cd 6972343
-bash-3.00$ pwd
/slot/ems2029/oracle/db/tech_st/11.1.0/patches/6972343
-bash-3.00$ ls
etc files README.txt
$ORACLE_HOME/OPatch/opatch apply

If you have created local inventory then you need to give the correct path for local inventory stored in oraInst.loc. If you have updated /etc/oraInst.loc with correct location of your inventory, then above command will work, If not you can give the correct location by giving local oraInst.loc file.

$ORACLE_HOME/OPatch/opatch apply -invPtrLoc /slot/ems2029/oracle/db/tech_st/11.1.0/oraInst.loc

Remember that when we are applying patch, patch number is NOT provided as an input to opatch command. So we have to to be in the directory of patch since opatch will pick the patch files from current directory.

Rolling back a patch

In case if a patch did not fix the issue or if you has applied wrong patch to oracle home, you can always rollback the patch using following comamnd.

opatch rollback -id

Applying bundle patches

Some times if you have to apply bundle patch having many patches, example lets say you want to a bundle patch 6778860 ( 11.1.0.6 BUNDLED PATCH FOR EBS 11I)) containing many other patches to RDBMS home, OPatch provides a facility to apply many patches to RDBMS home using a single comamnd. Following command should be used for applying bundle patches.

Download the patch and extract the same. You will see many directories (one for each patch) inside the main patch directory. Execute following command to apply all patches.

$ORACLE_HOME/OPatch/opatch util NApply -invPtrLoc /slot/ems2029/oracle/db/tech_st/11.1.0/oraInst.loc

NApply -> Will apply all patches to RDBMS home.

You can also use -phBaseDir to point to the directory where bundle patch is installed. So you can run opatch command from any location

$ORACLE_HOME/OPatch/opatch util NApply -phBaseDir /slot/ems2029/oracle/db/tech_st/11.1.0/patches/6778860 -invPtrLoc /slot/ems2029/oracle/db/tech_st/11.1.0/oraInst.loc

You can also apply specific patches using NApply

opatch util napply -id 1,2,3 -skip_subset -skip_duplicate

This will apply patches 1, 2, and 3 which are under < the patch_location> directory. OPatch will skip duplicate patches and subset patches (patches under that are subsets of patches installed in the ORACLE_HOME)

You can see all the options for NApply using following help command.

$ORACLE_HOME/OPatch/opatch util NApply -help

Query the inventory for patches applied

We can query the inventory to check various components installed for database as well as to check various patches applied to database. Use following command to get a minimum information from inventory about patches applied and components installed.

$ORACLE_HOME/OPatch/opatch lsinventory -invPtrLoc /slot/ems2029/oracle/db/tech_st/11.1.0/oraInst.loc

Patch 5763576 : applied on Wed May 28 03:20:53 PDT 2008
Created on 6 Feb 2008, 02:26:04 hrs PST8PDT
Bugs fixed:
5763576

If you are using central inventory then -invPtrLoc variable is not required.

You can also get the detailed view of inventory using following command. In case of detail information it gives which are the files that this patch have touched.

$ORACLE_HOME/OPatch/opatch lsinventory -detail -invPtrLoc /slot/ems2029/oracle/db/tech_st/11.1.0/oraInst.loc

Patch 6318357 : applied on Wed May 28 03:33:27 PDT 2008
Created on 4 Dec 2007, 22:02:16 hrs PST8PDT
Bugs fixed:
6318357
Files Touched:
udjvmrm.sql –> ORACLE_HOME/javavm/install/udjvmrm.sql
Patch Location in Inventory:
/slot/ems2029/oracle/db/tech_st/11.1.0/inventory/oneoffs/6318357
Patch Location in Storage area:
/slot/ems2029/oracle/db/tech_st/11.1.0/.patch_storage/6318357_Dec_4_2007_22_02_16

Which options are installed in Oracle Home?

You can check the options installed in RDBMS home using above command. The options installed will be listed at the start of output.
Example: $ORACLE_HOME/OPatch/opatch lsinventory -detail -invPtrLoc /slot/ems2029/oracle/db/tech_st/11.1.0/oraInst.loc

The complete output of this command can be seen at this location.

Opatch Log files

Log file for Opatch utility can be found at $ORACLE_HOME/cfgtoollogs/opatch

Advanced Option for OPatch

Here are some of the advanced options of OPatch utility.

UpdateRemoteNodes

It is used to propagate/remove files/directories to/from remote nodes using files under ORACLE_HOME/.patch_storage//rac/*.

The directories listed in copy_dirs.txt will be copied to remote nodes.
The files listed in copy_files.txt wil be copied to remote nodes.
The directories listed in remove_dirs.txt will be deleted from remote nodes.
The files listed in remove_files.txt will be deleted from remote nodes.
This is used where RAC setup is present.

Cleanup

It is used to clean up ‘restore.sh, make.txt’ files and ‘rac, scratch, backup’ directories in the ORACLE_HOME/.patch_storage directory. If -ps option is used, then it cleans the above specified areas only for that patch, else for all patches under ORACLE_HOME/.patch_storage. You will be still able to rollback patches after this cleanup.
Example: opatch util cleanup -ps 6121183_Ju _21_2007_04_19_42

CopyListedFiles

It is used to copy all files listed in ORACLE_HOME/.patch_storage//rac/copy_files.txt to remote nodes. If -fp option is used, then one can specify the path of the file containing the list of files to be copied. The files mentioned in this file will be copied to the remote nodes.
Example: opatch util copylistedfiles -fp a -remote_nodes ceintcb-a5
This is used where RAC setup is present.

CopyListedFilesTest

It is used to copy a single file to remote nodes. The usage remains the same as CopyListedFiles.
Example: opatch util copylistedfilestest -fp /home/oracle/a -remote_nodes ceintcb-a5
This is used where RAC setup is present.

CopyListedDirs

It is used to recursively copy all directories listed in ORACLE_HOME/.patch_storage//rac/copy_dirs.txt to remote nodes. If -dp option is used, then one can specify the path of the file containing the list of directories to be copied. The directories mentioned in this file will be copied to the remote nodes.
This is used where RAC setup is present.

CopyListedDirsTest

It is used to copy a single file to remote nodes. The usage remains the same as CopyListedDirs.
This is used where RAC setup is present.

RemoveListedFiles

It is used to remove files listed in ORACLE_HOME/.patch_storage//rac/remove_files.txt on remote nodes. If -fr option is used, then one can specify the path of the file containing the list of files to be removed. The files mentioned in this file will be removed from the remote nodes.
This is used where RAC setup is present.

RemoveListedFilesTest

It is used to remove a single file from remote nodes. The usage remains the same as RemoveListedFiles.
This is used where RAC setup is present.

RemoveListedDirs

It is used to recursively remove directories listed in ORACLE_HOME/.patch_storage//rac/remove_dirs.txt from remote nodes. If -dr option is used, then one can specify the path of the file containing the list of directories to be removed. The directories mentioned in this file will be removed from the remote nodes.
This is used where RAC setup is present.

RemoveListedDirsTest

It is used to remove a single directory from remote nodes. The usage remains the same as RemoveListedDirs.
This is used where RAC setup is present.

RunLocalMake

It is used to invoke re-link on the local node. The make commands are stored in ORACLE_HOME/.patch_storage//make.txt. You need to use the -ps option to specify the Patch ID with timestamp. A directory by this name will be present under ORACLE_HOME/.patch_storage. The make.txt file present under ORACLE_HOME/.patch_storage// will be used to perform the local make operation. This command cannot be run if you have already run Cleanup as it would have removed these make.txt files.

Example: opatch util runlocalmake -ps 6121250_ un_21_2007_04_16_11

RunRemoteMake

It is used to invoke re-link on remote nodes. The make commands are stored in
ORACLE_HOME/.patch_storage//rac/makes_cmd.txt. The usage remains the same as RunLocalMake.
This is used where RAC setup is present.

RunAnyCommand

It is used to run any command on remote nodes. The command should be specified using the -cmd option.
Example: opatch util runanycommand -remote_nodes ceintcb-a5 -cmd ls
This is used where RAC setup is present.

LoadXML

It is used to check the validity of an XML file. The -xmlInput option can be used to specify the path of the xml file.

@ Support can use this utility to verify the integrity of XML files contained in the local and central inventory.
Example: opatch util loadxml -xmlInput $ORACLE_HOME/inventory/ContentsXML/comps.xml

Verify

It is used to run the patch verification process to ensure that the patch was applied to the ORACLE_HOME. It uses the defined ORACLE_HOME and the given patch location via -ph, to run the check.

@ Support can use this utility to re-run the OPatch verification stage, if required for patch diagnosis. The patch staging area has to be present or created.

Example: opatch util verify -ph ~/6646853/6121183

Troubleshooting

Some times the inventory get corrupted because of some issues. In that case you need to repair the inventory. Following are the two methods which I know can be used to repair the inventory.

1) Development has released a utility (checkinv) which can be use to repair the inventory if it becomes corrupted for some reason.

Please note that this version of checkinv works only for 9.2 and 10.1 releases of oracle database. This doesn’t work for 10.2 releases of oracle database.

You can download the checkinv utility from Patch 4121946.

To cleanup the components, you should determine the unwanted or dangling components, then specify those using the option “-remcomp”, to cleanup the component list.

The utility detects and asks user’s permission to repair a bad inventory. In case you want to repair the inventory, the utility will first back up the files before any modification.

However, it is better that you backup the inventory before running the tool.

For more details, check metalink note ID 298906.1

2) Creating a new inventory using Oracle Universal Installer (OUI).

References:

Metalink note ID : 298906.1
Metalink note ID : 554417.1
Metalink note ID : 374092.1
Metalink note ID : 357221.1