PURPOSE:
This page contains useful information on administering Solaris. Most of this information was acquired as a result of some need on our system, so may also be helpful to other people administering a small Solaris installation.
Topics-
Martin Paul from the University of Vienna has created the Patch Check Advanced (PCA) perl script to make the process of patching Solaris much much easier. Here's the website for PCA. This has the script as well as information on using it.
Solaris provides some really great ways to keep your OS up-to-date in terms
of patches to the kernel, libraries and applications. It all revovles around
the smpatch command. The first thing to do is read the smpatch manpage.
Done reading? Ok.
To repeat some man info:
The smpatch command can be used to download, apply, and
remove patches. Also, use the smpatch command to configure
the patch management environment for your system.
The system on which you run Sun Patch Manager must be run-
ning at least Solaris 8 and have the Developer Software Sup-
port Group installed. If your system runs Solaris 8 or
Solaris 9, it must also have the Sun Patch Manager 2.0
software installed. If your system runs Solaris 10 and has
the Developer Software Support Group installed, the Sun
Patch Manager 2.0 software is included.
You need to configure the Patch Manager a little before you can use it.
This is done with the smpatch set command. Use the smpatch get
command to see how things are currently configured. Here's a list of
how things are configured on josie:
[dfoster@josie admin_info]% smpatch get
patchpro.backout.directory - ""
patchpro.download.directory - /var/sadm/spool
patchpro.install.types - rebootafter:reconfigafter:standard
patchpro.patch.source - https://getupdates1.sun.com/solaris/
patchpro.patchset - current
patchpro.proxy.host - ""
patchpro.proxy.passwd **** ****
patchpro.proxy.port - 8080
patchpro.proxy.user - ""
Please note that the download directory is set to /var/sadm/spool (this is the default
location for the download directory) and the backout directory is not set to anything,
but will default to /var/sadm/pkg. By default, backout data is stored in the default
locations used by patchadd. This is the save directory of each package that was modified
by the patch. For example, if a patch modifies the SUNWcsr package, the backout data
for that package is stored in the /var/sadm/pkg/SUNWcsr/save directory. Normally the
default locations are fine, but be aware that if you keep backout info on all your
patches, the backout directory can grow quite big. (see this about
keeping the size of /var manageable)
The install.types parameter determines the type of patches that smpatch can install.
patchpro.install.types
Your policy for applying patches. The value is a list of
zero or more colon-separated patch properties that are
permitted to be applied by an update operation (smpatch
update).
By default, patches that have the standard, rebootafter,
and reconfigafter properties can be applied.
Patches are classified as being standard or nonstandard. A
standard patch can be applied by smpatch update. Such a
patch is associated with the standard patch property. A non-
standard patch has one of the following characteristics:
o A patch that is associated with the rebootafter,
rebootimmediate, reconfigafter, reconfigimmediate, or
singleuser properties. Such a nonstandard patch can be
applied during an update operation if permitted by the
policy.
o A patch that is associated with the interactive pro-
perty. Such a patch cannot be applied by using the
smpatch command.
interactive A patch that cannot be applied by
running the usual patch management
tools (pprosvc, smpatch, or
patchadd). Before this patch is
applied, the user must perform spe-
cial actions. Such actions might
include checking the serial number
of a disk drive, stopping a critical
daemon, or reading the patch's
README file.
rebootafter The effects of this patch are not
visible until after the system is
rebooted.
rebootimmediate When this patch is applied, the sys-
tem becomes unstable until the sys-
tem is rebooted. An unstable system
is one in which the behavior is
unpredictable and data might be lost.
reconfigafter The effects of this patch are not
visible until after a reconfigura-
tion reboot (boot -r). See the
boot(1M) man page.
reconfigimmediate When this patch is applied, the sys-
tem becomes unstable until the sys-
tem gets a reconfiguration reboot
(boot -r). An unstable system is one
in which the behavior is unpredict-
able and data might be lost.
singleuser Do not apply this patch while the
system is in multiuser mode. You
must apply this patch on a quiet
system with no network traffic and
with extremely restricted I/O
activity.
standard This patch can be applied while the
system is in multiuser mode. The
effects of the patch are visible as
soon as it is applied unless the
application being patched is running
while the patch is applied. In this
case, the effects of the patch are
visible after the affected applica-
tion is restarted.
You can see that josie has patch install.types of rebootafter:reconfigafter:standard
This gets all the important patches we should need and the cron script we use alerts
us if one of the installed patches requires a reboot.
If you're running Solaris 10, set patchset parameter to "current".
If you're running Solaris 8/9, set patchset paramenter to "patchdb"
patch_josie.csh - This is the shell script I use to
check for, download and install patches automatically. It's not really commented,
but should be pretty self-explainitory. It creates a log of all the patches that
are needed and all that are installed (or why they aren't installed).
This is the root crontab entry that I use to run the script daily:
# Automate checking for, downloading & installing patches every morning at 7:30am
30 7 * * * /root/bin/patch_josie.csh 2>&1 > /dev/null
Important information about the /var directory and what to do if it gets too full !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! There is now a script ( clean_var_sadm_obsolete.csh) that will automatically clean out /var/crash and all the obsolete.Z files in /var/sadm/pkg !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Clean out /var/crash This can contain large files from crashes that are just taking up space. The software registry is located in the /var file system. /var may vary widely in its disk space consumption because it contains the /var/tmp directory where users can place temporary files, as well as /var/adm and /var/log, which collect accounting and logging information. The software registry requires space to grow as software is installed. Any patches that are installed without explicitly using the .do not backup the files. option will store the files needed for back out in the /var/sadm/pkg directory. Enough space should be available in the /var directory to hold user temporary files plus software installation requirements and patch management. For a large server, the software registry may require 50MB or more and will grow over time as patches are added. ---------------------------------------------------------------- If /var/sadm/pkg is really big (it holds patch backout info): ---------------------------------------------------------------- Check for the existence of "obsolete.Z" files in the /var/sadm/pkg hierarchy. If you've done a lot of patching these files can take up a lot of space. They can be removed without ill effect. For example: # find /var/sadm/pkg -type f -name obsolete.Z -exec rm {} + This can recover several hundreds of megabytes depending on the number of patches applied. If you don't want to backout the patches again you can just get rid of the undo backup files: find /var/sadm/pkg -name '1?????-??' -type d -exec rm -rf {} + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! There is now a csh script (clean_old_patch_backouts.csh) that will search the /var/sadm/pkg directory and remove ONLY patch backout files that are older than the most recent backout file. If there is only one backout file for a patch/pkg, then it will not be deleted. You would be surprised how much space this can clean up. I just got back 1.3GB of space running this on my system, which has been installing regular patches for close to two years. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Optionally, you could change the download/backout directory for smpatch: One thing you can do in this case is to clean out /var/sadm/spool, and then set up a patch download directory on another partition that does have the space. This only helps if /var/sadm has sufficient space for the patch and package files. ------------------------------------------------------------------------ You can use `smpatch set' to specify a different download directory, which you have to first create. Then `smpatch get' should show it, like this: $ smpatch get patchpro.backout.directory - "" patchpro.download.directory /opt/sadm/spool /var/sadm/spool Changing this setting adds a line to /etc/patch/patch.conf, like this: patchpro.download.directory=/opt/sadm/spool The real solution is to ensure that /var is large enough. 5 GB should be sufficient. ----------------------------------------------------------------------------- Do NOT create a symlink for /var/sadm/pkg or /var/sadm to somewhere else on a bigger partition. This will cause problems with patches as they are looking for a directory, not a symlink. If you want to do something like this, do it this way with a loopback mount. Therefor I prefer (if I cannot avoid them) loopback mounts on system directories. Say you want to migrate /var/sadm to /space/sadm: put an entry of the form /space/sadm - /var/sadm lofs - yes - in your /etc/vfstab somewhere after the line which mounts /space.
******************************************************************************** -------------------- Disk setup on josie -------------------- Josie is setup for RAID 1. This means all data is mirrored to two separate drives. In Josie's case, there are two IDE drives both installed as masters on the separate IDE channels (/dev/dsk/c0t0d0 and /dev/dsk/c0t2d0 respectively). Two IDE drives, each master of an independent IDE bus, are required in order for the RAID-1 mirror to boot if either drive fails. This is VERY important. If they aren't both masters, random kernel panics/core dumps/reboots can also occur as IDE bus errors occur during normal mirroring (I learned this the hard way) c0t0d0 is 80GB and c0t2d0 is 120GB. Each disk is "broken" into 7 "slices" (which we can think of as partitions) and the disks must be partitioned identically (or the second mirror has to have larger partitions than the master mirror). For josie, the smaller drive determines how much is mirrored (80GB, actually around 75GB after formatted). This means the 120GB drive will have an extra ~40GB of space (slice 4 of c0t2d0) that will not be mirrored, but can be mounted and used as a regular directory. For josie the disks were partitioned as follows: c0t0d0 slice name cylinders size blocks -------------------------------------------------------------------------------- 0 root wm 1062 - 38306 72.46GB (37245/0/0) 151959600 1 swap wu 0 - 513 1023.98MB (514/0/0) 2097120 2 backup wm 0 - 38306 74.53GB (38307/0/0) 156292560 3 var wm 548 - 1061 1023.98MB (514/0/0) 2097120 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 538 - 547 19.92MB (10/0/0) 40800 6 unassigned wm 528 - 537 19.92MB (10/0/0) 40800 7 unassigned wm 518 - 527 19.92MB (10/0/0) 40800 -------------------------------------------------------------------------------- c0t2d0 slice name cylinders size blocks -------------------------------------------------------------------------------- 0 root wm 2091 - 39335 72.46GB (37245/0/0) 151959600 1 swap wu 0 - 513 1023.98MB (514/0/0) 2097120 2 backup wm 0 - 57458 111.79GB (57459/0/0) 234432720 3 var wm 548 - 2090 3.00GB (1543/0/0) 6295440 4 unassigned wm 39336 - 57458 35.26GB (18123/0/0) 73941840 5 unassigned wm 538 - 547 19.92MB (10/0/0) 40800 6 unassigned wm 528 - 537 19.92MB (10/0/0) 40800 7 unassigned wm 518 - 527 19.92MB (10/0/0) 40800 -------------------------------------------------------------------------------- slice 2 is generally reserved and spans the entire disk, so is never mounted (but provides an easy way to back up an entire disk to tape) Only slice 0, 1 and 3 are mirrored. Slices 5, 6 & 7 are identical on both disks, but didn't need to be. Each of those slices (6 in all) are used to hold a state database replica, which is basically what the system uses to keep the mirrors all in sync. It is important to have multiple state database replicas on each disk, since if one disk fails, it still needs at least three to keep going (or something like that). These slices should be at least 4MB in size. Mirroring uses the concept of metadevice, where we create a metadevice for every partition we want to mirror. Then the two partitions that will actually be mirrored are made into sub-mirrors and attched to the metadevice as such. Below shows how the metadevices are setup on josie. d0 is / d10 is disk0 / (c0t0d0s0) d20 is disk1 / (c0t2d0s0) d1 is swap d11 is disk0 swap (c0t0d0s1) d21 is disk1 swap (c0t2d0s1) d3 is /var d13 is disk0 /var (c0t0d0s3) d23 is disk1 /var (c0t2d0s3) ------------------------------- To Create and setup metadevices ------------------------------- // Assumption: Solaris is initially installed on disk 0 before setting up RAID metadb -c 2 -a -f /dev/dsk/c0t0d0s5 //Create copies of state database metadb -c 2 -a -f /dev/dsk/c0t0d0s6 //replica, The "-c 2" will create metadb -c 2 -a -f /dev/dsk/c0t0d0s7 //two replicas per slice (That metadb -c 2 -a -f /dev/dsk/c0t2d0s5 //really isn't necessary) metadb -c 2 -a -f /dev/dsk/c0t2d0s6 metadb -c 2 -a -f /dev/dsk/c0t2d0s7 metainit -f d10 1 1 c0t0d0s0 //Initialize slices as "sub-mirrors" metainit -f d20 1 1 c0t2d0s0 //"-f" will force initialization of metainit -f d11 1 1 c0t0d0s1 //mounted devices metainit -f d21 1 1 c0t2d0s1 metainit -f d13 1 1 c0t0d0s3 metainit -f d23 1 1 c0t2d0s3 metainit d0 -m d10 //Set one of the submirrors as the metainit d1 -m d11 //master mirror for the metadevice metainit d3 -m d13 metaroot d0 //Updates /etc/vfstab to use metadevices The /etc/vfstab file must be updated at this point to reflect the changes made to the system. The / partition will have already been updated through the metaroot command run earlier, but the system needs to know about the new devices for swap and /var. The entries in the file will look something like the following: /dev/md/dsk/d1 - - swap - no - /dev/md/dsk/d3 /dev/md/rdsk/d3 /var ufs 1 yes - Notice that the device paths for the disks have changed from the normal style /dev/dsk/c#t#d#s# and /dev/rdsk/c#t#d#s# to the new metadevice paths, /dev/md/dsk/d# and /dev/md/rdsk/d#. REBOOT metattach d0 d20 //Attach the other submirror to the metadevice metattach d1 d21 //Now we should have a functioning RAID 1 system and metattach d3 d23 //it should begin syncing up the drives at this point //Do not do anything during initial resyncing // Make mirrored disk bootable installboot /usr/platform/'uname -i'/lib/fs/ufs/bootblk /dev/rdsk/c0t2d0s0 //Optional //To enable easy booting from second mirror device (in case first disk fails), //we can add an alias to the Open Boot Prom ls -l /dev/dsk/c0t0d0s0 //Returned value is full device name for first disk ls -l /dev/dsk/c0t1d0s0 //Returned value is full device name for second disk //at OBP prompt: ok nvalias disk0 /pci@1f,0/pci@1,1/ide@3/disk@0,0 //For disk0 in josie ok nvalias disk1 /pci@1f,0/pci@1,1/ide@3/disk@1,0 //For disk1 in josie (For some reason the ls -l command above returns /pci@1f,0/pci@1,1/ide@3/dad@0,0:a, but OBP won't seem to boot from that.) //Now you can boot the second device at the OBP prompt by //just typing "boot disk1" // To enable the system to boot from the other disk is one disk fails, // do the following at the OBP: ok setenv boot-device disk0 disk1 // Remember that the secondary disk needs to be made bootable using the // "installboot" command above. Now the system can boot from any disk, but // the problem will be that if one disk fails, only half the database replicas // will be available, and the system complains about insufficient database // replicas, write permissions, read permissions, etc. To overcome this // problem, modify the file /etc/system by appending the line: set md:mirrored_root_flag=1 // That way, it will allow the system to boot even if just half the // database replicas are available (Since we include 3 database replicas // on each disk, this may not be necessary... // Alternatively, if a disk dies and Solaris wants a quorum of DB replicas, // boot into single user mode, delete the database replicas on the failed disk // and reboot: boot -s metadb -d /dev/dsk/c0t?d0s5 metadb -d /dev/dsk/c0t?d0s6 metadb -d /dev/dsk/c0t?d0s7 reboot --------------------------------- Checking Metadevice (RAID) status --------------------------------- Use the metastat command. To see the status of all metadevices, use it without arguments. metastat To see the status of an individual metatdevice, specify the metadevice as an argument: metadevice d0 ----------------------------------------------------- To detach a submirror (disk) [i.e. for replacement] ----------------------------------------------------- // Assumption: We're replacing disk 1, do detach all submirrors on it metadetach d0 d20 //Detaches submirror for metadevice metadetach d1 d21 //May need to use the -f flag if the disk metadetach d3 d23 //"needs maintenance" metadb -d c0t1d0s5 //Remove state database replicas from the disk metadb -d c0t1d0s6 //(use metastat -i to verify) metadb -d c0t1d0s7 metaclear d20 //Remove submirror definition/instantiation metaclear d21 //(use metastat to verify) metaclear d23 --------------------- To Attach a submirror --------------------- // Assumption: We just replaced disk 1 with a new disk and partitioned and // formatted it correctly (see above) metadb -a -f /dev/dsk/c0t1d0s5 //Add database state replicas to new metadb -a -f /dev/dsk/c0t1d0s6 //hard drive (3 copies) metadb -a -f /dev/dsk/c0t1d0s7 metainit -f d20 1 1 c0t1d0s0 //Initialize new submirrors metainit -f d21 1 1 c0t1d0s1 metainit -f d23 1 1 c0t1d0s3 metattach d0 d20 //Attach submirrors to metadevice metattach d1 d21 //(disks are resynced at this point) metattach d3 d23 //Do nothing while disks are resyncing metaroot d0 //Updates /etc/vfstab to use metadevices //Make new mirrored disk bootable installboot /usr/platform/'uname -i'/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0 // If both submirrors are replaced with bigger drives (replacing one at a // time, resyncing data in between), and some of the slices are bigger now, // use the growfs command to enable the system to recognize the additional // space available. ------------------------------------------------------ To be able to boot with only 50% of metaDBs available (one disk failure in a two-disk mirror setup) ------------------------------------------------------ You can choose to force Solstice DiskSuite software to start if half of the state database replicas are available by setting the tunable mirrored_root_flag to 1 in /etc/system. The default value of this tunable is disabled, which requires that a majority of all replicas are available and are in sync before Solstice DiskSuite software will start. To enable this tunable, type the following: echo set md:mirrored_root_flag=1 >> /etc/system Caution Enable the mirrored_root_flag only if the following prerequisites are met: 1) The configuration only has two disks. 2) Unattended reboots of a system with only two disks are a requirement and the risk of booting from a stale replica, and therefore, stale volume management system state, is acceptable. This means that in your setup, system availability is more important than data consistency and integrity. Setting this tunable risks data corruption or data loss in the event of a transient disk failure, but allows unattended reboots. See the Solaris Volume Manager documentation on docs.sun.com for more information about state database replicas (metadbs). ********************************************************************************