当前位置: DBARoad > Unix | Shell, 故障案例, 本站推荐 > 文章正文

总结一个去年由于Paging Space不足导致系统挂起的案例。
监控报警，数据库挂起，无法连接，检查主机告警时，发现大量Paging Space不足的告警。

pgbb_db1./home/oracle$errpt | pg
......
C5C09FFA   0116193109 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116193109 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116193109 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116193109 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116193009 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116193009 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116192809 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116192809 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116192809 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116192809 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116192809 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0116192809 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
......

pgbb_db1./home/oracle$errpt -aj C5C09FFA | more
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Fri Jan 16 19:31:08 BEIST 2009
Sequence Number: 4625
Machine Id:      00C2C8B24C00
Node Id:         pgbb_db1
Class:           S
Type:            PERM
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
                627034
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
          36
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Fri Jan 16 19:31:08 BEIST 2009
Sequence Number: 4624
Standard input

检查Paging Space的使用情况，发现只有512M，且严重不足，有时敲个命令都无法得结果，返回内存不足的报错：

pgbb_db1./home/oracle$lsps -s
Total Paging Space   Percent Used
      512MB              96%
pgbb_db1./home/oracle$lsps -a
Page Space      Physical Volume   Volume Group    Size %Used Active  Auto  Type
hd6             hdisk0            rootvg         512MB    96   yes   yes    lv
pgbb_db1./home/oracle$

pgbb_db1./home/oracle$man chps
ksh: 0403-031 The fork function failed. There is not enough memory available.

由SA增加Paging Space的大小后，系统恢复正常：

[pgbb_db1]>lspv hdisk0
PHYSICAL VOLUME:    hdisk0                   VOLUME GROUP:     rootvg
PV IDENTIFIER:      00c2c8c279bcd0f5 VG IDENTIFIER     00c2c8c200004c00000011c
PV STATE:           active
STALE PARTITIONS:   0                        ALLOCATABLE:      yes
PP SIZE:            256 megabyte(s)          LOGICAL VOLUMES:  12
TOTAL PPs:          546 (139776 megabytes)   VG DESCRIPTORS:   2
FREE PPs:           166 (42496 megabytes)    HOT SPARE:        no
USED PPs:           380 (97280 megabytes)    MAX REQUEST:      256 kilobytes
FREE DISTRIBUTION:  00..00..00..57..109
USED DISTRIBUTION:  110..109..109..52..00

一个PP大小是256M,原大小为512M为2个PP，现改16G为64*256M，需增加62个PP

[pgbb_db1]>chps -s 62 hd6
[pgbb_db1]>lsps -s
Total Paging Space   Percent Used
      16384MB               4%

Paging Space使用率马上下降，系统恢复正常。
— The End —