1、內存熱點問題及案例分享企業客戶技術支持中心支持服務事業部2004.2檢查內存1. dmesg:Memory Information: physical page size = 4096 bytes, logical page size = 4096 bytes Physical: 1048576 Kbytes, lockable: 790684 Kbytes, available: 918096 Kbytes2.sam:samPerformance MonitorsSystem PropertiesMemory3.syslog.log4.glance2Memory的角色三個部分:1. Phys

2、ical memory2. Available MemoryThe amount of main memory not reserved for the kernel is termed available memory. Available memory is used by the system for executing user processes. 3. Lockable MemoryPages kept in memory for the lifetime of a process by means of a system call (such as mlock, plock, o

3、r shmctl) are termed locked memory. Locked memory cannot be paged and processes with locked memory cannot be deactivated. Kernel parameter: unlockable_mem. Total available system memory - unlockable_mem = lockable memory3檢查空閑內存1. TopMemory: 126108K (79812K) real, 419716K (324852K) virtual, 98184K fr

4、ee2. Glance3. Vmstat#vmstat 1 1 procs memory page faults cpu r b w avm free re at pi po fr de sr in sy cs us sy id 1 0 0 93405 5989 5 1 0 0 0 0 0 448 737 133 1 1 994與內存相關的內核參數Buffer cache related:nbuf and bufpages =0 表示使用動態內存 dbc_max_pct50 dbc_min_pct5 Buffer cache used by . 建議調低dbc_max_pct(尤其大量使用ra

5、w device)，否則占用太多內存。5與內存相關的內核參數(續)vx_ninode=0 表示使用動態內存,可能占用很大內存,造成系統內存資源耗盡，還可能造成superpagepool corruption Or fragmentation。unlockable_mem 使用缺省值(=0)6內存的分配內存分配的三種方式，需要相同數量的swap. plain memory as allocated with malloc(3C) system call. shared memory as allocated with shmget(2) system call. memory mapped fi

6、les as allocated withmmap(2) system call. 7內存的分配（續）一般內存的分配由malloc(3C) 系統調用完成a) To report process memory usage, by largest first, run: # ps efl | sort rnk 10 | moreAnd look at 10th column (SZ) to see the amount of memory used by this process for data/text and stack (as this value is in pages, you can

7、 multiply by 4096 to determine the size in bytes.) Anytime you see that the size (SZ) is a four-digit number, watch it over time and see if it continues to grow. Alternative ps command - Alternatively, you can use the UNIX95 options to look at both Virtual Size as well as the actual Size. Run:# UNIX

8、95=1 ps -efo vsz,sz,pid,args |grep -v grep | sort -rnk 1 | more 8內存的分配（續） b) And another way is to sort glances process list (or application list) by RSS (or Res Mem Virt Mem). For example: PROCESS LIST Users= 5 User CPU Util Cum Disk ThdProcess Name PID PPID Pri Name ( 100% max) CPU IO Rate RSS Cnt

9、pax 13819 13818 148 root 2.7/ 5.8 273.3 9.4/32.8 284kb 1glance 14464 1822 158 root 2.1/ 3.1 3.0 0.0/ 2.1 4.3mb 1scopeux 1715 1 127 root 1.7/ 0.2 518.4 1.5/ 0.0 4.1mb 1s 0 0 127 root 1.5/ 0.8 2213.0 0.3/ 0.0 16kb 1java 10095 1 168 root 1.0/ 2.7 348.7 0.0/ 4.2 42.0mb 28vxfsd 35 0 138 root 0.2/ 0.1 289

10、.4 1.9/ 1.3 352kb 16c) Unsupported tools: procsize & kmeminfo9內存的分配（續）2) 共享內存的分配: shmget(2) system call.a) To see shared memory usage, ipcs can be used. For example, run: # ipcs mb| more To totalthe shared memory usage, run: # ipcs -mb | sed -n /m/p | awk total+=$NFENDprintf(“%dn”, total)b) unsuppor

11、ted tool called shminfo 10內存的分配（續）3) 內存映像文件:mmap(2) system call. No tools available but unsupported shminfo.11用戶進程內存限制maxdsiz,maxssiz,maxtsizShared memory limit: shmmaxswap1232bit app13解釋對通常的程序來說：第一象限用來存放app text第二象限用來存放data & stack第三四象限用來存放共享的數據The default usage for the quadrants is:Quadrant 1 Proc

12、ess textQuadrant 2 Process dataQuadrant 3 Shared libraries Shared memory Memory mapped filesQuadrant 4 Shared libraries Shared memory Memory mapped filesThe last part of quadrant 4 is reserved for system I/O.1432bit app Private 部分Maxdsiz & maxssiz 所決定的數據段和堆棧段位于同一個象限中，總和不能大于1GB。Maxdsiz & maxssiz應該合理配

13、置，否則互相影響Maxssiz一般設置成64MB已足夠正常32bit應用，數據段最大為940MB安裝補丁，最大可以1.9GB。編譯選項 ld -NOR“chatr +q3p enable executable_name” to enable third quadrant private 1532bit app 共享部分第3、4象限用來做32bit app的共享尋址最大1.75GBSHMEM_MAGIC 將第二象限用于共享數據(這時最大為2.75GB)To get SHMEM_MAGIC, the executable needs to have been linked (ld) with EX

14、EC_MAGIC (the -N option) and chatred with SHMEM_MAGIC (the -M option).1664bit appFor 64 bit processes the quadrant sizes are 4tb, and therefore the maximum sizes for text, data, shared memory, etc, are quite large.For 64 bit 11.00/11i executables there is currently no need to have different memory m

15、aps available as the standard one allows up to 4TB for the program text, another 4TB for its private data and a total of 8TB for shared areas.17考慮一下16GB物理內存將多個數據庫置于同一臺機器 對32位數據庫，只有1.75GB內存可用怎么辦？18Memory Windows!19What does the Memory Windows feature do?Enabling memory windows does change the kernels

16、 default allocation policy. With only the global memory window configured, the kernel allocates shared addresses from the 4th quadrant first and then the 3rd quadrant. When max_mem_window is set to a non-zero value (enabling memory windows) , the kernel changes the default allocation policy such tha

17、t the 3rd quadrant is tried first followed by the 4th. 20內核設置To enable the use of memory windows the kernel tunable max_mem_window must be set to the desired amount of memory windows. The disabled value is 0. A good default is 256.Note: This tunable is not dynamic.21內核設置To enable 256 memory windows

18、in the kernel add the following line to the kernel configuration file (/stand/system): max_mem_window 256Build a new kernel and reboot.Setting max_mem_window to 256 results in creating 256 memory windows, plus the global memory window. Setting max_mem_window to 1 would create 1 memory window, plus t

19、he global window, for a total of 2.22Services.windowModify /etc/services.window.database1 20database2 30database3 4023相關命令getmemwindow is the command used to extract window ids of user processes from the /etc/services.window file.setmemwindow is the command that changes the window id of a running pr

20、ocess or starts a specified program in a particular memory window.24限制An application cannot dynamically switch between one window and another. Applications are only allowed to change their window at exec time. Applications are not required to change their source code, but instead use the memory wind

21、ow command setmemwindow to start an application in a specified memory window. 25例子For example: setmemwindow -i 10 myprog arg1 arg2 arg3 This would start the executable myprog with 3 arguments in the memory window corresponding to user key 10. There are other options to setmemwindow to control how me

22、mory windows are created.26例子（續）getmemwindow is used to extract the user ids/keys from the /etc/services.window a particular string.This avoids having to embed the keys in the scripts themselves.syntax: getmemwindow 27例子（續）Putting it all together in a real example:# cat startDB1.shWinId=$(getmemwind

23、ow database1)setmemwindow -i $WinId /home/dave/memwinbb/startDB1 DB1 says hello!28例子（續）# ./startDB1.shwriting to segment: DB1 says hello!Key is 137731256229例子（續）# ipcs -mobIPC status from /dev/kmem as of Thu Jul 10 08:52:34 2003T ID KEY MODE OWNER GROUP NATTCH SEGSZShared Memory:m 0 0 x4124288b -rw-

24、rw-rw- root root 0 348m 1 0 x4e000002 -rw-rw-rw- root root 1 61760m 2 0 x41280050 -rw-rw-rw- root root 1 8192m 5215 0 x52181f32 -rw-r-r- root sys 0 1024Our key of 1377312562 converted to hex is 52181f3230例子（續）# ./startDB2.shwriting to segment: DB2 says Bon Jour!Key is 1377312563# ./startDB3.shwritin

25、g to segment: DB3 says Ola!Key is 137704273431例子（續）# more stopDB1.shWinId=$(getmemwindow database1)setmemwindow -i $WinId /home/dave/memwinbb/stopDB1# ./stopDB1.sh# ./stopDB2.sh# ./stopDB3.sh32工具介紹pstopglance33ps# ps efl | sort rnk 10 | moreAnd look at 10th column (SZ) to see the amount of memory us

26、ed by this process for data/text and stack (as this value is in pages, you can multiply by 4096 to determine the size in bytes.) Anytime you see that the size (SZ) is a four-digit number, watch it over time and see if it continues to grow. Alternative ps command - Alternatively, you can use the UNIX

27、95 options to look at both Virtual Size as well as the actual Size. For example:# UNIX95=1 ps -efo vsz,sz,pid,args |grep -v grep | sort -rnk 1 | moreVSZ SZ PID COMMAND12252 627 2745 /opt/OV/bin/ovdbrun -c /var/opt/OV/share/databases/analysis/9060 1214 2362 /opt/omni/lbin/rds -d8808 1892 2677 /opt/hp

28、webjet-5.5/hpwebjetd34topLoad averages: 0.64, 0.57, 0.56 195 processes: 194 sleeping, 1 running . .Memory: 1444296K (1238320K) real, 1967080K (1468152K) virtual, 84908K free | | | 1 23 45Memory is not all of physcial, memory, it is: 分配給所有進程的物理內存DEDICATED to text, data or stack segments 所有runnable pr

29、ocesses的物理內存，不包括sleeping processes的。 分配給所有進程的虛擬內存DEDICATED to text, data or stack segments。不是所有都在物理內存，有部分在swap區中。分配給所有進程的虛擬內存，不包括sleeping or stopped processes新進程的可用物理內存，當比較低時，s. 35Top（續）Size is virtual memory sizeRES means the part in RAMWhat if too many CPUs?top hWhat if I want to get all process info?top n500 f top.out36glanceTotal VM : 121.1mb Sys Mem : 13.8mb User Mem: 91.3mb Phys Mem: 144.0mb Active VM: 73.7mb Buf Cache: 26.4mb Free Mem: 12.6mb Total VM: The total private virtual memory (in KBs unless otherwise specified) at t