1、內存泄漏問題調查XXX版本內存泄漏XXX版本內存泄漏長期掛機結果Mem-info:DMA per-cpu:CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0Active:3802 inactive:0 dirty:0 writeback:0 unstable:0 free:127 slab:1684 mapped:1 pagetables:124 bounce:0DMA free:508kB min:720kB low:900kB high:1080kB active:15208kB inactive:0kB presen

2、t:32512kB pages_scanned:3286541 all_unreclaimable? yeslowmem_reserve: 0 0DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 508kBFree swap: 0kB8192 pages of RAM0 pages of HIGHMEM1094 reserved pages30 pages shared0 pages swap cachedprintk: 12189 messages suppre

3、ssed.phone_voice invoked oom-killer: gfp_mask=0 x201d2, order=0, oomkilladj=0Call Trace: dump_stack+0 x8/0 x34 out_of_memory+0 x1f4/0 x214 _alloc_pages+0 x32c/0 x334 _do_page_cache_readahead+0 x1f4/0 x294 filemap_nopage+0 x2a4/0 x4d4 _handle_mm_fault+0 x3c8/0 x924 do_page_fault+0 x104/0 x3d0 ret_fro

4、m_exception+0 x0/0 x18表現現象：系統crash，不會自動重啟大量進程被killWeb和telnet無響應可以ping通看門狗進程還保持運行用戶態進程內存 32位操作系統中，每個進程擁有4G的虛擬內存空間，其中03GB是每個進程的私有用戶空間，這個空間對系統中其他進程是不可見的。34GB是linux內核空間，由系統所有的進程以及內核所共享的。 進程占用內存主要有如下幾個區域： 1、當前執行文件的代碼段，該代碼段稱為text段。 2、執行文件的數據段，主要存儲執行文件用到的全局變量，靜態變量。 3、存儲全局變量和動態產生的數據的堆。 4、用于保存局部變量和實現函數調用的棧。

5、5、采用mmap方式映射到虛擬地址空間中的內存段查看內存的幾個基本方法 Top 命令 /proc/meminfo /proc/pid/maps /proc/pid/statusTOP命令介紹 PID : 進程ID PPID : 父進程ID，1表示init進程, USER :當前用戶 STAT : 進程當前運行狀態 VSZ ：此進程一共占用了多大物理內存。 %MEM : 此進程占總內存消耗的百分比 %CPU : 此進程占總CPU消耗的百分比 COMMAND : 進程啟動時的命令行格式/PROC/MEMINFO $cat /proc/meminfoMemTotal: 2052440 kB /總內存

6、MemFree: 50004 kB /空閑內存Buffers: 19976 kB /給文件的緩沖大小Cached: 436412 kB /高速緩沖存儲器使用的大小SwapCached:19864 kB /被高速緩沖存儲用的交換空間大小Active: 1144512 kB /活躍使用中的高速緩沖存儲器頁面文件大小Inactive: 732788 kB /不經常使用的高速緩沖存儲器頁面文件大小 可用的物理內存=memfree+buffers+cached， 當memfree不夠時，內核會通過回寫機制(pdflush線程)把cached和buffered內存回寫到后備存儲器，從而釋放相關內存供進程使

7、用，或者通過手動方式顯式釋放cache內存/PROC/PID/MAPS# cat maps00400000-00414000 r-xp 00000000 1f:05 193 /bin/wanManager 代碼段00453000-00454000 rw-p 00013000 1f:05 193 /bin/wanManager 數據段00454000-0045d000 rwxp 00454000 00:00 0 heap 堆2aaa8000-2aaae000 r-xp 00000000 1f:05 366 /lib/ld-uClibc-0.9.28.so 程序連接的共享庫的內存地址2aaae00

8、0-2aaaf000 rw-p 2aaae000 00:00 0 2aaed000-2aaee000 r-p 00005000 1f:05 366 /lib/ld-uClibc-0.9.28.so2aaee000-2aaef000 rw-p 00006000 1f:05 366 /lib/ld-uClibc-0.9.28.so2aaef000-2ab4d000 r-xp 00000000 1f:05 319 /lib/libuClibc-0.9.28.so2ab4d000-2ab8c000 -p 2ab4d000 00:00 0 2ab8c000-2ab8d000 r-p 0005d000 1

9、f:05 319 /lib/libuClibc-0.9.28.so2ab8d000-2ab8e000 rw-p 0005e000 1f:05 319 /lib/libuClibc-0.9.28.so2ab8e000-2ab93000 rw-p 2ab8e000 00:00 0 2ab93000-2ab96000 r-xp 00000000 1f:05 313 /lib/libnvram-0.9.28.so2ab96000-2abd6000 -p 2ab96000 00:00 0 2abd6000-2abe4000 rw-p 00003000 1f:05 313 /lib/libnvram-0.

10、9.28.so2abe4000-2abe8000 r-xp 00000000 1f:05 365 /lib/libcfg.so2abe8000-2ac28000 -p 2abe8000 00:00 0 2ac28000-2ac29000 rw-p 00004000 1f:05 365 /lib/libcfg.so2ac29000-2ac38000 r-xp 00000000 1f:05 310 /lib/libpthread-0.9.28.so2ac38000-2ac77000 -p 2ac38000 00:00 0 2ac77000-2ac7c000 rw-p 0000e000 1f:05

11、310 /lib/libpthread-0.9.28.so2ac7c000-2ac7e000 rw-p 2ac7c000 00:00 0 線程棧7ff41000-7ff56000 rwxp 7ff41000 00:00 0 stack 進程棧1）代表內存段的虛擬地址2）代表執行權限，r，w，x，p=私有 s=共享3) 代表在進程地址里的偏移量4) 映射文件的主設備號和次設備號5) 映像文件的節點號，即inode6) 映像文件的路徑進程內存布局/PROC/PID/STATUS# cat statusName: wanManagerState: S (sleeping)SleepAVG: 97%T

12、gid: 2625Pid: 2625PPid: 1TracerPid: 0Uid: 0 0 0 0Gid: 0 0 0 0FDSize: 32Groups:VmPeak: 1836 kBVmSize: 1836 kBVmLck: 0 kBVmHWM: 492 kBVmRSS: 492 kBVmData: 1084 kBVmStk: 84 kBVmExe: 80 kBVmLib: 488 kBVmPTE: 16 kBThreads: 1SigQ: 1/256SigPnd: 00000000000000000000000000000000ShdPnd: 0000000000000000000000

13、0000000000SigBlk: 00000000000000000000000080000000SigIgn: 00000000000000000000000000001006SigCgt: 0000000000000000000000038001a000CapInh: 0000000000000000CapPrm: 00000000fffffeffCapEff: 00000000fffffeff VmSize:整個進程使用虛擬內存大小，是VmLib,VmExe,VmData,和VmStk的總和。 VmLck:虛擬內存鎖。進程當前使用的并且加鎖的虛擬內存總數 VmHWM: 表示進程所占用物

14、理內存的峰值 VmRSS:虛擬內存駐留集合大小。這是駐留在物理內存的一部分。它沒有交換到硬盤。它包括代碼，數據和棧。 VmData:虛擬內存數據。堆堆使用的虛擬內存。 VmStk:虛擬內存棧 棧棧使用的虛擬內存 VmExe:可執行的虛擬內存, 可執行的和靜態鏈接庫所使用的虛擬內存 VmLib:虛擬內存庫動態鏈接庫所使用的虛擬內存XX001版本內存監控實驗內存觀察：初始狀態# cat statusName: wanManagerState: S (sleeping)SleepAVG: 97%Tgid: 2625Pid: 2625PPid: 1TracerPid: 0Uid: 0 0 0 0Gid

15、: 0 0 0 0FDSize: 32Groups:VmPeak: 1836 kBVmSize: 1836 kBVmLck: 0 kBVmHWM: 492 kBVmRSS: 492 kBVmData: 1084 kBVmStk: 84 kBVmExe: 80 kBVmLib: 488 kBVmPTE: 16 kBThreads: 1SigQ: 1/256SigPnd: 00000000000000000000000000000000ShdPnd: 00000000000000000000000000000000SigBlk: 00000000000000000000000080000000Si

16、gIgn: 00000000000000000000000000001006SigCgt: 0000000000000000000000038001a000CapInh: 0000000000000000CapPrm: 00000000fffffeffCapEff: 00000000fffffeff# cat maps00400000-00414000 r-xp 00000000 1f:05 193 /bin/wanManager00453000-00454000 rw-p 00013000 1f:05 193 /bin/wanManager00454000-0045d000 rwxp 004

17、54000 00:00 0 heap2aaa8000-2aaae000 r-xp 00000000 1f:05 366 /lib/ld-uClibc-0.9.28.so2aaae000-2aaaf000 rw-p 2aaae000 00:00 0 2aaed000-2aaee000 r-p 00005000 1f:05 366 /lib/ld-uClibc-0.9.28.so2aaee000-2aaef000 rw-p 00006000 1f:05 366 /lib/ld-uClibc-0.9.28.so2aaef000-2ab4d000 r-xp 00000000 1f:05 319 /li

18、b/libuClibc-0.9.28.so2ab4d000-2ab8c000 -p 2ab4d000 00:00 0 2ab8c000-2ab8d000 r-p 0005d000 1f:05 319 /lib/libuClibc-0.9.28.so2ab8d000-2ab8e000 rw-p 0005e000 1f:05 319 /lib/libuClibc-0.9.28.so2ab8e000-2ab93000 rw-p 2ab8e000 00:00 0 2ab93000-2ab96000 r-xp 00000000 1f:05 313 /lib/libnvram-0.9.28.so2ab96

19、000-2abd6000 -p 2ab96000 00:00 0 2abd6000-2abe4000 rw-p 00003000 1f:05 313 /lib/libnvram-0.9.28.so2abe4000-2abe8000 r-xp 00000000 1f:05 365 /lib/libcfg.so2abe8000-2ac28000 -p 2abe8000 00:00 0 2ac28000-2ac29000 rw-p 00004000 1f:05 365 /lib/libcfg.so2ac29000-2ac38000 r-xp 00000000 1f:05 310 /lib/libpt

20、hread-0.9.28.so2ac38000-2ac77000 -p 2ac38000 00:00 0 2ac77000-2ac7c000 rw-p 0000e000 1f:05 310 /lib/libpthread-0.9.28.so2ac7c000-2ac7e000 rw-p 2ac7c000 00:00 0 7ff41000-7ff56000 rwxp 7ff41000 00:00 0 stack內存觀察：半小時后# cat statusName: wanManagerState: S (sleeping)SleepAVG: 97%Tgid: 2625Pid: 2625PPid: 1

21、TracerPid: 0Uid: 0 0 0 0Gid: 0 0 0 0FDSize: 32Groups:VmPeak: 1864 kBVmSize: 1864 kBVmLck: 0 kBVmHWM: 520 kBVmRSS: 520 kBVmData: 1112 kBVmStk: 84 kBVmExe: 80 kBVmLib: 488 kBVmPTE: 16 kBThreads: 1SigQ: 0/256SigPnd: 00000000000000000000000000000000ShdPnd: 00000000000000000000000000000000SigBlk: 0000000

22、0000000000000000080000000SigIgn: 00000000000000000000000000001006SigCgt: 0000000000000000000000038001a000CapInh: 0000000000000000CapPrm: 00000000fffffeffCapEff: 00000000fffffeff# cat maps00400000-00414000 r-xp 00000000 1f:05 193 /bin/wanManager00453000-00454000 rw-p 00013000 1f:05 193 /bin/wanManage

23、r00454000-00463000 rwxp 00454000 00:00 0 heap2aaa8000-2aaae000 r-xp 00000000 1f:05 366 /lib/ld-uClibc-0.9.28.so2aaae000-2aaaf000 rw-p 2aaae000 00:00 0 2aaed000-2aaee000 r-p 00005000 1f:05 366 /lib/ld-uClibc-0.9.28.so2aaee000-2aaef000 rw-p 00006000 1f:05 366 /lib/ld-uClibc-0.9.28.so2aaef000-2ab4d000

24、r-xp 00000000 1f:05 319 /lib/libuClibc-0.9.28.so2ab4d000-2ab8c000 -p 2ab4d000 00:00 0 2ab8c000-2ab8d000 r-p 0005d000 1f:05 319 /lib/libuClibc-0.9.28.so2ab8d000-2ab8e000 rw-p 0005e000 1f:05 319 /lib/libuClibc-0.9.28.so2ab8e000-2ab93000 rw-p 2ab8e000 00:00 0 2ab93000-2ab96000 r-xp 00000000 1f:05 313 /

25、lib/libnvram-0.9.28.so2ab96000-2abd6000 -p 2ab96000 00:00 0 2abd6000-2abe4000 rw-p 00003000 1f:05 313 /lib/libnvram-0.9.28.so2abe4000-2abe8000 r-xp 00000000 1f:05 365 /lib/libcfg.so2abe8000-2ac28000 -p 2abe8000 00:00 0 2ac28000-2ac29000 rw-p 00004000 1f:05 365 /lib/libcfg.so2ac29000-2ac38000 r-xp 00

26、000000 1f:05 310 /lib/libpthread-0.9.28.so2ac38000-2ac77000 -p 2ac38000 00:00 0 2ac77000-2ac7c000 rw-p 0000e000 1f:05 310 /lib/libpthread-0.9.28.so2ac7c000-2ac7e000 rw-p 2ac7c000 00:00 0 7ff41000-7ff56000 rwxp 7ff41000 00:00 0 stack結果 wanManager 的堆使用可能有問題，一直在增長 檢查wanManager代碼中任何關于對堆內存的使用 malloc allo

27、c realloc strdup（內部會調用malloc）西電版本錯誤（FIND_PID_BY_NAME函數） /* Buffer should contain a string like Name: binary_name */ sscanf(buffer, %*s %s, name); if (strcmp(name, pidName) = 0) /printf(name:%sn,name); pidList=realloc( pidList, sizeof(pid_t) * (i+2); pidListi+=strtol(next-d_name, NULL, 0); break; clo

28、sedir(dir); if (pidList) pidListi=0; /printf(pid:%ldn,pidList0); return pidList0;函數直接返回堆指針，如果調用者沒有釋放，就會造成內存泄漏。XXX002版本內存監控實驗/PROC/PID/FD fd目錄包含了所有該進程使用的文件描述符# ls -l fdlr-x- 1 0 0 64 6 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 5 - /tmp/session_vbugkhffltibixfzlrwx- 1 0 0 64 2 - /dev/consolelrwx-

29、1 0 0 64 1 - /dev/consolelr-x- 1 0 0 64 0 - /dev/null10分鐘后# ls -l /proc/1016/fdlr-x- 1 0 0 64 14 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 13 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 12 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 11 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 10 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 9 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 8 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 7 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 6 - /tmp/session_vbugkhffltibixfzlr-x- 1 0 0 64 5 - /tmp/session_vbugkhffltibixfzlrwx- 1 0 0 64 2 - /dev/consol