1、Oracle Databases on VMwareRAC 安裝手冊Contents1.Introduction52.Infrastructure Environment62.1Hardware62.2Software Environment62.3VMware Virtual Machines62.4Virtual Disk Layout72.5Oracle RAC Database Environment72.6Network Configuration72.7Physical Architecture102.8Logical Architecture113.Overview of Dep

2、loyment Steps124.Configure NTP Client on All ESX Hosts135.Prepare a Virtual Machine for the First RAC Node155.1Create a Virtual Machine155.2Add Two NICs155.3Add CRS and Voting Disk165.4Install the Guest Operating System195.5Install VMware Tools215.6Timekeeping with the Linux Guest Operating System29

3、6.Prepare Subsequent Virtual Machines/RAC Nodes306.1Clone the First Virtual Machine/RAC Node306.2Configure CSR and Voting Disks326.3Format the Added Disks Using fdisk347.Install Oracle Grid Infrastructure358.Verify the Grid Infrastructure Installation379.Add DATA and REDO Disks to Virtual Machine RA

4、C Nodes3910.Create DATA and REDO ASM Disk Groups as Grid User4311.Install and Create the Oracle Database 11g R2 RAC4412.References451. 介紹本文通過一個案例簡要介紹 Oracle Real Application Clusters (RAC) 在 VMware vSphere® 平臺上的安裝過程, 幫助 Oracle 數據庫管理員在VMware 平臺上規劃和實施Oracle RAC環境. 在VMware環境下,數據庫管理員通常需要和VMware的管理員

5、以及存儲廠商的管理員合作,以順利部署一個Oracle RAC環境.在虛擬化平臺上部署Oracle RAC在許多方面與純物理環境上的部署很類似,比如:· 在虛擬機創建完成之后,安裝操作系統和Oracle RAC軟件的過程與物理環境下基本一致· 存儲的規劃,包括用于數據存放的LUN數量的規劃和多路徑的設計與物理環境也基本一致.差別在于我們可以充分依靠虛擬化軟件, 簡化許多工作, 從而加快安裝進度,減少錯誤的發生,比如:· 利用事先準備好的模板, 多次安裝同版本的系統和數據庫,同時加強標準化· 利用快照技術, 很方便的進行測試和回退· 利用虛擬機復制

6、技術,簡化多個RAC節點的系統安裝、打補丁、配置工作由于安裝過程的細節與軟件版本（尤其是Linux版本和Oracle版本）有較強的關聯性，本文只能盡量把安裝步驟描述清楚.讀者在自己進行安裝的時候，還需要參考相應版本的官方安裝文檔.先將本次安裝中用到的主要軟件和版本羅列如下：· VMware ESXi 5.1 · VMware vCenter Server 5.1 · vSphere Client 5.1 · Oracle Database 11g . · Oracle Enterprise Linux x86_64 6.3.2. 軟硬件環境2.

7、1 硬件組件數量配置ESXi Hosts2Cisco UCS B4404x 10-core CPUs, 256GB RAM, and 4 NICsStorage1NetApp F3240 雙路控制器Ethernet Switches2Cisco FI6248FC Switches1 Cisco Nexus 7000Host Based Adapters4每個刀片兩個，與網卡共享流量2.2 軟件環境組件版本作用vSphere5.1Hypervisor hosting all virtual machinesVMware vCenter5.1管理 vSphereOracle Enterprise

8、Linux x86_646.3Guest operating system for virtual machines running Oracle Database serverOracle Database 11g (with Oracle RAC and Oracle Grid Infrastructure) Enterprise EditionMicrosoft Windows Server2008 R2用于 vCenter Server2.3 虛擬機配置虛擬機用途數量配置Oracle RAC nodes220 vCPU, 120GB RAM,Oracle Enterpr

9、ise Linux x86_64 6.3vCenter12 vCPUs, 8GB RAM, Windows Server 2008 R22.4 共享存儲配置Datastore NameVirtual Disk on ESXLinux路徑虛擬路徑驅動類型VMDK Size (GB)用途VMDATASTOREVMDK Hard disk 1/dev/sdaSCSI 0:0LSI Logic150Oracle Enterprise Linux 5.5 OS and Oracle binariesCRS1VMDK Hard disk 2Shared disk/dev/sdb1SCSI 1:0Parav

10、irtual20CRS and voting diskREDO1VMDK Hard disk 3Shared disk/dev/sdc1SCSI 1:1Paravirtual900REDO and ArchieveDATA01VMDK Hard disk 4Shared disk/dev/sdd1SCSI 2:0Paravirtual1000RAC database DATADATA02VMDK Hard disk 5Shared disk/dev/sde1SCSI 2:1Paravirtual1000RAC database DATADATA03VMDK Hard disk 6Shared

11、disk/dev/sdf1SCSI 3:0Paravirtual1000RAC database DATADATA04VMDK Hard disk 7Shared disk/dev/sdg1SCSI 3:1Paravirtual1000RAC database DATA2.5 Oracle RAC 數據庫環境Node NameASM InstanceRAC InstanceDatabaseASM StorageRAC1+ASM1RAC1RACASM file systemRAC2+ASM2RAC2RACASM file system2.6 網絡配置按照Oracle RAC的實施慣例，每個RAC

12、節點都提供一個Public網口用于承載業務，并在其上捆綁虛擬IP地址 (VIP) 用于HA切換. 各RAC節點之間還需要一個私有網絡進行節點之間的通信。在物理環境中，通常需要考慮鏈路的冗余以消除單點鼓掌。而在虛擬化環境中，冗余問題由底層虛擬化軟件處理，所以對于每個RAC節點，只是簡單地需要兩個網口就可以工作，大大簡化了配置和安裝工作.我們這次測試的網絡配置如下三:· UCS底層軟件配置出三個物理網口（均為10Gb） vmnic0 和 vmnic1用于Oracle Public and Oracle Private網絡, vmnic2用于vSphere® vMotion

13、74;工作.· 兩個分布式虛擬交換機 Oracle Public (VM Net) and Oracle Private (Oracle HB).· 每個RAC節點配置3個IP地址 以節點1 (RAC1) 為例:o One public static IP address (eth0): 162.1o One private static IP address (eth1):o One virtual static IP address(VIP): 162.1（由CRS軟件配置和管理）還有三個虛擬IP用于節點間負載均衡（由CRS配置和管理）物理網口 vmnic0 連接到分布

14、式交換機 VM Net,vmnic1連接到分布式交換機Oracle HB， vMotion 使用標準交換機vSwitch3. 這樣區分概念上比較清晰，也可以只建一個分布式網絡，用不同的端口組區分數據網絡和心跳網。Figure 1. Network ConfigurationOracle RAC Private2.7 2.8 邏輯結構圖 Figure 2. Logical Architecture of Two-Node Oracle RAC on vSphere3. 部署流程概覽部署流程大體按照下面的流程圖進行.Figure 3. Deployment Steps for Four-Node

15、Oracle RAC on vSphere with VMFS4. 為第一個RAC節點準備虛擬機環境本節安裝步驟簡介如下:· 使用vSphere Client創建第一個虛擬機。添加公網和心跳網卡，挑選適當的LUN創建仲裁盤CRS （VMDK模式）。· 安裝Redhat Linux操作系統，安裝VMware tools，按照Oracle RAC要求安裝相應補丁.· 在Linux中配置NTP 客戶端.4.1 創建第一個虛擬機1. 使用vSphere Client登錄vCenter.2. 選中一臺主機,在Getting Started 頁面單擊 Create a new

16、 virtual machine.3. In the Create New Virtual Machine wizard, select Typical for the Configuration, and click Next.4. On the Name and Location page, enter RAC1 for the Name, and click Next.5. On the Datastore page, select VMDATASTORE, and click Next.6. On the Guest Operating System page, select Linu

17、x from the Guest Operating System list. For the Version, select Oracle Linux 6 (64-bit) from the drop-down menu. Click Next.7. On the Create a Disk page, set the Virtual Disk Size to 20GB, and click Next.8. On the Ready to Complete page, select Edit the virtual machine settings before completion. Cl

18、ick Continue.9. On the Virtual Machine Properties page for the RAC1 virtual machine, select 8 CPUs and 128GB Memory.10. Click Finish.4.2 增加兩個虛擬網卡現在需要增加兩塊網卡. 一塊用于公共數據網絡, 另一塊用于心跳網絡. 添加這兩塊網卡的過程基本相同,差別僅在于添加公網網卡時選擇標簽: VM Net 而添加心跳網卡時選擇 Oracle HB.添加公網網卡的步驟：1. Select a virtual machine.2. Click Edit setting

19、s to display the Virtual Machine Properties pane.3. Click Add.4. Select Ethernet Adaptor and click Next to display the Add Hardware wizard.5. On the Network connection page, select VMXNET 3 from the Adapter Type drop-down menu.6. In the Network Connection section of the Network connection page, sele

20、ct Named network with specified label.7. In the Named network with specified label drop-down menu, select VM Net.8. In the Device Status section of the Network connection page, select Connect at power on.9. On the Ready to Complete page, click Finish, and then click OK.添加心跳網卡的步驟：1. Repeat steps 1 th

21、rough 6 from the preceding procedure.2. In the Named network with specified label drop-down menu, select Oracle HB.3. Continue with steps 8 and 9 from the preceding procedure.4.3 添加仲裁盤Oracle RAC需要至少一個共享的仲裁盤存放clusterware信息和節點投票信息。存儲管理員需要事先為仲裁盤分配好單獨的LUN，然后我們需要創建單獨的SCSI控制器并設為Paravirtual驅動. SCSI 總線 shar

22、ing policy 設為 none.下面描述創建共享存儲的步驟. 為虛擬機創建 20GB datastore：1. In vSphere Client, select a virtual machine.2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.3. Click Add to display the Add Hardware wizard.4. In the Add Hardware wizard

23、select Hard Disk for the Device Type, and click Next.5. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 20GB, select the Support clustering features such as Fault Tolerance check box, and select CRS1 to specify a datastore. Also, for vSphere 5, i

24、n the Disk Provisioning section select Thick Provision Eager Zeroed. Click Next.6. On the Compatibility Mode page, click Next.7. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In the Mode section, select Independent. Click Next.8. On the Ready to Complete page,

25、click Finish.9. Click OK.Add configuration parameters for each Oracle RAC virtual machine as described in Disabling simultaneous write protection provided by VMFS using the multi-writer flag ().手工修改虛擬機參數1. In vSphere Client, select a virtual machine.2. Right-click the name of the virtual machine and

26、 select Edit Settings to display the Virtual Machine Properties pane.3. Click the Options tab, click Advanced, and click General.4. Click Configuration Parameters.5. On the Configuration Parameters page, add rows and entries for the shared disks in the virtual machine's configuration parameters

27、as shown in the following screenshot.6. Repeat steps 1 through 5 for each of the remaining virtual machines.You can view the virtual machine properties for the RAC1 virtual machine in vCenter as in the following screenshot.web client 顯示如下圖：4.4 安裝操作系統將OEL6.4安裝介質ISO文件上傳到datastore，然后進行安裝。Note: This doc

28、ument is specific to Linux guest operating system. Oracle RAC and above versions can be deployed and supported on the list of guest operation systems using VMware multi-writer flag ().在虛擬機上安裝Linux操作系統1. In vSphere Client, select a virtual machine.2. Click Edit settings.3. In the Hardware ta

29、b of the Virtual Machine Properties pane, select CD/DVD Drive 1.4. In the Device Status section of this pane, select the Connect at power on check box.5. In the Device Type section of this pane, select Datastore ISO File and enter the path to your Linux ISO image file.6. In the Virtual Device Node s

30、ection of this pane, select CD/DVD Drive 1 from the drop-down menu.7. Click OK.8. Power on the virtual machine.9. Install Linux with the following settings.Custom partitioning scheme:/dev/sda1 ext3 150GB mounted on /Installed packages use the default package selection, plus the following:· kern

31、el-uek.x86_64· kmod-oracleasm· (download from .)··10. Use below script to install remaining packages:yum install binutils compat-libcap1 compat-libstdc+-33 compat-libstdc+-33.i686 gcc.x86_64 gcc-c+.x86_64 glibc.x86_64 glibc.i686 glibc-devel.x86_64 glibc-devel.i686 unixODBC.i686 u

32、nixODBC.x86_64 unixODBC-devel.x86_64 unixODBC-devel.i686 libgcc.x86_64 libgcc.i686 libstdc+.x86_64 libstdc+.i686 libstdc+-devel.x86_64 libstdc+-devel.i686 compat-libstdc+-33.x86_64 compat-libstdc+-33.i686 libaio.x86_64 libaio.i686 libaio-devel.x86_64 libaio-devel.i686 sysstat.x86_64 ksh libXtst-deve

33、l.x86_64 libXtst.i686 compat-db.x86_64 xorg-x11-utils.x86_64 xorg-x11-xinit.x86_64 make11. View the installed packages with this command:run command “rpm q <package name>12. Confirm that the following packages have been installed:· binutils-.2-5.11.el6 (x86_64)· compat-libca

34、p1-1.10-1 (x86_64)· compat-libstdc+-33-3.2.3-69.el6 (x86_64)· compat-libstdc· gcc-4.4.4-13.el6 (x86_64)· gcc-c+-4.4.4-13.el6 (x86_64)· glibc-2.12-1.7.el6 (i686)· glibc-2.12-1.7.el6 (x86_64)· glibc-devel-2.12-1.7.el6 (x86_64)·· ksh· libgcc-4.4.4-13.el

35、6 (i686)· libgcc-4.4.4-13.el6 (x86_64)· libstdc+-4.4.4-13.el6 (x86_64)· libstdc· libstdc+-devel-4.4.4-13.el6 (x86_64)· libstdc· libaio-0.3.107-10.el6 (x86_64)·· libaio-devel-0.3.107-10.el6 (x86_64)···· sysstat-9.0.4-11.el6 (x86_64)· un

36、ixODBC-2.2.14-11.el6 (x86_64) or later· unixODBC-2.2.14-11.el6.i686 or later· unixODBC-devel-2.2.14-11.el6 (x86_64) or later· unixODBC-devel-2.2.14-11.el6.i686 or later13. 最后安裝下面兩個包:· oracle-rdbms-server-11gR2-preinstall (從YUM安裝，這個包會配置Oracle RAC所需系統參數和環境變量)· cvuqdisk (Oracle

37、安裝介質中提供這個包，用于安裝前檢查)14. 檢查配置文件/etc/sysctl.confkernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 6815744 (512 x processes for example 6815744 for 13312 processes)net.ipv4.ip_local_port_range = 9000 65500  = 262144  = 4194304  = 262144  = 1048576 

38、kernel.shmall = physical RAM size / pagesize For most systems, this will be the value 2097152. See Note: 301830.1 for more information. eg:kernel.shmall=4160749568(62GB)kernel.shnmax=   34359738368(32GB)kernel.shmmax = RAM times 0.5 (or higher at customer's discretion - seeNo

39、te:567506.1) /etc/profile ulimit u 16384 ulimit n 6553515. 依照oracle傳統，創建clusterware主用戶 “grid”:rootracnode1 # groupadd -g 1000 oinstall rootracnode1 # groupadd -g 1200 asmadminrootracnode1 # groupadd -g 1201 asmdba rootracnode1 # groupadd -g 1202 asmoper rootracnode1 # useradd -m -u 1100 -g oins

40、tall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid rootracnode1 # id grid uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall), 1200(asmadmin), 1201(asmdba),1202(asmoper) 設置 grid 帳戶的口令：rootracnode1 # passwd grid Changing password for

41、 user grid. New UNIX password:xxxxxxxxxxx Retype new UNIX password:xxxxxxxxxxx passwd: all authentication tokens updated successfully. 為grid用戶創建初始化腳本(/home/grid/.bash_profile):# -# .bash_profile# -# OS User: grid# Application: Oracle Grid Infrastructure# Version: Oracle 11g release 2# -# Get the ali

42、ases and functionsif -f /.bashrc ; then . /.bashrcfialias ls="ls -FA"# -# ORACLE_SID# -# Specifies the Oracle system identifier (SID)# for the Automatic Storage Management (ASM)instance# running on this node.# Each RAC node must have a unique ORACLE_SID.# (i.e. +ASM1, +ASM2,.)# -ORACLE_SID

43、=+ASM1; export ORACLE_SID# -# JAVA_HOME# -# Specifies the directory of the Java SDK and Runtime# Environment.# -JAVA_HOME=/usr/local/java; export JAVA_HOME# -# ORACLE_BASE# -# Specifies the base of the Oracle directory structure# for Optimal Flexible Architecture (OFA) compliant# installations. The

44、Oracle base directory for the# grid installation owner is the location where# diagnostic and administrative logs, and other logs# associated with Oracle ASM and Oracle Clusterware# are stored.# -ORACLE_BASE=/u01/app/grid; export ORACLE_BASE# -# ORACLE_HOME# -# Specifies the directory containing the

45、Oracle# Grid Infrastructure software. For grid# infrastructure for a cluster installations, the Grid# home must not be placed under one of the Oracle base# directories, or under Oracle home directories of# Oracle Database installation owners, or in the home# directory of an installation owner. Durin

46、g # installation, ownership of the path to the Grid # home is changed to root. This change causes # permission errors for other installations.# -ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME# -# ORACLE_PATH# -# Specifies the search path for files used by Oracle# applications such as SQL*Plus.

47、 If the full path to# the file is not specified, or if the file is not# in the current directory, the Oracle application# uses ORACLE_PATH to locate the file.# This variable is used by SQL*Plus, Forms and Menu.# -ORACLE_PATH=/u01/app/oracle/common/oracle/sql; export ORACLE_PATH# -# SQLPATH# -# Speci

48、fies the directory or list of directories that# SQL*Plus searches for a login.sql file.# -# SQLPATH=/u01/app/common/oracle/sql; export SQLPATH# -# ORACLE_TERM# -# Defines a terminal definition. If not set, it# defaults to the value of your TERM environment# variable. Used by all character mode produ

49、cts. # -ORACLE_TERM=xterm; export ORACLE_TERM# -# NLS_DATE_FORMAT# -# Specifies the default date format to use with the# TO_CHAR and TO_DATE functions. The default value of# this parameter is determined by NLS_TERRITORY. The# value of this parameter can be any valid date# format mask, and the value must be surrounded by # double quotation marks. For example:# # NLS_DATE_FORMAT = "MM/DD/YYYY"# # -NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS" export NLS_DATE_FORMAT# -# TNS_ADMIN# -# Specifies the directory containing the Oracle Net# Servi