1、第四章 資源管理龔 斌山東大學(xué)計(jì)算機(jī)科學(xué)與技術(shù)學(xué)院山東省高性能計(jì)算中心Globus與資源規(guī)范語(yǔ)言RSLGlobus的資源管理Globus RMSGRAMGRAMGRAMLSFCondorSGEEEApplicationRSLSimple ground RSLInformation ServiceRSLspecializationBrokerGround RSLCo-allocatorQueries& InfoGRAM(Globus Resource Allocation Manager) Overview定位：資源管理的最低層功能：遠(yuǎn)程運(yùn)行作業(yè)，通過(guò)提供的API提交，檢測(cè)與終止作業(yè)GRAM的具

2、體職責(zé)處理Resource Specification Language (RSL)形式的作業(yè)請(qǐng)求對(duì)創(chuàng)建的作業(yè)進(jìn)行遠(yuǎn)程監(jiān)控與管理更新MDS的信息Globus Pre-WS Component Interaction DiagramGSIGSIGSIGSIFrom IBM Redbook SG24-6895-012003: Intro to Grid ComputingGRAM: Grid Resource Allocation Manager GASS: Global Access to Secondary Storage（輔助存儲(chǔ)全局訪問(wèn)）MDS: Monitoring and Discov

3、ery Service GRIS: Grid Resource Information ServiceGIIS: Grid Index Information ServiceGRAM ArchitectureFrom IBM Redbook SG24-6895-012003: Intro to Grid ComputingGRAM ElementsClientsGatekeeper daemon門戶監(jiān)護(hù)進(jìn)程Job ManagerGlobal Access to Secondary Storage (GASS)輔助存儲(chǔ)全局訪問(wèn)Dynamically-Updated Request Online

4、Coallocator (DUROC)動(dòng)態(tài)更新請(qǐng)求在線協(xié)同分配器User Resource Specification Language (RSL)GRAM ClientsThree clients:globusrunglobus-job-runglobus-job-submitGRAM管理流程圖示Client APIJob ManagerScheduler Specific PluginJob ProcessGatekeeperJob RequestJob cancelstate change callbackfork/su/execfork/exec/waitspsubmit/spqcon

5、dor,lsfgatekeeper的作用gatekeeper：A process, running as root, which begins the process of handling allocation requestsperforming mutual authentication of user and resource,determining a local user name for the remote user,starting a job manager which executes as that local user and actually handles the

6、 request.In order to start the job manager, the gatekeeper must run as a privileged program相關(guān)名詞解釋Resource An entity capable of running one or more processes on behalf of a userClient The process that is using the resource allocation client-side APIJob A process or set of processes resulting from a j

7、ob request. Job Request A request to gatekeeper to create one or more job processes, expressed in the supplied Resource Specification Language.Job Manager One job manager is created by the gatekeeper to fulfill every request submitted to the gatekeeper.對(duì)各個(gè)階段的解釋Unsubmitted ：The job has not yet been s

8、ubmitted to the schedulerStageIn ：The job manager is staging executable, input, or data files to the jobPending ：The job has been submitted to the scheduler, but resources have not yet been allocated for the job.Active ：The job has received all of its resources, and the application is executingSuspe

9、nded ：The job has been stopped temporarily by the schedulerStageOut ：The job manager is staging output files from the job manager host to remote storage. Done ：The job completed successfully.Failed ：The job terminated before completion, as a result of an error, or a user or system cancel.GRAM Compon

10、entsGlobus SecurityInfrastructureJob ManagerGRAM client API calls to request resource allocationand process creation.MDS client API callsto locate resourcesQuery current statusof resourceCreateRSL LibraryParseRequestAllocate &create processesProcessProcessProcessMonitor &controlSite boundaryClientMD

11、S: Grid Index Info ServerGatekeeperMDS: Grid Resource Info ServerLocal Resource ManagerMDS client API callsto get resource infoGRAM client API statechange callbacksGRAM ExamplesThe globus-job-run client is a sample GRAM client, using command-line arguments rather than RSL.% globus-job-run /bin/ls% g

12、lobus-job-run s myprog% globus-job-run s myprog stdin s in.txt stdout s out.txt GRAM ExamplesThe globusrun client is a more involved prototype that allows complicated RSL expressions.% globusrun r f myjob.rsl% globusrun r &(executable=myprog)Resource Management APIsGlobus Toolkit has APIs for RSL, G

13、RAM, and DUROC:globus_rslglobus_gram_clientglobus_gram_myjobglobus_duroc_controlglobus_duroc_runtimeResource Specification Language可以用于說(shuō)明作業(yè)要求的通用語(yǔ)言RSL是GRAM的核心部分，它提供了不同組件之間交換信息的手段，比如應(yīng)用與資源代理之間，資源協(xié)同分配與資源管理之間的信息交換形式 (attribute=value) 需要GRAM理解這些屬性attribute Globus提供使用RSL的API可以用于以上之外的更多場(chǎng)合RSL的一些屬性(executable

14、=string)Program to runA file path (absolute or relative) or URL(directory=string)Directory in which to run (default is $HOME)(arguments=arg1 arg2 arg3.)List of string arguments to program(environment=(E1 v1)(E2 v2)List of environment variable name/value pairsRSL的一些屬性(stdin=string)Stdin for programA

15、file path (absolute or relative) or URL(stdout=string)Stdout for programA file path (absolute or relative) or URL(stderr=string)Stdout for programA file path (absolute or relative) or URL(count=integer)Number of processes to run (default is 1)(hostCount=integer)On SMP multi-computers, number of node

16、s to distribute the “count” processes across(project=string)Project (account) against which to charge(queue=string)Queue into which to submit jobRSL Attributes For GRAM(jobType=value)Value is one of “mpi”, “single”, “multiple”, or “condor”mpi: Run the program using “mpirun -np ”single: Only run a si

17、ngle instance of the program, and let the program start the other count-1 processes.multiple: Start instances of the program using the appropriate scheduler mechanismcondor: Start a Condor processes running in “standard universe”RSL Attributes for GRAM(gramMyjob=value)Value is one of “collective”, “

18、independent”Defines how the globus_gram_myjob library will operate on the processescollective: Treat all processes as part of a single jobindependent: Treat each of the processes as an independent uniprocessor job (dryRun=true)Do not actually run job替代符示例This&(rslSubstitution=(URLBASE “ftp:/host:123

19、4”) (rslSubstitution=(URLDIR $(URLBASE)/dir) (executable=$(URLDIR)/myfile)is equivalent to this&(executable=ftp:/host:1234/dir/myfile)GRAM Defined RSL SubstitutionsGRAM defines a set of RSL substitutions before processing the job requestMachine InformationGLOBUS_HOST_MANUFACTURERGLOBUS_HOST_CPUTYPEG

20、LOBUS_HOST_OSNAMEGLOBUS_HOST_OSVERSIONGRAM Defined RSL SubstitutionsPaths to GlobusGLOBUS_INSTALL_PATHGLOBUS_TOOLS_PATHGLOBUS_SERVICES_PATHGLOBUS_DEPLOY_PATHMiscellaneousHOMELOGNAMEGLOBUS_ID用于DUROC的RSL屬性(subjobStartType=value)Alters the startup barrier mechanismvalues are “strict-barrier”, “l(fā)oose-ba

21、rrier”, “no-barrier”(subjobCommsType=value)values are “blocking-join” and “independent”if value is set to “independent”, the subjob wont be seen from the other subjobs when doing inter-subjob communication.(label=string)Identifier for this subjob(resourceManagerContact=string)(resourceManagerName=st

22、ring)Resource manager to which to submit a subjobExample: (single resource for now)$ globusrun -r chi/jobmanager-pbs & (executable=/home/abose/test.exe) (host_count=2) (count=4) (arguments=“-t 100 f out.dat) (email_address=“abose) (queue=cac) (pbs_stagein=“morpheus:/home/abose/test.exe) (pbs_stageou

23、t=“morpheus:/home/abose/out.dat) (pbs_stdout=/tmp/stdout) (pbs_stderr=/tmp/stderr) (maxwalltime=10)(jobtype=mpi”)“get test.exe from morpheus and run it on hypnos” - submitted by Globus gatekeeper on chi using PBS job managerRSL Example Resulting PBS Submission Script on Hypnos:#! /bin/sh# PBS batch

24、job script built by Globus job manager#PBS -S /bin/sh#PBS -M abose#PBS -m n#PBS -q cac#PBS -W stagein=/home/abose/test.exe:/home/abose/test.exe#PBS -W stageout=/home/abose/out.dat:/home/abose/out.dat#PBS -l walltime=10:00#PBS -o hypnos:/tmp/stdout#PBS -e hypnos:/tmp/stderr#PBS -l nodes=2#PBS -vX509_

25、USER_PROXY=/home/abose/.globus/.gass_cache/local/md5/1c/fd/d3/753b9028dfec2ddd6df84cd06c/md5/0a/4b/1d/599dac54863d650c2531cb92fc/data,GLOBUS_LOCATION=/usr/grid,GLOBUS_GRAM_JOB_CONTACT=:58963/575/1047861360/,GLOBUS_GRAM_MYJOB_CONTACT=URLx-nexus:/:58964/,HOME=/home/abose,LOGNAME=abose,LD_LIBRARY_PATH=

26、#Change to directory requested by usercd /home/abose/usr/gmpi.pgi/bin/mpirun np 4 /home/abose/test.exe t 100 f out.dat Slides taken from NPACI Training, 2003Programming with Globus API Command line programs syntax: grid_* or globus_* Function calls/APIs start with globus_* Library binaries start wit

27、h libglobus_*.a Includes:#include /defines most common data structures and others depending on which modules/functions are called in the program. Module Activation/Deactivation:- Functions are arranged in several modules. The corresponding modules must be activated before calling a function: globus_

28、module_activate(MODULE_NAME) globus_module_deactivate(MODULE_NAME) globus_module_deactivate_all()GLOBUS_SUCCESS (0) is returned if successful. Example Module Names: GLOBUS_GRAM_CLIENT_MODULEGLOBUS_IO_MODULEGLOBUS_GASS_COPY_MODULE Dependencies among module activations exist. Read API documentation. 評(píng)

29、 價(jià)優(yōu)點(diǎn)：增加了對(duì)JOB資源的描述定義了很多Attribute，支持GRAM、DUROC等多種資源管理方式缺點(diǎn)：也是偏重于對(duì)計(jì)算資源和資源請(qǐng)求的描述，不夠廣泛可擴(kuò)展性不好目前僅用于Globus，還不被其他Grid項(xiàng)目所支持WWW服務(wù)描述語(yǔ)言WSDLWSDLWeb Service Description Language用于描 述Web服務(wù)的技術(shù)調(diào)用語(yǔ)法。WSDL定義了一套基于 XML的語(yǔ)法，將Web服務(wù)描述為能夠進(jìn)行消息交換的服務(wù)訪問(wèn)點(diǎn)的集合，從而滿足了這種需求。WSDL服務(wù)定義為分布式系統(tǒng)提供了可機(jī)器識(shí)別的SDK文檔，并且可用于描述自動(dòng)執(zhí)行應(yīng)用程序通信中所涉及的細(xì)節(jié)。WSDL的當(dāng)前版本是1.

30、1，規(guī)范可以從/TR/wsdl獲得。 WSDLWSDL由Ariba、Intel、IBM和微軟等開發(fā)商提出。它用一種和具體語(yǔ)言無(wú)關(guān)的抽象方式定義了給定Web服務(wù)收發(fā)的有關(guān)操作和消息。 WSDL保持協(xié)議中立，但它確實(shí)內(nèi)建了綁定SOAP的支持，從而同SOAP建立了不可分割的聯(lián)系。WSDL的信息模型WSDL信息模型充分利用了抽象規(guī)范與規(guī)范具體實(shí)現(xiàn)的分離，也就是分離了服務(wù)接口定義(抽象接口)與服務(wù)實(shí)現(xiàn)定義(具體端點(diǎn))。 抽象接口規(guī)范描述了終端的處理能力，它在WSDL中表示為portType。束定機(jī)制 (binding mechanism)在WSDL中表示為binding元素，它使用特定的通信協(xié)議、數(shù)據(jù)編

31、碼模型和底層通信協(xié)議，將Web服務(wù)的抽象定義映射至特定實(shí)現(xiàn)。若束定結(jié)合了實(shí)現(xiàn)的訪問(wèn)地址，抽象端點(diǎn)也就成為可供服務(wù)請(qǐng)求者調(diào)用的具體端點(diǎn)(concrete endpoint)，WSDL的port元素表示了這一結(jié)合。抽象接口可以支持任何數(shù)量的操作(operations)。操作是由一組消息(messages)定義，消息定義了操作的交互定式。與抽象的消息、操作概念相對(duì)應(yīng)的具體實(shí)現(xiàn)是由binding元素指定。與XML應(yīng)用相同，WSDL模式定義了幾個(gè)高層元素，或稱為主要元素。 WSDL描述的基本屬性服務(wù)做些什么-服務(wù)所提供的操作(方法)。如何訪問(wèn)服務(wù)-數(shù)據(jù)格式詳情以及訪問(wèn)服務(wù)操作的必要協(xié)議。服務(wù)位于何處-由

32、特定協(xié)議決定的網(wǎng)絡(luò)地址，如URL。 WSDL基本元素的含義元素名含義types定義了Web服務(wù)使用的所有數(shù)據(jù)類型集合，可被元素的各消息部件所引用。 message通信消息數(shù)據(jù)結(jié)構(gòu)的抽象類型化定義。使用Types所定義的類型來(lái)定義整個(gè)消息的數(shù)據(jù)結(jié)構(gòu)。 operation對(duì)服務(wù)中所支持操作的抽象描述。 portType對(duì)于某個(gè)訪問(wèn)入口點(diǎn)類型所支持操作的抽象集合。 binding包含了如何將抽象接口的元素(portType)轉(zhuǎn)變?yōu)榫唧w表示的細(xì)節(jié)，具體表示也就是指特定的數(shù)據(jù)格式和協(xié)議的結(jié)合 port定義為協(xié)議/數(shù)據(jù)格式綁定與具體Web訪問(wèn)地址組合的單個(gè)服務(wù)訪問(wèn)點(diǎn)。 service定義服務(wù)WSDL信息模

33、型 WSDL對(duì)象結(jié)構(gòu)圖 WSDL文檔類型 typestypestypestypesmessagemessageoperationoperationportTypebindingbindingportportportserviceserviceWSDL文檔結(jié)構(gòu)WSDL工具Omniopera-圖形用戶界面的WSDI、XML和XSD編輯器。Microsoft的SOAP Toolkit-一種工具包，其中包括根據(jù)WSDL定義創(chuàng)建COM接口的向?qū)С绦颍€包括根據(jù)COM接口創(chuàng)建WSDL的向?qū)С绦颉BM的Web Services Toolkit-一種工具包，其中包括產(chǎn)生WSDL和SOAP部署說(shuō)明的向?qū)С绦颉?/p>

34、 資源描述框架RDFRDFResource Description Framework, RDFW3C的資源描述框架(RDF)的目的是提供一個(gè)訪問(wèn)網(wǎng)絡(luò)資源元數(shù)據(jù)(metadata)的標(biāo)準(zhǔn)，因此也提供了一個(gè)描述特定資源內(nèi)容的標(biāo)準(zhǔn)協(xié)議。W3C應(yīng)用元數(shù)據(jù)時(shí)的推薦標(biāo)準(zhǔn)是一個(gè)模型，一種句法(syntax(es)應(yīng)用在Web上時(shí)，RDF 通常用XML來(lái)編碼是語(yǔ)義萬(wàn)維網(wǎng) (semantic Web)的基礎(chǔ)、支撐W3C - Resource Description Framework (RDF)/RDF/RDF是一個(gè)用于表達(dá)關(guān)于萬(wàn)維網(wǎng)(World Wide Web)上的資源的信息的語(yǔ)言。專門用于表達(dá)關(guān)于Web

35、資源的元數(shù)據(jù), 比如Web頁(yè)面的標(biāo)題、作者和修改時(shí)間，Web文檔的版權(quán)和許可信息，某個(gè)被共享資源的可用計(jì)劃表等 為什么要使用 RDF?RDF提供共享元數(shù)據(jù)的模型(model)共享語(yǔ)義(meaning)元數(shù)據(jù)可以在相互了解不多或根本不了解的應(yīng)用之間共享例如一個(gè)基于RDF的書目應(yīng)用能夠吸收基于RDF的地理空間應(yīng)用的元數(shù)據(jù)并對(duì)其意義有所理解。用(X)HTML和XML置標(biāo)后，軟件應(yīng)用必須能夠理解復(fù)雜的編碼RDF的基本思想用Web標(biāo)識(shí)符（稱作統(tǒng)一資源標(biāo)識(shí)符，Uniform Resource Identifiers或URIs）來(lái)標(biāo)識(shí)事物，用簡(jiǎn)單的屬性（property）及屬性值來(lái)描述資源。這使得RDF可以將一個(gè)或多個(gè)關(guān)于資源的簡(jiǎn)單陳述表示為一個(gè)由結(jié)點(diǎn)和弧組成的圖（graph），其中的結(jié)點(diǎn)和弧代表資源、屬性或?qū)傩灾怠?舉例有一個(gè)人由/People/EM/cont