grid superscalar a programming paradigm for grid ：grid超标量体系结构为网格编程范式

GRID superscalar: a programming paradigm for GRID applications,CEPBA-IBM Research InstituteRal Sirvent, Josep M. Prez, Rosa M. Badia, Jess Labarta* Ivan Rodero* Roman Roset, Rogeli Grima, Jose M. Cela,Outline,ObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusions,Objective,Ease the programming of GRID applicationsBasic idea:,Grid,ns seconds/minutes/hours,Outline,ObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusions,The essence,Assembly language for the GRIDSimple sequential programming, well defined operations and operands C/C+, Perl, Automatic run time “parallelization”Use architectural concepts from microprocessor designInstruction window (DAG), Dependence analysis, scheduling, locality, renaming, forwarding, prediction, speculation,The essence,for (int i = 0; i MAXITER; i+) newBWd = GenerateRandom(); subst (referenceCFG, newBWd, newCFG); dimemas (newCFG, traceFile, DimemasOUT); post (newBWd, DimemasOUT, FinalOUT); if(i % 3 = 0) Display(FinalOUT);fd = GS_Open(FinalOUT, R);printf(Results file:n); present (fd);GS_Close(fd);,The essence,CIRI Grid,The essence,Subst,DIMEMAS,EXTRACT,Subst,DIMEMAS,EXTRACT,GS_open,Subst,DIMEMAS,EXTRACT,Subst,DIMEMAS,EXTRACT,Subst,DIMEMAS,EXTRACT,Subst,DIMEMAS,EXTRACT,Subst,DIMEMAS,EXTRACT,Display,Display,CIRI Grid,Outline,ObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusions,Three components:InterfaceMain programTasks,Applications structure from developers point of view,interface MC void subst(in File referenceCFG, in double seed, out File newCFG);void dimemas(in File newCFG, in File traceFile, out File DimemasOUT);void post(in File newCFG, in File DimemasOUT, inout File FinalOUT);void display(in File toplot);,Interface: CORBA-IDL Like,Applications structure from developers point of view,Main program:for (int i = 0; i MAXITER; i+) newBWd = GenerateRandom(); subst (referenceCFG, newBWd, newCFG); dimemas (newCFG, traceFile, DimemasOUT); post (newBWd, DimemasOUT, FinalOUT); if(i % 3 = 0) Display(FinalOUT);fd = GS_Open(FinalOUT, R);printf(Results file:n); present (fd);GS_Close(fd);,Applications structure from developers point of view,Applications structure from developers point of view,void display(in File toplot) char command500; sprintf(command, ./display.sh %s, toplot); GS_System(command);,void dimemas(in File newCFG, in File traceFile, out File DimemasOUT) char command200; putenv(DIMEMAS_HOME=/usr/local/cepba-tools); sprintf(command, /usr/local/cepba-tools/bin/Dimemas -o %s %s, DimemasOUT, newCFG ); GS_System(command);,Tasks,Outline,ObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusions,Automatic code generation,app.idl,app-worker.c,app.c,app-functions.c,gsstubgen,app.h,app-stubs.c,Automatic code generation,app.idl,gsstubgen,app.i,app-worker.pl,swig,app_wrapper.c,app.pm,app-stubs.c,app.so,client,app.pl,C compiler,app-functions.pm,server,Automatic code generation,#include int gs_result; void Filter(file referenceCFG, double latency, double bandwidth, file newCFG) char buff_latencyGS_GENLENGTH; char buff_bandwidthGS_GENLENGTH; /* Parameter marshalling */ sprintf(buff_latency, %.20g, latency); sprintf(buff_bandwidth, %.20g, bandwidth); Execute(FilterOp, 1, 2, 1, 0, referenceCFG, buff_latency, buff_bandwidth, newCFG); ,Automatic code generation,#include int main(int argc, char *argv) enum operationCode opCod = (enum operationCode)atoi(argv2); IniWorker(argc, argv); switch(opCod) case FilterOp: double latency; double bandwidth; latency = strtod(argv4, NULL); bandwidth = strtod(argv5, NULL); Filter(argv3, latency, bandwidth, argv6); break; EndWorker(gs_result, argc, argv); return 0; ,Automatic code generation,client,GRID superscalarruntime,app-stubs.c,app.c,GT2,.,GT2 services: gsiftp, gram,Outline,ObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusions,Current run-time features,Data dependence analysis Renaming File forwardingShared disks management and file transfer policy Resource brokering Task scheduling Task Submission End of task notification Results collection Explicit task synchronization File management primitives,Current run-time features,Data dependence analysisDetects RaW, WaR, WaW dependencies based on file parametersOriented to simulations, FET solvers, bioinformatic applicationsMain parameters are data filesTasks Directed Acyclic Graph is built based on these dependencies,“f1_2”,“f1_1”,Current run-time features,File renamingWaW and WaR dependencies are avoidable with renaming,T1_1,T2_1,T3_1,T1_2,T2_2,T3_2,T1_N,T1_N,T1_N,“f1”,“f1”,“f1”,While (!end_condition() T1 (, “f1”); T2 (“f1”, , ); T3 (,);,WaR,WaW,Current run-time features,T1,T2,f1,T1,T2,f1 (by socket),File forwarding reduces the impact of RaW data dependencies,Current run-time features,Shared disks management and file transfer policy,client,server1,server2,T1,f1,f4,T6,f4,f7,f1,f7,Working directories,Current run-time features,Shared disks management and file transfer policy,client,server1,server2,f1,f4,f7,f1,f7,T1,T6,Working directories,Current run-time features,Shared disks management and file transfer policy,client,server1,server2,Input directories,Current run-time features,Resource brokeringNot a project goal External broker preferredInterface between run-time and brokerMDS is not currently used:Machine LimitOfJobs SubmitQueue ExecutionPath pcboada.ac.upc.es 2 none /home/sc03/NAS/worker/kadesh.cepba.upc.es 4 short /user1/uni/upc/ac/rosab/NAS_dis/worker/ khafre.cepba.upc.es 1 none /scratch/rosab/NAS_dis/worker/ pcboada.ac.upc.es /home/sc03/NAS/client/,Current run-time features,LinkBW OpId0SimulatedTime OpId1SimulatedTime . (and MasterLinkBW)1310720 0.25 15.55 1.3 0. 1 0.2 0.3 0.2 0.21310720 0.5 5.0 0.3 0.01 0.1 0.1 0.1 0.11310720 0.15 7.5 0.5 0.07 0.15 0.2 0.15 0.151310720Broker receives requests from the run-timeResources selected taking into account:FT = File transfer time to resource rET = Execution time of task t on resource r,Current run-time features,Task schedulingDistributed between the Execute call, the callback function and the GS_Barrier callPossibilities The task can be submitted immediately after being createdTask waiting for resourceTask waiting for data dependencyGS_Barrier primitive before ending the program that waits for all tasks,Current run-time features,Task submissionTask submitted for execution as soon as the data dependencies are solved if resources are availableComposed of File transferTask submissionAll specified in RSL Temporal directory created in the server working directory for each taskCalls to globus:globus_gram_client_job_request globus_gram_client_callback_allowglobus_poll_blocking,Current run-time features,End of task notificationAsynchronous state-change callbacks monitoring system globus_gram_client_callback_allow()callback_func functionData structures update in Execute function, GRID superscalar primitives and GS_Barrier,Current run-time features,Results collectionCollection of output parameters which are not filesPartial barrier synchronizationSocket and file mechanisms provided,Current run-time features,Implicit task synchronization GS_BarrierInserted in the user main program when requiredMain program execution is blockedglobus_poll_blocking() called Once all tasks are finished the program may resume,Current run-time features,File management primitivesGS_FOpen GS_FCloseGS_OpenGS_CloseMandatory for file management operations in main programOpening a file with write optionData dependence analysis Renaming is appliedOpening a file with read optionPartial barrier until the task that is generating that file as output file finishesInternally file management functions are handled as local tasksTask node inserted Data-dependence analysis Function locally executed,Current run-time features,Previous prototype over Condor and MWCurrent prototype over Globus 2.x, using the APIFile transfer, security, provided by GlobusRun-time implemented primitives GS_on, GS_offExecuteGS_Open, GS_Close, GS_FClose, GS_FOpenGS_BarrierWorker side: GS_System,Outline,ObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusions,Programming experiences,Performance modelling (Dimemas, Paramedir)Algorithm flexibilityNAS Grid BenchmarksImproved component programs modularityReduced Grid level source code linesBioinformatics application (production)Improved portability (Globus vs just Loadleveler)Reduced Grid level source code linesPblade solution for bioinformatics,Programming experiences,Performance modelling,Programming experiences,NAS Grid Benchmarks,Programming experiences,NAS Grid BenchmarksAll of them implemented with GRID superscalarRun with classes S, W, AResults scale as expectedWhen several servers are used, ASCII mode required,Programming experiences,Performance analysisGRID superscalar run-time instrumentedParaver tracefiles from the client sideMeasures of ta