GRID SERVICES FOR COMPUTATIONAL CHEMISTRY AT THE RUSSIAN GRID POLYGONS: ANALYSIS OF THE STATE AND PERSPECTIVES Institute of Problem of Chemical Physics, - презентация

1 GRID SERVICES FOR COMPUTATIONAL CHEMISTRY AT THE RUSSIAN GRID POLYGONS: ANALYSIS OF THE STATE AND PERSPECTIVES Institute of Problem of Chemical Physics, Moscow district, Chernogolovka Varlamov Dmitry / Варламов Дмитрий Volokhov V.M., Volokhov A.V., Pivushkov A.V., Pokatovich G.A., Prokhorov A.I. Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

2 Chemistry – a science about materials, i.e. about the world in which we live… Constants of rates of reactions Industrial parameters of reactions Production of new and modified materials USERS: science, industry, business Features of synthesis of the new and changed materials The new and improved production technologies of materials Quantum-mechanical and molecular dynamics simulation Optimization of nanomaterials and nanostructures Quantum-chemical calculations: electronic structure of molecules, interaction potentials, substance structure etc Quantum-chemical calculations: electronic structure of molecules, interaction potentials, substance structure etc. Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

3 Quantum-statistical level Calculation of models taking into account the immediate environment of clusters of atoms and small molecules Hierarchical multilarge-scale approach to modeling of objects from quantum to macrolevel – all levels demand high efficience calculations Quantum-mechanical level Modeling by the ab initio methods of molecules and small clusters from atoms (possible phases in a material) Kinetic level System from 10 3 to 10 6 atoms, i.e. objects the size of nm 3 a.evolution of quantum non-equilibrium systems consisting of hundreds of clusters b.methods of molecular dynamics (equations of classical mechanics) for atoms and small molecules, considered as classical particles Mesoscopic level The viscosity, thermal conductivity, coefficient of friction, the wave processes in cells, meso-and macroscopic processes in materials, etc. The level of continuum media and construction level Defining the parameters of the finite element method, which implements a model of continuous media and the level of the model structures, based on the theory of mechanisms and machines, and the theory of complex systems Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

4 Typical needs of quantum chemistry for high- efficiency calculations 1.First two applied scientific calculations for Petaflops performance represented researches on molecular modeling: calculations of electronic structure of high-temperature superconductors (cuprates) by the quantum Monte-Carlo method and studying of effect of a huge magnetoresistance in magnetic nanoparticles (Jaguar, Oak Ridge, USA - 1,64 Pf) 2.Resource allocation (examples): researches of properties of water in low-dimensional systems – up to 8·10 6 CPU(core)-hours (Argonne National Laboratory); development of chemical catalysts – up to 30·10 6 CPU-hours (Jaguar) per year (2010); 3.Typical FULL docking of protein ligand: 200 atoms x configurations x 1000 core-hours – about 300 Pflops!!! 4.By some estimates practically most of the supercomputer centers of the USA (San Diego, Ohio, Illinois etc.) allocate to % of capacities for needs of biochemistry and molecular modeling, quantum chemistry, nanotechnological calculations Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

5 Research of catalitic hydrogen disintegration on nanostructures of Pt on a crystal surface of SnO 2 HH SKIF-MSU («Chebyshev»: VASP, 200 CPU, 15 hours, 10 steps of optimization, total requirement steps U(H 2) U(H+H) r H-H EXAMPLE OF ONE OF RESOURCE-INTENSIVE APPLICATION OF QUANTUM CHEMISTRY Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

6 Structure of the cluster Pt 19 put on a surface SnO 2, distances are given in Å, stability for separation of a cluster from a surface is resulted in eV VASP complex intended for carrying out of calculations taking into account translating symmetry is used. For each point of calculation it is necessary more than days of operation on 100 CPU, necessary quantity of points – more than hundred days Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

7 Dynamics of migration of a proton on a platinum cluster Dubna-2012, Distributed Computing and Grid-technologies in Science and Education VASP complex intended for carrying out of calculations taking into account translating symmetry is used. For each point of calculation it is necessary more than days of operation on 100 CPU, necessary quantity of points – more than hundred days

8 EXAMPLE OF ANOTHER OF RESOURCE-INTENSIVE APPLICATION Trajectory calculations of sections of chemical reactions H 2 +O 2 Σ=S n /N H+H+O+O H 2 +O+O H 2 O+O H 2 O 2 H+H+O 2 H+HO 2 HO+HO HO+O+H Start conditions: 4 angles of mutual orientation aim parameters 2 quantum oscillatory numbers 2 quantum rotational numbers energy of collision (in mass center system) Up to 1-10 millions independent trajectories – at least 2·10 6 CPU hours Chemical reactions are modeled by movement of atoms on potential surfaces within the limits of classical dynamics Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

9 Typical examples of supercomputer centers focused on quantum chemical and molecular dynamics calculations 1.The EPSRC (Engineering and Physical Sciences Research Council) UK National Service for Computational Chemistry Software (NSCCS, – >250 Tflops 2.The National Resource for Biomedical and Chemistry Supercomputing (NRBSC, - calculation of molecular systems from to atoms, provides resources to external users up to CPU- hours per structure; 3.Chemical Computing Group ( Quebec, Canada – >120 Tflops 4.Lawrence Berkeley National Laboratory (USA, chemical branch – > 450 Tflops; 5.Lehrstuhl für Theoretische Chemie der Technische Universität München, Germany ( - own Linux cluster (96 Tflops) plus constantly allocated resources of Supercomputer Center; 6.Swiss National Supercomputing Centre ( – from tenth Tflops to subPflops allocated resources for chemical calculations Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

10 Scientific directions of operations of IPCP IPCP RAS represents the largest in Russia academic institute which are carrying out researches in following areas: the theory of elementary chemical processes; structure of molecules and structure of solid state objects; creation of materials with in advance set properties; kinetics and mechanisms of complicate chemical reactions; nanotechnologies; creation of biologically active substances and medical drugs; Biotechnologies and biochemistry; and many other scientific directions…. Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

11 1.Государственный контракт от 23 сентября 2011 г. с ИПХФ РАН в рамках ФЦП «Исследования и разработки по приоритетным направлениям развития научно-технологического комплекса России на годы» по теме «Разработка технологии проведения высокопроизводительных расчетов в области вычислительной химии в различных распределенных средах с применением методов виртуализации приложений и ресурсов» – основной исполнитель 2.ФЦП "Развитие инфраструктуры наноиндустрии в Российской Федерации на годы", проект «Создание Национальной нанотехнологической сети (ГридННС)», государственный контракт от г. – соисполнитель 3. Государственный контракт в рамках ФЦП «Исследования и разработки по приоритетным направлениям развития научно-технологического комплекса России на годы» на тему «Развитие пилотной зоны российской грид- системы для высокопроизводительных вычислений, в том числе в интересах федеральных ядерных центров», тематика: «Адаптация пакетов прикладных программ для работы в пилотной инфраструктуре грид-системы. Разработка методов проведения грид-вычислений с использованием динамически формируемых сред исполнения грид-заданий» – соисполнитель УЧАСТИЕ ИПХФ В ГРИД-ПРОЕКТАХ Федеральные целевые программы Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

12 1.Программа 13 фундаментальных исследований Президиума РАН на годы «Проблемы создания национальной научной распределенной информационно- вычислительной среды на основе развития GRID технологий и современных телекоммуникационных сетей», проект «Исследование методов виртуализации вычислительных сред и приложений в области вычислительной химии. Динамическое формирование параллельных программных сред на распределенных ресурсах»; 2.Программа 27 фундаментальных исследований Президиума РАН на годы «Основы фундаментальных исследований наноматериалов», проект «Самоорганизация наноразмерных материалов и процессы их взаимодействия с адсорбируемыми соединениями: компьютерное моделирование в параллельных и распределенных GRID средах терафлопного уровня» Программы фундаментальных исследований Президиума РАН 1.Грант РФФИ а «Разработка методов динамического формирования параллельных вычислительных сред на различных типах грид ресурсов (на примере приложений квантовой химии)» Гранты Российского Фонда фундаментальных исследований Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

13 Historical sequence of usage of GRID technologies at IPCP г. Condor г. Globus 3/ г. LCG-2 – gLite (EGEE-RDIG) … Unicore (SKIF Polygon) …. Globus Toolkit 4 (GridNNN) The quantum-chemical applications adapted for usage in the distributed GRID environments: Gaussian 03 (parallel, with taking into account license restrictions) GAMESS US ( parallel – socket and MPI variants) CPMD ( parallel ) Dalton ( parallel ) NAMD (parallel) and many other: Authors programs – for example, investigations of tunnel properties of structures (non-stationary equation Schrödinger ) Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

14 On the basis of IPCP GRID center some Resource Sites of several Russian grid polygons (GridNNN, Minsvyaz GRID, SKIF-Polygon etc.) with support of a number of the established applied quantum-chemical packages (>10) and the Virtual Organizations (up to 3) oriented to computational chemistry constantly work Access to computing resources of a network of GridNNN (till Tf and to CPU) and to the large volume of the applied quantum and chemical software (Firefly, MolPro, LaMMPS, NAMD, AbInit etc.) supported by infrastructure of GridNNN is provided Dubna-2012, Distributed Computing and Grid-technologies in Science and Education Current achievements

15 Virtual Organizations (VO) in computational chemistry field within National Nanotechnological Network (GridNNN) Participants of GridNNN anв mentioned above VO НИВЦ МГУ («Чебышев») НИИЯФ МГУ ОИЯИ (Дубна) РНЦ Курчатовский Институт Казанский НЦ РАН ПИЯФ ИПХФ РАН ИМСС УрО РАН (Пермь) СПбГУ ВЦ ДВО РАН (Хабаровск) СПИИРАН и другие ВУЗы, НИИ РАН, инновационные и коммерческие предприятия The IPCP created and supports 3 Virtual organizations "NanoChem", "Gamess", "Gaussian" in the field of computational chemistry within infrastructure «National Nanotechnological Network» (GridNNN) for carrying out calculations with use applied quantum and chemical software (Gaussian, GAMESS, NAMD, etc.) on GRID polygon of the Russian National Nanotechnological Network (

16 Resource site of IPCP in SKIF grid polygon (category A site) ОС – Linux Ubuntu, Unicore 6.2 Middleware –Gateway ( –Server container (Unicore/X); –Target System Interface (TSI); –UAS (Unicore Atomic Services); –Unicore User Database – XUUDB; –PBS/Torque; –User Interface – Command-line client Access through the distributed environment to the main supercomputer resources of SKIF-Polygon is provided (for example, SKIF-MSU, different Russian sites) Quantum-chemical applications (Gamess-US, Gaussian, NAMD, authors multiparameter tasks) also are adapted for usage in Unicore environment Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

17 Use standard (from the point of view of the chemist and any end user) applied programs in grid environments is caused by several parameters: demand of applied packages scientific and industrial researchers, i.e. end users (there is no sense to carry out very labor-consuming adaptation and the subsequent control of the grid-environment for low-used packages); license restrictions (including existing for freely extended packages); for commercial applied packages of a condition of licensing can or limit extremely possibility of realization them in the grid-environment, or outright forbid it; availability of source code package that is highly desirable for some modules packages (eg, communication, or responsible for the implementation of parallel protocols); ability to work without an interactive user interaction and graphical display of information (which is still unlikely to realize in a grid environment). Quantum-chemical and molecular dynamic applied packages – features of use Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

18 Quantum-chemical and molecular dynamic applied packages on space of GridNNN and other grid polygons (all are established on a resource grid sites of IPCP) Quantum Chemistry: 1.Gaussian – including high-level web-interface 2.GAMESS-US – including high-level web-interface 3.FireFly 4.AbInit 5.NAMD 6.VASP 7.PWScf – including high-level web-interface 8.NWChem – including high-level web-interface Molecular dynamics 1.LAMMPS 2.OpenMX 3.GROMACS Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

19 Adaptation of the application package for use in grid environments 1.Grid Gateway settings for service of an applied package(s) and providing information on it to the monitor of resources 2.Creation of low-level interfaces for applied package(s) for solving of incoming jobs 3.Creation of high-level web-oriented user interfaces (POI, WIG) 4.In some cases – modification of source codes of applied packages Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

20 User certificate Simplified scheme of grid calculations with high-level interfaces to applied software packages (middleware GridNNN, gLite, Unicore) Data files Results Space of grid polygons Low-level interfaces to applied packages on Resource Sites ППП на ресурсных сайтах Authorised grid-user Authorization and certifications Loading, edition, storing the input data High-level formation of job Resource selection Job launch Jobs monitoring Return of the received results Formation of final result Web interfaces, POI, WIG Gateway to grid environment WWW-portal WIG(s) Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

21 Web-interfaces was created ( – grid portal of IPCP) to work with the quantum-chemical software packages GAMESS-US, Gaussian, NAMD, NWChem, PWscf etc. in virtual organizations of GridNNN and the pilot zone of the Russian grid system for high performance computing Friendly to the end user (researcher) Web interfaces to applied software packages considerably reduce labor input of work of the user with similar packages in the conditions of the distributed computing environments. The created interfaces allow the user to work via the Internet browser and to carry out the following actions: to authorize the user for start of a program complex and to carry out its certification in the Virtual Organization; to prepare a job (including creation and edition of initial data and configuration files) in compliance with package requirements; to start an applied package in grid-range infrastructure (if necessary – on any or chosen resource); to conduct monitoring of performance of a job (including stopping and restart); on termination of job – to receive results of the distributed calculations. Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

22 Example of module of Web-interface for quantum-chemical applied package GAMESS (for loading and edition initial files) Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

23 Module of the web interface for full-scale preparation of data and configuration files for a quantum-chemical GAMESS-US applied package in the distributed environments

24 Example of module of Web-interface for quantum-chemical applied package GAMESS (resource selection) Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

25 Example of module of Web-interface for quantum-chemical applied package GAMESS (monitoring of jobs) Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

26 The problems of creation and development of applied GRID services High economic costs of creation and operation of computing resources and infrastructure, absence of steady support of the Russian grid polygons; The absence (or low) interest from industry and business: –the general low culture of carrying out technological development in the majority of scientific and industrial organizations; –inability and unwillingness to use results of mathematical modeling and high- efficiency calculations. Absence of skilled professionals as from computer specialists (systems analysts, programmers), and by the users (chemists, engineers); Practical absence of domestic applied packages high software cost, the license restrictions, the reduced area of use, difficulty of development; Weak settlement of the legal and economic relations between owners of computing services (including GRID) and consumers Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

27 Suggested ways to solve problems Stimulation of the user(s) (industrial first of all) for use of computing GRID services: –absence or the minimum payment of using by GRID resources at the first stages of innovative development; –training of users as a full-time, and in a mode of interactive on-line - creation of information web resources on training to work with interfaces of applied packages and with GRID resources; –introduction in plans of training of applied experts (engineers, chemists, etc.) additional courses of work with applied software packages and carrying out by students of obligatory calculations of real scientific and technical and engineering tasks with use of supercomputers and GRID infrastructures; –creation friendly, with a large number of base templates, interfaces (WWW focused) to GRID resources; –introduction by the RF state as a certification condition when developing a part of commodity products and materials of obligatory application of means of mathematical modeling and testing Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

28 Suggested ways to solve problems (continued) Government policies and stimulation of developers of applied GRID services and resources –creation of the State (or equity), stable, constantly running computing infrastructure (supercomputers, clusters of medium-sized, high-speed communication channels, etc.); –compensation by the State of a part of operational costs of the supercomputer centers for support of the distributed calculations (grants or the budgetary financing); –tax privileges for development of the domestic applied software, creation of the innovative parks focused on it; –improvement of standard and legal and economic regulation in the field of strategic information technologies; –in the field of the State education: (a) expanding the number of budget places in universities for professions' system administration/programming, application programming etc.; (b) Use a wide range of undergraduate and graduate students of universities to develop the necessary system and application software in the learning process; (в) conduct training of university teachers in these specialties. Dubna-2012, Distributed Computing and Grid-technologies in Science and Education

29 Within the IPCP GRID Resource Center integrated into the National Nanotechnological Network and a pilot zone of the Russian grid-system for high-efficiency calculations, access to a number of quantum-chemical and molecular-dynamic applied packages for carrying out large-scale calculations on the Russian grid-ranges with use both simplified, and problem-oriented high-level web interfaces is realized Conclusions THANKS FOR YOUR ATTENTION!!!! Dubna-2012, Distributed Computing and Grid-technologies in Science and Education