Adios

Overview

The Adaptable IO System (ADIOS) provides a simple, flexible way for scientists to describe the data in their code that may need to be written, read, or processed outside of the running simulation. By providing an external to the code XML file describing the various elements, their types, and how you wish to process them this run, the routines in the host code (either Fortran or C) can transparently change how they process the data.

The in code IO routines were modeled after standard Fortran POSIX IO routines for simplicity and clarity. The additional complexity including organization into hierarchies, data type specifications, process grouping, and how to process the data is stored in an XML file that is read once on code startup. Based on the settings in this XML file, the data will be processed differently. For example, you could select MPI individual IO, MPI colletive IO, POSIX IO, an asynchronous IO technique, visualization engine, or even NULL for no output and cause the code to process the data differently without having to either change the source code or even recompile.

The real goal of this system is to give a level of adaptability such that the scientist can change how the IO in their code works simply by changing a single entry in the XML file and restarting the code. The ability to control at a per element basis and not just a data grouping such as a restart, diagnostic output, or analysis output makes this approach very flexible. Along with this detail level, a user can also just change which transport method is used for a data type such as a restart, analysis, or diagnostic write.

For the transport method implementer, the system provides a series of standard function calls to encode/decode data in the standardized .bp file format as well as “interactive” processing of the data by providing direct downcalls into the implementation for each data item written and also callbacks when processing a data stream once a data item has been identified along with its dimensions and a second callback once the data has been read giving the implementation the option to allocate memory and process the data as close to the data source as is reasonable.

Download & Revision History

Below are links for the Adios download and user manual. For additional support, please contact help@nccs.gov.

Download

Current stable version: ADIOS 1.4.1 (Download 1.4.1)

Revision History

1.4.1 Release Dec 2012
  • aggregated file reader method (from 1.3.1): use ADIOS_READ_METHOD_BP_AGGREGATE
  • memory limitation specified by user for chunked reading now taken into account in file reading
  • stage writer example code for staged I/O see examples/stage_write
  • code coupling example code for file-based/staged coupling see examples/coupling
  • bp2h5 supports converting complex and double complex types
  • new adios_write_byid() function to write multiple-blocks of the same global array from a process
  • fix: F90 modules: adios_write, adios_schedule_read
  • fix: invalid “out of bound” errors when reading multiple steps of multi-dimensional arrays
  • fix: double-free bug in support of old read API in Fortran
  • fix: backward compatibility of old read API to read multi-group files (fixed “invalid variable id…” errors)

1.4.0 Released July 2012
  • new read API for staging methods with step-by-step processing, also with non-blocking and chunking APIs
  • visualization schema added to ADIOS XML format
  • skel: I/O skeleton generator and evaluation tools
  • unified error/debug logging
  • hidden attributes (version, create/update times)
  • Java and Numpy bindings
  • F90 modules adios_write_mod and adios_read_mod to check syntax at compile time

1.3.1 Released Nov 2011

(Download 1.3.1)

  • fix: non-compliant MPI collective calls
  • fix: MPI_AMR method caused MPI_Finalize issues on some clusters
  • fix: histogram calculation: freeing memory too early

1.3 Released July 12, 2011
  • New read method BP_STAGED for reading files with improved performance
  • Changes in MPI_AMR and MPI_LUSTRE to support default parameters. Parameters for MPI_LUSTRE and MPI_AMR in XML file are not mandatory any more.

1.2.1 Released Aug 25, 2010
  • Bug fix in read API (to read arrays written into a file over several timesteps but without specifying time dimension)
  • Added DIMES method from Rutgers for tight-coupling applications

1.2 Released July 15, 2010
  • XML and non-XML write APIs are available
  • More statistics (min/max/avg/std.dev/histograms) available without overhead in write performance
  • Added MPI_AMR method for advanced mesh refinement codes and for aggregated writing (N procs to P writers to M disks)
  • Added support for subfiles
    • POSIX method many-file output can be read as one file
    • MPI_AMR method uses subfiles to improve write performance
  • Added NetCDF-4 transport method
  • Asynchronous, staging methods on Jaguar XT5 at ORNL
    • DataTap from Georgia Tech
    • DataSpaces from Rutgers
    • NSSI from Sandia
  • MPI_LUSTRE method for best performance on Lustre file system

1.0.1 Released on Dec 9, 2009
  • fix: builds read API on Mac (was broken at utils/bpdump)
  • fix: Fortran written BP files are correctly read by Fortran readers
  • added adios_mpi_stripe2 method for improved performance on Lustre file system

1.0 Released on Nov 18, 2009

Requirements and optional features
  • ADIOS requires MPI and MPI-IO.
  • ADIOS requires a version 2.5 or higher of Mini-XML.
    The Mini-XML library is used to parse XML configuration files. It can be downloaded from http://www.minixml.org/software.php
  • Python (required)
    The XML processing utility utils/gpp/gpp.py is a code written in python using xml.dom.minidom. It is used to generate C or Fortran code from the XML configuration files that can be included in the application source code. Examples and tests will not build without Python.
  • A Fortran 90 compiler (optional)
    The Fortran API is built only if a Fortran 90 compiler is available.
    Use --disable-fortran at configuration time to disable this feature.
  • NetCDF-3 (optional)
    • The bp2ncd converter utility to NetCDF format is built only if NetCDF is available. Currently ADIOS uses the NetCDF-3 library. Use the option --with netcdf= or ensure that the NETCDF_DIR environment variable is set before configuring ADIOS.
    • NetCDF version 3.6.2 can be downloaded from http://www.unidata.ucar.edu/downloads/netcdf
  • HDF5 (optional)
    • The bp2h5 converter utility to HDF5 format is built only if a HDF5 library is available. Currently ADIOS uses the 1.6 version of the HDF5 API but it can be built and used with the 1.8.x version of the HDF5 library too. Use the option --with hdf5= to configure ADIOS.
    • HDF5 can be downloaded from http://www.hdfgroup.org/downloads
  • Parallel HDF5 (optional)
    • The transport method writing files in the Parallel HDF5 format is built only if a parallel version of the HDF5 library is available. You need to use the option with phdf5= to build this transport method.If you define Parallel HDF5 and do not define serial HDF5, then bp2h5 will be built with the parallel library.Note that if you build this transport method, ADIOS will depend on PHDF5 when you link any application with ADIOS even if you the application does not intend to use this method.If you have problems compiling ADIOS with PHDF5 due to missing flags or libraries, you can define them using
        • --with-phdf5-incdir= ,
        • --with-phdf5-libdir= and
        • --with-phdf5-libs=
  • NetCDF-4 Parallel (optional)
  • DataSpaces staging (optional)
    • The DataSpaces model provides a separate server running on separate compute nodes, into/from which data can be written/read with a geometrical (3D) abstraction. It is an efficient way to stage data from one application to another in an asynchronous (and very fast) way. Multiple steps of data outputs can be stored, limited only by the available memory. DataSpaces can be downloaded from http://www.dataspaces.org.
    • DataSpaces is supported on Cray Gemini, Portals and Infiniband networks.
  • Matlab (optional)
    • If you have Matlab, you can build the ADIOS reader for Matlab separatly after building ADIOS. See details in the Matlab section of the INSTALL file before configuring adios.

Documents & Manuals

User’s Manual & Installation

User’s manual: ADIOS-UsersManual-1.4.1.pdf
Skel manual: Skel-manual.pdf
Visualization schema in ADIOS: ADIOS-vizschema.pdf
Developer’s manual: ADIOS-DevManual-1.4.1.pdf

Archived Manuals

Press & Publications

This page includes press releases and publications on Adios.

Press

ADIOS Ignites Combustion Simulations
http://www.hpcwire.com/features/ADIOS-Ignites-Combustion-Simulations-67321602.html

Fusion Gets Faster
http://www.hpcwire.com/features/Fusion-Gets-Faster-51820167.html?viewAll=y

Researchers Conduct Breakthrough Fusion Simulation
http://www.hpcwire.com/offthewire/Researchers_Conduct_Breakthrough_Fusion_Simulation.html

Publications

1. Cummings, Klasky, Podhorszki, Barreto, Lofstead, Schwan, Docan, Parashar, Sim, Shoshani, “EFFIS: and End-to-end Framework for Fusion Integrated Simulation”, submitted to PDP 2010, http://www.pdp2010.org/ <http://www.pdp2010.org/> .

2. N. Podhorszki, S. Klasky, Q. Liu, C. Docan, M. Parashar, H. Abbasi, J. Lofstead, K. Schwan, M. Wolf, F. Zheng, J. Cummings, “Plasma fusion code coupling using scalable I/O services and scientific workflows”, submitted to works08, http://www.isi.edu/works09/ <http://www.isi.edu/works09/> .

3. Bent, Gibson, Klasky, Lofstead, Parashar, Polte, Schwan, “… And eat it too: High read performance in write-optimized HPC I/O middlware file formats”, submitting to PDSW 2009.

4. Abbasi, H., Wolf, M., Eisenhauer, G., Klasky, S., Schwan, K., and Zheng, F. 2009. DataStager: scalable data staging services for petascale applications. In Proceedings of the 18th ACM international Symposium on High Performance Distributed Computing (Garching, Germany, June 11 – 13, 2009). HPDC ‘09. ACM, New York, NY, 39-48. DOI= http://doi.acm.org/10 <http://doi.acm.org/10> .

5. H. Abbasi, J. Lofstead, F. Zheng, S. Klasky, K. Schwan and M. Wolf, “Extending I/O through High Performance Data Services”, to appear at Cluster Computing 2009, New Orleans, LA, August 2009.