Data intense scientific domains use data compression to reduce the storage space needed. Lossless data compression preserves information accurately but lossy data compression can achieve much higher compression rates depending on the tolerable error margins. There are many ways of defining precision and to exploit this knowledge, therefore, the field of lossy compression is subject to active research. From the perspective of a scientist, the qualitative definition about the implied loss of data precision should only matter.With the Scientific Compression Library (SCIL), we are developing a meta-compressor that allows users to define various quantities for acceptable error and expected performance behavior. The library then picks a suitable chain of algorithms yielding the user's requirements, the ongoing work is a preliminary stage for the design of an adaptive selector. This approach is a crucial step towards a scientifically safe use of much-needed lossy data compression, because it disentangles the tasks of determining scientific characteristics of tolerable noise, from the task of determining an optimal compression strategy. Future algorithms can be used without changing application code. In this paper, we evaluate various lossy compression algorithms for compressing different scientific datasets (Isabel, ECHAM6), and focus on the analysis of synthetically created data that serves as blueprint for many observed datasets. We also briefly describe the available quantities of SCIL to define data precision and introduce two efficient compression algorithms for individual data points. This shows that the best algorithm depends on user settings and data properties.

@article{TDTSOCAFQC17,
	author	 = {Julian Kunkel and Anastasiia Novikova and Eugen Betke},
	title	 = {{Towards Decoupling the Selection of Compression Algorithms from Quality Constraints – an
      Investigation of Lossy Compression Efficiency}},
	year	 = {2017},
	month	 = {12},
	editor	 = {Jack Dongarra and Vladimir Voevodin},
	publisher	 = {Publishing Center of South Ural State University},
	address	 = {454080, Lenin prospekt, 76, Chelyabinsk, Russia},
	journal	 = {Supercomputing Frontiers and Innovations},
	series	 = {Volume 4, Number 4},
	pages	 = {17--33},
	doi	 = {http://dx.doi.org/10.14529/jsfi1704},
	abstract	 = {Data intense scientific domains use data compression to reduce the storage space
      needed. Lossless data compression preserves information accurately but lossy data compression
      can achieve much higher compression rates depending on the tolerable error margins. There are
      many ways of defining precision and to exploit this knowledge, therefore, the field of lossy
      compression is subject to active research. From the perspective of a scientist, the
      qualitative definition about the implied loss of data precision should only matter.With the
      Scientific Compression Library (SCIL), we are developing a meta-compressor that allows users
      to define various quantities for acceptable error and expected performance behavior. The
      library then picks a suitable chain of algorithms yielding the user's requirements, the
      ongoing work is a preliminary stage for the design of an adaptive selector. This approach is a
      crucial step towards a scientifically safe use of much-needed lossy data compression, because
      it disentangles the tasks of determining scientific characteristics of tolerable noise, from
      the task of determining an optimal compression strategy. Future algorithms can be used without
      changing application code. In this paper, we evaluate various lossy compression algorithms for
      compressing different scientific datasets (Isabel, ECHAM6), and focus on the analysis of
      synthetically created data that serves as blueprint for many observed datasets. We also
      briefly describe the available quantities of SCIL to define data precision and introduce two
      efficient compression algorithms for individual data points. This shows that the best
      algorithm depends on user settings and data properties.},
	url	 = {http://superfri.org/superfri/article/view/149},
}

* Toward Decoupling the Selection of Compression Algorithms from Quality Constraints (Julian Kunkel, Anastasiia Novikova, Eugen Betke, Armin Schaare), In High Performance Computing: ISC High Performance 2017 International Workshops, DRBSD, ExaComm, HCPM, HPC-IODC, IWOPH, IXPUG, P^3MA, VHPC, Visualization at Scale, WOPSSS, Lecture Notes in Computer Science (10524), pp. 1–12, (Editors: Julian Kunkel, Rio Yokota, Michaela Taufer, John Shalf), Springer, ISC High Performance, Frankfurt, Germany, ISBN: 978-3-319-67629-6, 2017
Publication details – DOI

Data intense scientific domains use data compression to reduce the storage space needed. Lossless data compression preserves the original information accurately but on the domain of climate data usually yields a compression factor of only 2:1. Lossy data compression can achieve much higher compression rates depending on the tolerable error/precision needed. Therefore, the field of lossy compression is still subject to active research. From the perspective of a scientist, the compression algorithm does not matter but the qualitative information about the implied loss of precision of data is a concern. With the Scientific Compression Library (SCIL), we are developing a meta-compressor that allows users to set various quantities that define the acceptable error and the expected performance behavior. The ongoing work a preliminary stage for the design of an automatic compression algorithm selector. The task of this missing key component is the construction of appropriate chains of algorithms to yield the users requirements. This approach is a crucial step towards a scientifically safe use of much-needed lossy data compression, because it disentangles the tasks of determining scientific ground characteristics of tolerable noise, from the task of determining an optimal compression strategy given target noise levels and constraints. Future algorithms are used without change in the application code, once they are integrated into SCIL. In this paper, we describe the user interfaces and quantities, two compression algorithms and evaluate SCIL’s ability for compressing climate data. This will show that the novel algorithms are competitive with state-of-the-art compressors ZFP and SZ and illustrate that the best algorithm depends on user settings and data properties.

@inproceedings{TDTSOCAFQC17,
	author	 = {Julian Kunkel and Anastasiia Novikova and Eugen Betke and Armin Schaare},
	title	 = {{Toward Decoupling the Selection of Compression Algorithms from Quality Constraints}},
	year	 = {2017},
	booktitle	 = {{High Performance Computing: ISC High Performance 2017 International Workshops, DRBSD,
      ExaComm, HCPM, HPC-IODC, IWOPH, IXPUG, P^3MA, VHPC, Visualization at Scale, WOPSSS}},
	editor	 = {Julian Kunkel and Rio Yokota and Michaela Taufer and John Shalf},
	publisher	 = {Springer},
	series	 = {Lecture Notes in Computer Science},
	number	 = {10524},
	pages	 = {1--12},
	conference	 = {ISC High Performance},
	location	 = {Frankfurt, Germany},
	isbn	 = {978-3-319-67629-6},
	doi	 = {https://doi.org/10.1007/978-3-319-67630-2},
	abstract	 = {Data intense scientific domains use data compression to reduce the storage space
      needed. Lossless data compression preserves the original information accurately but on the
      domain of climate data usually yields a compression factor of only 2:1. Lossy data compression
      can achieve much higher compression rates depending on the tolerable error/precision needed.
      Therefore, the field of lossy compression is still subject to active research. From the
      perspective of a scientist, the compression algorithm does not matter but the qualitative
      information about the implied loss of precision of data is a concern. With the Scientific
      Compression Library (SCIL), we are developing a meta-compressor that allows users to set
      various quantities that define the acceptable error and the expected performance behavior. The
      ongoing work a preliminary stage for the design of an automatic compression algorithm
      selector. The task of this missing key component is the construction of appropriate chains of
      algorithms to yield the users requirements. This approach is a crucial step towards a
      scientifically safe use of much-needed lossy data compression, because it disentangles the
      tasks of determining scientific ground characteristics of tolerable noise, from the task of
      determining an optimal compression strategy given target noise levels and constraints. Future
      algorithms are used without change in the application code, once they are integrated into
      SCIL. In this paper, we describe the user interfaces and quantities, two compression
      algorithms and evaluate SCIL’s ability for compressing climate data. This will show that the
      novel algorithms are competitive with state-of-the-art compressors ZFP and SZ and illustrate
      that the best algorithm depends on user settings and data properties.},
}