The IceProd framework : Distributed data processing for the IceCube neutrino observatory

Aartsen, M. G.; Abbasi, R.; Ackermann, M.; Adams, J.; Aguilar, J. A.; Ahlers, M.; Altmann, D.; Arguelles, C.; Auffenberg, J.; Bai, X.; Baker, M.; Barwick, S. W.; Baum, V.; Bay, R.; Beatty, J. J.; Tjus, J. B.; Becker, K. H.; BenZvi, S.; Berghaus, P.; Berley, D.; Bernardini, E.; Bernhard, A.; Besson, D. Z.; Binder, G.; Bindig, D.; Bissok, M.; Blaufuss, E.; Blumenthal, J.; Boersma, D. J.; Bohm, C.; Bose, D.; Boser, S.; Botner, O.; Brayeur, L.; Bretz, H. P.; Brown, A. M.; Bruijn, R.; Casey, J.; Casier, M.; Chirkin, D.; Christov, A.; Christy, B.; Clark, K.; Classen, L.; Clevermann, F.; Coenders, S.; Cohen, S.; Cowen, D. F.; Silva, A. H. C.; Danninger, M.; Daughhetee, J.; Davis, J. C.; Day, M.; De Clercq, C.; De Ridder, S.; Desiati, P.; de Vries, K. D.; de With, M.; DeYoung, T.; Diaz-Velez, J. C.; Dunkman, M.; Eagan, R.; Eberhardt, B.; Eichmann, B.; Eisch, J.; Euler, S.; Evenson, P. A.; Fadiran, O.; Fazely, A. R.; Fedynitch, A.; Feintzeig, J.; Feusels, T.; Filimonov, K.; Finley, C.; Fischer-Wasels, T.; Flis, S.; Franckowiak, A.; Frantzen, K.; Fuchs, T.; Gaisser, T. K.; Gallagher, J.; Gerhardt, L.; Gladstone, L.; Glusenkamp, T.; Goldschmidt, A.; Golup, G.; Gonzalez, J. G.; Goodman, J. A.; Gora, D.; Grandmont, D. T.; Grant, D.; Gretskov, P.; Groh, J. C.; Gross, A.; Ha, C.; Ismail, A. H.; Hallen, P.; Hallgren, A.; Halzen, F.; Hanson, K.; Hebecker, D.; Heereman, D.; Heinen, D.; Helbing, K.; Hellauer, R.; Hickford, S.; Hill, G. C.; Hoffman, K. D.; Hoffmann, R.; Homeier, A.; Hoshina, K.; Huang, F.; Huelsnitz, W.; Hulth, P. O.; Hultqvist, K.; Hussain, S.; Ishihara, A.; Jacobi, E.; Jacobsen, J.; Jagielski, K.; Japaridze, G. S.; Jero, K.; Jlelati, O.; Kaminsky, B.; Kappes, A.; Karg, T.; Karle, A.; Kauer, M.; Kelley, J. L.; Kiryluk, J.; Klas, J.; Klein, S. R.; Kohne, J. H.; Kohnen, G.; Kolanoski, H.; Kopke, L.; Kopper, C.; Kopper, S.; Koskinen, D. J.; Kowalski, M.; Krasberg, M.; Kriesten, A.; Krings, K.; Kroll, G.; Kunnen, J.; Kurahashi, N.; Kuwabara, T.; Labare, M.; Landsman, H.; Larson, M. J.; Lesiak-Bzdak, M.; Leuermann, M.; Leute, J.; Lunemann, J.; Macias, O.; Madsen, J.; Maggi, G.; Maruyama, R.; Mase, K.; Matis, H. S.; McNally, F.; Meagher, K.; Merck, M.; Merino, G.; Meures, T.; Miarecki, S.; Middell, E.; Milke, N.; Miller, J.; Mohrmann, L.; Montaruli, T.; Morse, R.; Nahnhauer, R.; Naumann, U.; Niederhausen, H.; Nowicki, S. C.; Nygren, D. R.; Obertacke, A.; Odrowski, S.; Olivas, A.; Omairat, A.; O'Murchadha, A.; Paul, L.; Pepper, J. A.; de los Heros, C. P.; Pfendner, C.; Pieloth, D.; Pinat, E.; Posselt, J.; Price, P. B.; Przybylski, G. T.; Quinnan, M.; Radel, L.; Rae, I.; Rameez, M.; Rawlins, K.; Redl, P.; Reimann, R.; Resconi, E.; Rhode, W.; Ribordy, M.; Richman, M.; Riedel, B.; Rodrigues, J. P.; Rott, C.; Ruhe, T.; Ruzybayev, B.; Ryckbosch, D.; Saba, S. M.; Sander, H. G.; Santander, M.; Sarkar, S.; Schatto, K.; Scheriau, F.; Schmidt, T.; Schmitz, M.; Schoenen, S.; Schoneberg, S.; Schonwald, A.; Schukraft, A.; Schulte, L.; Schultz, D.; Schulz, O.; Secke, D.; Sestayo, Y.; Seunarine, S.; Shanidze, R.; Sheremata, C.; Smith, M. W. E.; Soldin, D.; Spiczak, G. M.; Spiering, C.; Stamatikos, M.; Stanev, T.; Stanisha, N. A.; Stasik, A.; Stezelberger, T.; Stokstad, R. G.; Stossl, A.; Strahler, E. A.; Strom, R.; Strotjohann, N. L.; Sullivan, G. W.; Taavola, H.; Taboada, I.; Tamburro, A.; Tepe, A.; Ter-Antonyan, S.; Tesic, G.; Tilav, S.; Toale, P. A.; Tobin, M. N.; Toscano, S.; Tselengidou, M.; Unger, E.; Usner, M.; Vallecorsa, S.; van Eijndhoven, N.; van Overloop, A.; van Santen, J.; Vehring, M.; Voge, M.; Vraeghe, M.; Walck, C.; Waldenmaier, T.; Wallraff, M.; Weaver, C.; Wellons, M.; Wendt, C.; Westerhoff, S.; Whitehorn, N.; Wiebe, K.; Wiebusch, C. H.; Williams, D. R.; Wissing, H.; Wolf, M.; Wood, T. R.; Woschnagg, K.; Xu, D. L.; Xu, X. W.; Yanez, J. P.; Yodh, G.; Yoshida, S.; Zarzhitsky, P.; Ziemann, J.; Zierke, S.; Zoll, M.

polar.se

Simple search

Advanced search -
Research publications Advanced search -
Student theses Statistics

English Svenska Norsk

Change search

Cite ExportLink to record

Permanent link

https://urn.kb.se/resolve?urn=urn:nbn:se:polar:diva-1964

Direct link

http://polar.diva-portal.org/smash/record.jsf?pid=diva2:820703

Cite

Citation style

apa
ieee
modern-language-association-8th-edition
vancouver
Other style

More styles

Language

de-DE
en-GB
en-US
fi-FI
nn-NO
nn-NB
sv-SE
Other locale

More languages

Output format

html
text
asciidoc
rtf

The IceProd framework: Distributed data processing for the IceCube neutrino observatory

Aartsen, M. G.

Abbasi, R.

Ackermann, M.

Adams, J.

Show others and affiliations

Responsible organisation

Swedish Polar Research Secretariat

2015 (English)In: Journal of Parallel and Distributed Computing, ISSN 0743-7315, E-ISSN 1096-0848, Vol. 75Article in journal (Refereed) Published

Abstract [en]

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework. (C) 2014 Elsevier Inc. All rights reserved.

Place, publisher, year, edition, pages

2015. Vol. 75

Keywords [en]

Data management Grid computing Monitoring Distributed computing Computer Science

Research subject

SWEDARP, IceCube

Identifiers

URN: urn:nbn:se:polar:diva-1964DOI: 10.1016/j.jpdc.2014.08.001OAI: oai:DiVA.org:polar-1964DiVA, id: diva2:820703

Note

ISI Document Delivery No.: AW9BK Times Cited: 0 Cited Reference Count: 18 Aartsen, M. G. Abbasi, R. Ackermann, M. Adams, J. Aguilar, J. A. Ahlers, M. Altmann, D. Arguelles, C. Auffenberg, J. Bai, X. Baker, M. Barwick, S. W. Baum, V. Bay, R. Beatty, J. J. Tjus, J. Becker Becker, K. -H. BenZvi, S. Berghaus, P. Berley, D. Bernardini, E. Bernhard, A. Besson, D. Z. Binder, G. Bindig, D. Bissok, M. Blaufuss, E. Blumenthal, J. Boersma, D. J. Bohm, C. Bose, D. Boeser, S. Botner, O. Brayeur, L. Bretz, H. -P. Brown, A. M. Bruijn, R. Casey, J. Casier, M. Chirkin, D. Christov, A. Christy, B. Clark, K. Classen, L. Clevermann, F. Coenders, S. Cohen, S. Cowen, D. F. Silva, A. H. Cruz Danninger, M. Daughhetee, J. Davis, J. C. Day, M. De Clercq, C. De Ridder, S. Desiati, P. de Vries, K. D. de With, M. DeYoung, T. Diaz-Velez, J. C. Dunkman, M. Eagan, R. Eberhardt, B. Eichmann, B. Eisch, J. Euler, S. Evenson, P. A. Fadiran, O. Fazely, A. R. Fedynitch, A. Feintzeig, J. Feusels, T. Filimonov, K. Finley, C. Fischer-Wasels, T. Flis, S. Franckowiak, A. Frantzen, K. Fuchs, T. Gaisser, T. K. Gallagher, J. Gerhardt, L. Gladstone, L. Glusenkamp, T. Goldschmidt, A. Golup, G. Gonzalez, J. G. Goodman, J. A. Gora, D. Grandmont, D. T. Grant, D. Gretskov, P. Groh, J. C. Gross, A. Ha, C. Ismail, A. Haj Hallen, P. Hallgren, A. Halzen, F. Hanson, K. Hebecker, D. Heereman, D. Heinen, D. Helbing, K. Hellauer, R. Hickford, S. Hill, G. C. Hoffman, K. D. Hoffmann, R. Homeier, A. Hoshina, K. Huang, F. Huelsnitz, W. Hulth, P. O. Hultqvist, K. Hussain, S. Ishihara, A. Jacobi, E. Jacobsen, J. Jagielski, K. Japaridze, G. S. Jero, K. Jlelati, O. Kaminsky, B. Kappes, A. Karg, T. Karle, A. Kauer, M. Kelley, J. L. Kiryluk, J. Klaes, J. Klein, S. R. Koehne, J. -H. Kohnen, G. Kolanoski, H. Koepke, L. Kopper, C. Kopper, S. Koskinen, D. J. Kowalski, M. Krasberg, M. Kriesten, A. Krings, K. Kroll, G. Kunnen, J. Kurahashi, N. Kuwabara, T. Labare, M. Landsman, H. Larson, M. J. Lesiak-Bzdak, M. Leuermann, M. Leute, J. Luenemann, J. Macias, O. Madsen, J. Maggi, G. Maruyama, R. Mase, K. Matis, H. S. McNally, F. Meagher, K. Merck, M. Merino, G. Meures, T. Miarecki, S. Middell, E. Milke, N. Miller, J. Mohrmann, L. Montaruli, T. Morse, R. Nahnhauer, R. Naumann, U. Niederhausen, H. Nowicki, S. C. Nygren, D. R. Obertacke, A. Odrowski, S. Olivas, A. Omairat, A. O'Murchadha, A. Paul, L. Pepper, J. A. de los Heros, C. Perez Pfendner, C. Pieloth, D. Pinat, E. Posselt, J. Price, P. B. Przybylski, G. T. Quinnan, M. Raedel, L. Rae, I. Rameez, M. Rawlins, K. Redl, P. Reimann, R. Resconi, E. Rhode, W. Ribordy, M. Richman, M. Riedel, B. Rodrigues, J. P. Rott, C. Ruhe, T. Ruzybayev, B. Ryckbosch, D. Saba, S. M. Sander, H. -G. Santander, M. Sarkar, S. Schatto, K. Scheriau, F. Schmidt, T. Schmitz, M. Schoenen, S. Schoeneberg, S. Schoenwald, A. Schukraft, A. Schulte, L. Schultz, D. Schulz, O. Secke, D. Sestayo, Y. Seunarine, S. Shanidze, R. Sheremata, C. Smith, M. W. E. Soldin, D. Spiczak, G. M. Spiering, C. Stamatikos, M. Stanev, T. Stanisha, N. A. Stasik, A. Stezelberger, T. Stokstad, R. G. Stoessl, A. Strahler, E. A. Stroem, R. Strotjohann, N. L. Sullivan, G. W. Taavola, H. Taboada, I. Tamburro, A. Tepe, A. Ter-Antonyan, S. Tesic, G. Tilav, S. Toale, P. A. Tobin, M. N. Toscano, S. Tselengidou, M. Unger, E. Usner, M. Vallecorsa, S. van Eijndhoven, N. van Overloop, A. van Santen, J. Vehring, M. Voge, M. Vraeghe, M. Walck, C. Waldenmaier, T. Wallraff, M. Weaver, Ch. Wellons, M. Wendt, C. Westerhoff, S. Whitehorn, N. Wiebe, K. Wiebusch, C. H. Williams, D. R. Wissing, H. Wolf, M. Wood, T. R. Woschnagg, K. Xu, D. L. Xu, X. W. Yanez, J. P. Yodh, G. Yoshida, S. Zarzhitsky, P. Ziemann, J. Zierke, S. Zoll, M. Auffenberg, Jan/D-3954-2014 Auffenberg, Jan/0000-0002-1185-9094 U.S. National Science Foundation-Office of Polar Programs; U.S. National Science Foundation-Physics Division; University of Wisconsin Alumni Research Foundation; Grid Laboratory Of Wisconsin (GLOW) grid infrastructure at the University of Wisconsin Madison; Open Science Grid (OSG) grid infrastructure; U.S. Department of Energy, and National Energy Research Scientific Computing Center; Louisiana Optical Network Initiative (LONI) grid computing resources; Natural Sciences and Engineering Research Council of Canada; WestGrid and Compute/Calcul Canada; Swedish Research Council; Swedish Polar Research Secretariat; Swedish National Infrastructure for Computing (SNIC); Knut and Alice Wallenberg Foundation, Sweden; German Ministry for Education and Research (BMBF); Deutsche Forschungsgemeinschaft (DFG); Helmholtz Alliance for Astroparticle Physics (HAP); Research Department of Plasmas with Complex Interactions (Bochum), Germany; Fund for Scientific Research (FNRS-FWO); FWO Odysseus programme; Flanders Institute to encourage scientific and technological research in industry (IWT); Belgian Federal Science Policy Office (Belspo); University of Oxford, United Kingdom; Marsden Fund, New Zealand; Australian Research Council; Japan Society for Promotion of Science (JSPS); Swiss National Science Foundation (SNSF), Switzerland; National Research Foundation of Korea (NRF); Danish National Research Foundation, Denmark (DNRF) We acknowledge the support from the following agencies: U.S. National Science Foundation-Office of Polar Programs, U.S. National Science Foundation-Physics Division, University of Wisconsin Alumni Research Foundation, the Grid Laboratory Of Wisconsin (GLOW) grid infrastructure at the University of Wisconsin Madison, the Open Science Grid (OSG) grid infrastructure; U.S. Department of Energy, and National Energy Research Scientific Computing Center, the Louisiana Optical Network Initiative (LONI) grid computing resources; Natural Sciences and Engineering Research Council of Canada, WestGrid and Compute/Calcul Canada; Swedish Research Council, Swedish Polar Research Secretariat, Swedish National Infrastructure for Computing (SNIC), and Knut and Alice Wallenberg Foundation, Sweden; German Ministry for Education and Research (BMBF), Deutsche Forschungsgemeinschaft (DFG), Helmholtz Alliance for Astroparticle Physics (HAP), Research Department of Plasmas with Complex Interactions (Bochum), Germany; Fund for Scientific Research (FNRS-FWO), FWO Odysseus programme, Flanders Institute to encourage scientific and technological research in industry (IWT), Belgian Federal Science Policy Office (Belspo); University of Oxford, United Kingdom; Marsden Fund, New Zealand; Australian Research Council; Japan Society for Promotion of Science (JSPS); the Swiss National Science Foundation (SNSF), Switzerland; National Research Foundation of Korea (NRF); Danish National Research Foundation, Denmark (DNRF). 0 Academic press inc elsevier science San diego Computer Science, Theory & Methods

Available from: 2015-06-12 Created: 2015-05-13 Last updated: 2017-12-04

Open Access in DiVA

No full text in DiVA

Search outside of DiVA

Google Google Scholar

doi

urn-nbn

Total: 522 hits