WP6 VO Action Plan

From Soteria
Jump to: navigation, search

Contents

Scope

This document corresponds to deliverable 6.2 of the SOTERIA project called "Virtual Observatory action Plan". It contains an inventory of the SOTERIA datasets, a review of existing VO-technologies and an implementation plan to collectively transition to a chosen VO.

Top Level Requirements

TBW!!!

here we will repeat and enumerate at least all that we promised in the proposal


SOTERIA Data inventory

The data that we will take into account for the SOTERIA Virtual Observatory Plan
data origin SOTERIA contact content type size format status current total volume url
KULeuven Giovanni Lapenta output of MHD simulations data, images, movies data=1-10 GB, images=100MB, movies=100MB HDF5, netcdf, png insertion in VO TBD 10 TB  ?
KULeuven Giovanni Lapenta output of kinetic (PIC) simulations data, images, movies data=1-30 GB, images=100MB, movies=100MB HDF5, netcdf, png insertion in VO TBD  ? TB  ?
Kanzelhohe Wolfgang Otruba sunspot drawings images ~1.5k x 1.7k, once per day JPG for the time being available several 100 kB each, 300 per year since 1947 (total 18000 drawings = 6GB) description archive
Kanzelhohe Wolfgang Otruba full disk Halpha images various old scanned data is JPEG only, rest JPEG+FITS available, 1975-2000 scanned from film, 1 image/ 4min when sun = 0.5 M jpeg images 512x512; 2000-now CCD 1kx1k FITS/min,2003-now 1kx1k FITS/5s during flares, 2000-2008 2kx2k FITS/min , since 2008 new CCD 2kx2k FITS/min image quality TBC  ? TB http ?
Kanzelhohe Wolfgang Otruba full disk broadband (close but not identical to 'white light') images 3 images/day currently available on film, to be scanned to FITS during SOTERIA era available from 1990-mid2007 300 days/year http ?
Kanzelhohe Wolfgang Otruba full disk continuum images 1 images/minute during activity otherwise 1 image/hour 2kx2k CCD since 2007 300 days/year http ?
Kanzelhohe Wolfgang Otruba Ca K telescope images 1 images/minute during activity otherwise 1 image/hour 2kx2k CCD to be developed during SOTERIA 300 days/year http ?
Observatoire de Paris Jean Aboudarham full disk Halpha sets of 5 images at different positions in the line 1.5k x 1.34k, once per day FITS available File: 17 MB http://bass2000.obspm.fr/
Observatoire de Paris Jean Aboudarham full disk Ca II K Center line (K3) sets of 5 images at different positions in the line 1.5k x 1.34k, once per day FITS available File: 17 MB http://bass2000.obspm.fr/
Observatoire de Paris Jean Aboudarham full disk Ca II K line wing (K1) images 1.5k x 1.34k, once per day FITS available File: 3.7 MB http://bass2000.obspm.fr/
Observatoire de Paris Jean Aboudarham full disk Halpha prominences images with solar disk occulted with transparent disk 1.5k x 1.34k, once per day FITS available File: 3.5 MB http://bass2000.obspm.fr/
Observatoire de Paris Jean Aboudarham full disk Ca II prominences images with solar disk occulted by transparant disk 1.5k x 1.34k, once per day FITS available File: 3.5 MB http://bass2000.obspm.fr/
Observatoire de Paris, Nancay Jean Aboudarham full disk radioheliographs at 150.9 and 327 Mhz images 256 x 256, once per day (full time resolution (every 2 sec) on demand) FITS available File: 268 kB http://bass2000.obspm.fr/
Observatoire de Paris, Nancay Jean Aboudarham Nancay Decametric Array Observations dynamic spectra 830 x 610, once per day (full time resolution (sec?) on demand) FITS  ?available (sometimes observing jupiter instead of Sun) 510 kB http://bass2000.obspm.fr/
Observatoire de Paris, Pic du Midi Jean Aboudarham coronagraphic Halpha images 2k x 2k, twice a day (full time resolution (1 per sec?) on demand) FITS available (high time resolution in development) File: 8.4 MB http://bass2000.obspm.fr/
Observatoire de Paris, Pic du Midi Jean Aboudarham coronagraphic He I images 2k x 2k, twice a day (full time resolution on demand) FITS available File: 8.4 MB http://bass2000.obspm.fr/
Observatoire de Paris, Pic du Midi Jean Aboudarham Full disk Halpha broad band images 2k x 2k, twice a day (full time resolution on demand) FITS available File: 8.4 MB http://bass2000.obspm.fr/
Observatoire de Paris, Pic du Midi Jean Aboudarham Full disk Ca II K broad band images 2k x 2k, twice a day (full time resolution on demand) FITS available File: 8.4 MB http://bass2000.obspm.fr/
Lebedev Physical Institute, Russia; CORONAS-PHOTON/TESIS Sergey Bogachev (bogachev@sci.lebedev.ru), Vladimir Slemzin (slem@sci.lebedev.ru) Imaging spectroheliometer Mg XII, 8.42A images 2048k x 2048k, 1-10 per day FITS, JPEG launch early 2009 File: ~ 7 MB http://tesis.lebedev.ru/
Lebedev Physical Institute, Russia; CORONAS-PHOTON/TESIS Sergey Bogachev (bogachev@sci.lebedev.ru), Vladimir Slemzin (slem@sci.lebedev.ru) Hershel EUV telescope, FeXX-XXIII 132A images 2048k x 2048k, 1-10 per day FITS, JPEG launch early 2009 File: ~ 7 MB http://tesis.lebedev.ru/
Lebedev Physical Institute, Russia; CORONAS-PHOTON/TESIS Sergey Bogachev (bogachev@sci.lebedev.ru), Vladimir Slemzin (slem@sci.lebedev.ru) Hershel EUV telescope, HeII 304A images 2048k x 2048k, 1-10 per day FITS, JPEG launch early 2009 File: ~ 7 MB http://tesis.lebedev.ru/
Lebedev Physical Institute, Russia; CORONAS-PHOTON/TESIS Sergey Bogachev (bogachev@sci.lebedev.ru), Vladimir Slemzin (slem@sci.lebedev.ru) Hershel EUV telescope, FeIX 171A images 2048k x 2048k, 1-10 per day FITS, JPEG launch early 2009 File: ~ 7 MB http://tesis.lebedev.ru/
Lebedev Physical Institute, Russia; CORONAS-PHOTON/TESIS Sergey Bogachev (bogachev@sci.lebedev.ru), Vladimir Slemzin (slem@sci.lebedev.ru) Solar EUV coronagraph, HeII 304A images 2048k x 2048k, 1-10 per day FITS, JPEG launch early 2009 File: ~ 7 MB http://tesis.lebedev.ru/
Lebedev Physical Institute, Russia; CORONAS-PHOTON/TESIS Sergey Bogachev (bogachev@sci.lebedev.ru), Vladimir Slemzin (slem@sci.lebedev.ru) EUV spectroheliographs, 285-335A images 2048k x 2048k, 1-10 per day FITS, JPEG launch early 2009 File: ~ 7 MB http://tesis.lebedev.ru/
SRC-PAS, CORONAS-PHOTON Janusz Sylwester SphinX high cadence spectra 1 GB/week FITS launch early 2009 300 GB over rising phase cycle 24 http://tesis.lebedev.ru/
SRC-PAS, CORONAS-F Anna Kepa RESIK (all observations and selected data reduced to level 2) high cadence X-ray spectra 10 GB FITS 2001 - 2003 10 GB http://www.cbk.pan.wroc.pl
SRC-PAS, CORONAS-F Marek Siarkowski DIOGENESS X-ray spectra 75 MB FITS 2001 75 MB http://www.cbk.pan.wroc.pl
SRC-PAS, INTERBALL-Tail Marek Siarkowski, Szymon Gburek RF15-I X-ray lightcurves Lightcurves in energy range 2 keV - 240 keV, 8 channels IDL sav, gif 1995 - 2000 167 MB http://www.cbk.pan.wroc.pl/RF15-I_www/default.htm
Debrecen Observatory Andras Ludmany SOHO/MDI Debrecen Sunspot Data (SDD catalogue) formatted tables, continuum images, magnetograms data=300KB-50MB, full-disk images=2k x 2k, group images=30-300KB ASCII, FITS and JPG available 132GB data overview ftp repository
Debrecen Observatory Andras Ludmany Debrecen Photoheliographic Data (DPD catalogue) formatted tables, white-light images, magnetograms data=200KB-6MB, white-light images=4k x 4k, magnetograms=50KB-2MB, group images=30-900KB ASCII, FITS and JPG available 50GB data overview ftp repository
Debrecen Observatory Andras Ludmany Historical Solar Image Database (HSID) full-disc white-light drawings 200-400KB JPG available 3GB data overview ftp repository
University Gottingen Volker Bothmer STEREO archive images  ? FITS available, online?  ? TB url?
University Oulu Kalevi Mursula historical Dcx and Dxt indices and 4-16 stations Dt values hourly, daily and yearly data, images, movies variable ascii, png available 1932-2007 for 4 stations, 2000-2007 for 16 stations 1.15 GB http://dcx.oulu.fi/ (preliminary)
University Oulu Kalevi Mursula real time Dcx and Dxt indices and 4-16 stations Dt values hourly data, images variable ascii, png to be developed 100MB http://dcx.oulu.fi/ (preliminary)
ROB, USET telescopes Frédéric Clette full disc white light images 2k x 2k, 2 to 20 images/day FITS online (upgraded in June 2008) ~15 GB instrument pages
ROB, USET telescopes Frédéric Clette full disc H-alpha images 2k x 2k, 50 to 500 images/day FITS online (upgraded in January 2008) ~15GB instrument pages
ROB, USET telescopes Frédéric Clette full disc CaII-K images 2k x 2k, 20 to 100 images/day FITS Operational end 2009 - instrument pages
ROB, USET telescopes Frédéric Clette sunspot drawings images ~1.5k x 1.7k, few per day JPG for the time being available several 100 kB each, 200 per year [1]
ROB, Humain radiotelescopes Frédéric Clette radio spectrograph (CALLISTO) dynamic spectra 250 to 500 frequencies, 1 to 10 spectra/s FITS formatted standardized by CALLISTO network. on-line since July 2008. Potentially ROB=central CALLISTO network hub (TBC). ~ few GB data archive
ROB, SWAP David Berghmans full disc coronal, EUV (17.5nm) images 1k x 1k, 1 image/min FITS launch mid 2009  ? TB instrument pages
ROB, LYRA Jean-Francois Hochedez full disc irradiance, UV and EUV bandpasses time sequences 4 channels at > 1 Hz sampling FITS launch mid 2009  ? TB instrument pages
Hvar Observatory Bojan Vrsnak subfield white light during activity images 1024x1024, 5s cadence FITS, JPG launch mid 2009 File: 3Mb in preparation
Hvar Observatory Bojan Vrsnak subfield Halpha during activity images 1024x1024, 1 image/min FITS, JPG launch mid 2009 File: 3Mb in preparation
MTA KFKI RMKI Karoly Kecskemety energetic particle fluxes flux data 30+40 kB/day ASCII available 400MB in preparation

Review of existing VO-technologies

EGSO

EGSO, the "European Grid of Solar Observations", was developed as a Grid test-bed under the Information Society Technologies (IST) thematic programme of the European Commission's Fifth Framework Programme (Grant No. IST-2001-32409). The EC Grant for EGSO has ended in November 2006 and the it is supported by the partners on a best-efforts basis.

Strong Points

Weak Points

  • funding has ended, present infrastructure stays alive on best-effort basis
  • it is not clear who are presently the EGSO data providers or how to become one

Note that part of the EGSO consortium regrouped in the HELIO project that started this year. HELIO is mainly focused on developing a software framework of a Virtual Solar Observatory based on modern standards. It is based on the SOA (Service Oriented Architecture) allowing loosely coupling of every part of the whole infrastructure. It aims also to provides services to add value to observational data by allowing easy querying and analysis of data sets. For this reason, Helio is a complementary partner for Soteria

VSO

VSO

Strong Points

  • Working infrastructure with around 15 existing providers
  • Near future support is secured for the next few years.
  • Simple API

Weak Points

  • No enough man power to maintain it up to date.
  • Lot of "promised" feature not implemented or not documented.
  • Aging technology
  • Soteria and VSO have no direct partnership.

VSO is proven technology but suffers from its history and the lack of man power. Being VSO friendly doesn't seem to be a issue at the moment (we are doing the first tests) and so is the safest way to go. But it doesn't meet our expectations in terms of user friendliness and reliability. So we have to enhance it to meet our expectations in terms of speed, reliability and user friendliness

CESAR

BASS2000

Strong Points

  • point 1
  • point 2


Weak Points


General conclusion is ...

BASS2000

BASS2000

Strong Points

  • point 1
  • point 2


Weak Points


General conclusion is ...

Implementation Plan

The above problem will be addressed along two lines of attack:

1. In the coming months: start by adapting and extending the American "Virtual Solar Observatory" named herafter VSO. This means that all SOTERIA members with a data archive would become, in time, VSO data providers. Meanwhile at ROB we are making sure that we do not depend critically on the central VSO hub, but that SOTERIA has its own central hub that can work independently from it. VSO has the advantage that it is simple, it is up-and-running, and some of us (OBSPARIS) are already in there.

2. On a slower timescale: interact as much as possible with the HELIO consortium (successor of EGSO) and make sure that in the long run we are fully compatible with them. HELIO has the advantage that it is an FP7 project also (European based), that they are developing a state-of-the-art system far beyond VSO, and that they need a user community like SOTERIA to interact with.


TBW!!!

More detailed steps are needed here

Further notes

  • we concluded that the vast majority of the data consists of solar images, stored in FITS files. We will hence kick-off of the VO effort for these. We presume that once this is settled, timelines and dynamic spectra will be treated in a similar way.
  • we want to standardize the FITS keyword structure and filename convention of all SOTERIA data as much as possible. This standard will be based on the SOHO conventions and the Global Halpha Network conventions.
  • Obviously not every institute can be expected to reprocess all its historic data to comply with the new standard. We will develop an IDL SSW library for SOTERIA containing routines to convert SOTERIA image files into image objects. The read method of these objects will depend on the type/source of the image and handle any remaining non-standard aspects of the FITS file.
  • The WP6 will develop a template for these objects and read methods but it is up to the PI teams to write a read method for their particular instrument data.
  • The IDL SSW library will contain other routines that operate on the standard objects and extract usefull parameters as defined by the other work packages. These routines need to be developed by the other SOTERIA work packages and/or developed in coordination with WP6.
  • access to the image files across the network will be organised through a database. This database will contain all the standardized metadata as contained in the image objects. It will also contain a procedure to acquire the particular image across the network. In the simplest situation, this procedure is a fixed URL per image. In more complicated circumstances, this procedure is a SOAP call to a Virtual Observatory.
  • in practise the database is probably organised in different tables: one that describes static information (eg institute/instrument), one that describes data availibility and coverage. and one that describes the individual images.
  • it will be investigated how external virtual observatories can use the above described database to locate all SOTERIA data
  • the SWB-server will be based on the above image objects.
Retrieved from "http://soteria-space.eu/newwiki/index.php/WP6_VO_Action_Plan"
Personal tools