IC 3 Support EISCAT-3D users to reprocess data using user's algorithms

From
Revision as of 13:31, 18 March 2020 by ENVRIwiki (talk | contribs)
Jump to: navigation, search

Background

Short description

To reprocess environmental data with not default input parameters

Contact

Background Contact Person Organization Contact email
RI-ICT Ingemar Häggström EISCAT Ingemar.haggstrom@eiscat.se
e-Infrastructure_ICT Leonardo Candela CNR leonardo.candela@isti.cnr.it
Task 5.3/7.2 leader Zhiming Zhao
Paul Martin
University of Amsterdam (UvA) z.zhao@uva.nl
P.W.Martin@uva.nl

Use case type

Implementation case

Scientific domain and communities

Scientific domain

Atmosphere

Community

Data Service Provision | Data Usage {_}{+}http://confluence.envri.eu:8090/display/ERM/SV+Communities+_ for detailed explanation of those communities.>

Behavior

Detailed description

Objective and Impact

Most environmental RIs store data in different levels, and between them there is some sort of process. Most users are interested in the derived physical parameters, being the highest level of data. However, these normally depends on default input parameters for the processing of the data. There is often a need to reprocess the basic data with other input parameters, in the simple case to get a different spatial or temporal resolution, for details or statistical reasons. Depending on the data there can be a large number of more parameters to set and possibly alter.

Challenges

The challenge is to set up workflows how to select data and the process to do on it.

Detailed scenarios

Select data and process. Get a list of the default parameters, modify them as wanted. Run the process with the updated parameter list. This involves staging lower level data, processing it and delivering the results.

Technical status and requirements

The data and processes exists as such. One need to provide the basic tools for selection and setting up parameters in a generic way. It's needed to set up storage, several TB, to provide computation capabilities with basic software, here mainly Matlab.

Implementation plan and timetable

M0-3

  • decide the site for data, data transfer
  • discussions and defining workflows/techniques

M3-9

  • Setting up the services with interfaces to the data/metadata

M9-12

  • Testing, refining

Expected output and evaluation of output

A basic portal with human/machine interface, that can deliver updated physical parameters for a restricted dataset.

External Links

  1. IC_3 notebook: {+}https://envriplus.manageprojects.com/projects/wp9-service-validation-and-deployment-1/notebooks/631+