Editing IC 8 Catalogue, curation, provenance

== <span style="color: #BBCE00">Background</span> ==

=== <span style="color: #BBCE00">Short description</span> ===

The implementation case aims at fulfilling requirements for curation, cataloguing and provenance.<br>
The targeted usages are:

* support operations in RI (e.g. network of observation system maintainance with sensor registry)
* support downstream users (e.g. discover available datasets)
* support long term sustainability of the information (e.g. preserve datasets or software).

'''Catalogue''' is used for discovery (finding items of interest), contextualisation (determining relevance and quality) , access (connecting together users, datasets, software, resources to achieve the user end-objective).<br>
Items described in catalogues are among: datasets, systems and resources for observation and processing, observations event and results (e.g. samples), documents, persons, research objects.<br>
Provenance and curation functions rely on catalogue as a back-end repository, as input or output

'''Provenance''' relates to contextualisation. It provides functions writing, updating and reading catalogue to complete discovery and access with services determining relevance and quality of the items described in catalogues.<br>
Provenance being well covered in other implemention cases (IC_2 mostly, but IC_6 and IC_9 as well), the current implementation case will collaborate with them for requirements and fulfill them so to demonstrate a couple of provenance functions: '''to be listed (Barbara) 2 functions related to dataset's provenance'''.

'''Curation''' relates to the data management processes required to ensure availability of digital assets (datasets, software) through media migration to ensure physical readability, redundant copies to ensure availability, appropriate security and privacy measures to ensure reliability and appropriate catalogue content maintenance to ensure discovery, contextualisation and access to this digital assets.<br>
The current implementation case fulfill requirements for a couple of curation functions: (a) automated media migration of datasets to ensure continued availability and readability; (b) discovery of a curated dataset along with appropriate curated software and operating environment

[[File:Worddave2ff53e390870405f313192a74be6821.png]]

=== <span style="color: #BBCE00">Contact</span> ===

{|class="wikitable" style="background-color:#ffffff;"
|-
|'''Background'''
|'''Contact Person'''
|'''Organization'''
|'''Contact email'''
|-
|_<Choose one of the following roles: [RI-ICT
|RI-Domain
|ICT
|e-Infrastructure]>_
|-
|RI (Use Case proposer, Agile Group leader)
|Thomas Loubrieu
|IFREMER
|[mailto:Thomas.loubrieu@ifremer.fr Thomas.loubrieu@ifremer.fr]
|-
|RI
|Keith Jeffery
|
|[mailto:Keith.Jeffery@keithgjefferyconsultants.co.uk Keith.Jeffery@keithgjefferyconsultants.co.uk]
|-
|RI
|Chrstian Pichot, Andre Chanzy
|INRA (ANAEE)
|[mailto:christian.pichot@paca.inra.fr christian.pichot@paca.inra.fr]<br>[mailto:andre.chanzy@avignon.inra.fr andre.chanzy@avignon.inra.fr]
|-
|ITC
|Marco Rorro Giovanni Morelli The persons who managed<br>CKAN for EUDAT would be perfect here !
|EUDAT
|[mailto:M.rorro@cineca.it M.rorro@cineca.it]&nbsp;&nbsp;[mailto:g.morelli@cineca.it g.morelli@cineca.it]
|-
|RI
|Damien Boulanger
|IAGOS
|[mailto:damien.boulanger@obs-mip.fr damien.boulanger@obs-mip.fr]
|-
|RI
|Maggie Hellstrom
|ICOS
|[mailto:margareta.hellstrom@nateko.lu.se margareta.hellstrom@nateko.lu.se]
|-
|RI
|Barbara Magagna, Johannes Peterseil
|LTER
|[mailto:Barbara.magagna@umweltbundesamt.at Barbara.magagna@umweltbundesamt.at]<br>[mailto:Johannes.peterseil@umweltbundesamt.at Johannes.peterseil@umweltbundesamt.at]
|-
|Task 5.4
|Zhiming Zhao, Paul Martin
|UvA
|[mailto:z.zhao@uva.nl <span style="color: #222222">z.zhao@uva.nl</span>], [mailto:P.W.Martin@uva.nl <span style="color: #222222">P.W.Martin@uva.nl</span>]
|}

=== <span style="color: #BBCE00">Use case type</span> ===

'''''Implementation case'''''

''Conditions:''

* ''Each RI describes its portfolio of new and/or enhanced services that they expect from ENVRIPLUS results, derived from the ENVRIPLUS WPs;''
* ''ENVRIPLUS staff work with the RIs on these descriptions, which in the course of the project will be gradually updated with more details.''

''Implications:''

* ''Implementation cases selected and adopted by interested RIs;''
* ''Both RIs and ENVRIPLUS invest in the actual implementation and associated services.>''

=== <span style="color: #BBCE00">Scientific domain and communities</span> ===

'''<span style="color: #BBCE00">Scientific domain</span>'''

To be relevant in ENVRIPLUS context, the implementated functions must be validated by at least 2 RIs, preferably in 2 different spheres (bio, liquid, solid, gas):

* Atmosphere: IAGOS
* Biosphere: ANAEE
* Geosphere: EPOS
* Hydrosphere: SeaDataNet, Euro-ARGO


'''<span style="color: #BBCE00">Community</span>'''

* data acquisition community (e.g. observation system catalogue)
* data curation community maintain the catalogues and use them as a management tool for datasets (and other digital objects) inventory.
* data publication community uses catalogues to parameterize their actions (e.g. mint a DOI an an object requires metadata)
* data service provision community use catalogues to configure their services, including workflow composition to retrieve input datasets and process them, but they might not be considered as a first priority target.
* data usage community use catalogue for discovery and with contextualisation including provenance for asessment of relevance and quality and for traceability. Data curation community is targeted for curation as a provider but as well by the other community who uses curated information to work.


'''<span style="color: #BBCE00">Behavior</span>'''

The connected behaviours are:<br>
Data acquisition community:

* '''Instrument configuration and calibration''' need to be registered in catalogues.
* '''Data collection:''' Data collected from sensors need to be curated, at least safe guarded in early stage on replicated storage. The description of the observation need to be pushed in catalogue. Data curation community:
* '''Data quality checking:''' the quality assesment performed on dataset need to be documented in catalogue. Different versions of same datasets with different quality control performed need to be managed.
* '''Data preservation:''' catalogue and datasets need to be preserved for long term. With replicated copies, format maintenance and a data management plan (DMP)
* '''Data product generation:''' input and output datasets of the products need to be managed in catalogues and curated. For provenance the description of the product processing need to be managed in catalogue as well.
* '''Data replication''' is handle by curation sub-use case. Data publication community:
* '''Data publication:''' the information managed in catalogue and curated datasets should be clean enough for publication
* '''Semantic hamonisation:''' the content of catalogue and datasets will be homogeneous syntactically (format) and rely on harmonized thesaurus references (e.g .SKOS) to support semantic harmonisation.
* '''Data discovery and access:''' data discovery will be enabled in a harmonized way. Visualization and download access will be provided when available but not harmonized. The data access provided here will not take benefit of the provenance information (user tracking, profile, ...)
* '''Data citation:''' the metadata required for data citation will be available in catalogue. The DOIs or PID of datasets will be described in catalogue as well. However the function of creating a DOI is not manage by the use case.

Data Service provision community:

* '''service description and registration:''' the service should be described in a catalogue, however this may not be a first priority in this use case to manage this information.
* '''Service coordination and composition''' can use catalogues of datasets and services to schedule and organize service. However as said above this is not a priority in the current use case. Data usage community: behaviours of the community will be supported by the catalogue especially in aspects of discovery, contextualisation (including provenance), availablity (through curation).
* '''User profile management:''' Users are recorded in the catalogue and by matching processes harmonizes with other user directories (e.g. OrcID) managed in the use case.

== <span style="color: #BBCE00">Detailed description</span> ==

'''<span style="color: #BBCE00">Objective and Impact</span>'''

'''Catalogue'''<br>
The catalogue aims at providing functions cross-cutting RI, to edit and discover the following items:

* systems for observation and processing (processing is in lowest priority)
* observations event and results (e.g. samples)
* datasets
* documents
* persons
* research objects (lowest priority)

<u>Action  1</u>: Persons and documents will be described and federated in pre-existing e-infrastructures, to be defined (e.g. orcID, …) so to fulfill requirements for the provenance and curation functions.<br>
<u>Action  2</u>: Datasets description will be federated from harvesting the datasets catalogue (in whatever 'standard' metadata format) in each RI in a single entry point (metadata format to be chosen among: DC, DCAT, INSPIRE/ISO19115, geonetworks, CKAN, CERIF ) to be defined so to fulfill requirements for the provenance and curation functions.<br>
<u>Action  3</u>: Observation systems, events and results (including collected samples) edition and discovery functions will be implemented by a combination of RI specific tools and federated tools (e.g. for edition) so to fulfill requirements for the provenance and curation functions.


'''<span style="color: #BBCE00">Challenges</span>'''

The main challenge is the involvement of RI, from definition of the functions to the adoption of the solution.


'''<span style="color: #BBCE00">Detailed scenarios</span>'''

In the context of the 3 above actions:

# define curation and provenance functions to be provided, identify related requirement on catalogue (format and access API).
# define catalogue requirements  for discovery and access
# define metadata profile and access API
# implement the centralized or federated solution

As for AGILE, the steps can be iterative by having new iteration for new requirements identified or RI supported.
@@ Line 1: / Line 1: @@
-{{DISPLAYTITLE:IC_8 Catalogue, curation, provenance}}
 == <span style="color: #BBCE00">Background</span> ==
@@ Line 169: / Line 168: @@
 As for AGILE, the steps can be iterative by having new iteration for new requirements identified or RI supported.
-[[File:Worddav391ee37988e66979fe6cd5249485105f.png]]
-'''<span style="color: #BBCE00">Technical status and requirements</span>'''
-E-infrastructures which manage catalogues of persons and documents are existing, available through standard interfaces and cross-cutting RI.<br>
-Catalogues of datasets are generally provided by RI and their content is available through standard interfaces. Some tools are available on the shelf to implement the catalogue of datasets (DC, DCAT, INSPIRE/ISO19115, geonetworks, CKAN, CERIF). ENVRIPLUS need to federate them by utilising the richest available 'standard' and providing mappings to the others.<br>
-Catalogue of observation systems, events or samples may exists in RI. They are seldom or never accessible through standard interfaces. Some RI lack proper tools to manage these information which is however critical for the good quality and traceability of scientific results.
-'''<span style="color: #BBCE00">Implementation plan and timetable</span>'''
-'''Documents and persons'''<br>
-E-infrastructures which manage catalogues of persons and documents are existing.<br>
-The implementation case will define a list of official sustainable person and document repository which should be used by RI to describe their resources. and define mappings to/from the ENVRIPLUS catalogue metadata standard (when chosen)<br>
-''Expected result in Octobre 2016''
-'''Datasets'''<br>
-The implementation case will identify catalogues of datasets in RI and analyse their machine to machine interface for harvesting purpose. A single tool will harvest them centrally. Then their metadata will require conversion from local RI format to that of the ENVRIPLUS central catalogue as described above.<br>
-''Expected result in Octobre 2017''
-'''Observation systems, events or samples'''<br>
-An integrated system will shows observations systems, events and collected samples from 2 or 3 RI in liquid (EMSO, ARGO), solid (EPOS) and gas (ICOS) spheres.<br>
-Tools will be provided to easily edit the descriptions for RI which would not have their own system yet.<br>
-As before this will rquire mapping the metadata describing systems, events, samples at each RI to the common metadata standard of ENVRIPLUS.<br>
-''Expected result in Octobre 2018''
-'''<span style="color: #BBCE00">Expected output and evaluation of output</span>'''
-'''Documents and persons'''<br>
-number of RI actually using the chosen person and document e-infrastructure to identify their resources.
-'''Datasets'''<br>
-Number of RI which dataset results descriptions are available in the federated system.<br>
-Number of users of the federated dataset catalogue (inside or outside the RI).
-'''Observation systems, events or samples'''<br>
-Number of observation systems which events and results are actually available in the federated catalogue.<br>
-Number of users of the catalogues as support of the activities in the RI.
-== <span style="color: #BBCE00">External Links</span> ==
-# IC_8 notebook: [https://envriplus.manageprojects.com/projects/wp9-service-validation-and-deployment-1/notebooks/659 {+}][https://envriplus.manageprojects.com/projects/wp9-service-validation-and-deployment-1/notebooks/659+ <span style="color: #222222">https://envriplus.manageprojects.com/projects/wp9-service-validation-and-deployment-1/notebooks/659+</span>]
-[[Category:Use Cases]]