diff --git a/CHANGELOG.md b/CHANGELOG.md index 3aa4fa4..70b5f8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - [``GDI-SOP0009_dataset-withdrawal.md``](sops/european-level/GDI-SOP0009_dataset-withdrawal.md) - Withdrawal request GDI SOP. - [``.gitignore``](.gitignore) - File to ignore certain files in git commits. - [``GDI-SOP0011_withdraw-dataset-from-node-fair-data-point.md``](sops/node-specific/GDI-SOP0011_withdraw-dataset-from-node-fair-data-point.md) - Node-specific SOP template describing how a node FAIR Data Point should process soft withdrawal of a dataset or linked distributions. +- [``GDI-SOP0010_data-reception-procedure.md``](sops/node-specific/GDI-SOP0010_data-reception-procedure.md) - Node-specific SOP describing Data Reception. ### Modified - [``GDI-SOP_github-management.md``](docs/GDI-SOP_github-management.md) - Added reference to recorded session diff --git a/docs/GDI-SOP_charter.md b/docs/GDI-SOP_charter.md index 2c086a2..5dc5914 100644 --- a/docs/GDI-SOP_charter.md +++ b/docs/GDI-SOP_charter.md @@ -28,8 +28,10 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. | AI | Action Item | | AF | Application Form | | API | Application Programming Interface | +| BAM | Binary Alignment Map, file format for aligned sequencing reads mapped to a reference genome | | BEN | Beneficiary | | CC | Carbon Copy (used in communications to send a copy to others) | +| CRAM | Compressed Reference-oriented Alignment Map, a space-saving alternative to BAM | | CSC | Finnish IT Center for Science | | DAA | Data Access Agreement | | DAC | Data Access Committee | @@ -46,6 +48,8 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. | ELSI | Ethical, Legal, and Social Implications | | EMBL | European Molecular Biology Laboratory | | FAIR | Findability, Accessibility, Interoperability and Reusability | +| FASTA | Fast-A, file format for raw data output from sequencing machines | +| FASTQ | FASTA with Quality, file format for raw data output from sequencing machines | | FDP | FAIR Data Point | | FEGA | Federated EGA | | FTP | File Transfer Protocol | @@ -82,6 +86,7 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. | REMS | Resource Entitlement Management System | | RFC | Request For Comments | | RI | Research Infrastructure | +| SAM | Sequence Alignment/Map text file used in bioinformatics | | SDPC | Security and Data Protection Committee | | SLA | Service Level Agreement | | SP | Submitter Portal | @@ -95,6 +100,7 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. | URL | Uniform Resource Locator | | UU | University of Uppsala | | UX | User experience | +| VCF | Variant Call Format, file format used to store genetic variations | | VHD | Virtual Helpdesk | | VPN | Virtual Private Network | | UP | User portal | @@ -109,6 +115,10 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. | Beacon |A web-accessible service implementing the GA4GH Beacon specification, enabling federated discovery (and sometimes retrieval) of genomic variant and related biomedical data across distributed resources| | CHANGELOG | Log or record of all notable changes made to a project | | Data controller |The legal owner or organisation that determines the purposes and means of processing the dataset and can authorise its withdrawal| +| Data inclusion | Systematic integration of data into structured and machine-readable datasets | +| Data preparation | The process of structuring, annotating, and formatting data so that systems and users can discover, access and reuse it | +| Data reception | The process where GDI node securely acquires, ingests, and validates data supplied by a data provider | +| Data submission | The process of registering, publishing, and making data assets discoverable and accessible | | Dataset version |A specific released state of a dataset identified by a version label/number, used to distinguish minor or major changes between releases| | FitSM | Standards for lightweight IT Services Management; developed initially through The FedSM Project, funded by the European Commission, now maintained by ITEMO () | | Hard-deletion |Complete, irreversible erasure of the data from all storage media so it cannot be recovered| diff --git a/sops/README.md b/sops/README.md index 138a064..6735d40 100644 --- a/sops/README.md +++ b/sops/README.md @@ -7,4 +7,5 @@ Below you can find the full list of SOPs in this repository. | [GDI-SOP0007_sop-template-creation.md](./european-level/GDI-SOP0007_sop-template-creation.md) | GDI-SOP0007 | v1 | Helpdesk & operations | European-Level SOP | | | 7 | 2024.10.29 | | [GDI-SOP0008_node-helpdesk-ticket-classification.md](./node-specific/GDI-SOP0008_node-helpdesk-ticket-classification.md) | GDI-SOP0008 | v1.0.0 | Helpdesk & operations | Node-specific SOP | | | 4 | 2026.03.10 | | [GDI-SOP0009_dataset-withdrawal.md](./european-level/GDI-SOP0009_dataset-withdrawal.md) | GDI-SOP0009 | v1 | Data & metadata management | European-level SOP | | | 9 | 2026.03.10 | -| [GDI-SOP0011_withdraw-dataset-from-node-fair-data-point.md](./node-specific/GDI-SOP0011_withdraw-dataset-from-node-fair-data-point.md) | GDI-SOP0011 | v1 | Technical infrastructure & software development | Node-specific SOP | | | 5 | 2026.05.20 | +| [GDI-SOP0010_data-reception-procedure.md](./node-specific/GDI-SOP0010_data-reception-procedure.md) | GDI-SOP0010 | v1 | Data & metadata management | Node-specific SOP | | | 6 | 2026.08.31 | +| [GDI-SOP0011_withdraw-dataset-from-node-fair-data-point.md](./node-specific/GDI-SOP0011_withdraw-dataset-from-node-fair-data-point.md) | GDI-SOP0011 | v1 | Technical infrastructure & software development | Node-specific SOP | | | 5 | 2026.05.20 | diff --git a/sops/node-specific/GDI-SOP0010_data-reception-procedure.md b/sops/node-specific/GDI-SOP0010_data-reception-procedure.md new file mode 100644 index 0000000..8ae0518 --- /dev/null +++ b/sops/node-specific/GDI-SOP0010_data-reception-procedure.md @@ -0,0 +1,184 @@ +# European GDI - Data Reception Procedure + +| Metadata | Value | +|----------------------|-------------------| +| Template SOP number | GDI-SOP0010 | +| Template SOP version | v1 | +| Topic | Data & metadata management | +| Template SOP Type | Node-specific SOP | +| GDI Node | | +| Instance version | | + +## Index + +1. [Document History](#1-document-history) +2. [Glossary](#2-glossary) +3. [Roles and Responsibilities](#3-roles-and-responsibilities) +4. [Purpose](#4-purpose) +5. [Scope](#5-scope) +6. [Introduction and Background Information](#6-introduction-and-background-information) +7. [Summary or Context Diagram](#7-summary-or-context-diagram) +8. [Procedure](#8-procedure) +9. [References](#9-references) + +### 1. Document History + +| Template Version | Instance version | Author(s) | Description of changes | Date | +|---------|-----------|-----------|------------------------------|------------| +| `v1` | `v1.0.2` | Jorge Oliveira, Aleksandar Kmetov, Jacek Lebioda | Converting to SOP template and futher refinement | 2026.08.31 | +| `v1` | `v1.0.1` | Jorge Oliveira, Miguel Cisneiros | Further development of the first version of the SOP | 2025.08.22 | +| `v1` | `v1.0.0` | Jorge Oliveira, Pedro Ferreira | Created first version of the SOP | 2024.09.10 | + + +### 2. Glossary +Find GDI SOP common Glossary at the [**charter document**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_charter.md). + +| Abbreviation | Description | +|---------------|-----------------| +| BAM | Binary Alignment Map, file format for aligned sequencing reads mapped to a reference genome | +| CRAM | Compressed Reference-oriented Alignment Map, a space-saving alternative to BAM | +| ELSI | Ethical, Legal, and Social Issues | +| FASTA | Fast-A, file format for raw data output from sequencing machines | +| FASTQ | FASTA with Quality, file format for raw data output from sequencing machines | +| GDI | GDI European Genomic Data Infrastructure | +| LNDS | Luxembourg National Data Service | +| OK | All correct | +| SAM | Sequence Alignment/Map text file used in bioinformatics | +| SOP | Standard Operating Procedure | +| TB | Top to Bottom | +| TBD | To Be Determined | +| VCF | Variant Call Format, file format used to store genetic variations | + +| Term | Definition | +|------------------|-----------------| +| Data inclusion | Systematic integration of data into structured and machine-readable datasets | +| Data preparation | The process of structuring, annotating, and formatting data so that systems and users can discover, access and reuse it | +| Data reception | The process where GDI node securely acquires, ingests, and validates data supplied by a data provider | +| Data submission | The process of registering, publishing, and making data assets discoverable and accessible | +| Node Helpdesk | The support team is responsible for handling tickets related to data management issues associated with the node in the GDI Project. | +| Ticket | Incoming issues from the GDI Virtual Helpdesk, are to be processed within the Node Helpdesk and, after that, returned to the Virtual Helpdesk | + +### 3. Roles and Responsibilities +See qualifications and responsabilities of the roles at the [**Organisational Roles and Responsibilities**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_organisational-roles-and-responsibilities.md) document. + +| Role | Full name | GDI/node role | Organisation | +|------------|-----------------|-----------------|--------------| +| Author | Jorge Oliveira | Node technical coordinator | BioData.pt | +| Author | Jacek Lebioda | DevOps Engineer | LNDS | +| Reviewer | TBD | TBD | TBD | +| Approver | TBD | TBD | TBD | +| Authorizer | TBD | TBD | TBD | + + +### 4. Purpose +This SOP is an overarching procedure on how the data submissions made by the data suppliers will be handled. The purpose is to provide a clear, step-by-step guide for handling data submissions to the European Genomic Data Infrastructure, ensuring that helpdesk members can efficiently and accurately receive and process the submitted data and store it properly. Following, this SOP will streamline data management, maintain consistency, and ensure compliance with relevant protocols. + + +### 5. Scope +This SOP covers the process from receiving a data submission to the proper storage and acknowledgment of the submission. It defines how helpdesk members handle incoming submissions, ensuring they are stored accurately and appropriately for further use. + +### 6. Introduction and Background Information +This is a node-specific SOP template: each node can use this template as a basis and adapt it to their specific needs and procedures in their instances. + +Data Reception in GDI covers two steps of the Data Journey: Data Preparation and Data Inclusion. + + +### 7. Summary or Context Diagram + +```mermaid +graph TB + start( ) + input1[Submission request] + step1.1{Assessment of submission request - Data types verification} + step1.2{Assessment of submission request - ELSI compliance verification} + step2{Storage capacity check} + step3{Complete metadata} + step4(Review submission) + step5(Release datasets) + ending(" ") + + start -..- input1 + input1 -..->|Received by Role 1| step1 + step1 -->|Assessment OK| step2 + step1 -->|Assessment not OK| ending + step2 -->|Assessment OK| step3 + step2 -->|Assessment not OK| ending + step3 -->|Check OK| step4 + step3 -->|Check not OK| ending + step4 -->step5 + step5 -->step6 + step6 -..- ending +``` + +### 8. Procedure + + +#### 8.1 Assessment of submission request - data types evaluation + + +| Step identifier | When | Who | +|:------------------|:------|:----| +| 1 | When data is submitted | Helpdesk member assigned to the ticket (responsible for handling the submission request) | + +Check if the submitted data types are in a standardized format - SAM, BAM, CRAM, VCF, FASTA, FASTQ: + * Yes, continue the process + * No, contact the submitter and request appropriate changes. + +#### 8.2 Assessment of submission request - ELSI compliance verification + +| Step identifier | When | Who | +|:------------------|:----|:----| +| 2 | After data types verification | The Helpdesk member assigned to the ticket (responsible for handling the submission request) | + +Make compliance checks, i.e. make sure you have necessary ELSI documents (e.g. informed consent, ethical permits) and ELSI metadata (e.g. data controller). + * Yes, continue the process + * No, there is missing/incomplete/incorrect ELSI documentation or metadata: contact the submitter and request appropriate changes. + + +#### 8.3. Storage capacity check +| Step identifier | When | Who | +|:---------------------------|:-----------------|:----| +| 3 | After ticket assessment successful completion. | The Helpdesk member assigned to the ticket (responsible for handling the submission request) | + +After evaluating the submission, determine the data size and verify if sufficient storage is available to accommodate it. + * If yes, continue the process + * If not, (TBD) + + +#### 8.4. Storage capacity check +| Step identifier | When | Who | +|:---------------------------|:-----------------|:----| +| 4 | After completing the request evaluation and all related considerations. | The Helpdesk member assigned to the ticket (responsible for handling the submission request) | + +TBD + + +#### 8.5. Review submission +| Step identifier | When | Who | +|:---------------------------|:-----------------|:----| +| 5 | After metadata completion. | The Helpdesk member assigned to the ticket (responsible for handling the submission request) | + +After all, review the full submission: + * Data integrity check + - Validate checksums. + - For BAM/CRAM: samtools quickcheck and header validation. + - For VCF: bcftools view -h and vcftools/bcftools consistency checks. + - For FASTQ: basic read counts, header sanity checks. + * Metadata check + + + +#### 8.6. Release datasets +| Step identifier | When | Who | +|:---------------------------|:-----------------|:----| +| 6 | After submission review | The Helpdesk member assigned to the ticket (responsible for handling the submission request) | + +TBD + + +### 9. References + +| Reference | Description | +|-----------|------------------------------------------------------| +| [1](#) | European GDI - SOP Charter (including Glossary) | +| [2](#) | European GDI - Procedures for Information Service Management for SOP |