idscp horribly slow and memory demanding for large cases

[DavidPCoster]

The following version of idscp was used
IDStools/2.3.0-foss-2023b/bin/idscp

I used idscp to copy

total 20G
817M -r--r--r--. 1 g2dpc g2itmdev 817M Jun 3 00:05 core_profiles.h5
1.4G -r--r--r--. 1 g2dpc g2itmdev 1.4G Jun 3 00:05 core_sources.h5
608M -r--r--r--. 1 g2dpc g2itmdev 608M Jun 3 00:02 core_transport.h5
40K -r--r--r--. 1 g2dpc g2itmdev 40K Jun 3 00:05 dataset_description.h5
17G -r--r--r--. 1 g2dpc g2itmdev 17G Jun 3 00:04 equilibrium.h5
4.0K -r--r--r--. 1 g2dpc g2itmdev 2.4K Jun 3 00:05 master.h5
196K -r--r--r--. 1 g2dpc g2itmdev 193K Jun 3 00:05 summary.h5
540M -r--r--r--. 1 g2dpc g2itmdev 540M Jun 3 00:02 transport_solver_numerics.h5

to

total 14G
480M -rw-r--r--. 1 g2dpc g2itmdev 480M Jun 3 08:49 core_profiles.h5
1.2G -rw-r--r--. 1 g2dpc g2itmdev 1.2G Jun 3 09:14 core_sources.h5
548M -rw-r--r--. 1 g2dpc g2itmdev 548M Jun 3 09:55 core_transport.h5
40K -rw-r--r--. 1 g2dpc g2itmdev 40K Jun 3 09:55 dataset_description.h5
12G -rw-r--r--. 1 g2dpc g2itmdev 12G Jun 3 10:35 equilibrium.h5
4.0K -rw-r--r--. 1 g2dpc g2itmdev 2.4K Jun 3 10:50 master.h5
196K -rw-r--r--. 1 g2dpc g2itmdev 193K Jun 3 10:35 summary.h5
364M -rw-r--r--. 1 g2dpc g2itmdev 364M Jun 3 10:50 transport_solver_numerics.h5

This took 2 hours 6 minutes and 51.24 seconds, and required at least 95 GB of memory.

This seems very slow and expensive in terms of memory usage!

For comparison, the rsync time is 53.29 seconds.

I was also surprised to see that 6 GB was saved by copying the data using idscp.

The case contains 6000 time slices.

[olivhoenen]

idscp is probably not really the reason of this behaviour, we need to look at IMAS-Python + IMAS-Core.

Now we first need to understand a few things, especially to try understanding the reason for the different storage size:

• version of IMAS-Python, IMAS-Core and HDF5 used with idscp
• idscp arguments (and reason for using idscp to copy h5 pulse to h5 pulse)
• access to the files or version of DD and AL used for the source pulse (ids_properties/version_put)

[DavidPCoster]

IMAS-AL-Core/5.4.3-foss-2023b
IMAS-Python/2.0.1-foss-2023b
HDF5/1.14.4.3-gompi-2023b

idscp -s "imas:hdf5?path=IDS" -d "imas:hdf5?path=IDS_cp" -a
some plotting routines were unhappy with the files in IDS and were happier with the IDS_cp versions

source (IDS)
ids_properties
├── comment: 'ETS_PAF'
├── homogeneous_time: 2
├── provider: 'g2dpc'
├── creation_date: '2026-06-03T15:27:00Z'
└── ids_properties/version_put
├── data_dictionary: '3.42.0'
├── access_layer: '5.4.3'
└── access_layer_language: 'python-5.4.0'
OR
ids_properties
├── comment: 'Output from a transport solver'
├── homogeneous_time: 1
├── provider: 'g2dpc'
├── creation_date: '2026-06-03T07:00:04+0200'
└── ids_properties/version_put
├── data_dictionary: '3.42.0'
├── access_layer: '5.4.3'
└── access_layer_language: 'python-5.4.0'

destination (IDS_cp)
ids_properties
├── comment: 'ETS_PAF'
├── homogeneous_time: 2
├── provider: 'g2dpc'
├── creation_date: '2026-06-03T15:27:00Z'
└── ids_properties/version_put
├── data_dictionary: '3.42.0'
├── access_layer: '5.4.3'
└── access_layer_language: 'imas 2.0.1'

A copy of the files can be found at ITER under
/home/ITER/costerd/ETS-PAF_from_Gateway/run_ETS_MAIN_ENCAPSULATED_WITH_ETS_INIT_20260603_065931
in the directories IDS and IDS_cp

[olivhoenen]

First I want to understand the difference in size between the ref and the copied over data. I'm still puzzled there, as I see zero IDS data difference when looking at a single slice (I picked time=4.0) obtained with IMAS-Python/2.2.0 or with the IMAS-AL-Python/5.4.0 you seem to be using for the reference data:

As well as no size diff in e.g. core_profiles read and write from these same versions of IMAS-Python and IMAS-AL-Python, both have the same size that you have in your IDS_cp. I'm wondering how the reference data was written. Did you used any of the HDF5 backend options.

Maybe the size change is about internal space management from HDF5 file, I guess there may be extra space when you are growing the file in case it was written by appending slices vs written as a single put. I've tried with repack on the core_profiles.h5, get a decrease from 817MB to 795MB but nowhere near from 486MB from idscp (or manually get/put with IMAS-Core). Maybe @fleury-ai has a better understanding of this aspect.

[olivhoenen]

some plotting routines were unhappy with the files in IDS and were happier with the IDS_cp versions

Could you give the issues of the plotting routines? Maybe that would help getting clues on some reasons for difference between ref and copied data.

[DavidPCoster]

With regard to writing, I was not using any additional options: the URI was 'imas:hdf5?path=../../../IDS' and this was used in

DB = imas.DBEntry(uri_out, "w")
DB.create()

I don't remember which plots failed, and I have not been able to find failing plots on my recent data ...

[DavidPCoster]

And for a larger case, I now get

Segmentation fault

after

real 135m34.535s
user 126m50.875s
sys 4m28.123s

and no further information ...