ADD: calibration files from unit-level testing to resource/calibration

[fhagemann]

In this PR, we add the full ecal, taccal, depth splines and depth coefficients for the 16 flight detectors as determined from the unit-level calibrations, assuming the thicknesses from the table below ("E (sims)") and realized in cositools/massmodel-cosi-payload#3:

Also added are the combined files for all 16 detectors, derived from the individual unit-level files.

• I still need to run a final cross-check to make sure that all the files here are the FINAL calibration file as reported in the Unit Level Reports, which will require some extra time.
• I also still need to do a final cross-check to see if the DEE / forward pipeline accepts them based on the thicknesses updated in Adjust detector thicknesses to values used during unit-level testing massmodel-cosi-payload#3.

Note that for PAYLOAD calibrations, we will use updated thickness (and temperature) values (see cositools/massmodel-cosi-payload#4), that will be added to resource/calibration/COSISMEX/payload (or whatever we might call the repository) once they become available.

[fhagemann]

And this is the Julia code I used to generate the combined files

using DelimitedFiles

dets = [
    "lbl_HP52366-2_15.83mm_900V_3.53e9_83K",
    "lbl_HP52358-2_15.5mm_1050V_4.8e9_83K",
    "lbl_HP52432-1_15.83mm_600V_2.12e9_83K",
    "m61617-fm2_15.68mm_650V_2.52e9_83K",
    "lbl_HP52542-1_15.95mm_800V_2.78e9_83K",
    "lbl_HP52358-3_15.6mm_1000V_4.36e9_83K",
    "lbl_HP52434-1_15.7mm_1150V_5.38e9_83K",
    "lbl_HP52406-1_16.0mm_1000V_4.84e9_83K",
    "lbl_HP52542-2_16.1mm_800V_2.73e9_83K",
    "lbl_HP52407-3_15.8mm_700V_2.13e9_83K",
    "lbl_HP52379-1_15.71mm_700V_2.15e9_83K",
    "lbl_HP52358-1_15.67mm_850V_4.68e9_83K",
    "lbl_HP52434-2_16.3mm_1000V_3.99e9_83K",
    "lbl_HP52361-1_15.57mm_800V_2.92e9_83K",
    "m61618-fm3_15.2mm_750V_3.06e9_83K",
    "lbl_HP52406-3_15.93mm_1050V_4.53e9_83K"
];

sim_dir = "/home/hagemann/Software/COSItools/nuclearizer/resource/calibration/COSISMEX/unit/simulations/"
fn_out = "/home/hagemann/Software/COSItools/nuclearizer/resource/calibration/COSISMEX/payload/depth-splines-derived-from-unitlevel.csv"
open(fn_out, "w") do io
    for (i,det) in enumerate(dets)
        fn_in = joinpath(sim_dir, "HV32_LV32_59.5keV_"*det*"_ctd_dt_depth_mcSPICE.csv")
        write(io, "### $(i-1)\n")
        writedlm(io, readdlm(fn_in, ',', comments = true, comment_char = '#'), ',')
        write(io, "\n")
    end
end

dets_short = first.(split.(dets, "_1"))
depth_dir = "/home/hagemann/Software/COSItools/nuclearizer/resource/calibration/COSISMEX/unit/depth/"
fn_out = "/home/hagemann/Software/COSItools/nuclearizer/resource/calibration/COSISMEX/payload/depth-coeffs-derived-from-unitlevel.csv"
open(fn_out, "w") do io
    for (i,det) in enumerate(dets_short)
        fn = only(filter(x -> occursin(det, x), readdir(depth_dir)))
        fn_in = joinpath(depth_dir, fn)
        A::Matrix{Any} = readdlm(fn_in, ',', comments = true)
        A[:,1] .= Int.(A[:,1]) .+ (i-1)*10000
        writedlm(io, A, ',')
    end
end

ecal_dir = "/home/hagemann/Software/COSItools/nuclearizer/resource/calibration/COSISMEX/unit/ecal/"
dets_short = replace.(first.(split.(dets, "_1")), "lbl_" => "", "m" => "", r"-fm\d" => "")
fn_out = "/home/hagemann/Software/COSItools/nuclearizer/resource/calibration/COSISMEX/payload/full-ecal-derived-from-unitlevel.ecal"
open(fn_out, "w") do io
    for (d,det) in enumerate(dets_short)
        DetID = d - 1 # start DetID at 0
        fn = only(filter(x -> occursin(det, x), readdir(ecal_dir)))
        fn_in = joinpath(ecal_dir, fn)
        ecal_data = readdlm(fn_in)
        for i in findall(x -> x[3] isa Int, eachrow(ecal_data))
            @assert ecal_data[i,3] == 0
            ecal_data[i,3] = DetID
        end
        writedlm(io, join.(eachrow(ecal_data), " ")) 
    end
end

taccal_dir = "/home/hagemann/Software/COSItools/nuclearizer/resource/calibration/COSISMEX/unit/taccal/"
dets_short = replace.(first.(split.(dets, "_1")), "lbl_" => "", "m" => "", r"-fm\d" => "")
fn_out = "/home/hagemann/Software/COSItools/nuclearizer/resource/calibration/COSISMEX/payload/taccal-derived-from-unitlevel.csv"
open(fn_out, "w") do io
    for (d,det) in enumerate(dets_short)
        DetID = d - 1
        fn = only(filter(x -> occursin(det, x), readdir(taccal_dir)))
        fn_in = joinpath(taccal_dir, fn)
        raw_data, header = readdlm(fn_in, ',', header = true)
        ints   = Int64.(raw_data[:, 1:4])
        floats = Float64.(replace(raw_data[:, 5:end],"" => NaN))
        taccal_data = Any[ints floats]
        if DetID == 0
            write(io, "#" * join(header, ',')*'\n')
        end
        writedlm(io, taccal_data, ",")
    end
end

[fhagemann]

I had one more thorough round of cross-checking the calibration files in this PR against:

1. what's reported in the unit-level reports
2. what's reported in the ARM summary slides by Sean/Julian

Some notes:

• Marked in $\color{red}\textsf{red}$ is where the files deviate from what is reported in the unit-level reports.
  For HP52406-3, this is intentional: we took three TAC calibrations with timestamp 20250821, 20250826 and 20250829. It looks like the 20250821 was different from the 20250826/20250829 one:
  

  As the depth coefficients file was generated with 20250821, I chose to keep the 20250821 TAC calibration file in this PR instead of one of the "newer" 20250826/20250829 ones. If we want to keep a "newer" 20250826/20250829 TAC calibration file, we would need to re-run the depth calibration for this detector.

• Marked in $\color{blue}\textsf{blue}$ are the values where another file was used for the ARM calibration plots.

  • For HP52358-2, HP52432-1, and HP52358-1, the ARM plots were composed based on an energy calibration that did not apply TAC cuts. The energy calibrations were updated in September to include TAC cuts, and are also the ones reported in the unit-level reports.
  • For HP52542-2, HP52407-3, and HP52406-3, the ARM plots were composed using the quick ecal instead of the full ecal. Here, I'm using the energy calibration files reported in the unit-level reports.

One final thing to note: seems like for HP52358-3, the full energy calibration was not re-run to include TAC cuts.

[fhagemann]

I also seem to be able to run the DEE with all those files and the mass model in cositools/massmodel-cosi-payload#3

[ckierans]

This is a huge effort, @fhagemann !!! Thank you so much for the careful book keeping!