Skip to content

September 4 Final putbacks #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
iannevans merged 1 commit into from
Sep 4, 2026
Merged

September 4 Final putbacks #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Dataproducts-Summary-table.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ \subsubsection{Summary Table}
{\bf psf} &\raggedright Point Spread Function & A dataset that records the probability density function of spatial/angular spreading of incident particles from a point source caused by the instrument (detector and/or mirror and/or analysis) & \#response-function, \#pdf \cr
\noalign{\vspace{20pt}}
\hline
\multicolumn{4}{|r|}{{\bf Considered for addition to an IVOA Advanced Data Products Vocabulary}} \\ \hline
\multicolumn{4}{|r|}{{\bf Considered for addition to an IVOA Advanced Data Product Types Vocabulary}} \\ \hline
\noalign{\vspace{6pt}}
{\bf draws} & Draws & A dataset that records statistical draws computed from a probability distribution or a sample population, for example Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable, or the DeltaTS associated with a quantity from a frequentist analysis & none \cr
\noalign{\vspace{2pt}}
Expand Down
29 changes: 16 additions & 13 deletions HighEnergyObsCoreExt.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,11 +198,11 @@ \subsection{{\em dataproduct\_type}}
%\TODO{ mireille: Rewrite this paragraph to avoid measurements: no agreement exist for this term among ObsCore Implementors in practice. }
%%% There is no way to rewrite this paragraph without referring to measurements. The term "measurements" is defined in the ObsCore Recommendation and this recommendation modifies the wording of that definition somewhat. The measurements concept is actually quite important as it's use avoids having to dramatically increase the the number of data product type terms one would otherwise require (some of which may be facility dependent).

Many different types of data products are served by \gls{HEA} facilities, including datasets that consist of generic tabular data (an {\bf hea-event-list} is in fact a {\em specific type\/} of tabular data). Some \gls{HEA} facilities require a {\em dataproduct\_type\/} that represents such generic tabular data. We note in passing that such a {\em dataproduct\_type} actually falls at the same level as ({\em e.g.\/}) the {\bf image} {\em dataproduct\_type\/}: an {\bf image} has two spatial axes, but the observable axis is not defined; while commonly the observable axis for an {\bf image} is flux, it could easily be ({\em e.g.\/}) color, spectral-index, significance, and so on, and additional refinement such as {\em dataproduct\_subtype\/} may be required to more precisely specify the nature of a data product. Some of these data products are essential for further data analysis or important in their own right, but may be uncommon across multiple facilities possibly because of the way the observations are obtained.
Many different types of data products are served by \gls{HEA} facilities, including datasets that consist of generic tabular data (an {\bf hea-event-list} is in fact a {\em specific type\/} of tabular data). Some \gls{HEA} facilities require a {\em data\-product\_type\/} that represents such generic tabular data. We note in passing that such a {\em dataproduct\_type} actually falls at the same level as ({\em e.g.\/}\null) the {\bf image} {\em dataproduct\_type\/}: an {\bf image} has two spatial axes, but the observable axis is not defined; while commonly the observable axis for an {\bf image} is flux, it could easily be ({\em e.g.\/}\null) color, spectral-index, significance, and so on, and additional refinement such as {\em dataproduct\_subtype\/} may be required to more precisely specify the nature of a data product. Some of these data products are essential for further data analysis or important in their own right, but may be uncommon across multiple facilities possibly because of the way the observations are obtained.

In the ObsCore Recommendation Version 1.1, the {\em dataproduct\_type\/} {\bf measurements} is adequate for such generic tabular datasets. However, users of those products often may not be interested in the progenitor datasets, especially as some advanced data products may be derived from multiple observations, or multiple advanced data products may be extracted from the same single progenitor or a few progenitors ({\em e.g.\/}, advanced data products associated with multiple sources detected in a single observation field). Referring to the ObsCore Recommendation Version 1.1, we propose to delete the caveat associated with {\em dataproduct\_type\/} = ``measurements'' in the ObsCore IVOA Recommendation (\S~4.1.1) that requires the derived data products be exposed ``{\bf together} with the progenitor observation dataset''. The recovery of progenitor observation datasets may be achieved using provenance information, if desired.

In the \gls{IVOA} Data Product Type Vocabulary\footnote{\url{https://www.ivoa.net/rdf/product_type}.}, the term {\bf measurements} is described as ``Generic tabular data not fitting any of the other terms'' and that similarly matches what is needed here. While the description goes on to state ``Because of its lack of specificity, this term should generally be avoided, and new, more precise terms should be introduced instead.'', as indicated above there are circumstances under which doing so does not make sense. We are aware that there is currently a proposal to eliminate the {\bf measurements} data product type given it's previous loose definition. If that is done, then we would need a replacement, such as {\bf tabledata\/}, that can be used to represent generic tabular data, as there is no other data product type to represent this type of dataset.
In the \gls{IVOA} Data Product Type Vocabulary\footnote{\url{https://www.ivoa.net/rdf/product_type}.}, the term {\bf measurements} is described as ``Generic tabular data not fitting any of the other terms'' and that similarly matches what is needed here. While the description goes on to state ``Because of its lack of specificity, this term should generally be avoided, and new, more precise terms should be introduced instead.'', as indicated above there are circumstances under which doing so does not make sense. We are aware that there is currently a proposal to eliminate the {\bf measurements} data product type given its previous loose definition. If that is done, then we would need a replacement, such as {\bf tabledata\/}, that can be used to represent generic tabular data, as there is no other data product type to represent this type of dataset.

\subsection{{\em dataproduct\_subtype}}

Expand Down Expand Up @@ -392,7 +392,7 @@ \subsection{{\em messenger\_pdgid}}

We propose to add an optional attribute {\em messenger\_pdgid\/} that specifies the PDG ID of the messenger type for an observation. Constraints on {\em messenger\_pdgid\/} can provide a simple way to discover data sets for a specific messenger with a finer granularity than is possible with the {\em messenger\_name\/} attribute alone. The value should be a string that specifies the messenger particle's PDG ID as `{\em $\pm$nn\/}' (where the sign is mandatory). For example, a muon, $\mu^{-}$, may be specified as `+13'.

The difference between {\em messenger\_name\/} and {\em messenger\_pdgid} is that ({\em e.g.\/}) a data discovery query that includes {\tt messenger\_name = `neutrino'} would enable the user to identify datasets with a neutrino (any kind) messenger particle, whereas a data discovery query that includes {\tt messenger\_pdgid = `+16' OR messenger\_pdgid = `-16'} would restrict the search to include only $\tau$ neutrinos ($\nu_\tau$) or antineutrinos ($\nu_\tau^{-}$).
The difference between {\em messenger\_name\/} and {\em messenger\_pdgid} is that ({\em e.g.\/}\null) a data discovery query that includes {\tt messenger\_name = `neutrino'} would enable the user to identify datasets with a neutrino (any kind) messenger particle, whereas a data discovery query that includes {\tt messenger\_pdgid = `+16' OR messenger\_pdgid = `-16'} would restrict the search to include only $\tau$ neutrinos ($\nu_\tau$) or antineutrinos ($\nu_\tau^{-}$).

The downside of {\em messenger\_pdgid} is that PDG ID very unlikely to be recognized outside of the particle astrophysics and high energy particle physics communities. We choose to include both {\em messenger\_name\/} and {\em messenger\_pdgid} because the majority of astrophysicists --- even experienced high-energy astrophysicists --- are unlikely to recognize PDG ID\null. While any astrophysicist is likely to be able to write a query such as {\tt messenger\_name = `photon'}, very few would be able to cast that query as {\tt messenger\_pdgid = `+22'} from memory.

Expand Down Expand Up @@ -512,7 +512,7 @@ \subsubsection{Response Functions}
{\bf rmf}: A dataset that records the probability density function mapping from energy space into detector pulse height (or position) space \citep{ogip_spectrum_1998}.
\end{quote}

These terms have been declared in coordination with the IVOA Semantics working group in a specialized IVOA data product vocabulary dedicated to response functions and available at \url{https://www.ivoa.net/rdf/response-type/}.
These terms have been declared in coordination with the IVOA Semantics working group in a specialized IVOA data product vocabulary dedicated to response functions, available at \url{https://www.ivoa.net/rdf/response-type/}.

\subsubsection{Advanced Data Products}

Expand All @@ -528,7 +528,7 @@ \subsubsection{Advanced Data Products}
{\bf region}: A dataset that includes an encoding of (one or more) regions of parameter space, for example a spatial region or a region of phase space covered by a dataset. The set of dimensions represented by the region can be arbitrary.
\end{quote}

An advanced-dataproduct-type Vocabulary \footnote{\url{https://github.com/ivoa-std/VEPs/pull/21/changes\#diff-25a867536b53bdb6848baeba09cfc4df1e4ad42da919d440902e57b917dd9abc}.} is proposed to the Semantics WG to register these terms in a standard vocabulary, that can be extended to other spectral domains, e.g radio astronomy.
An advanced data product type vocabulary\footnote{\url{https://github.com/ivoa-std/VEPs/pull/21/changes\#diff-25a867536b53bdb6848baeba09cfc4df1e4ad42da919d440902e57b917dd9abc}.} is proposed to the Semantics WG to register these terms in a standard vocabulary, that can be extended to other spectral domains, {\em e.g.\/}, radio astronomy.


%mireille proposal for vocabularies
Expand All @@ -541,7 +541,7 @@ \subsubsection{Clarification of ``Flux'' in Data Product Type Vocabulary Definit

We propose to clarify the IVOA Data Product Type Vocabulary\footnote{\url{https://www.ivoa.net/rdf/product_type}.} definitions that use the word ``flux'' in the description of some terms, currently {\bf light-curve}, {\bf polarization-resolved-dataset}, {\bf polarized-spectrum}, and {\bf spectrum}, so that the definitions are applicable to \gls{HEA} data products.

The issue is that the term ``flux'' is not defined in the vocabulary, and the standard astronomical definition of ``flux'' is an energy flux (with SI units $\rm W\,m^{-2}$). This interpretation is bolstered by the statements ``flux or magnitude'' applied to several of the descriptions since optical/IR magnitude and energy flux density are tightly related. However, with this definition, many \gls{HEA} data products that may have fluxes defined using units of counts or particles ({\em e.g.\/}, photons), but not calibrated in units of energy, would not satisfy the current descriptions of ({\em e.g.\/}) {\bf light-curve} or {\bf spectrum}, even though that is how those products are used.
The issue is that the term ``flux'' is not defined in the vocabulary, and the standard astronomical definition of ``flux'' is an energy flux (with SI units $\rm W\,m^{-2}$). This interpretation is bolstered by the statements ``flux or magnitude'' applied to several of the descriptions since optical/IR magnitude and energy flux density are tightly related. However, with this definition, many \gls{HEA} data products that may have fluxes defined using units of counts or particles ({\em e.g.\/}, photons), but not calibrated in units of energy, would not satisfy the current descriptions of ({\em e.g.\/}\null) {\bf light-curve} or {\bf spectrum}, even though that is how those products are used.

For \gls{HEA}, because the mappings between observed counts, incident particles, and incident particle energy depend on the {\bf response-function}s (which in some cases may not be determinable without knowledge of the individual science use case) we often use several definitions of photometric properties such as flux, radiance, and flux density:

Expand All @@ -557,7 +557,7 @@ \subsubsection{Clarification of ``Flux'' in Data Product Type Vocabulary Definit
\item Energy flux density (differential flux), with units $\rm J\,m^{-2}\,s^{-1}\,\hbox{eV}^{-1}$.
\end{itemize}

The common \gls{HEA} flux, radiance, and flux density definitions listed above are not all covered by the UCD vocabulary. We propose to add new UCDs in \S~\ref{sec:UCDs} below to address this, and where appropriate refine the definitions of existing UCDs based on dimensionality. With the proposed additions and refinements, we can, for example, easily distinguish between a counts flux with UCD {\em phot.counts\/}, a photon flux with UCDs {\em phot.flux.particle;phys.particle.photon\/}, and an energy flux with UCD {\em phot.\-flux.energy\/}. These refinement will also be particularly useful for cases where a \gls{HEA} quantity should have the same UCD as ({\em e.g.\/}) a radio measurement but where very different units are used. For example, $\rm Jy$ ( Jansky) and $\rm erg\,cm^{-2}\,s^{-1}\,TeV$ are both spectral flux densities but the unit analysis doesn’t agree due to the frequency-energy equivalency.
The common \gls{HEA} flux, radiance, and flux density definitions listed above are not all covered by the UCD vocabulary. We propose to add new UCDs in \S~\ref{sec:UCDs} below to address this, and where appropriate refine the definitions of existing UCDs based on dimensionality. With the proposed additions and refinements, we can, for example, easily distinguish between a counts flux with UCD {\em phot.counts\/}, a photon flux with UCDs {\em phot.flux.particle;phys.particle.photon\/}, and an energy flux with UCD {\em phot.\-flux.energy\/}. These refinement will also be particularly useful for cases where a \gls{HEA} quantity should have the same UCD as ({\em e.g.\/}\null) a radio measurement but where very different units are used. For example, $\rm Jy$ (Jansky) and $\rm erg\,cm^{-2}\,s^{-1}\,TeV$ are both spectral flux densities but the unit analysis doesn’t agree due to the frequency-energy equivalency.

Restating the existing IVOA Data Product Type Vocabulary descriptions that use the term ``flux'' to explicitly state ``count/particle/energy flux or magnitude'' where appropriate would resolve the concern raised above with the current use of the term ``flux''.

Expand Down Expand Up @@ -749,12 +749,15 @@ \section{Contributions to the Note}

Further material can be found with those links:
\begin{itemize}
\item 2025-04-29: French ASOV workship ``High Energy in the VO'', \url{https://indico.obspm.fr/event/2674/},
\item 2024-11-16: IVOA Malta meeting, DM session with two High Energy presentations (B. Kh\'elifi/I. Evans), \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpNov2024DM},
\item 2024-11-15: IVOA Malta Plenary, CSP Plenary session, \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpNov2024CSPPlenary},
\item 2024-05-21: IVOA Sydney meeting, DM Session High Energy focus, \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpMay2024DM},
\item 2023-06-28: IVOA standards for High Energy Astrophysics (French VO Workshop), \url{https://indico.obspm.fr/event/1963/},
\item 2023-05-11: IVOA Bologna meeting: presentation (``DM for High Energy astrophysics'', M. Servillat) and first IVOA HE group meeting, \url{https://wiki.ivoa.net/internal/IVOA/IntropMay3023DM/2023-05-11_IVOA_meeting_-_VOHE.pdf},
\item 2026-06-09 IVOA Strasbourg meeting, ObsCore Extensions plenary with one High Energy presentation, 4 demonstrations, and panel discussion (I. Evans, M. Servillat, S. Hallman), \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpJune2026HEIG};
\item 2025-11-15 IVOA Goerlitz meeting, DM session with one High Energy presentation (M. Louys), \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpNov2025DM};
\item 2025-06-04: IVOA College Park meeting, DM session with ObsCore Extension progress report, \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpJune2025DM};
\item 2025-04-29: French ASOV workshop ``High Energy in the VO'', \url{https://indico.obspm.fr/event/2674/},
\item 2024-11-16: IVOA Malta meeting, DM session with two High Energy presentations (B. Kh\'elifi/I. Evans), \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpNov2024DM};
\item 2024-11-15: IVOA Malta Plenary, CSP Plenary session, \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpNov2024CSPPlenary};
\item 2024-05-21: IVOA Sydney meeting, DM Session High Energy focus, \url{https://wiki.ivoa.net/twiki/bin/view/IVOA/InterOpMay2024DM};
\item 2023-06-28: IVOA standards for High Energy Astrophysics (French VO Workshop), \url{https://indico.obspm.fr/event/1963/};
\item 2023-05-11: IVOA Bologna meeting: presentation (``DM for High Energy astrophysics'', M. Servillat) and first IVOA HE group meeting, \url{https://wiki.ivoa.net/internal/IVOA/IntropMay3023DM/2023-05-11_IVOA_meeting_-_VOHE.pdf};
\item 2022-10-11: Virtual Observatory and High Energy Astrophysics (French VO Workshop), \url{https://indico.obspm.fr/event/1489/}.
\end{itemize}

Expand Down
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ DOCNAME = HighEnergyObsCoreExt
DOCVERSION = 1.0

# Publication date, ISO format; update manually for "releases"
DOCDATE = 2026-09-01
DOCDATE = 2026-09-04

# What is it you're writing: NOTE, WD, PR, REC, PEN, or EN
DOCTYPE = PEN

# An e-mail address of the person doing the submission to the document
# repository (can be empty until a make upload is being made)
AUTHOR_EMAIL=ian.evans@ , bruno.khelifi@, mathieu.servillat@
AUTHOR_EMAIL=ian.evans@, bruno.khelifi@, mathieu.servillat@

# Source files for the TeX document (but the main file must always
# be called $(DOCNAME).tex)
Expand Down
Loading
Loading