Skip to content

updates about response tables and appendix B #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
iannevans merged 4 commits into from
Sep 1, 2026
Merged

updates about response tables and appendix B #67

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c21dd6d
updates about response tables and appendix B
loumir Aug 31, 2026
004a437
Refactor SQL query for response and observation join
loumir Sep 1, 2026
c0b5640
Update about keys used for join in response table
loumir Sep 1, 2026
6d7014e
Adjust URL formatting in ivoa.response table section
loumir Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Dataproducts-Summary-table.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,13 @@ \subsubsection{Summary Table}
{\bf psf} &\raggedright Point Spread Function & A dataset that records the probability density function of spatial/angular spreading of incident particles from a point source caused by the instrument (detector and/or mirror and/or analysis) & \#response-function, \#pdf \cr
\noalign{\vspace{8pt}}
\hline
\multicolumn{4}{|r|}{\bf Considered for addition to an IVOA Analysis Data Product Vocabulary\footnote{As noted in \S~\ref{sec:dataproduct_type}, we prefer the term ``Advanced Data Product'' rather than ``Analysis Data Product'' since the latter suggests that additional steps ({\em i.e.\/} some type of analysis) have taken place to construct these data products, which may not be the case.}$^,$\footnote{\url{https://github.com/ivoa-std/VEPs/pull/21/changes\#diff-25a867536b53bdb6848baeba09cfc4df1e4ad42da919d440902e57b917dd9abc}.}} \\ \hline
\multicolumn{4}{|r|}{\bf Considered for addition to an IVOA Advanced Data Product Vocabulary} \ \\ \hline

@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

\noalign{\vspace{2pt}}
{\bf draws} & Draws & A dataset that records statistical draws computed from a probability distribution or a sample population, for example Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable, or the DeltaTS associated with a quantity from a frequentist analysis & none \cr
{\bf pdf} &\raggedright Probability Density Function & A dataset that records the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable & none \cr
{\bf region} & Region & A dataset that encodes (one or more) regions of parameter space, for example a spatial region or a region of phase space covered by a dataset. The set of dimensions represented by the region can be arbitrary & none \cr
%\sptablerule
\caption{IVOA Vocabulary Extension for High energy data products. }
\caption{IVOA Vocabulary Extension for High energy data products.}
\label{tab:dp_vocabulary}
\end{longtable}
\end{landscape}
19 changes: 12 additions & 7 deletions HighEnergyObsCoreExt.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -412,21 +412,23 @@ \subsection{Summary}

{\centering \bf Column Name} &{\centering \bf UCD} &{\centering \bf Unit} &{\centering \bf Type} &{\centering \bf Description} &{\centering \bf MAN}\\
\hline
{\em ev\_xel\/} & \ucd{meta.number;instr.detection;phys.particle} & unitless & int & {Number of events in an {\bf hea-event-list}}& NO \\
{\em obs\_publisher\_did} & \ucd{meta.ref.ivoid} &unitless & String & Related dataset ID (foreign key) & YES \\
\hline
{\em s\_ref\_energy\/} & \ucd{meta.ref;em.energy;pos} & eV & float & {Energy at which the ObsCore spatial characterization attributes {\em s\_fov\/} , {\em s\_region\/}, {\em s\_resolution\/} are defined} & NO \\
{\em ev\_xel\/} & \ucd{meta.number;instr.detection;phys.particle} & unitless & integer & {Number of events in an {\bf hea-event-list}}& NO \\
\hline
{\em em\_ref\_energy\/} & \ucd{meta.ref;em.energy;em} & eV & float & {Energy at which the ObsCore spectral characterization attributes {\em em\_res\_power\/}, {\em em\_resolution\/} are defined} & NO \\
{\em s\_ref\_energy\/} & \ucd{meta.ref;em.energy;pos} & eV & double & {Energy at which the ObsCore spatial characterization attributes {\em s\_fov\/} , {\em s\_region\/}, {\em s\_resolution\/} are defined} & NO \\
\hline
{\em s\_ref\_oaa\/} & \ucd{pos.posAng;instr.offset;pos} & deg & float & {Off-axis angle ({\em i.e.\/}, the angular separation of the target or source from the telescope optical axis) at which the ObsCore spatial characterization attributes {\em s\_fov\/} , {\em s\_region\/}, {\em s\_resolution\/} are defined} & NO \\
{\em em\_ref\_energy\/} & \ucd{meta.ref;em.energy;em} & eV & double & {Energy at which the ObsCore spectral characterization attributes {\em em\_res\_power\/}, {\em em\_resolution\/} are defined} & NO \\
\hline
{\em em\_ref\_oaa\/} & \ucd{pos.posAng;instr.offset;em} & deg & float & {Off-axis angle ({\em i.e.\/}, the angular separation of the target or source from the telescope optical axis) at which the ObsCore spectral characterization attributes {\em em\_res\_power\/}, {\em em\_resolution\/} are defined} & NO \\
{\em s\_ref\_oaa\/} & \ucd{pos.posAng;instr.offset;pos} & deg & double & {Off-axis angle ({\em i.e.\/}, the angular separation of the target or source from the telescope optical axis) at which the ObsCore spatial characterization attributes {\em s\_fov\/} , {\em s\_region\/}, {\em s\_resolution\/} are defined} & NO \\
\hline
{\em em\_ref\_oaa\/} & \ucd{pos.posAng;instr.offset;em} & deg & double & {Off-axis angle ({\em i.e.\/}, the angular separation of the target or source from the telescope optical axis) at which the ObsCore spectral characterization attributes {\em em\_res\_power\/}, {\em em\_resolution\/} are defined} & NO \\
\hline
{\em t\_intervals\/} & \ucd{TBD}& unitless & TMOC & {List of observation intervals or stable/good time intervals describing the exact observation time coverage} & NO \\
\hline
{\em energy\_min\/} & \ucd{em.energy;stat.min} & float & eV & {Energy associated to the ObCcore attribute {\em em\_max\/}, describing the minimal energy of the dataset} & NO \\
{\em energy\_min\/} & \ucd{em.energy;stat.min} & eV & double & {Energy associated to the ObCcore attribute {\em em\_max\/}, describing the minimal energy of the dataset} & NO \\
\hline
{\em energy\_max\/} & \ucd{em.energy;stat.max} & float & eV & {Energy associated to the ObsCore attribute {\em em\_min\/}, describing the maximal energy of the dataset} & NO \\
{\em energy\_max\/} & \ucd{em.energy;stat.max} & eV & double & {Energy associated to the ObsCore attribute {\em em\_min\/}, describing the maximal energy of the dataset} & NO \\

@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching the swapped columns here.

\hline
{\em obs\_mode\/} & \ucd{meta.code;obs.param} & unitless & string &{Observation mode of the observation ({\em e.g.\/}, TBU)} & NO \\
\hline
Expand Down Expand Up @@ -526,6 +528,9 @@ \subsubsection{Advanced Data Products}
{\bf region}: A dataset that includes an encoding of (one or more) regions of parameter space, for example a spatial region or a region of phase space covered by a dataset. The set of dimensions represented by the region can be arbitrary.
\end{quote}

An advanced-dataproduct-type Vocabulary \footnote{\url{https://github.com/ivoa-std/VEPs/pull/21/changes\#diff-25a867536b53bdb6848baeba09cfc4df1e4ad42da919d440902e57b917dd9abc}.} is proposed to the Semantics WG to register these terms in a standard vocabulary, that can be extended to other spectral domains, e.g radio astronomy.

@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The footnote url string needs to be in a smaller font or manually split across lines. The long hash is causing the footnote to run off the right edge of the page.



%mireille proposal for vocabularies
\input{Dataproducts-Summary-table}

Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ DOCNAME = HighEnergyObsCoreExt
DOCVERSION = 1.0

# Publication date, ISO format; update manually for "releases"
DOCDATE = 2026-08-03
DOCDATE = 2026-08-31

# What is it you're writing: NOTE, WD, PR, REC, PEN, or EN
DOCTYPE = PEN
Expand Down
4 changes: 3 additions & 1 deletion extendedAccessTonewtypesOfproducts.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,9 @@ \subsection{Accessing Datasets via an Alternate Table Joined with the {\tt ivoa.

More generally, the cardinality of the relationship between a {\bf response-function} dataset and an {\bf hea-event-list} dataset will vary according to the facility and may be one-to-one, one-to-many, many-to-one, or even many-to-many. Compared to the set of queryable attributes included in {\tt ivoa.obscore} table, typically more but occasionally fewer attributes will be required to select an appropriate {\bf response-function} dataset of a given type. The set of attributes required to identify an appropriate {\bf response-function} dataset will typically depend on the type of {\bf response-function}.

Using this approach, {\bf response-function} data products could be described by one or more alternate {\tt ivoa.response$\{$\_xxx$\}$} tables, where {\tt $\{$\_xxx$\}$} is optional and would depend on the type of {\bf response-function} in the case that different sets of queryable attributes are required for different types of {\bf response-function}s. A conceptual example of a {\tt ivoa.response$\{$\_xxx$\}$} table is presented in Appendix~\ref{sec:accessoptionsappendix}.
Using this approach, {\bf response-function} data products could be described by a {\tt ivoa.response} table .
%{\tt ivoa.response$\{$\_xxx$\}$} tables, where {\tt $\{$\_xxx$\}$} is optional and would depend on the type of {\bf response-function} in the case that different sets of queryable attributes are required for different types of {\bf response-function}s.
A conceptual example of such a {\tt ivoa.response} table is presented in Appendix~\ref{sec:accessoptionsappendix}.

A similar strategy could apply for discovering data products for an observation that are associated with an {\bf hea-event-list} data product if those data products require a set of queryable attributes that is not present in the {\tt ivoa.obscore} table. A dedicated table would allow more specific properties of these data products to be listed in optional table columns ({\em e.g.\/}, if certain data products have dependencies on telescope altitude and azimuth, or off-axis and azimuthal angles). Of course, this approach would preclude them from being directly queryable in the {\tt ivoa.obscore} table, which may be undesirable from the user perspective.

Expand Down
61 changes: 43 additions & 18 deletions extendedAccessTonewtypesOfproductsAppendix.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
In this appendix we further illustrate the possible use of an alternate table joined with the {\tt ivoa.obscore} table as a method for accessing {\bf response-function}s and data products that require queryable attributes that are not present in the {\tt ivoa.obscore} table, as discussed in \S~\ref{sec:respaccess}. Further scientific input from \gls{HEIG} domain experts, developed from additional use cases that focus on exploring the appropriate set of attributes needed to support such queries, will be needed to further explore and generalize this solution.

In order to handle the various possible cardinality relationships between {\bf response-function} and {\bf hea-event-list} datasets, foreign keys must be defined in the {\tt ivoa.response$\{$\_xxx$\}$} tables that will allow {\tt JOIN} operations between those tables and the {\tt ivoa.obscore} table.
In order to handle the various possible cardinality relationships between {\bf response-function} and {\bf hea-event-list} datasets, foreign keys must be defined in the {\tt ivoa.response} tables that will allow {\tt JOIN} operations between those tables and the {\tt ivoa.obscore} table.

The {\tt ivoa.response} table uses a specific resp\_dataproduct\_type column which is compliant to the response type vocabulary : \scriptsize{\url{https://www.ivoa.net/rdf/response-type/}} and has its own resp\_obs\_publisher\_did column as well, in order to distinguish which response matches which ObsCore data set in the {\tt ivoa.response} table.

@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it seems that this table is more about identifying linkages between response functions and the datasets to which they are applied rather than identifying response functions based on query attributes needed to identify them, and that is the reason that only a single table is required (because those query attributes which will be different for different types of response functions) are not required to be present?

That is OK, but would not for example allow one to identify and retrieve a psf based on off-axis angle and azimuthal angle in the telescope-frame. One could add additional columns to the table, but then those columns would not be meaningful for other types of response functions.

I presume that this solves the cardinality problem by allowing you to have multiple records in this table with different pairs of obs_publisher_did and resp_publisher_did.

If this is the case I don't see that ra, dec, and region (and perhaps t_intervals, energy_min, energy_max) are useful columns to have separately in this table as they are very unlikely to be the set of attributes necessary to uniquely identify different types of responses independently. If you have to come in via the obs_publisher_did linkage it's OK, since in most cases a single response function will only apply to a single linked dataset. However, there may be some cases (with psf being the most likely example) where a single response function might apply multiply to multiple datasets. ra, dec, region etc. would work well for the HESS psf example, but this is not generalizable.

Also in this case, why are obs_id and obs_publisher_did both required as foreign keys, as the latter is unique and therefore imputes the former?

At this point, consideration from HEIG domain experts is still required.

@loumir loumir Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also in this case, why are obs_id and obs_publisher_did both required as foreign keys, as the latter is unique and therefore imputes the former?
I changed the text : only one foreign key is required.

If this is the case I don't see that ra, dec, and region (and perhaps t_intervals, energy_min, energy_max) are useful columns to have separately in this table as they are very unlikely to be the set of attributes necessary to uniquely identify different types of responses independently.

I use these fields in the SELECT to check them in the query response .

\subsection*{Proposal for a Response Table {\tt ivoa.response}}

@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not something that we have considered with sufficient due diligence that the HEIG can call this a "Proposal" at this time. At best for now we could say "Possible response table". Considerable extra discussion is required to come up with a "proposal" for table(s) that describe response functions.

@loumir loumir Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree there is still more exploration to stabilize this but isn't "Proposal" something we can discuss and amend later? may be proposal and possible have different flavors in French and English ...

@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the context of a note such as this from the HEIG to the TCG/WGs, a "proposal" would be something that HEIG has internally agreed is appropriate and is proposing to the TCG/WGs for adoption. In English, it would not be interpreted as being a proposal from some members of the HEIG for further internal review by the entire group. Maybe a difference in interpretation between English and French.

@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be willing to use "Preliminary Proposal" or "Possible Draft Proposal".

If one of those is OK, I can accept the PR and I'll work on the formatting (something is still wrong in that section - too much of the text is now appearing as \scriptsize so I suspect a "}" is misplaced somewhere) and folding in the comments from Karl and Matthias, while you work on your Use Cases PR.

@loumir loumir Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with both titles "Preliminary Proposal" or "Possible Draft Proposal".. . thanks


\subsection*{Example Response Table}
%\TODO{ check fields for this new table }
%\begin{table}[htbp]
%\begin{center}
{\small
\begin{longtable}{|m{0.25\linewidth}|m{0.1\linewidth}|m{0.10\linewidth}|m{0.41\linewidth}|}
\hline
Expand All @@ -32,15 +32,16 @@ \subsection*{Example Response Table}
{\em resp\_energy\_min} & eV & double & Energy band minimal value for response use \\\hline
{\em resp\_energy\_max} & eV & double & Energy band maximal value for response use \\\hline

\caption{Example Response Table. With appropriate further study the columns identified herein could form the basis for a recommended base set of columns for any {\tt ivoa.response$\{$\_xxx$\}$} table.}
\caption{Proposal for a standard response table : with appropriate further study the columns identified herein could form the basis for a recommended base set of columns for the {\tt ivoa.response} table.}

@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above.

@loumir loumir Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree and the sentence explains it is some work to be continued .
Thanks

\label{tab:response_table}
\end{longtable}
}
%\end{center}
%\end{table}

\noindent Implementing an {\tt ivoa.response$\{$\_xxx$\}$} table similar to this would allow queries such as the following: \\

\subsection*{Example queries}
\noindent Implementing an {\tt ivoa.response} table similar to this would allow queries such as the following: \\
\subsubsection*{Retrieve psf response for {\em obs\_id} and position criterium }
\noindent Find all datasets satisfying:
\begin{enumerate}[(i)]
\item Position inside 3 arcmin from (83.6324, $+22.0174$),
Expand All @@ -49,28 +50,52 @@ \subsection*{Example Response Table}
\item obs\_collection = ``HESS''.
\end{enumerate}

When using the response table defined in Table \ref{tab:response_table}, we can join the {\tt ivoa.obscore} table with the {\tt ivoa.response\_psf} table on
the two keys {\em obs\_id\/} and {\em obs\_publisher\_did\/} for instance, as in:
When using the response table defined in Table \ref{tab:response_table}, we can join the {\tt ivoa.obscore} table with the {\tt ivoa.response} table on
one of the two keys {\em obs\_id\/} or {\em obs\_publisher\_did\/} for instance, as in:
{\small
\begin{verbatim}
SELECT obs_publisher_did, s_ra, s_dec, s_fov, t_min, t_max,
SELECT o.obs_publisher_did, o.obs_id, s_ra, s_dec, s_fov, t_min, t_max,
energy_min, energy_max, access_url, access_format,
resp_publisher_did, resp_access_url, resp_access_format,
resp_publisher_did, r.obs_publisher_did, r.obs_id, resp_access_url, resp_access_format,
resp_energy_max, resp_energy_min
FROM ivoa.obscore
NATURAL JOIN ivoa.response_psf
FROM ivoa.obscore as o
NATURAL JOIN ivoa.response as r
WHERE
(target_name = ’Crab’ OR target_name = ’M1’ OR
CONTAINS(POINT(s_ra, s_dec), CIRCLE, 83.6324, +22.0174, 0.083333) = 1)
AND (resp_product_type = 'psf')
AND (obs_id = '1527')
AND (obs_collection = 'HESS')
AND (o.obs_id = '1527')
AND (o.obs_collection = 'HESS')
\end{verbatim}
}

If additional constraints are required, then we can use ({\em e.g.\/}) {\em resp\_t\_min\/}, and {\em resp\_t\_max\/} and constrain the time interval by adding a clause like:
If additional constraints are required, for instance on time, then we can add another set of columns in {\tt ivoa.response}({\em e.g.\/}) {\em resp\_t\_min\/}, and {\em resp\_t\_max\/} and constrain the time interval by adding a clause like:
{\small
\begin{verbatim}
AND (resp_t_min > 56000.0 and resp_t_max < 56001.5)
\end{verbatim}
}

\subsubsection*{Retrieve edisp response for data sets constrained by {\em obs\_id} or position and by {\em scan\_mode}}
{\small
\begin{verbatim}
SELECT o.obs_publisher_did, o.obs_id, s_ra, s_dec, s_fov, t_min, t_max,
energy_min, energy_max, access_url, access_format, scan_mode,
resp_publisher_did, resp.obs_id, resp_access_url, resp_access_format,
resp_energy_max, resp_energy_min,
FROM ( SELECT o.obs_publisher_did, o.obs_id, scan_mode
FROM ivoa.obscore
NATURAL JOIN ivoa.obscore-hea
WHERE
(o.obs_collection = 'IACT')
AND (scan_mode=raster-map)
AND (o.obs_id = '1976')
OR CONTAINS(POINT(s_ra, s_dec), CIRCLE, 83.6324, +22.0174, 0.083333) = 1) as s
JOIN ivoa.response ON
resp.obs_id =s.obs_id
WHERE (resp_product_type = 'edisp')
% s is the list of observations satisfying position or obs_id and scan mode criteria
\end{verbatim}
}% end \small



@iannevans iannevans Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is something wrong with this example. The number of opening and closing parentheses do not match. Did you intent to have an embedded SELECT on line 85?

Loading