fix(harvester): put CERN- report numbers in identifiers - #877
Conversation
3d911c8 to
bcec7d4
Compare
bcec7d4 to
80e574c
Compare
| report_numbers = src_metadata.get("report_numbers", []) | ||
| for rn in report_numbers: | ||
| report_number = rn.get("value") | ||
| if not report_number or report_number.startswith("CERN-"): |
There was a problem hiding this comment.
also, we need to monitor for EP numbers now... and the schema for EP numbers is different.
it should be taken into account when searching for existing records too - we didn't use the apprn until now, could you add matching by the EP approval too?
There was a problem hiding this comment.
Added. EP numbers now map to apprn (using the prefixes from CDS_COMMITTEE_APPROVAL_COMMUNITIES), and the matcher also searches by apprn when looking for existing records. Other CERN- report numbers still go to identifiers as cdsrn.
346ce48 to
3be508d
Compare
| report_number = self._retrieve_identifier(related_identifiers, "cdsrn") | ||
| report_number = self._retrieve_identifier(identifiers, "cdsrn") | ||
| related_report_number = self._retrieve_identifier(related_identifiers, "cdsrn") | ||
| approval_report_number = self._retrieve_identifier(identifiers, "apprn") |
There was a problem hiding this comment.
@zzacharo @palkerecsenyi guys, how are we serializing approval report number when sending it to datacite? in general, in export formats we should not have another identifier type for apprn, it should be indistinguishable from reportnumber, since apprn is our internal system implementation
There was a problem hiding this comment.
It is indistinguishable if I understand correctly. For example, https://dev-cds-rdm.web.cern.ch/records/vq903-1p243 is exported as:
"alternateIdentifiers": [
{
"alternateIdentifier": "https://dev-cds-rdm.web.cern.ch/records/vq903-1p243",
"alternateIdentifierType": "URL"
},
{
"alternateIdentifier": "oai:127.0.0.1:5000:vq903-1p243",
"alternateIdentifierType": "oai"
},
{
"alternateIdentifier": "CERN-EP-2024-309",
"alternateIdentifierType": "CDS"
},
{
"alternateIdentifier": "2917427",
"alternateIdentifierType": "CDS"
}
],All CDS internal identifiers are exported as type CDS:
Line 696 in ef723ca
3be508d to
cbf35a8
Compare
| RelatedReportNumberMatchFilter(value=related_report_number), | ||
| ApprovalReportNumberMatchFilter(value=approval_report_number), |
There was a problem hiding this comment.
I would set approval report number as a priority since it is more "controlled" by the system
Matching by related report number might be very error-prone - you will match records which are related to the record you are actually searching for and it might return multiple matches quite frequently. I think you might add some more restrictive criteria to this. Can you propose what would make it less error prone?
There was a problem hiding this comment.
the approach i used is to make the db search stricter by searching related identifiers with relation_type isvariantformof and scheme cdsrn, and if it still returns multiple records then treat it as ambiguous. also put apprn above related report number in the priority list
cbf35a8 to
6c6b48b
Compare
palkerecsenyi
left a comment
palkerecsenyi
left a comment
There was a problem hiding this comment.
Looks good to me, the approval number validation is correct as far as I can tell
| return False | ||
| if not schemes.is_approval_report_number(value): | ||
| return False | ||
| return any(value.startswith(prefix) for prefix in _committee_approval_prefixes()) |
There was a problem hiding this comment.
I'm not sure startswith is the best way to detect approval numbers since we might have CERN-EP-DRAFT-.... report numbers
There was a problem hiding this comment.
Hmmmm this is true, I am not sure what would be the best way to solve it though. We could specifically check for CERN-EP-DRAFT-* and exclude it, not sure if there is any other reliable way of doing it
There was a problem hiding this comment.
can we do a regex and check if the report number is in this format prefix-digit* so if we have prefix-letter we exclude?
There was a problem hiding this comment.
yup good catch, i have added the regex you suggested
6c6b48b to
143770e
Compare
5 participants
Closes #861
INSPIRE report numbers starting with CERN- (scheme cdsrn) were always mapped into metadata.related_identifiers but they now go to metadata.identifiers, while non-CERN- report numbers stay related. The record matcher reads and searches cdsrn in identifiers to match that placement.