Add $msg.hash filename parameter for content-based dedup of received files - #575
Conversation
…files Adds a $msg.hash.<algorithm>$ parameter to the message filename templates. The algorithm is md5, sha1, sha256 or sha512, optionally followed by an underscore and a truncation length (e.g. $msg.hash.sha256_16$ for the first 16 hex characters). It hex-encodes a digest of the message payload, computed lazily only when referenced. Including it in store_received_file_to (or the global received-file template) dedups by name and content: a re-delivery of the same file resolves to the same path and overwrites, while changed content gets a new name. SHA-256 truncated to 16 chars is a good default - 64 bits is far more than enough collision resistance per filename. Adds a unit test and a commented example in partnerships.xml.
|
Can you rerun that test build? I'm not able and it looks like a timeout issue |
|
@uhurusurfa this is the last PR I have planned if you want to cut a release |
|
I am not convinced about this one. Whilst I understand what you are trying to achieve, the AS2 Message-ID is supposed to be a unique identifier per AS2 message even if the message is resent. The resending of a payload with Message-ID usage is here: So if you store the file using the Message-ID you are effectively ensuring that you do not get duplicates. Whilst the 4.X version of OpenAS2 does not attempt to identify repeat resends that it believes it has received and acknowledged successfully, the 5.X version will add smarter message tracking capability identifying resends and flagging them when they occur allowing them to be ignored. Let me know if you still believe the hash feature is a necessary addition. |
|
The problem I'm trying to solve is when an entirely new AS2 message is sent. I have customers which occasionally send me an AS2 message where the message sends the same filename. There's two cases of toes:
This falls outside of AS2 spec and I understand if this is out of the scope of this project, it's easy enough to handle this outside of OpenAS2. |
|
This is also on the assumption that OpenAS2 will overwrite the original file if one comes with the same filename. Specifically for the same partnership so they'll have the same exact writepath. |
|
I am happy to add stuff outside the AS2 spec - just wanted to make sure it is not already covered. |
I will get a release done in the next day or 2. |
|
All your contributions have been released in 4.9.0 |
2 participants
Adds a$msg.hash.$ parameter to the message filename templates. The algorithm is md5, sha1, sha256 or sha512, optionally followed by an underscore and a truncation length (e.g. $msg.hash.sha256_16$ for the first 16 hex characters). It hex-encodes a digest of the message payload, computed lazily only when referenced.
Including it in store_received_file_to (or the global received-file template) dedups by name and content: a re-delivery of the same file resolves to the same path and overwrites, while changed content gets a new name. SHA-256 truncated to 16 chars is a good default - 64 bits is far more than enough collision resistance per filename.
Adds a unit test and a commented example in partnerships.xml.