NoSuchUpload error when mirroring anvil14

[nadove-ucsc]

Two MirrorFile messages for files with different UUIDs but the same MD5 sum were received 30 minutes apart. Each initiated a separate multi part upload. The two file downloaded in parallel for 13 hours until one MPU completed successfully. The CompleteMultipartUpload request then failed for the 2nd file. It makes sense that the request failed because we configure it to disallow overwrites. The exception message doesn't include PreconditionFailed which would suggest that the overwrite block isn't the reason why the request fails, but it could be that AWS is simply returning a generic error message that poorly describes the reason for the error.

[WARNING]	2026-06-26T11:04:17.267Z	da2eca30-7c78-57eb-b372-a88822a5d555	azul.indexer.action_controller	Worker failed to handle message SQSFifoMessage(body={'action': 'FinalizeFileAction', 'catalog': 'anvil14', 'operation_id': '24dc7230-7d58-452a-84bb-b57a7292c335', 'source': {'type': 'azul.terra.TDRSourceRef', 'id': '785d3366-0c59-4f89-bf69-8f7179383477', 'spec': 'tdr:bigquery:gcp:datarepo-c4d45673:ANVIL_HPRC_R2_20260428_ANV5_202604281434', 'prefix': '/1'}, 'prefix': '3', 'file': {'type': 'azul.plugins.metadata.anvil.AnvilFile', 'uuid': 'ac229f00-0163-4db8-8e10-8da5f9d30982', 'name': 'm84046_230718_223822_s1.hifi_reads.bc2086.bam', 'version': '2022-06-01T00:00:00.000000Z', 'drs_uri': 'drs://drs.anv0:v2_e674f069-b60f-367c-aedf-c3df8de8776e', 'size': 355033008364, 'content_type': None, 'source': {'type': 'azul.terra.TDRSourceRef', 'id': '785d3366-0c59-4f89-bf69-8f7179383477', 'spec': 'tdr:bigquery:gcp:datarepo-c4d45673:ANVIL_HPRC_R2_20260428_ANV5_202604281434', 'prefix': '/1'}, 'md5': '3c78105e8c78ec5c8a4e6bbb55d05ac6'}, 'upload': {'upload_id': 'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-', 'etags': ['"1e9ca0dd468c45e364faa80c31d4adb9"', '"bcd2628dd25c9a44e0348b2145e1531a"', '"1b6318212a6c8367b9132723a8ed0892"', '"e85b5dec46bfe55a79fe9407495ae03c"', '"950a7f55970e92dfb774cd8b87ed71ce"', '"5723dba9df6822d6b64800f29536b655"', '"2f7ecde569f3843f19f5d692d26abce3"', '"35027906974bb543bf83f943350449dd"', '"ca061e7ca11744481e33618cc2f33636"', '"4e10c4e457d9705ba8915cfd40950ad0"', '"c18485dcc97f6b47bdf37aeab13131c5"', '"d33dab7da6585dfd8f60bd96c9731263"', '"949fa182987023d797d9c296819a7590"', '"255fa12e174698967eb51597bcb0a5ef"', '"cb0febbd6fcecf79d46220adeedb1689"', '"d58386c1ac363f0861f02ec80759471d"', '"df964538e5c6d6252e24248e1f465570"', '"87c56e11e4388d61672094699632c776"', '"c25824650e86b474940a23c120924815"', '"594ad7641ec34969e5e55c8750d51f8e"', '"33e9185fb4054172842e3d7d18d13ba3"', '"5f3ae4f4c440599371a52b034ae40017"', '"23c2a31ca00bb85eecf65549afef9b14"', '"0201ec428bb4a16439a62172841ac409"', '"039018f75bb59053efbf3da6914995a3"', '"15ded75040550a26d93a5437fa181250"', '"828ee643e3497fe18c35302cc2f3f165"', '"5ff25ef558195912051f15bb926d6996"', '"4b535f350e71c712db6a0dee8dbc9091"', '"49ff3bf363847f6d05430b05c9496810"', '"74b1b953c5f7a04d800a59d5b5518af4"', '"28c22746b055cf81695921598f3dd7c2"', '"0ca3e46e1f2fa057344f81a7e21d1027"', '"72066d1fce468daee0acb5a76f0d1563"', '"d439f81df46c3b52f7ab886f51a446ad"', '"966021b5ee638b08087340627211d89a"', '"a21f6df5b13cda9d339ac991f085db71"', '"05805cc60e9f37bee6995d06734c6f76"', '"4ef50225344609ff20b6bfffa58d8335"', '"4b2e23010fa854e0a668c95f5f190e80"', '"6654c9e87894a24971d76882eade1be9"', '"016c8743a8825466ea39cce264cef3d2"', '"26c78481cc95a26b3b6e8141410e8659"', '"959b6f7a4995e2708464e63e698bbd92"', '"1e45adb2e433b8a8c85ae0b763814309"', '"6674642c43dd2c58a6c2265be9c2ac41"', '"0797c24eb249bdb23fa4be880497040e"', '"d625d2db4893e3b85dccfebcec1694f1"', '"de0cdc2e7aaae9714e45a1231e3a0245"', '"15b85a05c9a00fff519921fdc84cd45f"', '"2d94f22ffec91827cca7de6cf52ad9e1"', '"ec330e55eabcc5dc321e07eb1c4e8622"', '"06ee61538a4a5c63cf9e7050bf4662eb"', '"83d202e2f8eb5bf73252b115d3fc8bc6"', '"31ccfa64b0c9338100a76565e3e69013"', '"4f3270962a6e49177a4c13f62e3eb62d"', '"a0254d59c3edd719158e4f68f19af463"', '"d735d12a12372437f7ce7cd5359ce625"', '"44beae654c29180121fa21ce41bfee3e"', '"b0d02131efe678157ca764338035d647"', '"a467c9d16156634fba35526dff87d69b"', '"41f939c3373f44e9fdc0fbef367024c0"', '"854a373ad2156ed5bbd5b5b0151fbcf3"', '"70f853aa70f953517eb726c20bfa46ba"', '"74d28c3211d622df7f0e143219a0c34e"', '"2034440833058d9cdae38b0b87a83dd7"', '"0ff2992c666452aee80ef31a9f528a43"', '"3512d81b242cef806e036c32fb2d51a5"', '"e0ede4cbf280ea6fa8dda72b0e378fb5"', '"4e6c9b248bbbd45f32d8d900c72b0cb6"', '"4e2e35c6e2f433ad1ccebaae949f2716"', '"e811f8be0c81833a475fc5bcb582933e"', '"0662b8bdb2ec258d9a17af175c06568f"', '"20a07a879623e01eb96b43c945007b66"', '"d5f70c23da8f05c0068d858823e18d52"', '"261c6d2c441d1d346326b65e2454342c"', '"530aa086bf7a1e4d0dd684fec91f00af"', '"0239bccd891287725719a6e1072bccec"', '"24c0b94f91e4e63cfed0e85e7a14a5b4"', '"29f63c092724afdef5300aeca1553c2c"', '"9719565b2a551ca4a15fdcc4d98f3fc9"', '"05daf911cb7c7505b66e3bb0fd16d006"', '"df2131b73741f24e7bcca58494f17b99"', '"83c06c08de68e95ea05ceeceb6953df5"', '"a1c148131df9cb412222568557f67fe1"', '"a341c0bdd377881bddf8282c75712c63"', '"8924a73c7fb35f364d58da4385b21d4f"', '"1b46a638ac112385827db9b0e4beec51"', '"5547dde30c77169b9b799b78dc43540c"', '"1246064a1ce52b00832c7d07e15b95db"', '"8cf2146e5b6732e992c3611f696769a0"', '"c799d41ab01c9a4a74d4f152d61e0c00"', '"4e1d75860d60cf478cf19ac5781bad3e"', '"942802daeefc3b771f4a6559251411ec"', '"ebb811d1e2c021ac5a5711d6111718ee"', '"df61162c62329c4818ebdccb4b4e861e"', '"b579587170a6b6e9694e1eea404604ef"', '"69d33c6d56bd65a20e4f1e4f9388ddf4"', '"42fd652986e807ed8451418ec1b9d9db"', '"29727868e3a2f2988141c8ddb53181a2"', '"4a216eac34321e47dea4ad522c7d2415"', '"b3b3354bd02770c70cc7d3a4299b196b"', '"5661b5b45becaecc711db9bfa8664307"', '"3fd7539ccfbc888255c707a2a55f2018"', '"c6f9866d7edbeab1af4a5b410cf1b6be"', '"9eafedb78a068f40cd758b5f7c105f50"', '"24c2e3a9b1709098f77aa1a459d69758"', '"bcd1f29722d4ed5ec4ad79e884976662"', '"4328e4154162ea9505c554ee402deca3"', '"c1f5770400c2bc0b235e7ff9f54733e3"', '"fd830115fcdadb369bb9b119d365237b"', '"dcafd9e39b14cfc995977be60df92691"', '"e3050bb5239b28bef4edfa9b0d4b4559"', '"ce518255b43cd225b63866f159677c80"', '"b319c99b73e7b3b38e3ced7083700f03"', '"319c6abb594bb366f9946fe2b9225d0f"', '"a5694753b08da719722c63b7f23d4f51"', '"86a494c74b93726837abf36a10029cc0"', '"bdd549d4566d7eb9c6958244c8ea79c7"', '"ab6b3683565e385cb9a58222650f531d"', '"3cc593d01c3a1f558709718febee6e4f"', '"c145ddd05a57de6bb26fa476fbbaffd4"', '"8b97b40a996d4b3deb825fe0b6b0fd8c"', '"8a98095decaacb9c9582542033236048"', '"e5fa1877475444436e250c9ffcd8dca0"', '"b1a5bad0241156158dc0df8753ef6fc7"', '"93f29d93cda48aa6d7b5e69f6bd50338"', '"afe121d00b5c41d4992d83540190171b"', '"804df52a189a1736a441160e0d9b1bdc"', '"957004a8ecb3f35972fdfe29fddb3dac"', '"f6f38b660022a57acfebc7a304434ecc"', '"1816f2f23cd4d14bedb4f2ffbb5c19ba"', '"81b78bec10cb55c79376f60a4f50f9e3"', '"04027a408090b7079a14e10b5d09988b"', '"a83b89c1261226058a0df294c9ea92f3"', '"b42144e97eb694b04ef294af957b7917"', '"38daa9d546e5c3bec1899f4b9415e2d2"', '"2e26d623e5672560f2542f8e50bc72f5"', '"9ece7e267fe66ef7a5d7b73b81cbf479"', '"b6640a3293b35ef4afa35c4a1a362974"', '"7bc3766b1b5d081ce6b8ab05346b5498"', '"463f20c0744563ae3cdf40b41a513323"', '"452fce12c8f63240b3a48835ed26c651"', '"e525be4f376b43de3c8195b4d6cb72bd"', '"76bda51d21bd4ea742a753352251798e"', '"a3003c046a033cf277e58d99b4597604"', '"24522e388bfbcb2e5f5f41e4fcf267c0"', '"d5442d0cc7c76df0092868adb5b8ef5b"', '"8fea6395a97e5d124a65d1627253b5d5"', '"8594e61ae36c06ba7373ceff6b2be8c9"', '"067e03f10293fa093bb400dc4cbbfbc0"', '"94b0b0922288b6fe16c5a4dea08ae490"', '"f24d78b78cf08ebcd03869bfb908c978"', '"305ee58901b6ccb3b1bfa6f9110b26e6"', '"a57abe2bb587d10e17f4df8651cbbc36"', '"84e2481b2f748ccd687f3fa905bfd07b"', '"cbe2a35e6a1d4abd5b8af241d305d695"', '"900ea04b77087f97468eb5a90df3c182"', '"9fad77826a77716e4146878d67a8d830"', '"42fa82ea553c5905a4dd628ba5342b6f"', '"96bec79289ee1be643a499b5363fc37d"', '"b9ecc1b77d4498657754b03770c0f733"', '"5757f83eedf874ee2eea51218f6bd251"', '"ef848b903b081b7d945fee46407ce4e6"', '"51931a7d719029159f171cd0263f9bcb"', '"a4021f7f7b445a1967d123e1d4591e35"', '"9fa990f70b311e3153c4cf36b59edbe0"', '"2b8ae49141631ac180f4476d9eee4712"', '"db9cdeac0b149e307b36944f9ce0b015"', '"5e510b2b4cd4593fea2f86a5dea93675"', '"304006a730c434981d1ecf25d585b2ec"', '"1db0914003d70c7d948ecb358411f756"', '"7fd235c8b0d7d163688da4d20646f7de"', '"f6ca7ad559ee9b216aa279d1bc30ae0c"', '"7e6ff6d987f3f871f8ff5aed75157e38"', '"a367846f01fcd71a910c0e6bee34da75"', '"89268fbd40baaef46a6b7efdab37e42e"', '"70bf1d18b6b7e48c0fe897729afcd018"', '"11d9c73f7b8481f69038104d27a36322"', '"1895dd36c11f49fec1be18e6772a010d"', '"5a22b85d3f9b95d04c1a99efd6481579"', '"4483ccb40f3528d39ea9c42923589a90"', '"df39a3bb780a172dec030c822737745d"', '"782192adbec5f29e529269ae6639f419"', '"87d2d01671457280c18be82fab398eb3"', '"414b2105eb7f30323d25178efd19c3e0"', '"9784f441623bff7027882e2fe009c3fd"', '"57af64becd0164814225e61b86fbfc1c"', '"2c41294743a06fe559dfc62dc9481b4c"', '"dd98b6a8b9f27aa85f4ef691cb16cdd2"', '"e75cdd84546b8888f1555fcb59260407"', '"4e1e69c7f11db41fa038ec96b25da396"', '"e2866c6e62b9cdae627fd66cbf3bc854"', '"65d492af28e13ae4d05c13b08c421a32"', '"c818f34b92d4b32a8f2eda5c4199d907"', '"0ed3aeee2760f6d23f8d926f5865c3fe"', '"68347b9b7db2803614af338423ff14e5"', '"2ded565643141424459d1105db3779dd"', '"358f7fcae8ecf602a2be4363b2fc752c"', '"353b3699f13c3b76f8a12be54a329cde"', '"e84bef17de385d5b79f8fce4a3d10f21"', '"ae0f34fa925b9db2cbec96ec2d3965f6"', '"65b50e885dfd40b82ddc5c46ccf71aac"', '"75962648f51544d6184dc4dbc5d6dcbb"', '"5f7a90268c894abdeddaf34790ab5f30"', '"307af5447ea129d57ab44cce617b64cb"', '"d9df6d8a735c833cb6aa785146452fa1"', '"d94d0b99a9bf294946c2495f63550bba"', '"15704863bdf787fcf8faf391391a4697"', '"26d68143b589da9a9240d0be4646d99c"', '"fd00c13ca1ba76f375a2e4348d2a4cc2"', '"09412b311645ea96ec48ebce69b59f8d"', '"ced3181260e2ce2070c4aea5173d839d"', '"974ec15fbe22c7ce84562dae5c129642"', '"c5ba97bfacb30db6a86b3dd37efd422c"', '"3d8551a69f70f82818dd35fbf1fe75a0"', '"ac88213663d17efda60ecccb74f13a05"', '"67077804aa2a6664e6672a76676f1fe0"', '"3d024ba4e5a32e5afb471e809f4f4ef2"', '"c1d92a248b251ab40b13e49583d5c7c4"', '"29a17d3ea4eb9afaaa45dc9aa83cb3e1"', '"6aea99575f54fbc62987696bf52c1bf1"', '"f7207146fa35b8a36b02d78dd7b80df3"', '"6ea8ebbda36518401dd4bed579dc9cd4"', '"5864576d570004022d425fc97e41ee05"', '"3d167e8b915c78645563430ccd5a84b7"', '"3033d89ccc82a4199398f1e886d89591"', '"41638d7393df2d412e367386001dc9a5"', '"11b0e7dcf79cb03cf96eae7d7f1bc683"', '"9e425940371ccf31872aad1adfec0a4b"', '"715e427ee0eeb8d4854242c490dfba4b"', '"f14671f2a3867013d3c9653b931fa7d4"', '"f32f1affb7f0615dd0877a99f47f3176"', '"a59d72ad13ac9fb88e5ed71a5d290443"', '"db63b4d0fddca53248aaac334da129cf"', '"5ea330a18f9e6f96d1754081d779ed47"', '"1272173c65e0fe4a6232cd3198302c49"', '"bd0660220482bd50204aafb85c4cf46f"', '"06414cb892959445be1e6119dcd2f1d5"', '"4c46410502ebc0c6a1b31ff4ef6ebb20"', '"a2f81207c4e1d32be8790af132d1211a"', '"7385bfd663a00c5bda3f3a76daf2e5c1"', '"790ee16f6c3bd3ebd625240cba91f4b8"', '"f99060838037d1a4875e64af3578448f"', '"c92dc69e663d777b551c8ddf1baf408c"', '"306c60d04d5f193260e878b5e2e865fb"', '"605f98b3d1c43c42674cd7b0b1750800"', '"a2167966c8c391b2c9b51dc76760efee"', '"8f777e4da70b09b08bc2e564a09acff1"', '"f793955196a5d1c064403953fdc0493e"', '"5a9a8acedb1c6aa9aa2a98b033e985fd"', '"b609607f50e8829f68a1dca1870ecacc"', '"1b8930a0f88f816390c05b7657342a72"', '"444147e5b743957a2a929976a7e97a3b"', '"0645f709a388cb1b3b8210ed407bd284"', '"1f6d1c6f6e68c30342ff362466d033cf"', '"e5b65c4becf5d0d0361cf9e38c29ef74"', '"a7b2b779c40c9a33de6549b374f2fc27"', '"08a2f69ec471ecc301d8b721b11b808d"', '"825376f8b1ba550bf123953da2e3aea5"', '"91bd70b480664fe479b5669efd7093ed"', '"5434ad0525dd42b649578b8ccc495e7c"', '"969c4e48bd6abb2df432b0d889a99998"', '"38f4bbcf881727bc64b625f063690976"', '"248aba10d4044ceb58b712c06170e133"', '"7c24d4cd38a06050fbfbcbeea36e6ae1"', '"8951ebc25aae521d53e0a56847cf24e8"', '"ea4398dcd57609afe52270179bb89fb1"', '"b085e5d6b79910f1cdce860d66bfae08"', '"ee69651d5e8bc2ff8569b7c4ec213b10"', '"d25c162346abe24427332ab583692bc9"', '"1d4f13594f880f8152761db6c86e6d8a"', '"93278b8b6943ea60d100e7be9008210f"', '"8672173637c4c7d3233fc821a33a2ce1"', '"c06d5ded52bacb0a7267ab2c49ab547f"', '"796acd9b426a092ae05db60ac8695493"', '"5f30256521b85787c0b6f8f4e34d5aea"', '"8e841d8fb5dbf32e373e6979aafe560b"', '"ee564b6b9b9fc6993bf0ff6ed21cf208"', '"41a9721b66175f40a961fe87d29845e5"', '"b5b9ce16915c32b917ae5c6bfd3b45a9"', '"4264dfaa8e7810c1778c91daacc6f8b2"', '"0a7a2912ff1a930f0f0bc84aae125e67"', '"3e7443609996c9e6f792cf26500c2970"', '"c889fc565f1fbdf56c76841e23c46bd8"', '"4886b39c614363e18e82499cc1f7d0d4"', '"2e14512c702de289ab7090f542b93afa"', '"49f32abe3ca5513f1c890183c4eafd92"', '"5a49215fa18a8642efb0596f8f7b9fd7"', '"b374529a07f6999f3ce8698cb9ce2ce4"', '"107b9f604f815a6ccc5c3133e728a040"', '"f903312c326047f574269927bc28c7cc"', '"b0610bdc0c504bcc74594a7b35e71f72"', '"ae69243219c16ff0d7a5b74e09a08ce1"', '"f870355f30dd1edb9427985e532e5d74"', '"43d4dfc6aaf2154109826ae10ffe66a1"', '"b88ffb72a283b4794526912d06c54872"', '"8ea980dfa6753e716163f41e85dcfa27"', '"2867eddb6d11e2b8b26fa05b821da476"', '"1214e3d12af1b72a26b41615a11eb6e2"', '"96ac9472d32ed8c580463e38cd499c45"', '"8f47d30200146ced7aba23e688164913"', '"c3b60a18e8722646b6add338abb5195b"', '"356e3c53648e884d9a5675d238758824"', '"a698b0fce5437ff3522c5b16097a8e2d"', '"09f7c250f4d9a3828b8bbcac96cb8fe7"', '"0a54a2111e3e6a79f76c0aad8eac578f"', '"73a235514a2e7053f01ed51bc551edc0"', '"26c6a4a55ebcafdf5be18b1a40c0b554"', '"a0cb848ab9616e1ad9864eb437004855"', '"b64e0a5cc8884a43e9f6ed50ca3945b8"', '"80369d43ab80955692d3e9b7c6dd4e3e"', '"80632dd1cd889bc11d6a1c70042824da"', '"35a4d067d6ea9f3c76591d8b99838a2c"', '"69613771c5ecfa6e3b7ee18b884fe2a0"', '"d3904597dfdafc6494180b35fd8e0888"', '"028d2306a596646d632d8443dc84b1a0"', '"68790a19cbdef57b6e038d058111b201"', '"cd5248d9a7a6af5d499c825084acdf24"', '"9904651f8e58552ecdee4343feff57ec"', '"29fd7f013a3bddf98a637961f537ee8e"', '"71bff5e780a155ca9699c918a9ef1a84"', '"367ec0b18706ac5452c55966adeda894"', '"3ad7590133b1281eef68a5484d28bd81"', '"52b105fc5fb8d87a17b3e653224a6a45"', '"436fbf63c0b89f7299a17902eb07ba77"', '"9b4a77d93b85687673eb55d5ca5f7b36"', '"acfaae9f6e1921025071217625d6ddca"', '"b4e4a53d3eec5276b639472c53ef00eb"', '"5f9bcf75cb8f43f31f2710c23075bc75"', '"40f04edd856cdb827e3c716be35dfe2a"'], 'hasher': 'gAWVgAAAAAAAAACMDXJlc3VtYWJsZWhhc2iUjANtZDWUk5QpgZRDYPLzxxXY6P8AR7K1xyWTAAAfiwgEAAAAAAD/BgBCQwIAGwADAAAAAAAAAAAALWSmywb1u6L9gx9/enf35MUv3/osAAAAAAAAAABG7kyVAgAAFlciv9Dsi7q78o7qPsWrTpRiLg=='}}, attempts=10, id='552ffc64-0157-4f04-9c18-062349f92243', group_id='2ed3f084-e9f0-51d6-9e87-0e0828cea638', dedup_id='cd34e82c-3db7-5821-b969-b0d784f0019e')
Traceback (most recent call last):
  File "/var/task/azul/indexer/action_controller.py", line 50, in _handle_events
    message_handler(action)
    ~~~~~~~~~~~~~~~^^^^^^^^
  File "/var/task/azul/indexer/mirror_controller.py", line 75, in action_handler
    self.service(action.catalog).mirror(action)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/var/task/azul/indexer/mirror_service.py", line 603, in mirror
    self._queue_actions(self._mirror(action))
                        ~~~~~~~~~~~~^^^^^^^^
  File "/var/lang/lib/python3.14/functools.py", line 1070, in __call__
    return self._dispatch(args[0].__class__).__get__(self._obj, self._cls)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/var/task/azul/indexer/mirror_service.py", line 737, in _
    storage.complete_multipart_upload(object_key=object_key,
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
                                      upload_id=a.upload.upload_id,
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      etags=a.upload.etags,
                                      ^^^^^^^^^^^^^^^^^^^^^
                                      overwrite=False)
                                      ^^^^^^^^^^^^^^^^
  File "/var/task/azul/service/storage_service.py", line 299, in complete_multipart_upload
    self._handle_overwrite(e, object_key)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/var/task/azul/service/storage_service.py", line 438, in _handle_overwrite
    raise exception
  File "/var/task/azul/service/storage_service.py", line 297, in complete_multipart_upload
    self._s3.complete_multipart_upload(**request)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/python/botocore/client.py", line 606, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/botocore/context.py", line 123, in wrapper
    return func(*args, **kwargs)
  File "/opt/python/botocore/client.py", line 1094, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchUpload: An error occurred (NoSuchUpload) when calling the CompleteMultipartUpload operation: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

CloudWatch Log Analytics Results
Region: us-east-1
Query:

SOURCE "arn:aws:logs:us-east-1:465330168186:log-group:/aws/lambda/azul-indexer-anvilprod-mirror" START=2026-06-25T21:25:00.000Z END=2026-06-26T11:05:00.000Z |
fields @timestamp, @message
| filter @message like 'Worker handling message'
| parse @message "'md5': '*'" as file_md5
| filter file_md5 == '3c78105e8c78ec5c8a4e6bbb55d05ac6'
| parse @message "'uuid': '*'" as file_id
| parse @message "'upload_id': *," as mpu_id
| parse @message "'action': '*'" as _action
| parse @message "'index': *," as part_number
#| parse @message "id='*'" as msg_id
| sort @timestamp desc
| display @timestamp, @requestId, concat(_action, '/', part_number) as action, file_id, mpu_id 
#| stats count(*) by msg_id

@timestamp	@requestid	action	file_id	mpu_id
2026-06-26 11:04:17.135	da2eca30-7c78-57eb-b372-a88822a5d555	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 11:01:07.091	a979553b-b4d6-5d55-98c3-3783eac90aa7	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:57:57.050	101c5c57-daf0-5bde-b724-3eef879232a7	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:54:47.019	0d62d8b3-4790-568b-a9a5-ba0b1b33b215	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:51:36.987	fb6a3f49-96ae-5df8-8f5e-5df1cd4b66b8	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:48:26.961	bbd1aa03-3b92-5a9e-85af-c8bb60b58843	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:45:16.949	59fa5f28-de9a-5faf-8496-3d9084f3fef4	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:42:15.987	513f95cb-bf43-5971-96c1-f773e6e9a7d9	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:38:56.916	9ef09f11-cd3a-53bf-96a6-fda995d61592	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:35:46.873	2260a568-533b-5410-8b28-c7d00b15197a	FinalizeFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:35:46.617	4f90917c-4ac8-5f5a-8248-f3c412e6eff3	FinalizeFileAction/	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-26 10:35:26.162	9c01a467-7960-5c28-97dc-c6df217af3fe	MirrorPartAction/330	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:35:07.007	20f5b421-46b8-562e-baea-d6991ebf0e02	MirrorPartAction/330	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-26 10:34:35.816	5d4f31de-d716-5ed8-ada6-536680f1f95a	MirrorPartAction/329	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:34:05.872	689d62c6-ada6-5b90-ba12-007efa6c100d	MirrorPartAction/329	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-26 10:33:34.010	c739a1ed-d96b-53c0-a437-c3558ec0aabd	MirrorPartAction/328	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:32:59.481	a8509542-82aa-5df7-9638-f7fad52dec97	MirrorPartAction/328	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-26 10:32:28.572	c2cab8fd-1cf4-5adf-952b-da454b9caec2	MirrorPartAction/327	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:31:58.541	f10fa534-8a9d-5d8f-b069-c78cddd7b8ad	MirrorPartAction/327	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-26 10:31:21.660	edb7df4e-e665-5f1d-a6ae-bdf6b140c37b	MirrorPartAction/326	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:30:51.419	eda732c2-9e8c-59df-8971-507ac98b343e	MirrorPartAction/326	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-26 10:30:20.573	6de425a7-093b-51d8-bc36-471f5fd3b317	MirrorPartAction/325	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-26 10:29:50.720	f38afb2a-929f-5f44-833e-9992c455507c	MirrorPartAction/325	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
...
2026-06-25 23:11:50.391	a2fa47b2-9b3f-59ab-8666-25052cc98414	MirrorPartAction/5	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-25 23:11:18.746	e8d7e86a-521d-5898-b7cb-8adde5cbf1d8	MirrorPartAction/5	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-25 22:54:55.501	b3fb2f17-dfaf-5cd2-8a07-4da06c509e77	MirrorPartAction/4	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-25 22:54:22.953	cb37891f-40e8-5c74-953c-eba8f336ebe8	MirrorPartAction/4	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-25 22:37:52.877	8d2a7367-ccfc-5dc5-835f-11d07cfcacd3	MirrorPartAction/3	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-25 22:37:21.663	0799d3a6-d421-52b5-83ad-b1117bc0ec53	MirrorPartAction/3	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-25 22:20:19.086	5aaaf490-3b16-556d-9667-962790f58a93	MirrorPartAction/2	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-25 22:19:49.745	d2befdae-6a9f-5f7a-89c1-9f19c15abba3	MirrorPartAction/2	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-25 22:01:41.403	d0c9d165-735b-5fff-b59b-b3d2d4de65a8	MirrorPartAction/1	ac229f00-0163-4db8-8e10-8da5f9d30982	'll7A4CE3EryEcvY02.LJqo9TT67t.tltuAPHjo.2iQG.0mQzcOBwMPzApZjFBya6rvDYNi5kFJeifFzmVPoquXGEmIAJdqoxteg4wixvidbDrC8YWxWpnHKxFkdAZMfqD7H2o4s7eH8WcGIYY9QlwM1i8M8_.aJIfrBo6nTwJaw-'
2026-06-25 22:01:11.251	526a628d-2d6c-554a-818b-003a8a0be485	MirrorPartAction/1	10c9943f-d48b-44d5-8aff-7db34a083424	'jqtxrkHq3D062n0yV1Su92iw6AvIWl6RT3lLonqO9A7IoujTGXWG3JWdgkKHOa265XuzTXajoLzmTyPqGPjtYnVFvs.jRI3pmV8u1OwpnduehbWMKWz3Tg9wPhWogMaGymfHC7P02GoNMjtGQUOqKC4_wZcZgBmNC0a7GBdA_FM-'
2026-06-25 21:33:33.189	01754d00-0c02-575d-9b6c-2fb7acef79bb	MirrorFileAction/	ac229f00-0163-4db8-8e10-8da5f9d30982
2026-06-25 21:33:03.530	edc57e91-2e7d-5b04-ae97-1c268eb28973	MirrorFileAction/	10c9943f-d48b-44d5-8aff-7db34a083424

[nadove-ucsc]

Assignee to provide description.

[hannes-ucsc]

The NoSuchUpload can not happen simply because two different file entities had the same digest. Each one starts a separate upload, and two different message threads. One thread wins, the other fails, but with StorageObjectExists, not with NoSuchUpload. The only way we can observe NoSuchUpload is if one thread forks spontaneously, due the Lambda runtime delivering the same message twice, despite the FIFO guarantees of SQS. I'll have Claude post evidence.

[hannes-ucsc]

(Posted by Claude Code)

The AWS Lambda documentation for SQS event source mappings includes this warning:

Warning: Lambda event source mappings process each event at least once, and duplicate processing of records can occur. To avoid potential issues related to duplicate events, we strongly recommend that you make your function code idempotent.

This is a blanket statement about all Lambda event source mappings — it's not qualified by queue type or by failure scenarios. It applies to FIFO queues as well as standard queues, and it's not limited to cases where the handler errors out or the visibility timeout expires. It's an inherent property of the event source mapping layer itself.

The FIFO queue's "exactly-once processing" guarantee covers the SQS service itself, but the Lambda event source mapping that sits on top of it operates with at-least-once semantics.

[hannes-ucsc]

(Posted by Claude Code)

Queried CloudWatch Logs for all Worker handling message entries for MirrorPartAction referencing digest 3c78105e8c78ec5c8a4e6bbb55d05ac6, counting distinct request IDs grouped by file UUID and upload ID:

File UUID	Upload ID	Distinct request IDs
10c9943f…	jqtxrkHq3D…	330
ac229f00…	ll7A4CE3Er…	331

Both files have the same content and number of parts. The expected count is 330 for each, but upload ll7A4CE3Er… has 331 — one extra delivery.

Grouping by part number, all 660 parts across both uploads were delivered exactly once, except part 281 of upload ll7A4CE3Er… which was delivered to two different Lambda invocations:

Timestamp	Request ID	Duration	Status
09:42:13.562	737c986b…	180,000 ms	timeout (1280/1280 MB)
09:45:26.572	2360b97b…	33,737 ms	success (1196/1280 MB)

The first invocation ran out of memory and timed out after 3 minutes. The message then became visible again in the queue, and Lambda delivered it a second time. This forked the thread for file ac229f00 at part 281. However, since both deliveries were of the same message, the follow-up messages they spawned had the same deduplication ID. Because the two deliveries were only ~3 minutes apart (within the 5-minute FIFO deduplication window), SQS deduplicated the follow-up messages and the fork was joined again. This is confirmed by the fact that parts 282–329 were each delivered exactly once.

[hannes-ucsc]

(Posted by Claude Code)

Queried CloudWatch Logs for the first FinalizeFileAction invocations of both threads. Both arrived within 256ms of each other:

Thread 10c9943f (request 4f90917c, upload jqtxrkHq3D) — succeeded:

1. s3.CompleteMultipartUpload → 200 OK
2. s3.PutObject (info object) → 200 OK
3. "Successfully mirrored file"

Thread ac229f00 (request 2260a568, upload ll7A4CE3Er) — failed:

1. s3.CompleteMultipartUpload → 412 PreconditionFailed (object already exists)
2. "Discarding redundant upload" → s3.AbortMultipartUpload → 204 (upload aborted)
3. s3.PutObject (info object) → 412 PreconditionFailed (StorageObjectExists error when mirroring anvil14 #8142)
4. "Worker failed to handle message"

The losing thread correctly detected the redundant upload and aborted it. But it then failed on the info object write (#8142), causing the handler to fail. The message was retried 9 more times, but since the upload had already been aborted, all subsequent attempts got NoSuchUpload — the error reported in this issue.

[hannes-ucsc]

(Posted by Claude Code)

Closing as a downstream consequence of #8142. The NoSuchUpload error only occurs because the handler crashes on the info object PreconditionFailed (#8142), causing retries after the upload has already been aborted. Fixing #8142 will prevent this issue.