Tests for deleted/expired collection job creation

[tgeoghegan]

Adds test cases for trying to re-create collection jobs that have been either deleted explicitly by a DELETE or implicitly by their constituent data expiring. Along the way, we do some pretty significant refactoring of CollectionJobTest so that various test setup routines become methods on that type.

[tgeoghegan]

Draft because these tests are failing at the moment.

[tgeoghegan]

Now we have the tests we want so that we can assure that Janus behaves correctly. Next, we have to decide how we want Janus to behave. There's eight cases of interest:

• Leader selected, deleted job, re-create job with identical request
• Leader selected, deleted job, re-create job with same idempotency key but mutated request
• Leader selected, expired job, re-create job with identical request
• Leader selected, expired job, re-create job with same idempotency key but mutated request
• Time interval, deleted job, re-create job with identical request
• Time interval, deleted job, re-create job with same time interval but mutated request
• Time interval, expired job, re-create job with identical request
• Time interval, expired job, re-create job with same time interval but mutated request

I argue that:

Regardless of whether a job was deleted or expired, trying to re-create it with the same query (idempotency key or time interval, resp.) but varying other parameters (extensions, agg param) should fail with batchOverlap.

If a job was deleted and we try to re-create it with an identical request, we should serve HTTP 204 No Content, the same response we'd give if you tried to GET a deleted collection job. Note that's 204 No Content, not 404 Not Found! We are acknowledging that the collection job ever existed.

Finally, if a job expires and we try to re-create it with identical parameters, then we could serve HTTP 200 and the CollectionJobResp. It turns out that even if the data is expired, we still have a collection_jobs row in state Finished, with aggregate shares and a checksum and everything. But I think it would be better to serve 204 No Content, on the premise that the data has expired and thus is effectively deleted, especially since the garbage collector will eventually catch up and delete it for real.

To sum up, I am proposing:

• Leader selected, deleted job, re-create job with identical request -> 204 No Content
• Leader selected, deleted job, re-create job with same idempotency key but mutated request -> 400 Bad Request + batchOverlap problem document
• Leader selected, expired job, re-create job with identical request -> 204 No Content
• Leader selected, expired job, re-create job with same idempotency key but mutated request -> 400 Bad Request + batchOverlap problem document
• Time interval, deleted job, re-create job with identical request -> 204 No Content
• Time interval, deleted job, re-create job with same time interval but mutated request -> 400 Bad Request + batchOverlap problem document
• Time interval, expired job, re-create job with identical request -> 204 No Content
• Time interval, expired job, re-create job with same time interval but mutated request -> 400 Bad Request + batchOverlap problem document