[anish-mudaraddi]

Description

The current build jobs are complex and there's a lot of repeated code. Currently, to add a new image, a whole new CI/CD pipeline: test -> lint -> build dev image -> build release image would need to be written - ultimately slowing down development.

This PR:
Makes it easier and standardise on the way we add and test new docker images. By setting up "generic" build scripts that will automatically detect images in this repo, build and push these images without explicitly needing to write another CI action.

1. separate image build jobs from testing and linting jobs to promote code reuse.

   • "Create Dev Image" action runs when a PR is created - to build images and upload them to harbor in the stfc-cloud-staging project.

   • "Create Release Image" action runs when a PR is merged into master

2. read and export a buildcache when building images so CI finishes quicker

3. Provide a way to locally build images using ./github/scripts/build_image.sh - see updated readme on how to use the new CI jobs

4. Setup a periodic PR to rebuild images and bump patch versions so we pick up the latest package dependency updates

To add new docker images - we now only need to write an action to test and lint source code which is much easier to write and maintain. This is not easy to make generic since we have a mix of Go and Python and slightly different project/package configurations.

Changelog

simplified the CI such that:

1. have 2 scripts to handle building and pushing images - used by github action and can be invoked locally for testing. Easy to test the scripts outside of github actions

2. created a shared action to automatically detect changes to a image subdirectory and builds and pushes a dev image when a PR is opened: e.g. 1.0.0dev.1293a1

3. created an action that builds and pushes a new release image and github tag when a PR is merged to master

4. modified existing CI that would regularly publish a PR bumping the patch version for rabbit-consumer docker image. Made this apply to all docker images so that we can force dependencies to updated and the docker build step

5. simplified individual image CI jobs to just testing and linting - not also building the images. This reduces duplicated code.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.40%. Comparing base (b27dfaa) to head (52307f8).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #182      +/-   ##
==========================================
+ Coverage   97.24%   98.40%   +1.15%     
==========================================
  Files          16       48      +32     
  Lines        1198     3069    +1871     
==========================================
+ Hits         1165     3020    +1855     
- Misses         33       49      +16     

Flag	Coverage Δ
rabbit_consumer	97.24% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[anish-mudaraddi]

changes since last review:

1. simplify the build image scripts to use the docker/build_push_action
2. don't check the version bump was done properly, just that it was changed - allows setting a unique tag that doesn't necessarily match the major.minor.bump - some of our images don't use this