SRE-4027 build: provide ZSCALER_CA_FILE arg for local docker builds

[soumagne]

Fix leap 15 docker build

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[soumagne]

As mentioned in the ticket, the way I would have to currently build images locally is by doing:

docker build --platform linux/amd64 . --file utils/docker/Dockerfile.leap.15 --build-arg DEPS_JOBS=6 --tag daos.leap.15 --build-arg ZSCALER_CA_FILE="$(cat ~/Downloads/ZscalerRootCerts/ZscalerRootCertificate-2048-SHA256.crt)" --build-arg DAOS_JAVA_BUILD=no --build-arg DAOS_DEPS_BUILD=yes

which means passing the zscaler root certificate directly. If instead someone tells me that this certificate can be hosted somewhere and we can instead use some URL directly, this might work too for me.

[github-actions]

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-4027

[JohnMalmberg]

Wrong approach.

[JohnMalmberg]

Do not make this type of change, it is not going to help you at all long term.

All access to the dockerfiles has to come from Artifactory.
If you are running your own docker host, in the lab you need to have it setup to use:
hub.daos.hpc.amslabs.hpecorp.net or it will not work reliably.

Docker is blocking most access from the corporate proxy from anonymous users. We have setup the CI docker builders to pull from alternative sources because of this and we use a trick to make it look like it is using images from the dockerhub.

Any use of HTTPS_PROXY to avoid using the internal Artifact Servers is a bug, that needs to be removed.

[soumagne]

Just to be clear, this is not something I want to use in CI builds. I only want to use that option for my local docker builds on my laptop.

[JohnMalmberg]

Then install the ZSCALER CA on laptop on what ever is running docker. It does not need to be in the dockerfile.

I have updated the ticket for this with how to access the internal docker hub.

[JohnMalmberg]

I understand more of what you are trying to do now, I think.

Now when you are doing this build on your laptop, are you using any lab resources at all?

If you are not using lab resources than you use:

DAOS_LAB_CA_FILE_URl to pass your zscaler cert contents.

And then the repo-helper scripts can see of DAOS_LAB_CA_FILE_URI starts with HTTP:// or HTTPS:// and then install the CA accordingly.

If you are using LAB resources like Artifactory, then you should be able to get what you need without the Zscaler CA.

[soumagne]

That's right, I'm not using any of the lab resources in this case. I think I can try doing what you just proposed with DAOS_LAB_CA_FILE_URI.
Zscaler intercepts all HTTP traffic on HPE laptops so afaik there is no other solution than installing the Zscaler certificate, this is further described in https://docs.docker.com/guides/zscaler/

[daosbuild3]

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19045/1/testReport/

[soumagne]

Wrong approach.

I'm trying to build docker images locally on my laptop, in this particular case a Mac, what is your solution then ? To me this is a quick solution for testing things without having to go through CI.

[JohnMalmberg]

Can you just reuse the DAOS_LAB_CA_FILE_URI parameter instead of adding a new one?

[JohnMalmberg]

I understand more of what you are trying to do now, I think.

Now when you are doing this build on your laptop, are you using any lab resources at all?

If you are not using lab resources than you use:

DAOS_LAB_CA_FILE_URl to pass your zscaler cert contents.

And then the repo-helper scripts can see of DAOS_LAB_CA_FILE_URI starts with HTTP:// or HTTPS:// and then install the CA accordingly.

If you are using LAB resources like Artifactory, then you should be able to get what you need without the Zscaler CA.

[JohnMalmberg]

Right now, I have determined that when in the lab environment the HTTP_PROXY/HTTPS_PROXY are not needed for the Dockerfiles.

And so far I have not found any CI tests that need them set either when in the lab.

I will change my PR a bit when I get a chance to remove the use of those experimental repos.

[soumagne]

Right now, I have determined that when in the lab environment the HTTP_PROXY/HTTPS_PROXY are not needed for the Dockerfiles.

And so far I have not found any CI tests that need them set either when in the lab.

I will change my PR a bit when I get a chance to remove the use of those experimental repos.

If I recall @knard38 was the one who added those HTTP_PROXY stuff so you might want to see with him :) (this is outside the scope of this PR)

[JohnMalmberg]

If I recall @knard38 was the one who added those HTTP_PROXY stuff so you might want to see with him :) (this is outside the scope of this PR)

Looks like a document is needed to explain how to properly use proxy and artifact servers, and why using proxy environment variables should be avoided when ever possible.

Currently we seem to be using HTTP_PROXY where we should not be, and this is causing some of the instability in our CI.