Gradle S3 Plugin

Table of Contents

• Build Configuration
  • Groovy DSL (build.gradle)
  • Kotlin DSL (build.gradle.kts)
  • Version Catalog (gradle/libs.versions.toml)
• Configuration
  • Region
  • Authentication
  • Default Bucket
• Usage
  • Jobs
  • Task Types
• Uploads
  • Common Properties
  • Single File
  • Directory
  • File Collection
• Downloads
  • Common Properties
  • Single File
  • Recursive
• Progress Reporting
• Build the Project
• License

A Gradle plugin that uploads files to Amazon S3 and downloads files from Amazon S3.

This project is a fork of mygrocerydeals/gradle-s3-plugin. The original project is no longer maintained.

This fork works with Gradle 9 and later, and it is written in Java.

Build Configuration

Apply the plugin in your build script.

Groovy DSL (build.gradle)

plugins {
    id 'org.infosmid.plugins.s3' version '2.0.0'
}

Kotlin DSL (build.gradle.kts)

plugins {
    id("org.infosmid.plugins.s3") version "2.0.0"
}

Version Catalog (gradle/libs.versions.toml)

Declare the plugin in the catalog:

[versions]
s3-plugin = "2.0.0"

[plugins]
infosmid-s3 = { id = "org.infosmid.plugins.s3", version.ref = "s3-plugin" }

Then use the generated libs accessor:

plugins {
    alias(libs.plugins.infosmid.s3)
}

See the Gradle Plugin Portal page for other versions.

Configuration

All settings are in the s3 extension. Each task can override them.

Region

Set region to the AWS region of your bucket. Use this setting when the authentication profile has no region, or to replace the region of the profile.

s3 {
    region = 'us-east-1'
}

Authentication

By default, the plugin uses the AWS default credentials provider chain. One way to supply credentials is to set the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. See the AWS documentation for other methods.

You can also set a profile, or an access key and a secret, in the s3 extension. An access key and secret have priority over a profile.

s3 {
    profile = 'my-profile'
    awsAccessKeyId = '12345678'
    awsSecretAccessKey = 'my-secret'
}

Warning

Do not commit secrets to version control. Read them from a property or an environment variable instead.

Default Bucket

Set bucket to define a bucket for all tasks. A task can use a different bucket.

s3 {
    bucket = 'my.default.bucketname'
}

Usage

You can define jobs in a configuration block, or you can create tasks of a specific type.

Jobs

A job in s3Uploads or s3Downloads creates a task. The task name is the job name plus a suffix:

• The job jobName in s3Uploads creates the task jobNameUploadTask.

• The job dlJob in s3Downloads creates the task dlJobDownloadTask.

s3Uploads {
    jobName {
        key = 'target-filename'
        file = 'source-filename'
    }
}

s3Downloads {
    dlJob {
        keyPrefix = 'folder'
        destDir = 'targetDir'
    }
}

Task Types

import org.infosmid.gradle.s3.S3Upload
import org.infosmid.gradle.s3.S3Download

task myDownload(type: S3Download) {
    keyPrefix = 'folder'
    destDir = 'targetDir'
}

task myUpload(type: S3Upload) {
    key = 'target-filename'
    file = 'source-filename'
    compareContent = true
}

task myUploadFiles(type: S3Upload) {
    keyPrefix = 'target-folder'
    files = fileTree('my-folder').matching {
        include('**/*.zip')
    }
    compareContent = false
}

Uploads

An upload task has three modes:

• Single file upload

• Directory upload, including all subdirectories

• FileCollection upload

Common Properties

Property	Default	Description
bucket	s3.bucket	S3 bucket to use.
awsAccessKeyId	s3.awsAccessKeyId	AWS access key.
awsSecretAccessKey	s3.awsSecretAccessKey	AWS secret access key.
overwrite	false	If true, the task replaces an S3 object that already exists.
compareContent	false	If true, the task downloads the S3 object and compares it with the local file. The task uploads the file only when the content is different.
skipError	false	If true, the task does not fail when an object is missing.
batchSize	100	Number of files to upload in one batch. A larger batch can make the upload faster.

Single File

Property	Description
key	Key of the S3 object to create.
file	Path of the file to upload.

By default, the task does not replace an S3 object that already exists. Set overwrite to true to replace it.

Directory

Property	Description
keyPrefix	S3 prefix under which the task creates the uploaded objects.
sourceDir	Local directory to upload.

A directory upload always replaces existing objects under the prefix.

File Collection

Property	Description
keyPrefix	S3 prefix under which the task creates the uploaded objects.
files	A FileCollection. Create it with files() or fileTree().

Downloads

A download task has two modes:

• Single file download

• Recursive download

Common Properties

Property	Default	Description
bucket	s3.bucket	S3 bucket to use.
awsAccessKeyId	s3.awsAccessKeyId	AWS access key.
awsSecretAccessKey	s3.awsSecretAccessKey	AWS secret access key.

Single File

Property	Description
key	Key of the S3 object to download.
file	Local path where the task saves the file.

Recursive

Property	Description
keyPrefix	S3 prefix of the objects to download.
destDir	Local directory where the task saves the objects.

A recursive download keeps the full keyPrefix in the local path. For example, a bucket has these object keys:

top/foo/bar
top/README

This download:

s3Downloads {
    downloadRecursive {
        keyPrefix = 'top/foo/'
        destDir = 'local-dir'
    }
}

creates this local tree:

local-dir/
└── top
    └── foo
        └── bar

The task downloads only the objects under top/foo, but it keeps the full S3 path below destDir. The command aws s3 cp --recursive behaves in a different way: it removes the prefix.

To remove the prefix, use a Copy task after the download:

def localTree = 'path/to/some/location'

task downloadRecursive(type: org.infosmid.gradle.s3.S3Download) {
    bucket = 's3-bucket-name'
    keyPrefix = "${localTree}"
    destDir = "${buildDir}/download-root"
}

// Copy the downloaded tree to a new location and remove the keyPrefix.
task copyDownload(type: Copy, dependsOn: downloadRecursive) {
    from "${buildDir}/download-root/${localTree}"
    into "${buildDir}/pruned-tree"
}

Progress Reporting

A download task reports progress in percent at the Gradle INFO log level. Run Gradle with the -i option to see it.

Build the Project

Use Java 17 or later.

./gradlew build

This command compiles the plugin and runs the unit tests and the functional tests. The functional tests do not connect to AWS.

To build the documentation and the API docs, run:

./gradlew docs

License

Released under the MIT License.