diff --git a/.agents/skills/localize-rustfs-docs/references/de.md b/.agents/skills/localize-rustfs-docs/references/de.md index ebbf7e379..cd5992be9 100644 --- a/.agents/skills/localize-rustfs-docs/references/de.md +++ b/.agents/skills/localize-rustfs-docs/references/de.md @@ -46,6 +46,8 @@ Treat RustFS source code, APIs, and exact Console labels as authoritative for pr | rebalancing | Rebalancing / Neuverteilung | Preserve the RustFS operation name when it maps to a Console action. | | decommission | Außerbetriebnahme | Explain that data is drained before the storage pool is deactivated. | | replication | Replikation | Qualify as `Bucket-Replikation` or `Site-Replikation`. | +| on-demand migration | On-demand Migration | Keep the Console label. Explain once as bedarfsgesteuerte Migration (eingehender Abruf). Do not call it Replikation. | +| backfill | Backfill | Keep Console labels such as **Start backfill** and **Dry run**. Explain once as Hintergrundübernahme. | | versioning | Versionierung | Use for S3 Object versions, not source control. | | Object Lock | Object Lock / Objektsperre | Preserve exact API and Console labels. | | legal hold | Legal Hold | Preserve API/UI wording; explain the retention effect in German. | diff --git a/.agents/skills/localize-rustfs-docs/references/fr.md b/.agents/skills/localize-rustfs-docs/references/fr.md index c27b09d07..be63dc039 100644 --- a/.agents/skills/localize-rustfs-docs/references/fr.md +++ b/.agents/skills/localize-rustfs-docs/references/fr.md @@ -47,6 +47,8 @@ Treat RustFS source code, APIs, and exact Console labels as authoritative for pr | rebalancing | rééquilibrage | Preserve exact Console action names. | | decommission | mise hors service | Explain that data is drained before the pool is deactivated. | | replication | réplication | Qualify as `réplication de compartiment` or `réplication de site`. | +| on-demand migration | migration à la demande | Keep the Console label **On-demand migration**. Inbound pull on a read miss or a backfill. Do not call it réplication. | +| backfill | backfill | Keep Console labels such as **Start backfill** and **Dry run**. Explain once as remplissage en arrière-plan. | | versioning | gestion des versions | Use for S3 Object versions, not source control. | | Object Lock | verrouillage d’objet | Preserve exact API and Console labels. | | legal hold | conservation légale | Preserve API identifiers such as `ObjectLockLegalHold`. | diff --git a/.agents/skills/localize-rustfs-docs/references/ja.md b/.agents/skills/localize-rustfs-docs/references/ja.md index c8b375339..23bd22213 100644 --- a/.agents/skills/localize-rustfs-docs/references/ja.md +++ b/.agents/skills/localize-rustfs-docs/references/ja.md @@ -47,6 +47,8 @@ Treat RustFS source code, APIs, and exact Console labels as authoritative for pr | rebalancing | リバランス | Preserve exact Console action names. | | decommission | デコミッション / 使用停止 | Explain that data is drained before the pool is deactivated. | | replication | レプリケーション | Qualify as `バケットレプリケーション` or `サイトレプリケーション`. | +| on-demand migration | オンデマンド移行 | Keep the Console label **On-demand migration**. Inbound pull on a read miss or a backfill. Do not translate it as レプリケーション. | +| backfill | バックフィル | Keep Console labels such as **Start backfill** and **Dry run**. | | versioning | バージョニング | Use for S3 Object versions, not source control. | | Object Lock | オブジェクトロック | Preserve exact API and Console labels. | | legal hold | リーガルホールド | Preserve API identifiers such as `ObjectLockLegalHold`. | diff --git a/.agents/skills/localize-rustfs-docs/references/terminology.md b/.agents/skills/localize-rustfs-docs/references/terminology.md index 2a7374c75..5c42df69b 100644 --- a/.agents/skills/localize-rustfs-docs/references/terminology.md +++ b/.agents/skills/localize-rustfs-docs/references/terminology.md @@ -40,6 +40,8 @@ Treat these references as project decisions, not substitutes for current upstrea | rebalance | 数据再平衡 | Use “再平衡” for the operation name when context is already clear. | | decommission | 退役 | Use the established RustFS operation name “存储池退役”; explain that the pool is drained and deactivated rather than substituting a generic “下线”. | | replication | 复制 | Qualify as “站点复制” or “存储桶复制” according to the RustFS feature. | +| on-demand migration | 按需迁移 | Keep the Console label **On-demand migration**. Inbound pull on a local read miss or a backfill. Do not translate it as 复制. | +| backfill | 回填 | Keep Console labels such as **Start backfill**, **Background backfill**, and **Dry run**. | | versioning | 版本控制 | Use specifically for S3 object versioning. | | object lock | 对象锁定 | Do not conflate with distributed locks. | | legal hold | 依法保留 | Use the established Amazon S3 Chinese term; preserve API names such as `ObjectLockLegalHold`. | diff --git a/content/de/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png b/content/de/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png new file mode 100644 index 000000000..b14205dd7 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png b/content/de/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png new file mode 100644 index 000000000..dc246d6dd Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png b/content/de/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png new file mode 100644 index 000000000..1a9824956 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png b/content/de/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png new file mode 100644 index 000000000..cd34928bb Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png b/content/de/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png new file mode 100644 index 000000000..596b1ce4c Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png b/content/de/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png new file mode 100644 index 000000000..3b383986d Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png new file mode 100644 index 000000000..53d6ff190 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png new file mode 100644 index 000000000..847456845 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png new file mode 100644 index 000000000..47a093fc4 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png new file mode 100644 index 000000000..46428d357 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png new file mode 100644 index 000000000..5afe4166f Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png new file mode 100644 index 000000000..f835f4019 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png new file mode 100644 index 000000000..d50889e64 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png new file mode 100644 index 000000000..9ea4b18fe Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png new file mode 100644 index 000000000..eb0dd81d6 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png new file mode 100644 index 000000000..74c3048f7 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png differ diff --git a/content/de/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png b/content/de/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png new file mode 100644 index 000000000..09f4cacf3 Binary files /dev/null and b/content/de/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png differ diff --git a/content/de/administration/data/bucket/meta.json b/content/de/administration/data/bucket/meta.json index bcd7c6cc0..7afb7b386 100644 --- a/content/de/administration/data/bucket/meta.json +++ b/content/de/administration/data/bucket/meta.json @@ -5,6 +5,7 @@ "deletion", "quota", "replication", + "on-demand-migration", "policy" ] } diff --git a/content/de/administration/data/bucket/on-demand-migration.md b/content/de/administration/data/bucket/on-demand-migration.md new file mode 100644 index 000000000..ae90c94b5 --- /dev/null +++ b/content/de/administration/data/bucket/on-demand-migration.md @@ -0,0 +1,221 @@ +--- +title: "On-demand Migration aus Cloudflare R2" +description: "Hängen Sie einen Cloudflare-R2-Bucket als reine Lesequelle an und ziehen Sie Objekte bei einem Lese-Miss oder per Backfill nach RustFS." +--- + +Die On-demand Migration zieht Objekte aus einem externen Bucket in einen RustFS-Bucket, wenn ein Client einen fehlenden Schlüssel liest oder wenn Sie einen Backfill starten. **Dieses Dokument verwendet Cloudflare R2 als Beispiel** von Anfang bis Ende. Die folgenden Schritte folgen einer Console-v1.0.3-Durchführung: Ein R2-Bucket wurde an einen leeren lokalen Bucket gehängt, die Verbindung geprüft und danach drei Abrufe ausgeführt — ein Objekt mit `rc cp`, ein Präfix-Backfill und ein Backfill des gesamten Quell-Namespace mit Testlauf. + +RustFS liest die Quelle. Das Aktivierungsformular sagt, dass RustFS dort weder schreibt noch löscht. Die Migrationsseite sagt, dass RustFS Objekte nur beim Lesen oder beim Backfill zieht, dass neue `PUT`- und `DELETE`-Operationen in RustFS nicht zur Quelle synchronisiert werden und dass spätere Änderungen an der Quelle lokale Kopien nicht aktualisieren. + +Die Screenshots dieser Durchführung verwenden Platzhalter für die R2-Konto-ID und den Zugriffsschlüssel. Der Host in der Terminal-Eingabeaufforderung ist `localhost`. + +## Quell-Provider + +**Source connection** → **Provider** listet diese Quellen. Die Namen sind die Console-Beschriftungen. Google Cloud Storage erscheint zweimal: HMAC und Native. + +1. Amazon S3 +2. S3-compatible storage +3. MinIO +4. RustFS +5. Cloudflare R2 +6. Google Cloud Storage (HMAC) +7. Azure Blob Storage (Native) +8. Google Cloud Storage (Native) + +![Provider-Menü der RustFS-Console für die On-demand Migration mit Amazon S3, S3-compatible storage, MinIO, RustFS, Cloudflare R2, Google Cloud Storage (HMAC), Azure Blob Storage (Native) und Google Cloud Storage (Native)](./images/rustfs-console-on-demand-migration-source-providers.png) + +Eine Migration von einem anderen Provider in dieser Liste folgt demselben Ablauf: einen leeren lokalen Bucket erstellen, die On-demand Migration aktivieren, **Provider** wählen, **Test connection** auswählen, **Enable migration** auswählen und danach bei einem GET-Miss oder per Backfill ziehen. Nur Endpunkt, Zugangsdaten und Bucket-Felder unterscheiden sich je nach Provider. Die R2-Durchführung unten füllt diese Felder für Cloudflare R2. + +## On-demand Migration und Bucket-Replikation + +| Ziel | Verwenden | +| --- | --- | +| Objekte aus einem Provider der Liste oben nach RustFS ziehen, bei einem Lese-Miss oder per Backfill | On-demand Migration | +| Objektversionen aus einem RustFS-Bucket in einen anderen Bucket schieben | [Bucket-Replikation](./replication.md) | +| Standortkonfiguration und Identitätsdaten zwischen Bereitstellungen synchronisieren | [Site-Replikation](/operations/high-availability/site-replication) | + +Die On-demand Migration ist ein eingehender Abruf. Die [Bucket-Replikation](./replication.md) ist eine ausgehende Kopie von Objektversionen und setzt Versionierung auf beiden Buckets voraus. Nachdem ein Schlüssel lokal gespeichert ist, folgt die On-demand Migration der Quelle nicht weiter. + +:::warning[Schließen Sie keine Replikationsschleife] +Wählen Sie keinen Bucket als Quelle der On-demand Migration, der bereits Replikationsziel dieses lokalen Buckets ist. +::: + +## Bevor Sie beginnen + +Zugangsdaten und Endpunkt in diesem Abschnitt gelten für das Cloudflare-R2-Beispiel. Für einen anderen Provider tragen Sie Endpunkt, lesende Zugangsdaten und Bucket-Namen dieses Systems in dieselben Console-Felder ein. + +- Eine RustFS-Console mit **On-demand migration**. Diese Durchführung nutzte Console v1.0.3 auf Port `9001`. +- Ein Console-Konto, das die **Settings** des lokalen Buckets öffnen kann. +- Die RustFS-CLI [`rc`](/operations/rc) auf dem Host, der den lokalen Bucket liest. +- Ein Cloudflare-R2-API-Token, das Objekte im Quell-Bucket lesen kann. Begrenzen Sie das Token nach Möglichkeit auf diesen Bucket. Öffentlicher Zugriff auf den R2-Bucket gehört nicht zu diesem Ablauf. +- Der R2-S3-Endpunkt des Quell-Buckets, `https://.r2.cloudflarestorage.com`. Wenn der Bucket eine Jurisdiction nutzt, kopieren Sie den Endpunkt aus dem R2-Dashboard. + +Richten Sie `rc` auf die lokale RustFS-S3-API, nicht auf R2: + +```bash +rc alias set rustfs http://localhost:9000 --region us-east-1 --bucket-lookup path +``` + +Der lokale Bucket in dieser Durchführung heißt `migration`. Der R2-Bucket heißt `dl-rustfs`. Für `migration` war die Versionierung aktiviert und Object Lock deaktiviert. Object Lock war zum Aktivieren der On-demand Migration nicht nötig, und die Versionierung war keine Voraussetzung für das Aktivieren. Mit aktivierter Versionierung kann ein lokaler Löschmarker den Quellschlüssel verdecken. + +:::warning[Verwenden Sie ein lesendes Quell-Token] +Erstellen Sie ein R2-Token, das den Quell-Bucket lesen, dort aber nicht schreiben oder löschen kann. Das Formular sagt, dass RustFS das Geheimnis speichert und es nicht an die Console zurückgibt. +::: + +:::note[Das Aktivieren der Quelle kopiert keine Daten] +**Migration active** bedeutet, dass der Bucket an die Quelle gehängt ist. Die Laufzeitzähler bleiben bei null, bis ein Client einen fehlenden Schlüssel liest oder Sie einen Backfill starten. +::: + +## 1. Einen leeren lokalen Bucket erstellen + +Öffnen Sie in der Console **Browser** → **Buckets**, erstellen Sie einen Bucket namens `migration` und prüfen Sie, dass er keine Objekte enthält. Öffnen Sie **Settings** dieses Buckets, wenn Sie die Quelle anhängen. + +![RustFS-Console-Browser mit dem leeren Bucket migration vor der On-demand Migration](./images/rustfs-console-empty-bucket-before-on-demand-migration.png) + +## 2. On-demand Migration öffnen + +1. Öffnen Sie den Bucket `migration` und wählen Sie **Settings**. +2. Wählen Sie **Automation**. +3. Suchen Sie **On-demand migration**. +4. Wenn noch nichts konfiguriert ist, wählen Sie **Open migration settings** oder **+ Enable on-demand migration**. Wenn die Migration bereits aktiv ist, wählen Sie **Manage migration**. + +Die unkonfigurierte Seite in dieser Durchführung meldet, dass keine Migrationsquelle konfiguriert ist, und bietet **+ Enable on-demand migration**. + +![Seite für die On-demand Migration in der RustFS-Console, bevor eine Quelle konfiguriert ist, mit Enable on-demand migration](./images/rustfs-console-enable-on-demand-migration.png) + +Nachdem die Quelle aktiviert ist, zeigt **Automation** den Status **Migration active** und **Manage migration**. + +![Automation-Einstellungen des Buckets migration in der RustFS-Console mit aktiver On-demand Migration](./images/rustfs-console-bucket-automation-on-demand-migration.png) + +## 3. Den Bucket auf Cloudflare R2 richten + +Setzen Sie die Quelle in **Enable on-demand migration** wie folgt. Ersetzen Sie Konto-ID und Schlüssel durch Ihr eigenes R2-Token. Lassen Sie das Sitzungstoken leer, sofern das Token keines verlangt. Lassen Sie **Local prefix** leer, damit der gesamte Quell-Namespace sichtbar ist. Diese Durchführung speicherte kein zusätzliches Präfix (**All keys**, **No added prefix**). + +| Feld | Wert in dieser Durchführung | +| --- | --- | +| Source | Cloudflare R2 | +| Endpoint | `https://.r2.cloudflarestorage.com` | +| Region | `us-east-1` (das Formular akzeptiert auch `auto`) | +| Source Bucket | `dl-rustfs` | +| Addressing style | Automatic | +| Access Key | R2-Zugriffsschlüssel-ID | +| Secret Key | R2-Geheimnis | +| Session token | Leer | +| Local prefix | Leer | + +![Formular Enable on-demand migration für eine Cloudflare-R2-Quelle, Endpunkt und Zugriffsschlüssel durch Platzhalter ersetzt](./images/rustfs-console-on-demand-migration-r2-source-form.png) + +## 4. Die Verbindung testen und die Migration aktivieren + +Wählen Sie **Test connection**. Eine erfolgreiche Prüfung meldet **Connection test passed** und zeigt einen Beispiel-Objektschlüssel aus dem Quell-Bucket. Wählen Sie danach **Enable migration**. + +![Connection test passed im Dialog Enable on-demand migration, einschließlich eines Beispiel-Objektschlüssels](./images/rustfs-console-on-demand-migration-connection-test-passed.png) + +Dasselbe Ergebnis erscheint auch als Toast: + +![Toast Connection test passed im Dialog der On-demand Migration](./images/rustfs-console-on-demand-migration-connection-test-toast.png) + +Der Konfigurationsstatus wird **Migration active**. Bis etwas gelesen oder per Backfill gezogen wird, bleibt **Runtime status** bei null migrierten Objekten und null migrierten Daten. **Latest job** bleibt **No backfill yet**. + +![On-demand Migration für den Bucket migration aktiv, Laufzeitzähler noch bei null](./images/rustfs-console-on-demand-migration-active-zero-runtime.png) + +## 5. Ein Objekt mit rc cp abrufen + +Lesen Sie den lokalen Bucket mit `rc`. Senden Sie diese Anfrage nicht an den R2-Endpunkt. In dieser Durchführung war der lokale Bucket noch leer, und ein fehlendes Objekt wurde gezogen. Der erste Miss liefert das Objekt an den Client und speichert es lokal. Ein späteres Lesen desselben Schlüssels bedient RustFS. + +```bash +rc cp rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb /tmp/ +``` + +```text +rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb -> /tmp/rustfs-nightly-2026-11-02.deb (71.34 MiB) +``` + +Verwenden Sie einen anderen Schlüssel, wenn der Quell-Bucket dieses Objekt nicht enthält. Die Terminalsitzung zeigt dieses `rc cp` und spätere `rc ls`-Prüfungen des lokalen Nightly-Präfixes. Die Liste ist das lokale Präfix, nachdem Objekte vorhanden waren. Das einzelne `rc cp` oben ist der Abruf von 71,34 MiB. + +![Terminal mit rc cp, das während der On-demand Migration ein Objekt von 71,34 MiB über den lokalen Bucket migration zieht](./images/rustfs-rc-cp-on-demand-migration-single-object.png) + +Dasselbe Nightly-Präfix im R2-Bucket, wie es im Cloudflare-Dashboard erschien: + +![Cloudflare-R2-Objektliste des Nightly-Präfixes, das als Quelle der On-demand Migration diente](./images/rustfs-cloudflare-r2-nightly-objects-before-migration.png) + +Die Bucket-Größe in diesem Dashboard ist die Summe von Cloudflare für den Bucket. Die späteren RustFS-Zähler auf dieser Seite sind das, was der Migrationsjob gemeldet hat. + +Nach dem Abruf des einzelnen Objekts zeigt der Console-**Browser** dieses Objekt im lokalen Bucket: + +![RustFS-Console-Browser mit rustfs-nightly-2026-11-02.deb nach einem Lesevorgang der On-demand Migration](./images/rustfs-console-browser-single-object-on-demand-migration.png) + +## 6. Ein Präfix per Backfill übernehmen + +**Start backfill** durchsucht die Quelle und zieht Schlüssel, die lokal noch nicht vorhanden sind. Der Dialog in dieser Durchführung verwendete diese Einstellungen: + +| Option | Bedeutung | Wert in der Durchführung | +| --- | --- | --- | +| Source listing prefix | Nur dieses Präfix auflisten | `artifacts/rustfs/packages/nightly/` | +| Existing local objects | Verhalten, wenn der Schlüssel bereits lokal ist | **Always skip** | +| Dry run | Treffer auflisten, ohne Objekte einzureihen oder zu schreiben | Für die Probe ein, für den Abruf aus | + +**Dry run** listet passende Objekte auf, ohne ein Objekt einzureihen oder zu schreiben. Prüfen Sie das Präfix und schalten Sie **Dry run** aus, bevor Sie ziehen. + +![Dialog Start background backfill mit Nightly-Präfix, Always skip und aktiviertem Dry run](./images/rustfs-console-on-demand-migration-backfill-dry-run.png) + +Es folgt ein während der Durchführung aufgezeichnetes Präfix-Listing. Der erste Schlüssel war in der Laboraufzeichnung abgeschnitten. Der vollständige Schlüssel hat die Form `artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-29.deb`. Die Zeitstempel sind die Laboraufzeichnung, etwa UTC. Die vollständige Liste ist länger als dieser Auszug. + +```text +…ghtly-2026-10-29.deb +[2026-11-02 03:25:08] 70.36 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-30.deb +[2026-11-02 03:25:08] 70.65 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-31.deb +[2026-11-02 03:25:08] 71.15 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-01.deb +[2026-11-02 03:14:39] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb +[2026-11-02 03:25:08] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb +``` + +Während ein Backfill Objekte schreibt, zeigt die Console aufgelistete und gezogene Anzahlen. Solange der Quell-Scan noch läuft, erscheint kein Fertigstellungsprozentsatz. + +![Dialog Backfill in progress mit aufgelisteten Objekten, gezogenen Objekten und migrierten Daten](./images/rustfs-console-on-demand-migration-backfill-in-progress.png) + +## 7. Einen Testlauf ausführen und dann den Quell-Namespace übernehmen + +Lassen Sie **Source listing prefix** leer, um den konfigurierten Quell-Namespace zu durchsuchen. Führen Sie zuerst **Dry run** aus und prüfen Sie die aufgelistete Menge. Schalten Sie **Dry run** danach aus und starten Sie den Backfill. + +Ein Testlauf in dieser Durchführung endete mit aufgelisteten Objekten und ohne Abruf: + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 0 | 0 | 0 | 0 B | + +![Background backfill nach einem Testlauf abgeschlossen, 128 Objekte aufgelistet und keines gezogen](./images/rustfs-console-on-demand-migration-backfill-listed-only.png) + +Der anschließende vollständige Backfill mit leerem Präfix endete so: + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 127 | 1 | 0 | 8.0 GiB | + +![Background backfill für den Quell-Namespace abgeschlossen, 127 Objekte gezogen und keine fehlgeschlagenen Objekte](./images/rustfs-console-on-demand-migration-backfill-completed.png) + +## 8. Den Bucket prüfen + +Prüfen Sie auf der Seite der On-demand Migration: + +1. **Runtime status** — migrierte Objekte und migrierte Daten sind nicht mehr null, und es sind keine Abruffehler verzeichnet. +2. **Background backfill** — der letzte Job zeigt die Anzahlen für aufgelistet, gezogen, übersprungen und fehlgeschlagen. +3. **Browser** und `rc` — listen Sie das Präfix mit `rc ls` auf, wie in der Terminalsitzung oben, und lesen Sie einen lokalen Schlüssel erneut mit `rc cp`. + +In dieser Durchführung zeigte das Laufzeitfenster **Migrated data** 8,1 GiB und **Migrated objects** 128. Der letzte Backfill zeigte Listed 128, Pulled 127, Skipped 1, Failed objects 0 und Data 8,0 GiB. + +![Laufzeit der On-demand Migration und abgeschlossener Backfill nach dem Abruf aus Cloudflare R2](./images/rustfs-console-on-demand-migration-runtime-after-backfill.png) + +Die Bucket-Liste nach der Durchführung zeigt `migration` mit 128 Objekten und 8,1 GiB: + +![RustFS-Console-Bucket-Liste mit dem Bucket migration nach der On-demand Migration](./images/rustfs-console-bucket-list-after-on-demand-migration.png) + +:::warning[Die lokale Kopie folgt der Quelle nicht] +Eine spätere Änderung desselben Schlüssels in R2 wird nicht über das lokale Objekt gezogen. Ein lokales `PUT` dieses Schlüssels wird nicht zur Quelle zurückgeschrieben und bleibt vor dem Quellschlüssel. Die On-demand Migration ist keine fortlaufende Replikation von der Quelle. +::: + +## Nächste Schritte + +- [Bucket-Replikation konfigurieren](./replication.md), wenn Sie eine ausgehende Kopie brauchen +- [rc installieren und verwenden](/operations/rc) +- [Site-Replikation konfigurieren](/operations/high-availability/site-replication) für Standortkonfiguration und Identitätsdaten diff --git a/content/en/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png b/content/en/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png new file mode 100644 index 000000000..b14205dd7 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png b/content/en/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png new file mode 100644 index 000000000..dc246d6dd Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png b/content/en/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png new file mode 100644 index 000000000..1a9824956 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png b/content/en/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png new file mode 100644 index 000000000..cd34928bb Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png b/content/en/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png new file mode 100644 index 000000000..596b1ce4c Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png b/content/en/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png new file mode 100644 index 000000000..3b383986d Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png new file mode 100644 index 000000000..53d6ff190 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png new file mode 100644 index 000000000..847456845 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png new file mode 100644 index 000000000..47a093fc4 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png new file mode 100644 index 000000000..46428d357 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png new file mode 100644 index 000000000..5afe4166f Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png new file mode 100644 index 000000000..f835f4019 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png new file mode 100644 index 000000000..d50889e64 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png new file mode 100644 index 000000000..9ea4b18fe Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png new file mode 100644 index 000000000..eb0dd81d6 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png new file mode 100644 index 000000000..74c3048f7 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png differ diff --git a/content/en/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png b/content/en/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png new file mode 100644 index 000000000..09f4cacf3 Binary files /dev/null and b/content/en/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png differ diff --git a/content/en/administration/data/bucket/meta.json b/content/en/administration/data/bucket/meta.json index 39583c1f8..a56087f79 100644 --- a/content/en/administration/data/bucket/meta.json +++ b/content/en/administration/data/bucket/meta.json @@ -5,6 +5,7 @@ "deletion", "quota", "replication", + "on-demand-migration", "policy" ] } \ No newline at end of file diff --git a/content/en/administration/data/bucket/on-demand-migration.md b/content/en/administration/data/bucket/on-demand-migration.md new file mode 100644 index 000000000..f75d2bd85 --- /dev/null +++ b/content/en/administration/data/bucket/on-demand-migration.md @@ -0,0 +1,221 @@ +--- +title: "On-demand Migration from Cloudflare R2" +description: "Attach a Cloudflare R2 bucket as a read-only source and pull objects into RustFS on a read miss or with a backfill." +--- + +On-demand migration pulls objects from an external bucket into a RustFS bucket when a client reads a missing key, or when you start a backfill. **This document uses Cloudflare R2 as the example** end to end. The steps below follow a Console v1.0.3 walkthrough that attached an R2 bucket to an empty local bucket, tested the connection, and then ran three pulls: one object with `rc cp`, a prefix backfill, and a full-namespace backfill with a dry run. + +RustFS reads the source. The enable form states that RustFS never writes or deletes objects there. The migration page states that RustFS pulls objects only when they are read or backfilled, that new `PUT` and `DELETE` operations in RustFS are not synchronized to the source, and that later source changes do not update local copies. + +Screenshots in this walkthrough use placeholders for the R2 account id and access key. The terminal prompt host is `localhost`. + +## Source providers + +**Source connection** → **Provider** lists these sources. The names below are the Console labels. Google Cloud Storage is two entries, HMAC and Native. + +1. Amazon S3 +2. S3-compatible storage +3. MinIO +4. RustFS +5. Cloudflare R2 +6. Google Cloud Storage (HMAC) +7. Azure Blob Storage (Native) +8. Google Cloud Storage (Native) + +![RustFS Console Provider menu for on-demand migration, listing Amazon S3, S3-compatible storage, MinIO, RustFS, Cloudflare R2, Google Cloud Storage (HMAC), Azure Blob Storage (Native), and Google Cloud Storage (Native)](./images/rustfs-console-on-demand-migration-source-providers.png) + +A migration from any other provider in that list uses the same flow: create an empty local bucket, enable on-demand migration, choose **Provider**, select **Test connection**, select **Enable migration**, then pull on a GET miss or with a backfill. Only the endpoint, credential, and bucket fields differ by provider. The R2 walkthrough below fills those fields for Cloudflare R2. + +## On-demand migration and bucket replication + +| Goal | Use | +| --- | --- | +| Pull objects into RustFS from a provider in the list above, on a read miss or by backfill | On-demand migration | +| Push object versions from a RustFS bucket to another bucket | [Bucket replication](./replication.md) | +| Synchronize site configuration and identity data across deployments | [Site replication](/operations/high-availability/site-replication) | + +On-demand migration is an inbound pull. [Bucket replication](./replication.md) is an outbound copy of object versions and requires versioning on both buckets. After a key is stored locally, on-demand migration does not keep following the source. + +:::warning[Do not close a replication loop] +Do not select a bucket that is already a replication target of this local bucket as the on-demand migration source. +::: + +## Before you begin + +The credential and endpoint here are for the Cloudflare R2 example. For another provider, enter that system's endpoint, read credential, and bucket name in the same Console fields. + +- A RustFS Console that includes **On-demand migration**. This walkthrough used Console v1.0.3, on port `9001`. +- A Console account that can open the local bucket's **Settings**. +- The RustFS CLI, [`rc`](/operations/rc), on the host that will read the local bucket. +- A Cloudflare R2 API token that can read objects in the source bucket. Prefer a token scoped to that bucket. Public access on the R2 bucket is not part of this flow. +- The R2 S3 endpoint for the source bucket, `https://.r2.cloudflarestorage.com`. If the bucket uses a jurisdiction, copy the endpoint from the R2 dashboard. + +Point `rc` at the local RustFS S3 API, not at R2: + +```bash +rc alias set rustfs http://localhost:9000 --region us-east-1 --bucket-lookup path +``` + +The local bucket in this walkthrough is `migration`. The R2 bucket is `dl-rustfs`. Versioning was enabled on `migration`, and Object Lock was disabled. The walkthrough did not need Object Lock to enable on-demand migration, and it did not treat versioning as a requirement for enabling it. With versioning enabled, a local delete marker can hide the source key. + +:::warning[Use a read-only source token] +Create an R2 token that can read the source bucket and cannot write or delete there. The form states that RustFS stores the secret and does not return it to the Console. +::: + +:::note[Enabling the source does not copy data] +**Migration active** means the bucket is attached to the source. Runtime counters stay at zero until a client reads a missing key or you start a backfill. +::: + +## 1. Create an empty local bucket + +In the Console, open **Browser** → **Buckets**, create a bucket named `migration`, and confirm that it has no objects. Open **Settings** from that bucket when you are ready to attach the source. + +![RustFS Console Browser showing the empty migration bucket before on-demand migration](./images/rustfs-console-empty-bucket-before-on-demand-migration.png) + +## 2. Open on-demand migration + +1. Open bucket `migration` and select **Settings**. +2. Select **Automation**. +3. Find **On-demand migration**. +4. When it is not configured, select **Open migration settings** or **+ Enable on-demand migration**. When it is already active, select **Manage migration**. + +The unconfigured page in this walkthrough says that no migration source is configured and offers **+ Enable on-demand migration**. + +![RustFS Console on-demand migration page before a source is configured, with Enable on-demand migration](./images/rustfs-console-enable-on-demand-migration.png) + +After the source is enabled, **Automation** shows **Migration active** and **Manage migration**. + +![RustFS Console bucket Automation settings with on-demand migration active for the migration bucket](./images/rustfs-console-bucket-automation-on-demand-migration.png) + +## 3. Point the bucket at Cloudflare R2 + +In **Enable on-demand migration**, set the source as follows. Replace the account id and keys with your own R2 token. Leave the session token empty unless the token requires one. Leave **Local prefix** empty to expose the whole source namespace. This walkthrough saved no extra prefix (**All keys**, **No added prefix**). + +| Field | Value in this walkthrough | +| --- | --- | +| Source | Cloudflare R2 | +| Endpoint | `https://.r2.cloudflarestorage.com` | +| Region | `us-east-1` (the form also accepts `auto`) | +| Source Bucket | `dl-rustfs` | +| Addressing style | Automatic | +| Access Key | R2 access key id | +| Secret Key | R2 secret | +| Session token | Empty | +| Local prefix | Empty | + +![Enable on-demand migration form for a Cloudflare R2 source, with the endpoint and access key replaced by placeholders](./images/rustfs-console-on-demand-migration-r2-source-form.png) + +## 4. Test the connection and enable migration + +Select **Test connection**. A successful check reports **Connection test passed** and shows a sample object key from the source bucket. Then select **Enable migration**. + +![Connection test passed inside the enable on-demand migration dialog, including a sample object key](./images/rustfs-console-on-demand-migration-connection-test-passed.png) + +The same result also appears as a toast: + +![Connection test passed toast on the on-demand migration dialog](./images/rustfs-console-on-demand-migration-connection-test-toast.png) + +The configuration status becomes **Migration active**. Until something is read or backfilled, **Runtime status** stays at zero migrated objects and zero migrated data. **Latest job** stays **No backfill yet**. + +![On-demand migration active for bucket migration, with runtime counters still at zero](./images/rustfs-console-on-demand-migration-active-zero-runtime.png) + +## 5. Pull one object with rc cp + +Read the local bucket with `rc`. Do not send this request to the R2 endpoint. In this walkthrough the local bucket was still empty, and one missing object was pulled. The first miss returns the object to the client and stores it locally. A later read of the same key is served from RustFS. + +```bash +rc cp rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb /tmp/ +``` + +```text +rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb -> /tmp/rustfs-nightly-2026-11-02.deb (71.34 MiB) +``` + +Use another key if the source bucket does not contain this object. The terminal session shows that `rc cp` and later `rc ls` checks of the local nightly prefix. The listing is the local prefix after objects were present. The single `rc cp` above is the 71.34 MiB pull. + +![Terminal showing rc cp pulling one 71.34 MiB object through the local migration bucket during on-demand migration](./images/rustfs-rc-cp-on-demand-migration-single-object.png) + +The same nightly prefix on the R2 bucket, as it appeared in the Cloudflare dashboard: + +![Cloudflare R2 object list for the nightly prefix used as the on-demand migration source](./images/rustfs-cloudflare-r2-nightly-objects-before-migration.png) + +The bucket-size figure on that dashboard is Cloudflare's total for the bucket. The RustFS counters later on this page are what the migration job reported. + +After the single-object pull, the Console **Browser** shows that object in the local bucket: + +![RustFS Console Browser showing rustfs-nightly-2026-11-02.deb after an on-demand migration read](./images/rustfs-console-browser-single-object-on-demand-migration.png) + +## 6. Backfill a prefix + +**Start backfill** scans the source and pulls keys that are not already local. The dialog in this walkthrough used these settings: + +| Option | Meaning | Walkthrough value | +| --- | --- | --- | +| Source listing prefix | List only this prefix | `artifacts/rustfs/packages/nightly/` | +| Existing local objects | What to do when the key is already local | **Always skip** | +| Dry run | List matches without queueing or writing objects | On for the rehearsal, then off for the pull | + +**Dry run** lists matching objects without queueing or writing any object. Confirm the prefix, then turn **Dry run** off before you pull. + +![Start background backfill dialog with a nightly prefix, Always skip, and Dry run enabled](./images/rustfs-console-on-demand-migration-backfill-dry-run.png) + +A prefix listing recorded during the walkthrough follows. The first key was truncated in the lab record. The full key has the form `artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-29.deb`. Timestamps are the lab record, about UTC. The full list is longer than this excerpt. + +```text +…ghtly-2026-10-29.deb +[2026-11-02 03:25:08] 70.36 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-30.deb +[2026-11-02 03:25:08] 70.65 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-31.deb +[2026-11-02 03:25:08] 71.15 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-01.deb +[2026-11-02 03:14:39] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb +[2026-11-02 03:25:08] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb +``` + +While a backfill is writing objects, the Console shows listed and pulled counts. It does not show a completion percentage while the source scan is still running. + +![Backfill in progress dialog with objects listed, objects pulled, and migrated data](./images/rustfs-console-on-demand-migration-backfill-in-progress.png) + +## 7. Run a dry run, then backfill the source namespace + +Leave **Source listing prefix** empty to scan the configured source namespace. Run **Dry run** first and confirm the listed set. Then turn **Dry run** off and start the backfill. + +A dry run in this walkthrough finished with objects listed and nothing pulled: + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 0 | 0 | 0 | 0 B | + +![Background backfill completed after a dry run, with 128 objects listed and none pulled](./images/rustfs-console-on-demand-migration-backfill-listed-only.png) + +The full backfill that followed, with an empty prefix, finished as follows: + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 127 | 1 | 0 | 8.0 GiB | + +![Background backfill completed for the source namespace, with 127 objects pulled and no failed objects](./images/rustfs-console-on-demand-migration-backfill-completed.png) + +## 8. Verify the bucket + +On the on-demand migration page, check: + +1. **Runtime status** — migrated objects and migrated data are no longer zero, and no pull failures are recorded. +2. **Background backfill** — the latest job shows the listed, pulled, skipped, and failed counts. +3. **Browser** and `rc` — list the prefix with `rc ls`, as in the terminal session above, and read a local key again with `rc cp`. + +In this walkthrough the runtime panel showed **Migrated data** 8.1 GiB and **Migrated objects** 128. The latest backfill showed Listed 128, Pulled 127, Skipped 1, Failed objects 0, and Data 8.0 GiB. + +![On-demand migration runtime and completed backfill after the Cloudflare R2 pull](./images/rustfs-console-on-demand-migration-runtime-after-backfill.png) + +The bucket list after the walkthrough shows `migration` with 128 objects and 8.1 GiB: + +![RustFS Console bucket list with the migration bucket after on-demand migration](./images/rustfs-console-bucket-list-after-on-demand-migration.png) + +:::warning[The local copy does not track the source] +A later change to the same key on R2 is not pulled over the local object. A local `PUT` of that key is not written back to the source, and it stays in front of the source key. On-demand migration is not continuous replication from the source. +::: + +## Next steps + +- [Configure bucket replication](./replication.md) when you need an outbound copy +- [Install and use rc](/operations/rc) +- [Configure site replication](/operations/high-availability/site-replication) for site configuration and identity data diff --git a/content/fr/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png b/content/fr/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png new file mode 100644 index 000000000..b14205dd7 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png b/content/fr/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png new file mode 100644 index 000000000..dc246d6dd Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png b/content/fr/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png new file mode 100644 index 000000000..1a9824956 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png b/content/fr/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png new file mode 100644 index 000000000..cd34928bb Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png b/content/fr/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png new file mode 100644 index 000000000..596b1ce4c Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png b/content/fr/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png new file mode 100644 index 000000000..3b383986d Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png new file mode 100644 index 000000000..53d6ff190 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png new file mode 100644 index 000000000..847456845 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png new file mode 100644 index 000000000..47a093fc4 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png new file mode 100644 index 000000000..46428d357 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png new file mode 100644 index 000000000..5afe4166f Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png new file mode 100644 index 000000000..f835f4019 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png new file mode 100644 index 000000000..d50889e64 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png new file mode 100644 index 000000000..9ea4b18fe Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png new file mode 100644 index 000000000..eb0dd81d6 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png new file mode 100644 index 000000000..74c3048f7 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png differ diff --git a/content/fr/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png b/content/fr/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png new file mode 100644 index 000000000..09f4cacf3 Binary files /dev/null and b/content/fr/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png differ diff --git a/content/fr/administration/data/bucket/meta.json b/content/fr/administration/data/bucket/meta.json index 48d5e82b2..de0857eeb 100644 --- a/content/fr/administration/data/bucket/meta.json +++ b/content/fr/administration/data/bucket/meta.json @@ -5,6 +5,7 @@ "deletion", "quota", "replication", + "on-demand-migration", "policy" ] } \ No newline at end of file diff --git a/content/fr/administration/data/bucket/on-demand-migration.md b/content/fr/administration/data/bucket/on-demand-migration.md new file mode 100644 index 000000000..215bae616 --- /dev/null +++ b/content/fr/administration/data/bucket/on-demand-migration.md @@ -0,0 +1,221 @@ +--- +title: "Migration à la demande depuis Cloudflare R2" +description: "Attachez un compartiment Cloudflare R2 en lecture seule et importez les objets dans RustFS lors d’une lecture manquante ou par un backfill." +--- + +La migration à la demande importe des objets d’un compartiment externe vers un compartiment RustFS lorsqu’un client lit une clé absente, ou lorsque vous lancez un backfill. **Ce document utilise Cloudflare R2 comme exemple** de bout en bout. Les étapes suivent un parcours sur la Console v1.0.3 : un compartiment R2 a été attaché à un compartiment local vide, la connexion a été testée, puis trois imports ont été exécutés — un objet avec `rc cp`, un backfill de préfixe, et un backfill de tout le namespace source avec un dry run. + +RustFS lit la source. Le formulaire d’activation indique que RustFS n’y écrit ni n’y supprime jamais d’objets. La page de migration indique que RustFS n’importe des objets que lorsqu’ils sont lus ou remplis par un backfill, que les nouvelles opérations `PUT` et `DELETE` dans RustFS ne sont pas synchronisées vers la source, et que les modifications ultérieures de la source ne mettent pas à jour les copies locales. + +Les captures de ce parcours utilisent des marqueurs pour l’identifiant de compte R2 et la clé d’accès. L’hôte de l’invite du terminal est `localhost`. + +## Fournisseurs source + +**Source connection** → **Provider** liste ces sources. Les noms sont les libellés exacts de la Console. Google Cloud Storage apparaît deux fois : HMAC et Native. + +1. Amazon S3 +2. S3-compatible storage +3. MinIO +4. RustFS +5. Cloudflare R2 +6. Google Cloud Storage (HMAC) +7. Azure Blob Storage (Native) +8. Google Cloud Storage (Native) + +![Menu Provider de la Console RustFS pour la migration à la demande, avec Amazon S3, S3-compatible storage, MinIO, RustFS, Cloudflare R2, Google Cloud Storage (HMAC), Azure Blob Storage (Native) et Google Cloud Storage (Native)](./images/rustfs-console-on-demand-migration-source-providers.png) + +Une migration depuis un autre fournisseur de cette liste suit le même flux : créer un compartiment local vide, activer la migration à la demande, choisir **Provider**, sélectionner **Test connection**, sélectionner **Enable migration**, puis importer lors d’un GET manquant ou par un backfill. Seuls les champs de point de terminaison, d’identifiants et de compartiment changent selon le fournisseur. Le parcours R2 ci-dessous remplit ces champs pour Cloudflare R2. + +## Migration à la demande et réplication de compartiment + +| Objectif | Utiliser | +| --- | --- | +| Importer des objets dans RustFS depuis un fournisseur de la liste ci-dessus, lors d’une lecture manquante ou par backfill | Migration à la demande | +| Pousser des versions d’objets d’un compartiment RustFS vers un autre compartiment | [Réplication de compartiment](./replication.md) | +| Synchroniser la configuration de site et les données d’identité entre des déploiements | [Réplication de site](/operations/high-availability/site-replication) | + +La migration à la demande est un import entrant. La [réplication de compartiment](./replication.md) est une copie sortante des versions d’objets et exige la gestion des versions sur les deux compartiments. Une fois une clé stockée localement, la migration à la demande ne suit plus la source. + +:::warning[Ne fermez pas une boucle de réplication] +Ne choisissez pas comme source de la migration à la demande un compartiment qui est déjà la cible de réplication de ce compartiment local. +::: + +## Avant de commencer + +Les identifiants et le point de terminaison de cette section concernent l’exemple Cloudflare R2. Pour un autre fournisseur, saisissez le point de terminaison, l’identifiant de lecture et le nom de compartiment de ce système dans les mêmes champs de la Console. + +- Une Console RustFS qui contient **On-demand migration**. Ce parcours utilisait la Console v1.0.3, sur le port `9001`. +- Un compte de Console autorisé à ouvrir **Settings** du compartiment local. +- La CLI RustFS [`rc`](/operations/rc) sur l’hôte qui lit le compartiment local. +- Un jeton d’API Cloudflare R2 capable de lire les objets du compartiment source. Limitez de préférence le jeton à ce compartiment. L’accès public du compartiment R2 ne fait pas partie de ce parcours. +- Le point de terminaison S3 R2 du compartiment source, `https://.r2.cloudflarestorage.com`. Si le compartiment utilise une juridiction, copiez le point de terminaison depuis le tableau de bord R2. + +Pointez `rc` vers l’API S3 RustFS locale, pas vers R2 : + +```bash +rc alias set rustfs http://localhost:9000 --region us-east-1 --bucket-lookup path +``` + +Le compartiment local de ce parcours est `migration`. Le compartiment R2 est `dl-rustfs`. La gestion des versions était activée sur `migration`, et le verrouillage d’objet était désactivé. Le verrouillage d’objet n’était pas nécessaire pour activer la migration à la demande, et la gestion des versions n’en était pas un prérequis. Lorsque la gestion des versions est activée, un marqueur de suppression local peut masquer la clé source. + +:::warning[Utilisez un jeton source en lecture seule] +Créez un jeton R2 qui peut lire le compartiment source et ne peut ni y écrire ni y supprimer. Le formulaire indique que RustFS conserve le secret et ne le renvoie pas à la Console. +::: + +:::note[Activer la source ne copie pas les données] +**Migration active** signifie que le compartiment est attaché à la source. Les compteurs d’exécution restent à zéro tant qu’un client ne lit pas une clé absente ou que vous ne lancez pas un backfill. +::: + +## 1. Créer un compartiment local vide + +Dans la Console, ouvrez **Browser** → **Buckets**, créez un compartiment nommé `migration` et vérifiez qu’il ne contient aucun objet. Ouvrez **Settings** de ce compartiment lorsque vous êtes prêt à attacher la source. + +![Navigateur de la Console RustFS affichant le compartiment migration vide avant la migration à la demande](./images/rustfs-console-empty-bucket-before-on-demand-migration.png) + +## 2. Ouvrir la migration à la demande + +1. Ouvrez le compartiment `migration` et sélectionnez **Settings**. +2. Sélectionnez **Automation**. +3. Repérez **On-demand migration**. +4. S’il n’est pas configuré, sélectionnez **Open migration settings** ou **+ Enable on-demand migration**. S’il est déjà actif, sélectionnez **Manage migration**. + +Dans ce parcours, la page non configurée indique qu’aucune source de migration n’est configurée et propose **+ Enable on-demand migration**. + +![Page de migration à la demande de la Console RustFS avant la configuration d’une source, avec Enable on-demand migration](./images/rustfs-console-enable-on-demand-migration.png) + +Une fois la source activée, **Automation** affiche **Migration active** et **Manage migration**. + +![Paramètres Automation du compartiment migration dans la Console RustFS, avec la migration à la demande active](./images/rustfs-console-bucket-automation-on-demand-migration.png) + +## 3. Pointer le compartiment vers Cloudflare R2 + +Dans **Enable on-demand migration**, renseignez la source comme suit. Remplacez l’identifiant de compte et les clés par votre propre jeton R2. Laissez le jeton de session vide, sauf si le jeton l’exige. Laissez **Local prefix** vide pour exposer tout le namespace source. Ce parcours n’a enregistré aucun préfixe supplémentaire (**All keys**, **No added prefix**). + +| Champ | Valeur dans ce parcours | +| --- | --- | +| Source | Cloudflare R2 | +| Endpoint | `https://.r2.cloudflarestorage.com` | +| Region | `us-east-1` (le formulaire accepte aussi `auto`) | +| Source Bucket | `dl-rustfs` | +| Addressing style | Automatic | +| Access Key | ID de clé d’accès R2 | +| Secret Key | Secret R2 | +| Session token | Vide | +| Local prefix | Vide | + +![Formulaire Enable on-demand migration pour une source Cloudflare R2, le point de terminaison et la clé d’accès étant remplacés par des marqueurs](./images/rustfs-console-on-demand-migration-r2-source-form.png) + +## 4. Tester la connexion et activer la migration + +Sélectionnez **Test connection**. Un contrôle réussi affiche **Connection test passed** et une clé d’objet d’exemple issue du compartiment source. Sélectionnez ensuite **Enable migration**. + +![Connection test passed dans la boîte de dialogue Enable on-demand migration, avec une clé d’objet d’exemple](./images/rustfs-console-on-demand-migration-connection-test-passed.png) + +Le même résultat apparaît aussi sous forme de toast : + +![Toast Connection test passed sur la boîte de dialogue de migration à la demande](./images/rustfs-console-on-demand-migration-connection-test-toast.png) + +L’état de configuration devient **Migration active**. Tant qu’aucune lecture ni aucun backfill n’a lieu, **Runtime status** reste à zéro objet migré et zéro donnée migrée. **Latest job** reste **No backfill yet**. + +![Migration à la demande active pour le compartiment migration, les compteurs d’exécution étant encore à zéro](./images/rustfs-console-on-demand-migration-active-zero-runtime.png) + +## 5. Récupérer un objet avec rc cp + +Lisez le compartiment local avec `rc`. N’envoyez pas cette requête au point de terminaison R2. Dans ce parcours, le compartiment local était encore vide et un objet absent a été importé. Le premier échec de lecture renvoie l’objet au client et le stocke localement. Une lecture ultérieure de la même clé est servie par RustFS. + +```bash +rc cp rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb /tmp/ +``` + +```text +rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb -> /tmp/rustfs-nightly-2026-11-02.deb (71.34 MiB) +``` + +Utilisez une autre clé si le compartiment source ne contient pas cet objet. La session de terminal montre ce `rc cp` et les contrôles `rc ls` ultérieurs du préfixe nightly local. La liste correspond au préfixe local une fois les objets présents. Le `rc cp` ci-dessus est l’import de 71.34 MiB. + +![Terminal montrant rc cp important un objet de 71.34 MiB via le compartiment local migration pendant la migration à la demande](./images/rustfs-rc-cp-on-demand-migration-single-object.png) + +Le même préfixe nightly sur le compartiment R2, tel qu’il apparaissait dans le tableau de bord Cloudflare : + +![Liste d’objets Cloudflare R2 du préfixe nightly utilisé comme source de la migration à la demande](./images/rustfs-cloudflare-r2-nightly-objects-before-migration.png) + +Le chiffre de taille de compartiment sur ce tableau de bord est le total Cloudflare du compartiment. Les compteurs RustFS plus loin sur cette page sont ce que le travail de migration a indiqué. + +Après l’import de l’objet unique, le **Browser** de la Console affiche cet objet dans le compartiment local : + +![Browser de la Console RustFS affichant rustfs-nightly-2026-11-02.deb après une lecture de migration à la demande](./images/rustfs-console-browser-single-object-on-demand-migration.png) + +## 6. Lancer un backfill sur un préfixe + +**Start backfill** parcourt la source et importe les clés qui ne sont pas déjà locales. La boîte de dialogue de ce parcours utilisait ces réglages : + +| Option | Signification | Valeur du parcours | +| --- | --- | --- | +| Source listing prefix | Ne lister que ce préfixe | `artifacts/rustfs/packages/nightly/` | +| Existing local objects | Que faire si la clé est déjà locale | **Always skip** | +| Dry run | Lister les correspondances sans mettre en file ni écrire d’objets | Activé pour la répétition, puis désactivé pour l’import | + +**Dry run** liste les objets correspondants sans mettre en file ni écrire aucun objet. Vérifiez le préfixe, puis désactivez **Dry run** avant d’importer. + +![Boîte de dialogue Start background backfill avec un préfixe nightly, Always skip et Dry run activé](./images/rustfs-console-on-demand-migration-backfill-dry-run.png) + +Voici un extrait de liste de préfixe enregistré pendant le parcours. La première clé était tronquée dans les notes de laboratoire. La clé complète a la forme `artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-29.deb`. Les horodatages sont ceux des notes, environ UTC. La liste complète est plus longue que cet extrait. + +```text +…ghtly-2026-10-29.deb +[2026-11-02 03:25:08] 70.36 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-30.deb +[2026-11-02 03:25:08] 70.65 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-31.deb +[2026-11-02 03:25:08] 71.15 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-01.deb +[2026-11-02 03:14:39] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb +[2026-11-02 03:25:08] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb +``` + +Pendant qu’un backfill écrit des objets, la Console affiche les nombres listés et importés. Elle n’affiche pas de pourcentage d’avancement tant que le parcours de la source est en cours. + +![Boîte de dialogue Backfill in progress avec les objets listés, les objets importés et les données migrées](./images/rustfs-console-on-demand-migration-backfill-in-progress.png) + +## 7. Lancer un dry run, puis le backfill du namespace source + +Laissez **Source listing prefix** vide pour parcourir le namespace source configuré. Lancez d’abord **Dry run** et vérifiez l’ensemble listé. Désactivez ensuite **Dry run** et démarrez le backfill. + +Un dry run de ce parcours s’est terminé avec des objets listés et aucun import : + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 0 | 0 | 0 | 0 B | + +![Background backfill terminé après un dry run, avec 128 objets listés et aucun import](./images/rustfs-console-on-demand-migration-backfill-listed-only.png) + +Le backfill complet qui a suivi, avec un préfixe vide, s’est terminé ainsi : + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 127 | 1 | 0 | 8.0 GiB | + +![Background backfill terminé pour le namespace source, avec 127 objets importés et aucun objet en échec](./images/rustfs-console-on-demand-migration-backfill-completed.png) + +## 8. Vérifier le compartiment + +Sur la page de migration à la demande, vérifiez : + +1. **Runtime status** — les objets migrés et les données migrées ne sont plus à zéro, et aucun échec d’import n’est enregistré. +2. **Background backfill** — le dernier travail affiche les nombres listés, importés, ignorés et en échec. +3. **Browser** et `rc` — listez le préfixe avec `rc ls`, comme dans la session de terminal ci-dessus, puis relisez une clé locale avec `rc cp`. + +Dans ce parcours, le panneau d’exécution affichait **Migrated data** 8.1 GiB et **Migrated objects** 128. Le dernier backfill affichait Listed 128, Pulled 127, Skipped 1, Failed objects 0 et Data 8.0 GiB. + +![Exécution de la migration à la demande et backfill terminé après l’import depuis Cloudflare R2](./images/rustfs-console-on-demand-migration-runtime-after-backfill.png) + +La liste des compartiments après le parcours montre `migration` avec 128 objets et 8.1 GiB : + +![Liste des compartiments de la Console RustFS avec le compartiment migration après la migration à la demande](./images/rustfs-console-bucket-list-after-on-demand-migration.png) + +:::warning[La copie locale ne suit pas la source] +Une modification ultérieure de la même clé sur R2 n’est pas importée par-dessus l’objet local. Un `PUT` local de cette clé n’est pas réécrit vers la source et reste devant la clé source. La migration à la demande n’est pas une réplication continue depuis la source. +::: + +## Étapes suivantes + +- [Configurer la réplication de compartiment](./replication.md) lorsque vous avez besoin d’une copie sortante +- [Installer et utiliser rc](/operations/rc) +- [Configurer la réplication de site](/operations/high-availability/site-replication) pour la configuration de site et les données d’identité diff --git a/content/ja/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png b/content/ja/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png new file mode 100644 index 000000000..b14205dd7 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png b/content/ja/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png new file mode 100644 index 000000000..dc246d6dd Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png b/content/ja/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png new file mode 100644 index 000000000..1a9824956 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png b/content/ja/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png new file mode 100644 index 000000000..cd34928bb Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png b/content/ja/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png new file mode 100644 index 000000000..596b1ce4c Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png b/content/ja/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png new file mode 100644 index 000000000..3b383986d Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png new file mode 100644 index 000000000..53d6ff190 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png new file mode 100644 index 000000000..847456845 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png new file mode 100644 index 000000000..47a093fc4 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png new file mode 100644 index 000000000..46428d357 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png new file mode 100644 index 000000000..5afe4166f Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png new file mode 100644 index 000000000..f835f4019 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png new file mode 100644 index 000000000..d50889e64 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png new file mode 100644 index 000000000..9ea4b18fe Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png new file mode 100644 index 000000000..eb0dd81d6 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png new file mode 100644 index 000000000..74c3048f7 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png differ diff --git a/content/ja/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png b/content/ja/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png new file mode 100644 index 000000000..09f4cacf3 Binary files /dev/null and b/content/ja/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png differ diff --git a/content/ja/administration/data/bucket/meta.json b/content/ja/administration/data/bucket/meta.json index 7cc23abfc..4a318bf58 100644 --- a/content/ja/administration/data/bucket/meta.json +++ b/content/ja/administration/data/bucket/meta.json @@ -5,6 +5,7 @@ "deletion", "quota", "replication", + "on-demand-migration", "policy" ] } \ No newline at end of file diff --git a/content/ja/administration/data/bucket/on-demand-migration.md b/content/ja/administration/data/bucket/on-demand-migration.md new file mode 100644 index 000000000..650b0fca8 --- /dev/null +++ b/content/ja/administration/data/bucket/on-demand-migration.md @@ -0,0 +1,221 @@ +--- +title: "Cloudflare R2 からのオンデマンド移行" +description: "Cloudflare R2 バケットを読み取り専用のソースとして接続し、読み取りミスまたはバックフィルでオブジェクトを RustFS に取り込みます。" +--- + +オンデマンド移行は、クライアントが存在しないキーを読み取るとき、またはバックフィルを開始したときに、外部バケットのオブジェクトを RustFS バケットへ取り込みます。**この文書は Cloudflare R2 を最初から最後までの例として使います。** 以降の手順は Console v1.0.3 で行った作業です。空のローカルバケットへ R2 バケットを接続し、接続を確認したあと、3 種類の取り込みを実行しました。`rc cp` による単一オブジェクト、プレフィックスのバックフィル、そしてドライランを含むソース名前空間全体のバックフィルです。 + +RustFS はソースを読み取ります。有効化フォームには、RustFS がソースへ書き込まず、削除もしないと書かれています。移行ページには、読み取りまたはバックフィルのときにだけオブジェクトを取り込むこと、RustFS 上の新しい `PUT` と `DELETE` はソースへ同期されないこと、ソース側のその後の変更はローカルコピーを更新しないことが書かれています。 + +この手順のスクリーンショットでは、R2 のアカウント ID とアクセスキーをプレースホルダーに置き換えています。端末プロンプトのホストは `localhost` です。 + +## ソースプロバイダー + +**Source connection** → **Provider** には次のソースが表示されます。名称は Console のラベルそのものです。Google Cloud Storage は HMAC と Native の 2 項目です。 + +1. Amazon S3 +2. S3-compatible storage +3. MinIO +4. RustFS +5. Cloudflare R2 +6. Google Cloud Storage (HMAC) +7. Azure Blob Storage (Native) +8. Google Cloud Storage (Native) + +![オンデマンド移行の RustFS Console Provider メニュー。Amazon S3、S3-compatible storage、MinIO、RustFS、Cloudflare R2、Google Cloud Storage (HMAC)、Azure Blob Storage (Native)、Google Cloud Storage (Native) を表示](./images/rustfs-console-on-demand-migration-source-providers.png) + +この一覧の他のプロバイダーからの移行も、同じ流れです。空のローカルバケットを作成し、オンデマンド移行を有効にし、**Provider** を選び、**Test connection** を選び、**Enable migration** を選び、その後 GET ミスまたはバックフィルで取り込みます。プロバイダーごとに変わるのは、エンドポイント、認証情報、バケットのフィールドだけです。以下の R2 手順では、これらのフィールドを Cloudflare R2 用に入力します。 + +## オンデマンド移行とバケットレプリケーション + +| 目的 | 使う機能 | +| --- | --- | +| 上の一覧のプロバイダーから、読み取りミスまたはバックフィルでオブジェクトを RustFS へ取り込む | オンデマンド移行 | +| RustFS バケットのオブジェクトバージョンを別のバケットへ送る | [バケットレプリケーション](./replication.md) | +| デプロイ間でサイト設定と ID データを同期する | [サイトレプリケーション](/operations/high-availability/site-replication) | + +オンデマンド移行は受信方向の取り込みです。[バケットレプリケーション](./replication.md)はオブジェクトバージョンの送信方向のコピーであり、両方のバケットでバージョニングが必要です。キーがローカルに保存されたあと、オンデマンド移行はソースの変更を追い続けません。 + +:::warning[レプリケーションの循環を作らない] +このローカルバケットのレプリケーションターゲットになっているバケットを、オンデマンド移行のソースに選ばないでください。 +::: + +## 開始する前に + +この節の認証情報とエンドポイントは、Cloudflare R2 の例です。他のプロバイダーでは、同じ Console フィールドに、そのシステムのエンドポイント、読み取り用の認証情報、バケット名を入力します。 + +- **On-demand migration** がある RustFS Console。この手順ではポート `9001` の Console v1.0.3 を使いました。 +- ローカルバケットの **Settings** を開ける Console アカウント。 +- ローカルバケットを読み取るホスト上の RustFS CLI [`rc`](/operations/rc)。 +- ソースバケットのオブジェクトを読める Cloudflare R2 API トークン。トークンは可能ならそのバケットに限定します。R2 バケットのパブリックアクセスはこの手順では使いません。 +- ソースバケットの R2 S3 エンドポイント `https://.r2.cloudflarestorage.com`。バケットが管轄区域にある場合は、R2 ダッシュボードに表示されるエンドポイントをコピーします。 + +`rc` は R2 ではなく、ローカル RustFS の S3 API を向けます。 + +```bash +rc alias set rustfs http://localhost:9000 --region us-east-1 --bucket-lookup path +``` + +この手順のローカルバケットは `migration`、R2 バケットは `dl-rustfs` です。`migration` ではバージョニングが有効、オブジェクトロックは無効でした。オンデマンド移行の有効化にオブジェクトロックは不要で、バージョニングも有効化の前提ではありませんでした。バージョニングが有効なとき、ローカルの削除マーカーでソースキーを隠せます。 + +:::warning[読み取り専用のソーストークンを使う] +ソースバケットを読めて、そこへ書き込みや削除ができない R2 トークンを作ります。フォームには、RustFS がシークレットを保存し、Console へは返さないと書かれています。 +::: + +:::note[ソースを有効にしてもデータはコピーされない] +**Migration active** は、バケットがソースに接続された状態です。クライアントが欠落キーを読むか、バックフィルを開始するまで、ランタイムのカウンタは 0 のままです。 +::: + +## 1. 空のローカルバケットを作成する + +Console で **Browser** → **Buckets** を開き、`migration` というバケットを作成して、オブジェクトがないことを確認します。ソースを接続する準備ができたら、そのバケットの **Settings** を開きます。 + +![オンデマンド移行の前に空になっている migration バケットを示す RustFS Console の Browser](./images/rustfs-console-empty-bucket-before-on-demand-migration.png) + +## 2. オンデマンド移行を開く + +1. バケット `migration` を開き、**Settings** を選びます。 +2. **Automation** を選びます。 +3. **On-demand migration** を見つけます。 +4. 未設定なら **Open migration settings** または **+ Enable on-demand migration** を選びます。すでに有効なら **Manage migration** を選びます。 + +この手順の未設定ページには、移行ソースが未設定であることと、**+ Enable on-demand migration** が表示されます。 + +![ソース設定前の RustFS Console オンデマンド移行ページ。Enable on-demand migration がある](./images/rustfs-console-enable-on-demand-migration.png) + +ソースを有効にすると、**Automation** に **Migration active** と **Manage migration** が表示されます。 + +![オンデマンド移行が有効な migration バケットの RustFS Console Automation 設定](./images/rustfs-console-bucket-automation-on-demand-migration.png) + +## 3. バケットを Cloudflare R2 に向ける + +**Enable on-demand migration** でソースを次のように設定します。アカウント ID とキーは自分の R2 トークンに置き換えます。トークンがセッショントークンを要求しない場合、セッショントークンは空にします。ソース名前空間全体を対象にするには **Local prefix** を空にします。この手順では追加のプレフィックスを保存していません(**All keys**、**No added prefix**)。 + +| フィールド | この手順での値 | +| --- | --- | +| Source | Cloudflare R2 | +| Endpoint | `https://.r2.cloudflarestorage.com` | +| Region | `us-east-1`(フォームは `auto` も受け付けます) | +| Source Bucket | `dl-rustfs` | +| Addressing style | Automatic | +| Access Key | R2 のアクセスキー ID | +| Secret Key | R2 のシークレット | +| Session token | 空 | +| Local prefix | 空 | + +![Cloudflare R2 ソース向けの Enable on-demand migration フォーム。エンドポイントとアクセスキーはプレースホルダー](./images/rustfs-console-on-demand-migration-r2-source-form.png) + +## 4. 接続をテストして移行を有効にする + +**Test connection** を選びます。成功すると **Connection test passed** と、ソースバケットのサンプルオブジェクトキーが表示されます。その後 **Enable migration** を選びます。 + +![Enable on-demand migration ダイアログ内の Connection test passed。サンプルオブジェクトキーを含む](./images/rustfs-console-on-demand-migration-connection-test-passed.png) + +同じ結果はトーストでも表示されます。 + +![オンデマンド移行ダイアログ上の Connection test passed トースト](./images/rustfs-console-on-demand-migration-connection-test-toast.png) + +設定状態は **Migration active** になります。読み取りかバックフィルが起きるまで、**Runtime status** の移行済みオブジェクトと移行済みデータは 0 のままです。**Latest job** は **No backfill yet** のままです。 + +![migration バケットでオンデマンド移行が有効、ランタイムカウンタはまだ 0](./images/rustfs-console-on-demand-migration-active-zero-runtime.png) + +## 5. rc cp でオブジェクトを 1 つ取得する + +`rc` でローカルバケットを読みます。このリクエストを R2 エンドポイントへ送らないでください。この手順ではローカルバケットがまだ空で、欠落していたオブジェクトを 1 つ取り込みました。最初のミスはオブジェクトをクライアントへ返し、ローカルへ保存します。同じキーの次の読み取りは RustFS が返します。 + +```bash +rc cp rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb /tmp/ +``` + +```text +rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb -> /tmp/rustfs-nightly-2026-11-02.deb (71.34 MiB) +``` + +ソースバケットにこのオブジェクトがない場合は、別のキーを使います。端末セッションには、この `rc cp` と、その後のローカル nightly プレフィックスに対する `rc ls` が写っています。一覧はオブジェクトがローカルに存在したあとのプレフィックスです。上の 1 回の `rc cp` が 71.34 MiB の取り込みです。 + +![オンデマンド移行中に、ローカルの migration バケット経由で rc cp が 71.34 MiB のオブジェクトを 1 つ取得している端末](./images/rustfs-rc-cp-on-demand-migration-single-object.png) + +Cloudflare ダッシュボードに表示されていた、ソース側の同じ nightly プレフィックスです。 + +![オンデマンド移行のソースに使った nightly プレフィックスの Cloudflare R2 オブジェクト一覧](./images/rustfs-cloudflare-r2-nightly-objects-before-migration.png) + +そのダッシュボードのバケットサイズは、Cloudflare が示すバケット全体の合計です。このページ後半の RustFS カウンタは、移行ジョブが報告した値です。 + +単一オブジェクトの取り込み後、Console の **Browser** にローカルバケットのそのオブジェクトが表示されます。 + +![オンデマンド移行の読み取り後、RustFS Console の Browser に表示された rustfs-nightly-2026-11-02.deb](./images/rustfs-console-browser-single-object-on-demand-migration.png) + +## 6. プレフィックスをバックフィルする + +**Start backfill** はソースを走査し、まだローカルにないキーを取り込みます。この手順のダイアログでは次の設定を使いました。 + +| オプション | 意味 | この手順での値 | +| --- | --- | --- | +| Source listing prefix | このプレフィックスだけを一覧する | `artifacts/rustfs/packages/nightly/` | +| Existing local objects | キーがすでにローカルにあるときの扱い | **Always skip** | +| Dry run | 一致を一覧し、キューにも書き込みにもしない | 事前確認ではオン、取り込みではオフ | + +**Dry run** は一致するオブジェクトを一覧するだけで、キューに入れず、オブジェクトも書きません。プレフィックスを確認してから **Dry run** をオフにして取り込みます。 + +![nightly プレフィックス、Always skip、Dry run 有効の Start background backfill ダイアログ](./images/rustfs-console-on-demand-migration-backfill-dry-run.png) + +手順中に記録したプレフィックス一覧の抜粋です。ラボの記録では先頭のキーが切れています。完全なキーは `artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-29.deb` の形です。タイムスタンプは当時の記録で、おおよそ UTC です。完全な一覧はこの抜粋より長くなります。 + +```text +…ghtly-2026-10-29.deb +[2026-11-02 03:25:08] 70.36 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-30.deb +[2026-11-02 03:25:08] 70.65 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-31.deb +[2026-11-02 03:25:08] 71.15 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-01.deb +[2026-11-02 03:14:39] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb +[2026-11-02 03:25:08] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb +``` + +バックフィルがオブジェクトを書き込んでいる間、Console は一覧済み件数と取り込み済み件数を表示します。ソースの走査が続いている間は、完了割合を表示しません。 + +![一覧済みオブジェクト、取り込み済みオブジェクト、移行データを示す Backfill in progress ダイアログ](./images/rustfs-console-on-demand-migration-backfill-in-progress.png) + +## 7. ドライランのあと、ソース名前空間をバックフィルする + +設定済みのソース名前空間を走査するには、**Source listing prefix** を空にします。先に **Dry run** を実行して一覧を確認します。その後 **Dry run** をオフにしてバックフィルを開始します。 + +この手順のドライランは、オブジェクトを一覧しただけで取り込みはありませんでした。 + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 0 | 0 | 0 | 0 B | + +![ドライラン後に完了したバックグラウンドバックフィル。128 件を一覧し、取り込みは 0](./images/rustfs-console-on-demand-migration-backfill-listed-only.png) + +その後、プレフィックスを空にして実行した全体のバックフィルは次の結果でした。 + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 127 | 1 | 0 | 8.0 GiB | + +![ソース名前空間のバックグラウンドバックフィル完了。127 件を取り込み、失敗オブジェクトは 0](./images/rustfs-console-on-demand-migration-backfill-completed.png) + +## 8. バケットを確認する + +オンデマンド移行のページで確認します。 + +1. **Runtime status** — 移行済みオブジェクトと移行済みデータが 0 ではなく、取り込み失敗が記録されていない。 +2. **Background backfill** — 最新ジョブに、一覧、取り込み、スキップ、失敗の件数が表示されている。 +3. **Browser** と `rc` — 上の端末セッションのように `rc ls` でプレフィックスを一覧し、ローカルにあるキーを `rc cp` で再度読む。 + +この手順のランタイムパネルは **Migrated data** が 8.1 GiB、**Migrated objects** が 128 でした。最新のバックフィルは Listed 128、Pulled 127、Skipped 1、Failed objects 0、Data 8.0 GiB でした。 + +![Cloudflare R2 からの取り込み後のオンデマンド移行ランタイムと完了したバックフィル](./images/rustfs-console-on-demand-migration-runtime-after-backfill.png) + +手順後のバケット一覧では、`migration` が 128 オブジェクト、8.1 GiB になっています。 + +![オンデマンド移行後の RustFS Console バケット一覧に表示された migration バケット](./images/rustfs-console-bucket-list-after-on-demand-migration.png) + +:::warning[ローカルコピーはソースを追従しない] +R2 上の同じキーを後から変更しても、ローカルオブジェクトの上には取り込まれません。そのキーへのローカル `PUT` はソースへ書き戻されず、ソースキーより前に残ります。オンデマンド移行は、ソースからの継続的なレプリケーションではありません。 +::: + +## 次のステップ + +- 送信方向のコピーが必要なときは、[バケットレプリケーションを設定する](./replication.md) +- [rc をインストールして使う](/operations/rc) +- サイト設定と ID データには、[サイトレプリケーションを設定する](/operations/high-availability/site-replication) diff --git a/content/zh/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png b/content/zh/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png new file mode 100644 index 000000000..b14205dd7 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-cloudflare-r2-nightly-objects-before-migration.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png b/content/zh/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png new file mode 100644 index 000000000..dc246d6dd Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-browser-single-object-on-demand-migration.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png b/content/zh/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png new file mode 100644 index 000000000..1a9824956 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-bucket-automation-on-demand-migration.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png b/content/zh/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png new file mode 100644 index 000000000..cd34928bb Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-bucket-list-after-on-demand-migration.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png b/content/zh/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png new file mode 100644 index 000000000..596b1ce4c Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-empty-bucket-before-on-demand-migration.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png b/content/zh/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png new file mode 100644 index 000000000..3b383986d Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-enable-on-demand-migration.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png new file mode 100644 index 000000000..53d6ff190 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-active-zero-runtime.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png new file mode 100644 index 000000000..847456845 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-completed.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png new file mode 100644 index 000000000..47a093fc4 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-dry-run.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png new file mode 100644 index 000000000..46428d357 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-in-progress.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png new file mode 100644 index 000000000..5afe4166f Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-backfill-listed-only.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png new file mode 100644 index 000000000..f835f4019 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-passed.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png new file mode 100644 index 000000000..d50889e64 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-connection-test-toast.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png new file mode 100644 index 000000000..9ea4b18fe Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-r2-source-form.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png new file mode 100644 index 000000000..eb0dd81d6 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-runtime-after-backfill.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png new file mode 100644 index 000000000..74c3048f7 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-console-on-demand-migration-source-providers.png differ diff --git a/content/zh/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png b/content/zh/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png new file mode 100644 index 000000000..09f4cacf3 Binary files /dev/null and b/content/zh/administration/data/bucket/images/rustfs-rc-cp-on-demand-migration-single-object.png differ diff --git a/content/zh/administration/data/bucket/meta.json b/content/zh/administration/data/bucket/meta.json index 00af9f84e..77c2c3af0 100644 --- a/content/zh/administration/data/bucket/meta.json +++ b/content/zh/administration/data/bucket/meta.json @@ -5,6 +5,7 @@ "deletion", "quota", "replication", + "on-demand-migration", "policy" ] } \ No newline at end of file diff --git a/content/zh/administration/data/bucket/on-demand-migration.md b/content/zh/administration/data/bucket/on-demand-migration.md new file mode 100644 index 000000000..cd69883c5 --- /dev/null +++ b/content/zh/administration/data/bucket/on-demand-migration.md @@ -0,0 +1,221 @@ +--- +title: "从 Cloudflare R2 按需迁移" +description: "将 Cloudflare R2 存储桶配置为只读源,并在读取未命中或回填时把对象拉入 RustFS。" +--- + +按需迁移在客户端读取本地缺失的对象键时,或在你启动回填时,把外部存储桶中的对象拉入 RustFS 存储桶。**本文以 Cloudflare R2 作为贯穿全文的示例。** 下面的步骤按 Console v1.0.3 的一次实操编写:把 R2 存储桶挂到空的本地存储桶上,测试连接并启用,然后做三次拉取——用 `rc cp` 拉单个对象、按前缀回填,以及对整个源命名空间先试运行再全量回填。 + +RustFS 只读源。启用表单写明,RustFS 不会在源上写入或删除对象。迁移页写明,只有在对象被读取或被回填时才会拉取;RustFS 中新的 `PUT` 和 `DELETE` 不会同步回源,源上之后的变更也不会更新已落本地的副本。 + +本页截图中的 R2 账户 ID 和访问密钥均已换成占位符。终端提示符中的主机是 `localhost`。 + +## 源提供方 + +**Source connection** → **Provider** 列出这些源。下面的名称是控制台原文。Google Cloud Storage 有两项:HMAC 和 Native。 + +1. Amazon S3 +2. S3-compatible storage +3. MinIO +4. RustFS +5. Cloudflare R2 +6. Google Cloud Storage (HMAC) +7. Azure Blob Storage (Native) +8. Google Cloud Storage (Native) + +![RustFS 控制台按需迁移的 Provider 菜单,列出 Amazon S3、S3-compatible storage、MinIO、RustFS、Cloudflare R2、Google Cloud Storage (HMAC)、Azure Blob Storage (Native) 和 Google Cloud Storage (Native)](./images/rustfs-console-on-demand-migration-source-providers.png) + +从该列表中的其他提供方迁移时,流程相同:创建空的本地存储桶,启用按需迁移,选择 **Provider**,选择 **Test connection**,选择 **Enable migration**,然后在 GET 未命中时拉取,或通过回填拉取。随提供方变化的只有端点、凭证和存储桶字段。下面的 R2 实操为 Cloudflare R2 填写这些字段。 + +## 按需迁移与存储桶复制 + +| 目标 | 使用 | +| --- | --- | +| 从上面列出的提供方把对象拉入 RustFS,在读取未命中时拉取,或通过回填拉取 | 按需迁移 | +| 把 RustFS 存储桶中的对象版本推到另一个存储桶 | [存储桶复制](./replication.md) | +| 在部署之间同步站点配置和身份数据 | [站点复制](/operations/high-availability/site-replication) | + +按需迁移是入站拉取。[存储桶复制](./replication.md)是对象版本的出站复制,并且要求源存储桶和目标存储桶都启用版本控制。对象键写入本地之后,按需迁移不会继续跟随源。 + +:::warning[不要形成复制环路] +不要把已经是此本地存储桶复制目标的存储桶,再选成按需迁移的源。 +::: + +## 开始之前 + +这里的凭证和端点属于 Cloudflare R2 示例。如果使用列表中的其他提供方,在同样的控制台字段中填写该系统的端点、只读凭证和存储桶名称。 + +- 控制台中有 **On-demand migration(按需迁移)**。本次实操使用 Console v1.0.3,端口为 `9001`。 +- 控制台账号可以打开本地存储桶的 **Settings**。 +- 读取本地存储桶的主机上已安装 RustFS CLI [`rc`](/operations/rc)。 +- 一个可以读取源存储桶对象的 Cloudflare R2 API 令牌。建议把令牌限制在该存储桶。此流程不使用 R2 存储桶的公共访问。 +- 源存储桶的 R2 S3 端点 `https://.r2.cloudflarestorage.com`。存储桶若使用管辖区,请从 R2 控制台复制端点。 + +把 `rc` 指向本地 RustFS 的 S3 API,不要指向 R2: + +```bash +rc alias set rustfs http://localhost:9000 --region us-east-1 --bucket-lookup path +``` + +本次实操的本地存储桶是 `migration`,R2 存储桶是 `dl-rustfs`。`migration` 已启用版本控制,对象锁定为关闭。启用按需迁移不需要对象锁定,这次也没有把版本控制当作启用条件。启用版本控制后,本地删除标记可以挡住源上的同名键。 + +:::warning[使用只读的源令牌] +创建只能读取源存储桶、不能在源上写入或删除的 R2 令牌。表单写明,RustFS 会保存秘密密钥,并且不会把它返回到控制台。 +::: + +:::note[启用源不会开始搬数据] +**Migration active** 只表示存储桶已经挂上源。在客户端读到缺失的键,或你启动回填之前,运行时计数保持为 0。 +::: + +## 1. 创建空的本地存储桶 + +在控制台打开 **Browser** → **Buckets**,创建名为 `migration` 的存储桶,并确认其中没有对象。准备挂源时,从该存储桶进入 **Settings**。 + +![RustFS 控制台 Browser 中,按需迁移前为空的 migration 存储桶](./images/rustfs-console-empty-bucket-before-on-demand-migration.png) + +## 2. 打开按需迁移 + +1. 打开存储桶 `migration`,选择 **Settings**。 +2. 选择 **Automation**。 +3. 找到 **On-demand migration**。 +4. 尚未配置时,选择 **Open migration settings** 或 **+ Enable on-demand migration**。已经启用时,选择 **Manage migration**。 + +本次实操中,未配置页面会说明还没有迁移源,并提供 **+ Enable on-demand migration**。 + +![尚未配置源时的 RustFS 按需迁移页,页面上有 Enable on-demand migration](./images/rustfs-console-enable-on-demand-migration.png) + +源启用之后,**Automation** 会显示 **Migration active** 和 **Manage migration**。 + +![RustFS 控制台中 migration 存储桶的 Automation 设置,按需迁移已处于活动状态](./images/rustfs-console-bucket-automation-on-demand-migration.png) + +## 3. 把存储桶指向 Cloudflare R2 + +在 **Enable on-demand migration** 中按下面填写。把账户 ID 和密钥换成你自己的 R2 令牌。令牌不需要会话令牌时,会话令牌留空。**Local prefix** 留空即可暴露整个源命名空间。本次保存的配置没有额外前缀(**All keys**,**No added prefix**)。 + +| 字段 | 本次实操的值 | +| --- | --- | +| Source | Cloudflare R2 | +| Endpoint | `https://.r2.cloudflarestorage.com` | +| Region | `us-east-1`(表单也接受 `auto`) | +| Source Bucket | `dl-rustfs` | +| Addressing style | Automatic | +| Access Key | R2 访问密钥 ID | +| Secret Key | R2 秘密密钥 | +| Session token | 空 | +| Local prefix | 空 | + +![为 Cloudflare R2 源填写的启用按需迁移表单,端点和访问密钥已换成占位符](./images/rustfs-console-on-demand-migration-r2-source-form.png) + +## 4. 测试连接并启用迁移 + +选择 **Test connection**。检查通过时显示 **Connection test passed**,并给出源存储桶中的一个样例对象键。然后选择 **Enable migration**。 + +![启用按需迁移对话框内的连接测试通过结果,其中包含一个样例对象键](./images/rustfs-console-on-demand-migration-connection-test-passed.png) + +同一结果也会以 toast 出现: + +![按需迁移对话框上的连接测试通过 toast](./images/rustfs-console-on-demand-migration-connection-test-toast.png) + +配置状态变为 **Migration active**。在发生读取或回填之前,**Runtime status** 中的已迁移对象和已迁移数据都为 0。**Latest job** 保持 **No backfill yet**。 + +![migration 存储桶的按需迁移已启用,运行时计数仍为 0](./images/rustfs-console-on-demand-migration-active-zero-runtime.png) + +## 5. 用 rc cp 拉取单个对象 + +用 `rc` 读取本地存储桶。不要把这个请求发到 R2 端点。本次实操开始时本地存储桶仍为空,并拉取了一个缺失对象。第一次未命中会把对象返回给客户端,并写入本地。之后再读同一个键,由 RustFS 本地提供。 + +```bash +rc cp rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb /tmp/ +``` + +```text +rustfs/migration/artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb -> /tmp/rustfs-nightly-2026-11-02.deb (71.34 MiB) +``` + +源存储桶中没有这个对象时,请改用你自己的键。终端记录了这次 `rc cp`,以及之后用 `rc ls` 查看本地 nightly 前缀。列表是对象已经在本地之后的前缀内容。上面这一条 `rc cp` 是 71.34 MiB 的单对象拉取。 + +![终端中 rc cp 在按需迁移期间通过本地 migration 存储桶拉取一个 71.34 MiB 对象](./images/rustfs-rc-cp-on-demand-migration-single-object.png) + +Cloudflare 控制台中,作为源的同一个 nightly 前缀: + +![作为按需迁移源的 Cloudflare R2 nightly 前缀对象列表](./images/rustfs-cloudflare-r2-nightly-objects-before-migration.png) + +该控制台上的存储桶大小是 Cloudflare 统计的存储桶总量。本页后面的 RustFS 计数是这次迁移任务的结果。 + +单个对象拉取之后,控制台 **Browser** 会显示本地存储桶中的该对象: + +![按需迁移读取之后,RustFS 控制台 Browser 中的 rustfs-nightly-2026-11-02.deb](./images/rustfs-console-browser-single-object-on-demand-migration.png) + +## 6. 按前缀回填 + +**Start backfill** 会扫描源,并拉取本地还没有的键。本次对话框使用了这些设置: + +| 选项 | 含义 | 本次实操的值 | +| --- | --- | --- | +| Source listing prefix | 只列出该前缀 | `artifacts/rustfs/packages/nightly/` | +| Existing local objects | 键已在本地时如何处理 | **Always skip** | +| Dry run | 只列出匹配对象,不排队、不写入 | 演练时打开,正式拉取时关闭 | + +**Dry run(试运行)** 只列出匹配对象,不排队也不写入任何对象。确认前缀后,先关闭 **Dry run** 再拉取。 + +![启动后台回填对话框,前缀为 nightly,Existing local objects 为 Always skip,并且启用了 Dry run](./images/rustfs-console-on-demand-migration-backfill-dry-run.png) + +下面是实操中记录的前缀列表节选。实验室记录里第一行键被截断了,完整键的形式是 `artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-29.deb`。时间戳来自当时的记录,大约为 UTC。完整列表比这段节选更长。 + +```text +…ghtly-2026-10-29.deb +[2026-11-02 03:25:08] 70.36 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-30.deb +[2026-11-02 03:25:08] 70.65 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-10-31.deb +[2026-11-02 03:25:08] 71.15 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-01.deb +[2026-11-02 03:14:39] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-2026-11-02.deb +[2026-11-02 03:25:08] 71.34 MiB artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb +``` + +回填正在写入对象时,控制台会显示已列出和已拉取的数量。源仍在扫描时,不显示完成百分比。 + +![回填进行中对话框,显示已列出对象、已拉取对象和已迁移数据](./images/rustfs-console-on-demand-migration-backfill-in-progress.png) + +## 7. 先试运行,再回填源命名空间 + +**Source listing prefix** 留空,即扫描已配置的源命名空间。先做 **Dry run**,确认列出的范围,再关闭 **Dry run** 并启动回填。 + +这次实操中的一次试运行只列出了对象,没有拉取: + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 0 | 0 | 0 | 0 B | + +![试运行结束后的后台回填,列出 128 个对象,拉取数量为 0](./images/rustfs-console-on-demand-migration-backfill-listed-only.png) + +随后以前缀为空做的全量回填结果如下: + +| Listed | Pulled | Skipped | Failed objects | Data | +| --- | --- | --- | --- | --- | +| 128 | 127 | 1 | 0 | 8.0 GiB | + +![源命名空间的后台回填已完成,拉取 127 个对象,失败对象为 0](./images/rustfs-console-on-demand-migration-backfill-completed.png) + +## 8. 验证存储桶 + +在按需迁移页面检查: + +1. **Runtime status** — 已迁移对象和已迁移数据不再为 0,并且没有记录到拉取失败。 +2. **Background backfill** — 最近一次任务显示列出、拉取、跳过和失败的数量。 +3. **Browser** 和 `rc` — 像上面的终端记录那样用 `rc ls` 列出前缀,再用 `rc cp` 重新读取一个已在本地的键。 + +本次运行时面板显示 **Migrated data** 为 8.1 GiB,**Migrated objects** 为 128。最近一次回填为 Listed 128、Pulled 127、Skipped 1、Failed objects 0、Data 8.0 GiB。 + +![从 Cloudflare R2 拉取之后的按需迁移运行时状态和已完成回填](./images/rustfs-console-on-demand-migration-runtime-after-backfill.png) + +实操结束后的存储桶列表中,`migration` 为 128 个对象、8.1 GiB: + +![按需迁移之后,RustFS 控制台存储桶列表中的 migration 存储桶](./images/rustfs-console-bucket-list-after-on-demand-migration.png) + +:::warning[本地副本不会跟随源] +R2 上同名键之后的变更不会覆盖本地对象。对该键的本地 `PUT` 不会写回源,并且会挡在源键前面。按需迁移不是从源持续复制。 +::: + +## 后续步骤 + +- 需要出站副本时,[配置存储桶复制](./replication.md) +- [安装并使用 rc](/operations/rc) +- 需要同步站点配置和身份数据时,[配置站点复制](/operations/high-availability/site-replication)