s3ctl

A tool for managing S3 bucket versions and tombstones. It can restore a bucket to a specific point in time and clean up stale delete markers.

Contributing

We welcome contributions! This quick guide shows how to build and run the integration-style tests locally.

• Prerequisites: Go 1.25+ and access to an S3-compatible endpoint (use the MinIO stack below for local dev).

Build

• Using Makefile (recommended):
  • Build: make build (outputs bin/s3ctl). You can override GOOS/GOARCH, e.g. make build GOOS=linux GOARCH=amd64.
  • Run from source: make run (equivalent to go run .).
  • Install to GOPATH/bin: make install.
  • Clean artifacts: make clean.
• Plain Go: go build -o bin/s3ctl .

Test

Helpers under test/main.go exercise restore and tombstone-cleanup flows.

1. Start a local S3-compatible service (optional): see "Local MinIO with Docker Compose". Export env vars for the endpoint: S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, S3_USE_SSL.
2. Prepare test data and capture a restore timestamp: go run test/main.go setup (note the printed bucket and RESTORE_TIMESTAMP).
3. Build, then run commands against the test bucket:
   • Restore: export S3_BUCKET="<bucket>" RESTORE_TIMESTAMP="<timestamp>" && bin/s3ctl restore
   • Cleanup tombstones: bin/s3ctl cleanup-tombstones
4. Optional verifications: go run test/main.go verify-restore and go run test/main.go verify-tombstone.

Please update this README if you change the flow.

Overview

This tool provides two main functions:

1. Restore a bucket to a specific point in time by deleting object versions created after that timestamp.
2. Remove stale delete markers (tombstones) that have no live versions.

Prerequisites

• An S3-compatible storage system (e.g., MinIO, AWS S3)
• Properly configured environment variables (see below)

Configuration (Environment variables)

Variable	Description
S3_ENDPOINT	S3 endpoint (e.g., localhost:9000)
S3_ACCESS_KEY	S3 access key
S3_SECRET_KEY	S3 secret key
S3_BUCKET	Target bucket name
S3_USE_SSL	Use SSL (true/false, default: false)

Usage

Commands

• restore — Restore a bucket to a specific point in time.
  • Options: -dry-run (default: true). Example to actually apply changes: -dry-run=false.
• cleanup-tombstones — Delete stale delete markers (no live versions).
  • Options: -dry-run (default: true).

Examples

• Restore to a timestamp:

  export S3_BUCKET="your-bucket" RESTORE_TIMESTAMP="2023-01-01T00:00:00Z"
  bin/s3ctl restore -dry-run=false

• Cleanup tombstones (preview only):

  bin/s3ctl cleanup-tombstones

CI hints

A minimal CI can:

• Run go run test/main.go setup to create a versioned test bucket and produce a RESTORE_TIMESTAMP.
• Execute bin/s3ctl restore -dry-run=false with the printed timestamp, then go run test/main.go verify-restore.
• Execute bin/s3ctl cleanup-tombstones and then go run test/main.go verify-tombstone.

---

Local MinIO with Docker Compose

A basic MinIO server is included for local testing.

Quick start

1. Copy the environment template and adjust credentials if desired:

   cp .env.example .env

2. Start MinIO:

   docker compose up -d

3. Access endpoints:
   • MinIO Console: http://localhost:9001 (login with MINIO_ROOT_USER/MINIO_ROOT_PASSWORD)
   • S3 API endpoint: http://localhost:9000

Using with this project

Export variables so the tools/tests can talk to MinIO:

export S3_ENDPOINT=localhost:9000
export S3_ACCESS_KEY=${MINIO_ROOT_USER:-minioadmin}
export S3_SECRET_KEY=${MINIO_ROOT_PASSWORD:-minioadmin123}
export S3_USE_SSL=false

Then follow the steps in the Usage section (e.g., go run test/main.go setup).

To stop/remove the stack:

docker compose down