diff --git a/README.md b/README.md index 2cef068..11209bf 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ Command-line interface tools/scripts - [install_kubectl.sh](docs/install_kubectl.md) - [install_loki.sh](docs/install_loki.md) - [install_promtail.sh](docs/install_promtail.md) +- [install_s3_mount.sh](docs/install_s3_mount.md) - [install_ssm_agent.sh](docs/install_ssm_agent.md) - [install_ssm_plugin.sh](docs/install_ssm_plugin.md) - [install_terraform.sh](docs/install_terraform.md) @@ -39,6 +40,7 @@ Refer to the respective files for detailed usage instructions: - **[install_kubectl.sh](docs/install_kubectl.md)**: Install or uninstall `kubectl` on a Linux system. - **[install_loki.sh](docs/install_loki.md)**: Install or uninstall Grafana Loki log aggregation system. - **[install_promtail.sh](docs/install_promtail.md)**: Install or uninstall Grafana Promtail log shipper with flexible configuration options. +- **[install_s3_mount.sh](docs/install_s3_mount.md)**: Install or uninstall AWS Mountpoint for Amazon S3 to mount S3 buckets as local filesystems. - **[install_ssm_agent.sh](docs/install_ssm_agent.md)**: Install or uninstall the AWS Systems Manager (SSM) Agent on various Linux distributions, macOS, and Windows. - **[install_ssm_plugin.sh](docs/install_ssm_plugin.md)**: Install or uninstall the AWS SSM Session Manager Plugin for remote EC2 access. - **[install_terraform.sh](docs/install_terraform.md)**: Install or uninstall Terraform on supported Linux distributions. diff --git a/docs/install_s3_mount.md b/docs/install_s3_mount.md new file mode 100644 index 0000000..ad0c018 --- /dev/null +++ b/docs/install_s3_mount.md @@ -0,0 +1,247 @@ +# install_s3_mount.sh + +This script installs, configures, and manages AWS Mountpoint for Amazon S3 on RPM-based Linux distributions (Amazon Linux, RHEL, CentOS, Fedora). AWS Mountpoint allows you to mount an S3 bucket as a local filesystem. + +## Usage + +```bash +./install_s3_mount.sh [install|uninstall] +``` + +- **install** (default): Installs AWS Mountpoint and configures the S3 mount. +- **uninstall**: Removes the S3 mount configuration (does not remove AWS Mountpoint package). + +## Required Environment Variables + +- **S3_BUCKET_NAME**: Name of the S3 bucket to mount (required for install) +- **MOUNT_USER**: User to set ownership for the mount (required for install) + +## Optional Environment Variables + +- **MOUNT_POINT**: Mount point directory (default: `/mnt/s3_config`) +- **MOUNTPOINT_VERSION**: Pin specific version or leave empty for latest + +## Example Usage + +To install S3 mount with default settings: + +```bash +S3_BUCKET_NAME=my-bucket MOUNT_USER=ec2-user ./install_s3_mount.sh install +``` + +To install with a custom mount point: + +```bash +S3_BUCKET_NAME=my-bucket MOUNT_POINT=/mnt/my-s3 MOUNT_USER=ubuntu ./install_s3_mount.sh install +``` + +To install a specific version of AWS Mountpoint: + +```bash +S3_BUCKET_NAME=my-bucket MOUNT_USER=ec2-user MOUNTPOINT_VERSION=1.0.0 ./install_s3_mount.sh install +``` + +To uninstall S3 mount: + +```bash +MOUNT_POINT=/mnt/s3_config ./install_s3_mount.sh uninstall +``` + +## Running Without Cloning + +```bash +S3_BUCKET_NAME=my-bucket MOUNT_USER=ec2-user bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_s3_mount.sh) install +``` + +## Verification + +After installation, check if AWS Mountpoint is installed: + +```bash +mount-s3 --version +``` + +Check the service status: + +```bash +sudo systemctl status s3-mount.service +``` + +Verify the mount point: + +```bash +mountpoint /mnt/s3_config +``` + +List mounted filesystems: + +```bash +df -h | grep s3 +``` + +## Supported Operating Systems + +- **Amazon Linux 2023, RHEL 8/9, Fedora** (uses `dnf` for package management) +- **Amazon Linux 2, RHEL 7, CentOS 7** (uses `yum` for package management) + +**Note**: AWS Mountpoint is currently only available as RPM packages, so this script supports RPM-based distributions only. + +## Supported Architectures + +- **x86_64** (amd64) +- **aarch64** (arm64) + +## Features + +- Automatically detects package manager and operating system +- Installs `wget` as a dependency if not present +- Downloads and installs AWS Mountpoint RPM package +- Creates systemd service for automatic mounting on boot +- Configures mount with read-only access and proper user permissions +- Automatically starts and enables the mount service +- Skips installation if AWS Mountpoint is already installed +- Handles version pinning for reproducible installations + +## Error Handling + +- If AWS Mountpoint is already installed, the script will skip reinstallation and display the current version. +- If an unsupported OS or architecture is detected, the script exits with an error message. +- The script validates that required environment variables are set before proceeding. +- Service file and binary existence checks are performed before starting the service. +- Missing dependencies are automatically installed. +- User validation ensures the specified mount user exists before configuration. + +## Cleanup + +- Temporary installation files (`mount-s3.rpm`) are automatically removed after execution. +- The script uses `trap cleanup EXIT` to ensure cleanup even if interrupted. + +## Configuration + +The script automatically configures the S3 mount with: + +- **Systemd service** at `/etc/systemd/system/s3-mount.service` +- **Read-only mount** for safety +- **User permissions** based on the specified `MOUNT_USER` +- **File and directory modes** (0644 for files, 0755 for directories) +- **Automatic mount** on system boot +- **Proper unmount** on service stop + +### Configuration Details + +- **Mount type**: Read-only (`--read-only`) +- **Allow other users**: Enabled (`--allow-other`) +- **File permissions**: 0644 (readable by all, writable by owner) +- **Directory permissions**: 0755 (readable/executable by all, writable by owner) +- **User/Group**: Set based on `MOUNT_USER` UID/GID +- **Service type**: oneshot with RemainAfterExit + +## Service Management + +After installation, you can manage the S3 mount service using: + +```bash +sudo systemctl start s3-mount.service +sudo systemctl stop s3-mount.service +sudo systemctl restart s3-mount.service +sudo systemctl enable s3-mount.service +sudo systemctl disable s3-mount.service +``` + +### Service Commands + +- **Start**: `sudo systemctl start s3-mount.service` +- **Stop**: `sudo systemctl stop s3-mount.service` (unmounts the bucket) +- **Restart**: `sudo systemctl restart s3-mount.service` +- **Status**: `sudo systemctl status s3-mount.service` +- **Enable**: `sudo systemctl enable s3-mount.service` (mounts automatically on boot) +- **Disable**: `sudo systemctl disable s3-mount.service` (prevents auto-mount on boot) + +## Prerequisites + +The script automatically installs required dependencies: + +- `wget` - for downloading AWS Mountpoint RPM + +### AWS Credentials + +AWS Mountpoint requires AWS credentials to access S3 buckets. Ensure one of the following is configured: + +- **IAM instance role** (recommended for EC2 instances) +- **AWS credentials file** (`~/.aws/credentials`) +- **Environment variables** (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) +- **AWS credentials chain** (IAM roles, credentials file, environment variables) + +## Troubleshooting + +### Common Issues + +1. **Service fails to start**: Check logs with `sudo journalctl -u s3-mount.service -f` +2. **Permission denied**: Ensure AWS credentials are properly configured +3. **Bucket not found**: Verify the bucket name is correct and accessible +4. **Mount point busy**: Ensure no processes are using the mount point +5. **User not found**: Verify the `MOUNT_USER` exists on the system + +### Logs and Debugging + +- **Service logs**: `sudo journalctl -u s3-mount.service` +- **Check if mounted**: `mountpoint /mnt/s3_config` or `df -h | grep s3` +- **Check mount details**: `mount | grep s3` +- **Manual mount test**: `sudo mount-s3 --read-only my-bucket /mnt/test` + +### AWS Permissions + +The IAM role or user needs the following S3 permissions: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:GetObject" + ], + "Resource": [ + "arn:aws:s3:::my-bucket", + "arn:aws:s3:::my-bucket/*" + ] + } + ] +} +``` + +### Manual Unmount + +If the service fails to unmount, you can manually unmount: + +```bash +sudo umount /mnt/s3_config +``` + +### Removing AWS Mountpoint Package + +The uninstall script does not remove the AWS Mountpoint package. To remove it: + +```bash +# For dnf-based systems +sudo dnf remove mountpoint-s3 + +# For yum-based systems +sudo yum remove mountpoint-s3 +``` + +## Security Considerations + +- **Read-only mount**: The script configures the mount as read-only for safety +- **IAM permissions**: Use least-privilege IAM policies +- **Network security**: Consider VPC endpoints for S3 access to avoid internet traffic +- **User permissions**: Only the specified user and group can write to the mount (if write access is enabled) + +## Limitations + +- **RPM-only**: AWS Mountpoint is currently only available as RPM packages +- **Read-only**: The script configures read-only mounts by default +- **Network dependency**: Requires network connectivity to S3 +- **Performance**: File operations may be slower than local filesystems due to network latency diff --git a/scripts/install_s3_mount.sh b/scripts/install_s3_mount.sh new file mode 100755 index 0000000..0850530 --- /dev/null +++ b/scripts/install_s3_mount.sh @@ -0,0 +1,295 @@ +#!/bin/bash + +set -e + +S3_BUCKET_NAME="${S3_BUCKET_NAME:-}" +MOUNT_POINT="${MOUNT_POINT:-/mnt/s3_config}" +MOUNTPOINT_VERSION="${MOUNTPOINT_VERSION:-}" # Optional: pin version, otherwise uses latest +MOUNT_USER="${MOUNT_USER:-}" # User to set ownership for mount (required) + +cleanup() { + rm -f /tmp/mount-s3.rpm +} +trap cleanup EXIT + +detect_package_manager() { + if command -v dnf &>/dev/null; then + PACKAGE_MANAGER="dnf" + elif command -v yum &>/dev/null; then + PACKAGE_MANAGER="yum" + elif command -v apt &>/dev/null || command -v apt-get &>/dev/null; then + PACKAGE_MANAGER="apt" + else + echo "Unsupported package manager. Exiting." + exit 1 + fi +} + +install_dependencies() { + detect_package_manager + echo "Detected package manager: $PACKAGE_MANAGER" + + # Check what's missing + local missing_packages=() + + if ! command -v wget &>/dev/null; then + missing_packages+=("wget") + fi + + if [ ${#missing_packages[@]} -eq 0 ]; then + echo "All dependencies are already installed." + return 0 + fi + + echo "Installing missing packages: ${missing_packages[*]}" + + case "$PACKAGE_MANAGER" in + apt | apt-get) + sudo $PACKAGE_MANAGER update && sudo $PACKAGE_MANAGER install -y "${missing_packages[@]}" + ;; + dnf) + sudo dnf install -y "${missing_packages[@]}" + ;; + yum) + sudo yum install -y "${missing_packages[@]}" + ;; + *) + echo "Unsupported package manager. Exiting." + exit 1 + ;; + esac +} + +install_mountpoint() { + if command -v mount-s3 &>/dev/null; then + echo "AWS Mountpoint is already installed. Skipping installation." + echo "Current version: $(mount-s3 --version 2>&1 | head -n 1 || echo 'unknown')" + exit 0 + fi + + install_dependencies + + echo "Installing AWS Mountpoint for Amazon S3..." + cd /tmp + + # Detect architecture + ARCH=$(uname -m) + if [ "$ARCH" = "x86_64" ]; then + ARCH="x86_64" + elif [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then + ARCH="arm64" + else + echo "Unsupported architecture: $ARCH" + exit 1 + fi + + # Get version (use pinned version if set, otherwise use latest) + if [ -z "$MOUNTPOINT_VERSION" ]; then + MOUNTPOINT_URL="https://s3.amazonaws.com/mountpoint-s3-release/latest/${ARCH}/mount-s3.rpm" + echo "Downloading AWS Mountpoint RPM (latest) from ${MOUNTPOINT_URL}..." + else + MOUNTPOINT_URL="https://s3.amazonaws.com/mountpoint-s3-release/releases/${MOUNTPOINT_VERSION}/${ARCH}/mount-s3-${MOUNTPOINT_VERSION}-1.${ARCH}.rpm" + echo "Downloading AWS Mountpoint RPM (version ${MOUNTPOINT_VERSION}) from ${MOUNTPOINT_URL}..." + fi + + if wget "${MOUNTPOINT_URL}" -O mount-s3.rpm 2>&1; then + echo "Download successful. Installing AWS Mountpoint RPM..." + # Use rpm directly to avoid dnf Python module issues (preferred method) + if rpm -Uvh mount-s3.rpm 2>&1; then + echo "AWS Mountpoint installed successfully" + rm -f mount-s3.rpm + else + echo "Warning: rpm install failed, trying with ${PACKAGE_MANAGER}..." + # Try dnf/yum as fallback + if sudo ${PACKAGE_MANAGER} install -y ./mount-s3.rpm 2>&1; then + echo "AWS Mountpoint installed successfully via ${PACKAGE_MANAGER}" + rm -f mount-s3.rpm + else + echo "Error: Failed to install AWS Mountpoint RPM with both rpm and ${PACKAGE_MANAGER}" + echo "Checking if package is already installed..." + if rpm -q mountpoint-s3 2>/dev/null; then + echo "AWS Mountpoint appears to be already installed" + rm -f mount-s3.rpm + else + echo "Error: Failed to install AWS Mountpoint RPM" + exit 1 + fi + fi + fi + else + echo "Error: Failed to download AWS Mountpoint RPM" + exit 1 + fi + + cd / + + # Verify mount-s3 is installed + if ! command -v mount-s3 &>/dev/null; then + echo "Error: Failed to install AWS Mountpoint" + exit 1 + fi + + echo "Mountpoint version: $(mount-s3 --version 2>&1 || echo 'unknown')" +} + +configure_s3_mount() { + if [ -z "$S3_BUCKET_NAME" ]; then + echo "Error: S3_BUCKET_NAME is required. Cannot configure S3 mount." + exit 1 + fi + + if [ -z "$MOUNT_USER" ]; then + echo "Error: MOUNT_USER is required. Cannot configure S3 mount." + exit 1 + fi + + echo "Creating mount point ${MOUNT_POINT}..." + sudo mkdir -p "${MOUNT_POINT}" + sudo chmod 755 "${MOUNT_POINT}" + + # Get user UID and GID + if id "$MOUNT_USER" &>/dev/null; then + MOUNT_UID=$(id -u "$MOUNT_USER") + MOUNT_GID=$(id -g "$MOUNT_USER") + echo "${MOUNT_USER} UID: ${MOUNT_UID}, GID: ${MOUNT_GID}" + else + echo "Error: User ${MOUNT_USER} not found." + exit 1 + fi + + # Create systemd service for S3 mount + echo "Creating systemd service for S3 mount..." + sudo tee /etc/systemd/system/s3-mount.service > /dev/null </dev/null; then + echo "Error: AWS Mountpoint binary not found. Please run installation first." + exit 1 + fi + + echo "Enabling S3 mount service..." + sudo systemctl daemon-reload + sudo systemctl enable s3-mount.service + + echo "Starting S3 mount service..." + if sudo systemctl start s3-mount.service; then + echo "S3 mount service started successfully" + sleep 3 + if mountpoint -q "${MOUNT_POINT}"; then + echo "S3 bucket ${S3_BUCKET_NAME} is mounted at ${MOUNT_POINT}" + else + echo "Warning: Mount point exists but may not be mounted correctly" + sudo systemctl status s3-mount.service --no-pager -l || true + fi + else + echo "Error: Failed to start S3 mount service" + sudo systemctl status s3-mount.service --no-pager -l || true + exit 1 + fi +} + +uninstall_s3_mount() { + if [ ! -f /etc/systemd/system/s3-mount.service ]; then + echo "S3 mount is not configured. Skipping uninstallation." + exit 0 + fi + + echo "Uninstalling S3 mount..." + + # Stop and disable the service (Linux only) + if [[ "$(uname -s)" == "Linux" ]]; then + if systemctl is-system-running &>/dev/null; then + sudo systemctl stop s3-mount.service 2>/dev/null || true + sudo systemctl disable s3-mount.service 2>/dev/null || true + fi + fi + + # Unmount if mounted + if mountpoint -q "${MOUNT_POINT}" 2>/dev/null; then + sudo umount "${MOUNT_POINT}" 2>/dev/null || true + fi + + # Remove service file + sudo rm -f /etc/systemd/system/s3-mount.service + + # Reload systemd + if [[ "$(uname -s)" == "Linux" ]]; then + sudo systemctl daemon-reload + fi + + echo "S3 mount has been uninstalled." + echo "Note: AWS Mountpoint package is not removed. Use package manager to remove if needed." +} + +usage() { + echo "Usage: $0 [install|uninstall]" + echo "" + echo "Required environment variables:" + echo " S3_BUCKET_NAME - Name of the S3 bucket to mount (required for install)" + echo " MOUNT_USER - User to set ownership for mount (required for install)" + echo "" + echo "Optional environment variables:" + echo " MOUNT_POINT - Mount point directory (default: /mnt/s3_config)" + echo " MOUNTPOINT_VERSION - Pin specific version or leave empty for latest" + echo "" + echo "Examples:" + echo " S3_BUCKET_NAME=my-bucket MOUNT_USER=ec2-user $0 install" + echo " S3_BUCKET_NAME=my-bucket MOUNT_POINT=/mnt/my-s3 MOUNT_USER=ubuntu $0 install" + echo " MOUNT_POINT=/mnt/s3_config $0 uninstall" + exit 1 +} + +if [ "$#" -eq 0 ]; then + if [ -z "$S3_BUCKET_NAME" ] || [ -z "$MOUNT_USER" ]; then + echo "Error: S3_BUCKET_NAME and MOUNT_USER are required." + usage + fi + install_mountpoint + configure_s3_mount + start_s3_mount +elif [ "$1" == "install" ]; then + if [ -z "$S3_BUCKET_NAME" ] || [ -z "$MOUNT_USER" ]; then + echo "Error: S3_BUCKET_NAME and MOUNT_USER are required." + usage + fi + install_mountpoint + configure_s3_mount + start_s3_mount +elif [ "$1" == "uninstall" ]; then + uninstall_s3_mount +else + usage +fi + +if [ "$1" != "uninstall" ]; then + echo "" + echo "S3 mount installation completed successfully." + echo "S3 bucket ${S3_BUCKET_NAME} is mounted at ${MOUNT_POINT} (read-only)" + echo "Run 'mountpoint ${MOUNT_POINT}' to verify the mount." +fi