Add a security GCP log analyzer

.github/workflows/beam_Infrastructure_SecurityLogging.yml

@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This workflow works with the GCP security log analyzer to
+# generate weekly security reports and initialize log sinks
+
+name: GCP Security Log Analyzer
+
+on:
+  workflow_dispatch:
+  schedule:
+    # Once a week at 9:00 AM on Monday
+    - cron: '0 9 * * 1'
+  push:
+    paths:
+      - 'infra/security/config.yml'
+
+# This allows a subsequently queued workflow run to interrupt previous runs
+concurrency:
+  group: '${{ github.workflow }} @ ${{ github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.sender.login }}'
+  cancel-in-progress: true
+
+#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
+permissions:
+  contents: read
+
+jobs:
+  beam_GCP_Security_LogAnalyzer:
+    name: GCP Security Log Analysis
+    runs-on: [self-hosted, ubuntu-20.04, main]
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.13'
+
+      - name: Install Python dependencies
+        working-directory: ./infra/security
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Setup gcloud
+        uses: google-github-actions/setup-gcloud@v2
+
+      - name: Initialize Log Sinks
+        if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
+        working-directory: ./infra/security
+        run: python log_analyzer.py --config config.yml initialize
+
+      - name: Generate Weekly Security Report
+        if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+        working-directory: ./infra/security
+        env:
+          SMTP_SERVER: smtp.gmail.com
+          SMTP_PORT: 465
+          EMAIL_ADDRESS: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_ADDRESS }}
+          EMAIL_PASSWORD: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_PASSWORD }}
+          EMAIL_RECIPIENT: "dev@beam.apache.org"
+        run: python log_analyzer.py --config config.yml generate-report --dry-run

infra/security/README.md

@@ -0,0 +1,84 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# GCP Security Analyzer
+
+This document describes the implementation of a security analyzer for Google Cloud Platform (GCP) resources. The analyzer is designed to enhance security monitoring within our GCP environment by capturing critical events and generating alerts for specific security-sensitive actions.
+
+## How It Works
+
+1.  **Log Sinks**: The system uses [GCP Log Sinks](https://cloud.google.com/logging/docs/export/configure_export_v2) to capture specific security-related log entries. These sinks are configured to filter for events like IAM policy changes or service account key creation.
+2.  **Log Storage**: The filtered logs are routed to a dedicated Google Cloud Storage (GCS) bucket for persistence and analysis.
+3.  **Report Generation**: A scheduled job runs weekly, executing the `log_analyzer.py` script.
+4.  **Email Alerts**: The script analyzes the logs from the past week, compiles a summary of security events, and sends a report to a configured email address.
+
+## Configuration
+
+The behavior of the log analyzer is controlled by a `config.yml` file. Here’s an overview of the configuration options:
+
+-   `project_id`: The GCP project ID where the resources are located.
+-   `bucket_name`: The name of the GCS bucket where logs will be stored.
+-   `logging`: Configures the logging level and format for the script.
+-   `sinks`: A list of log sinks to be created. Each sink has the following properties:
+    -   `name`: A unique name for the sink.
+    -   `description`: A brief description of what the sink monitors.
+    -   `filter_methods`: A list of GCP API methods to include in the filter (e.g., `SetIamPolicy`).
+    -   `excluded_principals`: A list of service accounts or user emails to exclude from monitoring, such as CI/CD service accounts.
+
+### Example Configuration (`config.yml`)
+
+```yaml
+project_id: your-gcp-project-id
+bucket_name: your-log-storage-bucket
+
+sinks:
+  - name: iam-policy-changes
+    description: Monitors changes to IAM policies.
+    filter_methods:
+      - "SetIamPolicy"
+    excluded_principals:
+      - "ci-cd-account@your-project.iam.gserviceaccount.com"
+```
+
+## Usage
+
+The `log_analyzer.py` script provides two main commands for managing the security analyzer.
+
+### Initializing Sinks
+
+To create or update the log sinks in GCP based on your `config.yml` file, run the following command:
+
+```bash
+python log_analyzer.py --config config.yml initialize
+```
+
+This command ensures that the log sinks are correctly configured to capture the desired security events.
+
+### Generating Weekly Reports
+
+To generate and send the weekly security report, run this command:
+
+```bash
+python log_analyzer.py --config config.yml generate-report
+```
+
+This is typically run as a scheduled job (GitHub Action) to automate the delivery of weekly security reports.
+
+
+

infra/security/config.yml

@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+project_id: testing-me-460223
+
+# Logging
+logging:
+  level: DEBUG
+  format: "[%(asctime)s] %(levelname)s: %(message)s"
+
+# gcloud storage bucket
+bucket_name: "testing-me-460223-tfstate"
+
+# GCP Log sinks
+sinks:
+  - name: iam-policy-changes
+    description: Monitors changes to IAM policies, excluding approved CI/CD service accounts.
+    filter_methods:
+      - "SetIamPolicy"
+    excluded_principals:
+      - beam-github-actions@apache-beam-testing.iam.gserviceaccount.com
+      - github-self-hosted-runners@apache-beam-testing.iam.gserviceaccount.com
+
+  - name: sa-key-management
+    description: Monitors creation and deletion of service account keys.
+    filter_methods:
+      - "google.iam.admin.v1.IAM.CreateServiceAccountKey"
+      - "google.iam.admin.v1.IAM.DeleteServiceAccountKey"
+    excluded_principals:
+      - beam-github-actions@apache-beam-testing.iam.gserviceaccount.com
+      - github-self-hosted-runners@apache-beam-testing.iam.gserviceaccount.com