r7insight_lambdaS3

Follow the instructions below to send logs stored on AWS S3 to Rapid 7.

IntelliJ Setup

1. Open the project in IntelliJ

2. Go to project settings and configure python sdk. You can create a new python sdk by creating a virtual environment option which creates a project level sdk rather than a global one. You end up with a venv folder in the project

3. Language level, select sdk default

Example use cases:

• Forwarding AWS ELB and CloudFront logs
  • (make sure to set ELB/CloudFront to write logs every 5 minutes)
  • When forwarding these logs, the script will format the log lines according to Rapid 7 KVP or JSON spec to make them easier to analyze
• Forwarding OpenDNS logs

Obtain log token

1. Log in to your Rapid 7 account

2. Add a new token based log

Deploy the script to AWS Lambda using AWS Console

1. Create a new Lambda function

2. Choose the appropriate Python blueprint for S3 objects

   

3. Configure function:

   • Give your function a name
   • Assign or create a new role that is allowing GetObject on necessary s3 bucket(s). Note: Creating a new one automatically from this UI adds required permissions automatically`

4. Configure triggers:

   • Choose the bucket log files are being stored in
   • Set event type "All object create events (All)"
   • Tick "Enable Trigger" checkbox

5. Click "Create function"

6. Upload function code:

   • Create a .ZIP file, containing r7insight_lambdas3.py and the folder certifi
     • Make sure the files and certifi folder are in the root of the ZIP archive
     • From the root folder run zip -r deploy.zip . -x '*.git*' '*.idea*' 'venv*' 'doc*' '.gitignore' 'README.md' 'LICENSE'
   • Go to "Function code" section of lambda setup
   • Choose "Upload a .zip file" in "Code entry type" dropdown and upload the archive created in previous step
   • Choose your python runtime version from "Runtime" dropdown
   • Set handler name to r7insight_lambdas3.lambda_handler

   Working example file structure:

    (root) - current AWS sets root directory name to your lambda function name automatically
    ├── certifi/
    │   ├── 
    │   ├── __init__.py
    │   ├── __main__.py
    │   ├── cacert.pem
    │   ├── core.py
    │   ├── old_root.pem
    │   └── weak.pem
    ├── r7insight_lambdas3.py`
    ``
   
   `Note: Zip files downloaded via github puts all the files under a subdirectory; thefore, uploading the downloaded zip directly to AWS will not work.`
   
   

7. Set Environment Variables:

   • Token value should match UUID provided by Rapid 7 UI or API
   • Region should be that of your R7 account

   Key	Value
   region	InsightOps region(eu,us etc)
   token	Token UUID

8. Advanced settings:

   • Set memory limit to a high enough value to facilitate log parsing and sending - adjust to your needs
   • Set timeout to a high enough value to facilitate log parsing and sending - adjust to your needs
   • Leave VPC value to "No VPC" as the script only needs S3 access
     • If you choose to use VPC, please consult Amazon Documentation

9. Enable function:

   • Click "Create function"

Gotchas

• The "Test" button execution in AWS Lambda will ALWAYS fail as the trigger is not provided by the built in test function. In order to verify, upload a sample file to source bucket

3rd party dependencies

• python-certifi