Sentry Breakpad Server

Hello, this a set of scripts used to act as an intermediate processor between a minidump file uploaded to a bucket, to Sentry, an open-source Web UI for storing and displaying error reports. It’s intended purpose is to be used to allow for a unified source to access, sort, and query crash reports/stack traces across multiple devices and builds. So when a device integrated with Google Breakpad crashes, it will create a minidump file and send it to an Amazon Bucket via Antenna, a web collector app developed by Mozilla. Antenna has been modified to then call this set of scripts which downloads the file, processes it into a stack trace with Breakpad (explained below), and then parses the stack trace and sends it to Sentry.

What you’ll need:

• Antenna (https://github.com/mozilla-services/antenna)

• An Amazon Bucket for Antenna to send files to (the default instance of Antenna sends it to a localstack, but we have the bucket set to breakpad-vivint-com)

• These Scripts

• The Google Breakpad library (https://github.com/google/breakpad) The Breakpad Library includes a library and executable to process the minidump files into stack traces

  • NOTE: I(Matthew) stripped down and deleted a lot of the files after cloned it into the this folder as we only needed the processer part, if you wish to move this folder outside or somewhere else or want a fresh install of Breakpad, you will have to change some directories within the scripts (should only be the ‘process_dump.py’ right now)

• A working Instance of Sentry
  Ours runs on a Centos Amazon Server, these were needed:
  - Postgresql (I implemented with 9.5)
  - Redis
  - A bunch of other dev packages that I can’t remember the name of…
  For help with Sentry creation…look online…here’s a general overview of what I did

  • Get Postgresql and make database with superuser

    $ node app
    $ psql template1
    $ create user [USERNAME] with password ['PWD']
    $ alter user [USERNAME] to superuser
    $ create database [DATEBASE] with owner [USERNAME]
    

  • Get Redis and start it (sudo systemctl start redis.service)
  • make python virtualenv

    pip install -U sentry
    

    - sentry init
    

    (THIS CREATES 2 CONFIG FILES (default /.sentry/) where most of Sentry should be configured)
    • In sentry.conf.py, Databases should be edited to correspond with the Postgresql info created above (default host and port are 127.0.0.1 and 5432 by default)
    • Both sentry.conf.py and config.yml should be edited for various purposes (file storage, email, etc.)

General Architecture/Overview/TO-DO:

1. Antenna is listening (default port 8000)

2. Devices crashes and sends compressed dump file to brkpad.vivint.com/submit via curl requests
   -TO DO [SUGGESTED]: add some authentication to this

3. Antenna calls breakpad_resource.py (in /antenna/, which calls crashstorage.py (in /antenna/ext/s3)

4. crashstorage.py saves files to bucket
   -TO DO [SUGGESTED]: review file naming and path method for personalized use

5. breakpad_resource.py then calls boto.py in sentry_breakpad_server

6. boto.py downloads file and calls process_dump.py

7. process_dump.py uncompresses file and uses Breakpad to process into readable stacktrace, and sends it to
   parse_for_sentry.py
   -TO DO [SUGGESTED]: Review/enable/disable how the code deletes file and/or moves them to another folder
   -TO DO [URGENT]: Figure how to integrate symbols, right now it just tries to look into empty folder ‘symfiles’
   -TO DO [URGENT]: Decide what to do with metadata, it might be in the stack trace itself, you may want to have other files in the tarfile which we will have to adjust for, have it in the filename, etc.
   -TO DO [IMPORTANT]: Project/DSN configuration: Right now, errors are just sent to a fixed project/DSN, but based on the various parameters of the crash, the build it came from, the model of camera, etc. you may want to create multiple projects or even teams
   -I suggest using the Sentry API (https://docs.sentry.io/api/) to handle this. There are plenty of documentation in order to DELETE, RETRIEVE, or CREATE new projects/DSNs
   -I’ve made a script (sentry_project_info.py), but it is no where near finished

8. parse_for_sentry.py parses stack trace for info, message, tags, extra, stack trace
   -TO DO [IMPORTANT]: Review…like all of this… how you want tags to be assigned, what you want the main message to be, etc. Basically how the info is displayed on Sentry
   -TO DO [IMPORTANT]: Do you want to have Sentry display where in the code the crash occurred, if so, you’ll need to make the source code accessible to the server and definitely review the parse_stacktrace() method

9. The info is uploaded to Sentry (once again, the DSN is right now fixed to an test Project, please change above)

TO DO [SUGGESTED]: make a supervisord.service file to automatically run sentry (I already made one in
/etc/supervisord.d/sentry.conf but please review)

TO DO [IMPORTANT]: Creating Tests/Projects, viewing error and most features seem to work fine, but when I try to change the settings of a project or delete it, it goes to another URL and times out. You can carry through with the process just by adding ‘https://’ to the URL, but this is not ideal haha. Since it involves the https:// thing I believe it has something to do with our encryption process (see nginx stuff) but I could be wrong. If problem is not a priority and/or it has yet to fix, you can delete project via terminal with API no problem

TO DO [IMPORTANT]: Configure file storage for Sentry, and like everything else to be able to handle production

Up and Running:

1. First set environment variable for Antenna and bucket:
   ANTENNA_ENV="/home/matthew.tung/antenna/dev.env" (subject to change if you move everything)
   AWS_ACCESS_KEY_ID="**************YO5Q"
   AWS_SECRET_ACCESS_KEY="**************xTdY"
   (you can permanently set the last two with ‘aws configure’

2. cd into /antenna/ and start Antenna with: “./bin/run_web.sh”
   -if it throws and error you may have to: “sudo make build”

3. Antenna should now be running, periodically giving heartbeat messages

4. If you haven’t configured the server for ‘sudo systemctl start supervisord.service’, then in 3
   terminals source the virtualenv:

   source /sentry_app/bin/activate
   

   Then run:

     sentry run cron
     sentry run worker
     sentry run web
   

5. Everything should be good to go now! For us go to "https://brkpad.vivint.com/sentry/" to see!

Random Stuff:

Wanna change how stuff looks and is sorted in the bucket?
look at /antenna/antenna/ext/s3/crashstorage.py !

Wanna download or wipe everything from said bucket?
/sentry_breakpad/server/boto.py has functions for that

Wanna change where dump files are stored in server before and after?
Look at boto.py and process_dump.py

Dealing with dump file meta data, not producing a processed stacktrace?
Look at process_dump.py

Stuff is going to Sentry, but it’s not in the right project?
Look at process_dump.py and maybe parse_for_sentry.py

Stuff is going to Sentry, but it’s showing the data how you like it?
Look at parse_for_sentry.py

Sentry starting to crash or not be able to handle all the reports?
Look at config files in ‘/.sentry/’

In general, I think Sentry is a great tool to use for this purpose because it’s well-fleshed in terms of product and documentation and it is fairly versatile in how you can display reports. There is talk on the forums about sentry soon be able to support minidumps better, but right now, you’re stuck with parse_for_sentry.py :)