Ansible able to migrate database on upgrade

.gitignore

@@ -12,8 +12,6 @@ __pycache__/
 
 # C extensions
 *.so
-postmaster.db
-db/
 # Distribution / packaging
 .Python
 env/

.travis.yml

@@ -12,6 +12,7 @@ after_success:
   - coveralls
 
 before_deploy:
+  - python manage.py clean
   - gem install fpm
   - chmod 777 build_release.sh
   - ./build_release.sh

build_release.sh

@@ -1,4 +1,21 @@
 #!/bin/bash
-REVISION=`git describe --abbrev=0 --tags | cut -c 2-`
-fpm -s dir -t deb -n "postmaster" -v $REVISION --prefix /opt/postmaster/git --description 'PostMaster is a beautiful web application to manage domains, users, and aliases on a Linux mail server' --url 'https://github.com/StackFocus/PostMaster' --after-install ops/ansible/run_ansible.sh -d git -d python -d python-pip -d python-dev -d python-virtualenv -d libldap2-dev -d libssl-dev -d libsasl2-dev -d libffi-dev -d apache2 -d libapache2-mod-wsgi -d libmysqlclient-dev ./
+#REVISION=`git describe --abbrev=0 --tags | cut -c 2-`
+REVISION=1.4.0
+fpm -s dir -t deb -n "postmaster" -v $REVISION \
+--prefix /opt/postmaster/git \
+--description 'PostMaster is a beautiful web application to manage domains, users, and aliases on a Linux mail server' \
+--url 'https://github.com/StackFocus/PostMaster' \
+--maintainer 'StackFocus <hello@stackfocus.org>' \
+--vendor 'StackFocus' \
+--license 'AGPL' \
+--deb-changelog './docs/ChangeLog.md' \
+--after-install ops/ansible/ansible_install.sh \
+--after-upgrade ops/ansible/ansible_upgrade.sh \
+--exclude env \
+--exclude .git \
+--exclude .vagrant \
+--deb-no-default-config-files \
+-d git -d python -d python-pip -d python-dev -d python-virtualenv \
+-d libldap2-dev -d libssl-dev -d libsasl2-dev -d libffi-dev \
+-d apache2 -d libapache2-mod-wsgi -d libmysqlclient-dev ./
 ls -la | grep *.deb

config.py

@@ -16,7 +16,6 @@ class BaseConfiguration(object):
     SQLALCHEMY_TRACK_MODIFICATIONS = True
     SQLALCHEMY_DATABASE_URI = 'mysql://root:vagrant@localhost:3306/servermail'
     basedir = path.abspath(path.dirname(__file__))
-    SQLALCHEMY_MIGRATE_REPO = path.join(basedir, 'db/migrations')
     LOG_LOCATION = '/opt/postmaster/logs/postmaster.log'
 
 

docs/ChangeLog.md

@@ -2,19 +2,22 @@
 
 BACKWARDS INCOMPATIBILITIES / NOTES:
 
-* 'Log File' config option is now baked into the application config and cannot be set in the api / webui / database. Use `python manage.py setlogfile <path>` or edit config.py to change the log file location. See [GH-128].
+* The 'Log File' config option is now baked into the application config and cannot be set in the API/UI/database. Use `python manage.py setlogfile <path>` or edit config.py to change the log file location. See [GH-128].
+* `python manage.py createdb` has been replaced with `python manage.py upgradedb` [GH-138]
 
 Features:
 
 * Added the ability to install PostMaster via a deb package [GH-111]
+* Database upgrades/migrations are automatic during ugrades via the deb package and Docker [GH-138]
 
 Improvements:
 
+* Database migrations are now tracked via source control to ensure consistency across installations [GH-138]
 * Added additional documentation [GH-115]
 * Improved Active Directory authentication performance [GH-118]
 * Cleaned up JavaScript event listeners [GH-120]
 * Vagrant now uses Ansible for configuration instead of a bash script [GH-111]
-* Updated the Python packages to the latest versions [GH-127]
+* Updated the Python packages to the latest versions [GH-135]
 
 Bug Fixes:
 

docs/Configuration/CommandLineConfiguration.md

@@ -26,14 +26,14 @@ Use the following commands to restore the proper permissions on the PostMaster f
 
 **setdburi** sets the MySQL database URI that PostMaster uses to connect to the MySQL server used by your mail server.
 
-**createdb** runs a database migration, and configures the default configuration settings if they are missing on the database specified using the "setdburi" command.
-This is used during the installation of PostMaster.
+**upgradedb** upgrades the existing database to the latest schema version and adds the default configuration items if they are missing.
 
 **generatekey** replaces the secret key in config.py which is used by Flask (the Python framework used for PostMaster) for cryptographic functions.
 After the initial installation, this command should not be run again as all current logins would become invalid upon the next restart of the PostMaster.
 
 **runserver -d -h 0.0.0.0** runs PostMaster in debug mode on port 5000. This is useful if you are having issues as it bypasses the webserver
 and displays failing errors in HTML.
 
-**db [command] --directory 'db/migrations'** runs advanced database migration commands using the db/migrations directory.
+**db [command]** runs advanced database migration commands.
+It is recommended to use the wrapper commands listed above instead, however, in rare and advanced circumstances, these sets of commands may be necessary.
 For more information, visit the [Alembic API documentation](https://alembic.readthedocs.org/en/latest/api/commands.html).

docs/Installation/Docker.md

@@ -1,9 +1,4 @@
 ### Overview
-PostMaster is a Docker friendly application, however, it does require a data volume.
-This is because PostMaster uses database migrations to safely upgrade the database schema.
-The data volume contains auto-generated database migration scripts that allow you to revert back if a database migration ever failed.
-If this folder is missing, PostMaster can't tell what state your database is in, and therefore, cannot revert back.
-In this documentation, the data volume will be hosted in /opt/postmaster_data on the Docker host, but you can alter this in anyway you see fit.
 
 ### Prerequisites
 1. An Ubuntu 14.04 mail server configured with the guide from [DigitalOcean](https://www.digitalocean.com/community/tutorials/how-to-configure-a-mail-server-using-postfix-dovecot-mysql-and-spamassassin) or [Linode](https://www.linode.com/docs/email/postfix/email-with-postfix-dovecot-and-mysql).
@@ -32,35 +27,22 @@ bind-address is set 0.0.0.0 and not 127.0.0.1 in:
 
 ### Installation
 
-1. Switch to the Docker host, and login as root:
-
-        sudo su -
-
-2. As root, create the following folder on the Docker host to hold PostMaster's data volume:
-
-        mkdir /opt/postmaster_data
-
-3. Provide the appropriate permissions to PostMaster's data volume:
-
-        chown root:root /opt/postmaster_data && chmod 770 /opt/postmaster_data
-
-4. Download the PostMaster sourcecode from GitHub:
+1. Download the PostMaster sourcecode from GitHub:
 
         git clone https://github.com/StackFocus/PostMaster.git ~/postmaster
 
-5. Build the Docker image:
+2. Build the Docker image:
 
         cd ~/postmaster
         docker build -t postmaster .
 
-6. Run a PostMaster Docker container from the created image.
-The -v points to the data volume that was created in step 2.
+3. Run a PostMaster Docker container from the created image.
 The -p has the Docker host serve port 80 of the PostMaster container. Change this to what suits your environment.
 The -e specifies the value of the DB_URI environment variable, which is the URI that PostMaster will use to connect to your mail server's MySQL server.
 Make sure to replace 'password_changeme' and 'docker.postmaster.local' with what you configured in step 2 of MySQL Preparation:
 
-        docker run -p 0.0.0.0:80:8082 -v /opt/postmaster_data:/opt/postmaster/git/db \
+        docker run -p 0.0.0.0:80:8082 \
              -e DB_URI=mysql://postmasteruser:password_changeme@docker.postmaster.local:3306/servermail -d postmaster
 
-7. PostMaster should now be running. Simply use the username "admin" and the password "PostMaster" to login.
+4. PostMaster should now be running. Simply use the username "admin" and the password "PostMaster" to login.
 You can change your username and password from Manage -> Administrators.

docs/Installation/Ubuntu1404.md

@@ -51,7 +51,7 @@ replace '127.0.0.1' with the IP address or DNS specified in step 2 of MySQL Prep
 which means that only the necessary changes to the database are made, and these changes are reversible if something went wrong.
 To start the migration, run the following command:
 
-        python manage.py createdb
+        python manage.py upgradedb
 
 6. PostMaster uses a secret key for certain cryptographic functions. To generate a random key, run the following command:
 
@@ -117,7 +117,7 @@ replace '127.0.0.1' with the IP address or DNS specified in step 2 of MySQL Prep
 which means that only the necessary changes to the database are made, and these changes are reversible if something went wrong.
 To start the migration, run the following command:
 
-        python manage.py createdb
+        python manage.py upgradedb
 
 12. PostMaster uses a secret key for certain cryptographic functions. To generate a random key, run the following command:
 

docs/Installation/WindowsServer2012R2.md

@@ -60,7 +60,7 @@ bind-address is set 0.0.0.0 and not 127.0.0.1 in:
 9. Create a directory to contain PostMaster:
 
         New-Item "$env:SystemDrive\PostMaster" -Type Directory
-        
+
 9. Create a directory to contain PostMaster logs:
 
         New-Item "$env:SystemDrive\PostMaster\logs" -Type Directory
@@ -114,7 +114,7 @@ This is done with the following command:
 20. Install "mysqlclient":
 
         pip install "$env:SystemDrive\mysqlclient-1.3.7-cp27-none-win32.whl"
-        
+
 19. Remove the "mysqlclient" installation file:
 
         Remove-Item "$env:SystemDrive\mysqlclient-1.3.7-cp27-none-win32.whl" -Force
@@ -129,7 +129,7 @@ This is done with the following command:
 
 22. At this point, PostMaster requires an IIS site. You can either use the "Default Web Site" and change the virtual directory to C:\PostMaster\git,
 or create a new site that points to that directory. This tutorial will use the Default Web Site. To change the virtual directory, use the following commands:
-        
+
         Import-Module WebAdministration
         Set-ItemProperty 'IIS:\Sites\Default Web Site\' -Name physicalPath -Value "$env:SystemDrive\PostMaster\git"
 
@@ -149,12 +149,12 @@ replace '127.0.0.1' with the IP address or DNS specified in step 2 of MySQL Prep
 
         cd C:\Postmaster\git
         python manage.py setdburi 'mysql://postmasteruser:password_changeme@127.0.0.1:3306/servermail'
- 
+
 25. PostMaster needs to create a few tables under the servermail database. This is done via a database migration,
 which means that only the necessary changes to the database are made, and these changes are reversible if something went wrong.
 To start the migration, run the following command:
 
-        python manage.py createdb
+        python manage.py upgradedb
 
 26. PostMaster uses a secret key for certain cryptographic functions. To generate a random key, run the following command:
 

manage.py

@@ -26,13 +26,23 @@
 
 
 @manager.command
-def createdb():
-    """Runs the db init, db migrate, db upgrade commands automatically,
-    and adds the default configuration settings if they are missing"""
-    if not os.path.isdir('db/migrations'):
-        flask_migrate.init(directory=app.config['SQLALCHEMY_MIGRATE_REPO'])
-    flask_migrate.migrate(directory=app.config['SQLALCHEMY_MIGRATE_REPO'])
-    flask_migrate.upgrade(directory=app.config['SQLALCHEMY_MIGRATE_REPO'])
+def upgradedb():
+    """Upgrades the existing database to the latest schema and adds the
+    default configuration items if they are missing"""
+    alembic_version_table_exists = db.engine.dialect.has_table(db.session.connection(), 'alembic_version')
+
+    if not alembic_version_table_exists:
+        virtual_domains_table_exists = db.engine.dialect.has_table(db.session.connection(), 'virtual_domains')
+        virtual_users_table_exists = db.engine.dialect.has_table(db.session.connection(), 'virtual_users')
+        virtual_aliases_table_exists = db.engine.dialect.has_table(db.session.connection(), 'virtual_aliases')
+
+        # If the alembic_version table doesn't exist and the virtual_* tables exist, that means the database is
+        # in the default state after following the mail server guide on Linode or DigitalOcean.
+        if virtual_domains_table_exists and virtual_users_table_exists and virtual_aliases_table_exists:
+            # This marks the first revision as complete, which is the revision that creates the virtual_* tables
+            flask_migrate.stamp(revision='bcc85aaa7896')
+
+    flask_migrate.upgrade()
     add_default_configuration_settings()
 
 
@@ -47,14 +57,13 @@ def clean():
     """Cleans the codebase, including database migration scripts"""
     if os.name == 'nt':
         commands = ["powershell.exe -Command \"@('*.pyc', '*.pyo', '*~', '__pycache__') |  Foreach-Object { Get-ChildItem -Filter $_ -Recurse | Remove-Item -Recurse -Force }\"",  # pylint: disable=anomalous-backslash-in-string, line-too-long
-                    "powershell.exe -Command \"@('postmaster.db', 'db', 'postmaster.log') |  Foreach-Object { Get-ChildItem -Filter $_ | Remove-Item -Recurse -Force }\""]  # pylint: disable=anomalous-backslash-in-string, line-too-long
+                    "powershell.exe -Command \"@('postmaster.log') |  Foreach-Object { Get-ChildItem -Filter $_ | Remove-Item -Recurse -Force }\""]  # pylint: disable=anomalous-backslash-in-string, line-too-long
     else:
         commands = ["find . -name '*.pyc' -exec rm -f {} \;",  # pylint: disable=anomalous-backslash-in-string
                     "find . -name '*.pyo' -exec rm -f {} \;",  # pylint: disable=anomalous-backslash-in-string
                     "find . -name '*~' -exec rm -f {} \;",  # pylint: disable=anomalous-backslash-in-string
                     "find . -name '__pycache__' -exec rmdir {} \;",  # pylint: disable=anomalous-backslash-in-string
-                    "rm -f postmaster.db postmaster.log",
-                    "rm -rf db"]
+                    "rm -f postmaster.log"]
     for command in commands:
         os.system(command)
 

migrations/README

@@ -0,0 +1 @@
+Generic single-database configuration.

migrations/alembic.ini

@@ -0,0 +1,45 @@
+# A generic, single database configuration.
+
+[alembic]
+# template used to generate migration files
+# file_template = %%(rev)s_%%(slug)s
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

migrations/env.py

@@ -0,0 +1,87 @@
+from __future__ import with_statement
+from alembic import context
+from sqlalchemy import engine_from_config, pool
+from logging.config import fileConfig
+import logging
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers basically.
+fileConfig(config.config_file_name)
+logger = logging.getLogger('alembic.env')
+
+# add your model's MetaData object here
+# for 'autogenerate' support
+# from myapp import mymodel
+# target_metadata = mymodel.Base.metadata
+from flask import current_app
+config.set_main_option('sqlalchemy.url',
+                       current_app.config.get('SQLALCHEMY_DATABASE_URI'))
+target_metadata = current_app.extensions['migrate'].db.metadata
+
+# other values from the config, defined by the needs of env.py,
+# can be acquired:
+# my_important_option = config.get_main_option("my_important_option")
+# ... etc.
+
+
+def run_migrations_offline():
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL
+    and not an Engine, though an Engine is acceptable
+    here as well.  By skipping the Engine creation
+    we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the
+    script output.
+
+    """
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(url=url)
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online():
+    """Run migrations in 'online' mode.
+
+    In this scenario we need to create an Engine
+    and associate a connection with the context.
+
+    """
+
+    # this callback is used to prevent an auto-migration from being generated
+    # when there are no changes to the schema
+    # reference: http://alembic.readthedocs.org/en/latest/cookbook.html
+    def process_revision_directives(context, revision, directives):
+        if getattr(config.cmd_opts, 'autogenerate', False):
+            script = directives[0]
+            if script.upgrade_ops.is_empty():
+                directives[:] = []
+                logger.info('No changes in schema detected.')
+
+    engine = engine_from_config(config.get_section(config.config_ini_section),
+                                prefix='sqlalchemy.',
+                                poolclass=pool.NullPool)
+
+    connection = engine.connect()
+    context.configure(connection=connection,
+                      target_metadata=target_metadata,
+                      process_revision_directives=process_revision_directives,
+                      **current_app.extensions['migrate'].configure_args)
+
+    try:
+        with context.begin_transaction():
+            context.run_migrations()
+    finally:
+        connection.close()
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()