Documentation of Configuration Management

[Drupaldise]

Hi, where can I find some docs about the configuration management?
How can i work with the staging dir in files/config/staging - or what is this?
What is the best practice to work with GIT (or other version control) with my exported config files?
In other words how can I deploy my config changes without uploading a file - instead using the filesystem?

Thanks for enlighten me. ;-)

[quicksketch]

Hi @Drupaldise! We have an article coming out in Drupal Watchdog next month that includes some details, but I think generally speaking we don't have documentation on managing config via the file system. We need to figure out even where this documentation would go between the API docs and the Backdrop handbook, but generally I think I can sum it up. Perhaps once we have a place to put it, we can get something official written up.

---

If you're going to be managing configuration outside the user interface (such as through version control or even FTP), you should move the config directories outside of the Backdrop "files" directory. Backdrop by default will put the configuration files in a randomly generated /files/config_[hash] directory, and points settings.php at this path like this:

$config_directories['active'] = 'files/config_b7c6057f5199d7aa921445a3d31d30df/active';
$config_directories['staging'] = 'files/config_b7c6057f5199d7aa921445a3d31d30df/staging';

Because the "files" directory is generally not in version control, you can make managing your config easier (and more secure) if this directory is moved outside the web directory. Generally that means up one level in the file system:

$config_directories['active'] = '../config/active';
$config_directories['staging'] = '../config/staging';

And you'll need to organize your Git repository or site directory to match this structure, so it would be something like:

• .git
• .gitignore
• config
  • active
  • staging
  • versioned (more on this later)
• docroot
  • .htaccess
  • core
  • files
  • layouts
  • modules
  • ... etc. (the Backdrop installation).

I included the .gitignore file in here because you'll probably want to make a few entries in this file. I'd suggest ignoring both the "files" directory and the active and staging config directories individually:

/config/active
/config/staging
/docroot/files

The key to all management of config is the "staging" directory. This directory will be empty all the time unless you're actively moving configuration. We're also introducing our own "versioned" directory that we'll manage in Git.

On your localhost or development machine. Configure your site as you would want to deploy to live. Then to deploy your changes simply copy the "active" files into the "versioned" directory, then check it into Git.

cp config/active/* config/versioned/
git add versioned -A
git commit -m "Updating config files."
git push origin master

Then on your live server, copy the files back out of "versioned" directory into the "staging" (NOT the "active") directory:

ssh me@example.com
git pull origin master
cp config/versioned/* config/staging/

You've now "staged" your configuration for deployment, but it's not yet active. You have to let Backdrop import the configuration to allow it to make necessary database changes and validate the configuration is acceptable (e.g. won't accidentally delete in-use fields).

So visit admin/config/development/configuration in your Backdrop site, where you should now see the staged changes. You can view the differences in each file that was modified, deleted, or added. Click the "Import all" button. If no validation errors occurred, the config will be imported and active. Now all menus, content types, theme settings, etc. that you configured on your localhost will be reflected on the live site.

In the future, we'll probably have a command-line tool (#47) to do this last step without the UI, but it's a good idea to use the UI to view the file diffs anyway.

Note that after you do the import, your "staged" directory will be empty again. Ready for the next time you do a deployment and copy from the "versioned" directory. If you would like to pull down the configuration from live, you can simply do the same process of copying the active directory to versioned, checking it in, pulling down on your local, copying to staging, then importing.

---

There may be other approaches that work as well. We've theorized ideas like managing the "staging" directory all the time, or a combination of symlinks, but so far I think the "just make another directory" for versioned config is the easiest approach we've encountered.

[jenlampton]

@quicksketch That's an interesting approach. I'm going to outline this approach below, along with the two others, in more detail for a better compare / contrast for anyone curious.

The extra directory approach

Configuration of config directory

config/
config/active
config/staging
config/versioned <-- track via Git

Additional set up

1. Move config outside of files (and preferably outside the docroot)
2. Update settings.php to point at the new locations of active & staging

Deploying to prod - moving upstream

1. local: overwrite versioned directory with active (deletions too!)
2. local: add, commit, push via Git
3. production: pull changes
4. production: copy files
5. production: run config importer (via UI)

Sync local from prod - moving downstream

1. production: overwrite versioned directory with active (deletions too!)
2. production: add, commit, push via Git
3. local: pull changes
4. local: copy files
5. local: run config importer (via UI)

Upsides:

1. All files tracked via Git will remain "up to date" after importing staged config

Downsides:

1. Accidental reverts are possible - see (1) below

There are two other approaches we've also been considering:

The versioned staging directory approach

This an idea from @populist and @davidstrauss based on how the config directory is being managed for Drupal 8 on Pantheon.

Configuration of config directory

config/
config/active
config/staging <-- track via Git

Additional set up

1. Move config outside of files (and preferably outside the docroot)
2. Update settings.php to point at the new locations of active & staging

Deploying to prod - moving upstream

1. local: overwrite staging directory with active (deletions too!)
2. local: add, commit, push via Git
3. production: pull changes
4. production: run config importer (via UI)

Sync local from prod - moving downstream

1. production: overwrite staging directory with active (deletions too!)
2. production: add, commit, push via Git
3. local: pull changes
4. local: run config importer (via UI)

Upsides:

1. No extra directories needed
2. One less step (than above) when moving upstream & downstream
3. When deployed, the config changes would "land" in your staging directory on production, ready to be imported.

Downsides:

1. Accidental reverts are possible - see (1) below
2. Git reports staging directory as always "out of date" - see (2) below

And this approach is my personal favorite (and what we'll be recommending in the Drupal Watchdog article, out this May):

The figure-8 approach

This approach was recommended by @heyrocker after creating CMI for Drupal 8. It involves using your settings.php file to define different active and staging directories for each environment.

Configuration of config directory

config/
config/dev-active <-- track via Git
config/prod-active <-- track via Git

Additional set up

1. Move config outside of files (and preferably outside the docroot)
2. Rename config/active to something like config/prod-active
3. rename config/staging to something like config/dev-active.
4. On your local site, update settings.php to point at dev-active for active, and prod-active for staging.
5. On your production site, update settings.php to point at prod-active for active, and dev-active for staging.

Deploying to prod - moving upstream

1. local: add & commit all files in config exactly as they are. push.
2. production: pull changes
3. production: run config importer (via UI)

When you pull upstream, the config files from what was the active directory on local land in what is now the staging directory on production - ready for import.

Sync local from prod - moving downstream

1. production: add & commit all files in config exactly as they are. push.
2. local: pull changes
3. local: run config importer (via UI)

When you pull downstream, the config files from what was the active directory on production land in what is now the staging directory on local - ready for import.

Upsides:

1. Once set up, you never need to think about which directory is which!
2. No extra directories needed
3. One more less step to move both upstream & downstream
4. Commit things as they are, no extra moving files around.

Downsides:

1. Git reports the (staging) directory as always "out of date" - see (2) below
2. Figuring this out may make your head explode

Here is a more detailed explanation of the two downsides mentioned above:

(1) ACCIDENTAL REVERTS ARE POSSIBLE: If there was ever a config change made on production and not actively copied out of the active directory and committed wherever it's supposed to be tracked, the next time code is deployed that production change would be reverted.

(2) GIT SHOWS STATUS: OUT OF DATE. Git is tracking and reporting all the deleted config files in the staging directory. Running a Git status check on any environment will show the deleted files. Developers may get used to seeing deleted files in staging, and train themselves to ignore those warnings. When a config file is actually deleted BY MISTAKE this is very likely to be overlooked!

[Drupaldise]

Hey Guys, thanks for pointing this out. It makes it more clear for me now. In my opinion there must be a topic for that on backdropcms.org Handbook. This feature is so important.

[heyrocker]

Note that my 'figure 8' idea stems from the very early stages of CMI, and I don't think it translates well to the way that the modern system (in Drupal or Backdrop) is architected and the thought-work that has gone into it since then. I hadn't considered the third directory option. It's a solid solution but I wish we had something a little more seamless. The whole thing of git status being out of date never really bothered me.

[jenlampton]

@heyrocker what is it about the figure-8 that doesn't translate well to how Drupal 8 & Backdrop are architected? What is it about the other two options that translates better? I still prefer the figure-8 approach since it removes the need for copying anything anywhere (and needing to carefully think about where to copy it). Also, this set-up lends itself nicely to having more than 2 environments (like on Pantheon or Acquia) just by adding additional directories.

Config	Local	Dev	Test	Live
jen-active	deploy
dev-active		deploy
test-active			deploy
live-active				deploy

[mikemccaffrey]

Move config outside of files (and preferably outside the docroot)

I'm pretty sure this is not possible when you are hosting on Pantheon, since the docroot is also the root of the git repository so you can't put it outside of that, and files is the only area that the web server has permission to update.

So, am I correct in assuming that with Pantheon, you need to keep the config files in the hashed directory in files? Has anyone figured out a way around that?

[quicksketch]

So, am I correct in assuming that with Pantheon, you need to keep the config files in the hashed directory in files? Has anyone figured out a way around that?

You could just add an .htaccess file to prevent it from being accessed directly. We're planning on adding one by default in the future over in #706. However I think the hash will stay either way, for sites that are not running Apache for basic security.

[mikemccaffrey]

FYI, the approach I ended up using on Pantheon was to leave the active config directory in its default location under the files folder where the pantheon web server can make changes, but to put the staging config directory in the site root under version control.

This actually enforces a workflow that I prefer to the default one where the contents of the staging directory are emptied after import. Since backdrop can't delete the files in the staging directory if they are under version control, I can go to the config management page at any time and see all the active config changes that have been made by users since the last code release, just like how with features you can see what has been overridden from the version in code.

The only messy part is how it generates dozens of "Warning: unlink() permission denied" errors after you hit Import All. We should get CMI to play nice with version controlled config directories, perhaps as part of the 2.x plans in #435.

[quicksketch]

The only messy part is how it generates dozens of "Warning: unlink() permission denied" errors after you hit Import All.

We could gracefully handle this one situation. Just suppress errors from unlink with the @ symbol and display a single message instead.

$deleted_all_files = TRUE;
foreach ($config_files as $file) {
  $deleted_all_files = @unlink($file) && $deleted_all_files;
}
if (!$deleted_all_files) {
  backdrop_set_message(t('Failed to deleted some configuration files from the staging directory.'));
}

[heyrocker]

Is there an argument to be made that you shouldn't delete after import at all? I know this argument was made during dev on d8 and i was against it, but i might be warming to it now.

[heyrocker]

I was trying to remember back to how this whole thing came about, and here's a discussion of the problem.

https://www.drupal.org/node/1697256#comment-6641656

It really stems from "How do you stage deletes?" When a config file previously existed and then disappears, the config system sees this as a delete. There are situations where this can get totally foobar'd if your import directory is not cleaned out after every use. Here is an example:

1. You import a bunch of stuff, and then your import is left behind.
2. In your active directory you delete a field.
3. You copy your config from active to import.
4. Your deleted field is still there, because you just copied files from one place to another.
5. When you import your field doesn't get deleted.

Now sure, you can be extremely careful when doing that copy from dir to dir, but we were trying to figure out how to manage that in a more foolproof manner. Note that the management of how to stage deletes was easily the most complicated problem we had to solve in the base system. You can read that entire thread for tons of alternate approaches we tried and finally abandoned.

Just wanted to throw that background out there, for people who might be thinking about trying to work around that directory emptying behavior.

[mikemccaffrey]

We could gracefully handle this one situation. Just suppress errors from unlink with the @ symbol and display a single message instead.

Or we could just check if the system has write permission in the staging directory, and present a single clear notice such as "backdrop cannot automatically remove the staging files".

It really stems from "How do you stage deletes?" When a config file previously existed and then disappears, the config system sees this as a delete. There are situations where this can get totally foobar'd if your import directory is not cleaned out after every use.

This is definitely a problem if the user is manually copying config files around, but should not come up if the config is in version control. We may want to think about how to handle each of these two very different use cases in their own fashion.