Update manager in D7: What's done and what's next

It's been quiet on the 3281d blog for a while, but not because of a lack of interesting things happening. Quite the contrary, I've been so busy that I haven't made time to write. To break the silence, I'm going to start with a status report about the Update manager in D7 core -- Drupal can now (securely) update itself when new releases are available if you tell it to! It's an incredible new feature that will remove a major source of pain for a large set of Drupal users and help keep more Drupal sites running the latest, most secure releases. However, there are a lot of rough edges to fix before the 7.0 release, and help is urgently needed to get this feature fully fleshed out in time.

In this report, you'll find:

Screenshot of the Update manager update selection page

Background

Although this effort has been brewing for months (depending on how you count, years) with a lot of different people contributing, it's really only due to the heroic efforts of Jacob Singh that it's in core today. Jacob shepherded the feature through numerous issues in the core issue queue, many different iterations of the architecture, and even weathered a pretty serious last minute storm. However, Jacob was running out of steam, and not getting the help he kept asking for, and things were starting to look bleak for the code slush deadline (this was one of the 10 critical features on the code freeze exception list). That's when I fully jumped in.

As the maintainer of the Update status module in Drupal 6 core (and the update_status contributed module for Drupal 5), and as a member of the Drupal Security Team that has a keen interest in seeing sites keep up to date with security releases, I wanted to make sure this feature made it in, was done right, was as secure as possible, and was integrated properly with the rest of Update status. I had partly reviewed earlier iterations (even throwing some of the curve balls that Jacob had to deal with), but I never took the issue and ran with it until October 11th, 4 days before the code slush freeze. At that point, I set out to figure out exactly how the feature needed to work, closely review every line of the patch, identify what all the moving parts where, how they should fit together, and where all the code should live. I ended up reorganizing or rewriting most of the patch over the next 4 days, and the result is something to be proud of. With much fanfare (although I'm not on Twitter, sources told me "the tweet-o-sphere went wild", and IRC was certainly ablaze with cheers), webchick committed it to Drupal 7 on the afternoon of the final freeze deadline on October 15th. Phew!

How it works

The module formerly known as "Update status" is now the "Update manager" in Drupal 7. The previous "Available updates" report you know and love (or maybe hate) is still there. However, there are now two new screens, one for updating your site, and another for installing new modules or themes.

Updating existing modules and themes

To update existing modules on your site, there is now a page that only lists missing updates with checkboxes to select which ones to update (see the screenshot at the top of this post). After you select what to update or install, the Update manager downloads the necessary releases, extracts them into a temporary folder, and verifies them. If all goes well, you see:

Screenshot of the Update manager landing page before upgrading
Note, this screenshot is based on a patch at #617588 -- what's in core at the time of this writing is a bit different).

Once you confirm you're ready to do the update, you're redirected out of your Drupal site and onto the new authorize.php script which performs the actual updates (see below).

Installing new modules and themes

The other new interface from the Update manager is a page to install new modules and themes:

Screenshot of the Update manager new installation page
Note: this screenshot is likely to change based on #614288

Once you upload an archive or paste in the URL, the Update manager downloads, extracts, and verifies the release, and if all goes well, redirects you to the authorize.php script to install the new module or theme.

The new authorize.php script

The authorize.php script lives at the top of the web root directory (next to index.php, update.php, etc) and plays a few very critical roles in the Update manager workflow. When you first land on this script, it prompts you for the credentials to connect to your site (either via SSH or FTP) so that it can perform the updates as you, the user that owns all of the source code files for your Drupal site, not the user that the web server is running as. In some configurations, these are actually the same user, which is less secure and generally a bad idea, but in that case, the Update manager skips the step of prompting you for connection details and will just write the files directly. Your password is never stored, just used while authorize.php is moving the new releases into place. If you're using OSX, when Software Update runs, it usually needs to prompt you for the administrative password on your machine to install the updates. This is the equivalent step for your Drupal installation.

Screenshot of the authorize.php connect settings page

The other important thing about authorize.php is that it runs at a very low Drupal bootstrap level, meaning that only a small set of core functionality is loaded, and no contributed modules are loaded at all. This way, if a new release is broken, the update manager can still operate and could potentially help you recover from the failed update. It's also more secure, since it's nearly impossible with this system for any cross site scripting attacks to capture your FTP/SSH credentials.

Once you've authorized the Update manager to upgrade your site, it connects to the filesystem as you, moves the new files into place, and then you see a landing page that tells you what happened and reports any errors that occurred.

How to make it go away

Although this feature is going to be a life-saver for many users with a single site on shared hosting, it's not a great thing if you have a "real" website with development, staging, production servers and a careful deployment workflow. It's not what you want if you're managing your sites via Aegir or if you're using drush. For all of these people, there's an easy way to turn off the Update manager entirely. There's a setting you can uncomment in the default.settings.php file that ships with Drupal 7 core:

$conf['allow_authorize_operations'] = FALSE;

If that is set to FALSE, the Available updates report (what I still think of as "Update status") will be there, but none of this new functionality will be accessible anywhere on the site. The new Update manager pages won't be in the site navigation at all, and authorize.php will be unavailable. If you try to visit any of these, even as the super user account (user ID 1), you'll get access denied.

What's left?

Sounds great, huh? Well, there's still a lot of work left to do... We're tracking all the issues with the Update manager issue tag. While that might look like a pretty impressive "sea of green" (and in some ways, it is), some of the hardest issues are the ones that are left. Of particular interest:

  • #606592: Allow upgrading core with the update manager. Currently, Update manager can only update contributed modules and themes. There are a lot of additional complications for updating core itself. I've written up most of the problems in the issue, but this needs interested parties to start trying to solve them.
  • 606190#: Fix handling of database schema updates in update manager workflow. Once upon a time, the idea was that after your code was updated, if the new release contained any database schema changes, the Update manager would automatically redirect you to update.php or perhaps even just use the same functions to run the updates for you. There were some complications with this approach that threatened to derail getting the feature into core at all by the deadline, but if we can work those out now, we can still fix this before the December 1st UI freeze deadline.
  • #616744: Add a UI for if the site supports https. Thankfully, #607008: Fix bugs in https support and force using https for authorize.php if available is already committed, but the bigger problem of D7's https handling still needs to be tackled. Sending your SSH password over http is a really bad idea, so it'd be great if D7 auto-detected if https is available and configured itself to use it for things like authorize.php, the user login forms, etc.
  • #605276: Add tests for the update manager. Unfortunately, some parts of this system are going to be very hard or impossible to test automatically given our current testing framework. If each test was running in a sandboxed virtual host and we could do crazy things as root, we'd be able to simulate a lot more of the cases with file ownership, run an ftpd and an sshd, and so on. So, functional tests aren't going to work very well here. However, there are a number of new classes that drive a lot of this functionality, and we can write tests that extend these classes to allow unit testing of individual pieces. See #602544: Write unit tests for the Updater classes for more. So, even though we won't test a connection over SSH, we'll at least test everything that's depending on a connection.
  • #609772: authorize.php connection settings form doesn't degrade without JS. This form went through a few different designs and approaches, and the code that finally landed in core doesn't actually work if you have JS disabled. :(
  • #605272: Use a better directory when installing code via the update manager. Right now, the newly installed files are always placed in the site-specific configuration directory (e.g. sites/default or sites/[hostname]) not in sites/all. This is because things get very complicated if you're trying to use the Update manager on a multi-site installation. A healthy discussion is brewing in the issue, but this needs more thought and code.
  • #418302: Securely copy default.settings.php to settings.php during install. Although this isn't technically part of the Update manager itself, this is an issue to reuse a lot of the same code and mechanisms to fix one of the worst Drupal UI WTFs we have: the completely bizarro step during the installer where it asks you to manually log into your site copy a file to a new location, make it world-writable, and then after the installer writes something to it, it asks you to chmod the file back to a more secure file mode. W.T.F...? There are (sort of) "good" reasons why it does this, but now that we have a way to safely access files as the user, we could do all this craziness for them.
  • #509010: Build a Package signing system for d.o. so that people can securely and simply download and verify packages. All the places in this post where I say "... and verifies them", I really just mean "invokes a hook that would let modules verify them. :( We don't actually have a package signing system in place, and verifying the md5 hashes already provided by drupal.org isn't happening, either.

Of course, there are a lot of smaller issues in the queue, and probably some others that are going to be opened over the next few weeks and months. But, the goal is to fix as much of this as possible before the December 1st UI freeze, since after that, it's going to be much harder to make any more improvements in any of this.

Even if you're not a developer or UI designer and you want to help -- please try out the system on a test site and let us know your experiences. As always, please search for an existing issue before submitting a new one.

See you in the issue queue or in IRC!

Thanks,
-Derek (dww)

Comments

Now that's what I call Update

Now that's what I call Update manager. Awesome!

Public Key SSH

What if your server has FTP disabled and SSH is set to public key pair encryption only? Entering a username and password are meaningless.

Public key still needs access to your private key...

looks promising

Thanks for update on this, I was curious about it.

I didn't know you could install themes/modules through this also, that will surely help out a lot of new people, or people without the right acces, or not knowing how to use an ftp.

Great stuf!

A great step foward...

This feature hasn't got enough praise for what it is. I've been using it to do updates for some of my D7 experiments, and especially at the moment as modules are moving fast to keep up with core, it is really great. It also cuts down the time that the site needs to be in maintenance mode because it is all automated!

Great work.