Tue, 31 Jan 2012

GNOME Shell 3.2 in wheezy: a retrospective

When you read this, GNOME Shell 3.2 will (hopefully!) have finally transitioned to Debian’s testing suite.

Planet GNOME readers might think Debian now has outdated versions of software even in their development versions, or the distribution’s development marches at glacial pace. Wheezy GNOME users will finally have a Shell that matches the rest of their GNOME components, something that works with the Shell extensions website and much less problems and limitations compared to 3.0.2.

The reality is that GNOME 3.2’s packaging was quite ready back when it was released in late September, but a number of not-so-desirable situations held GNOME Shell from transitioning to testing until today, four months later. So, what happened?

TL;DR: transitioning from GNOME 2 → GNOME 3 is not so easy if you want to keep testing in a sane state, and when you need to deal with dozens of indirectly related packages, for more than 10 architectures… but it shouldn’t take nearly a full year, either…

Let’s go back to the last months of 2010. Debian squeeze is in very deep freeze, and the release team and many Debian developers are focusing on squashing as many release critical bugs as they can, in order to make Debian 6.0 the great release it ended up being. The GNOME project has recently delayed the big launch of GNOME 3.0 again, until March 2011; Debian has already settled on GNOME 2.28 for its release, although it will end up cherry-picking many updates from the 2.30 release modules.

With most of the stabilization work being done, many Debian GNOME team members were at that time working on packaging very early versions of what would end up being GNOME 3.0 technology: GTK+3.0, GNOME Shell, Mutter… and some brave users even tried to use it via the experimental archive.

On February 6th, Debian 6.0 was released, and soon after, on April 6th, GNOME made a huge step forward with the much anticipated release of GNOME 3.0. At that time, Debian developers were busy breaking unstable as much as they could, as it’s tradition on the weeks following a major release, and the Debian GNOME team was able to start moving some GNOME 3.0 libraries (those which were parallel-installable with their GTK+2.0 versions) to unstable.

However, moving the bulk of GNOME 3.0 to unstable wasn’t so easy. When you start doing that, you need to be sure you’re ready to have all affected packages in a “transitionable” state as soon as possible, to minimise the chances of blocking transitions of unrelated packages via the dependencies they pick up with rebuilds. All the packages involved in a transition need to be ready to go in the same “testing run”, for all supported architectures. When you’re dealing with dozens of GNOME source packages at the same time, many of which introduce new libraries, or worse, introduce incompatible APIs that affect many more unrelated packages, things get hairy, and you need a plan.

So, Joss outlined what a sane approach to this monster transition could look like. The amount of work to do was what we call “fun” on #debian-gnome. In a nutshell, we had to deal with quite a few transitions, starting with having a newer version of libnotify in unstable, and a pre-requisite for that was making sure all the packages using libnotify1 were ready to use the source-incompatible libnotify4, and this meant preparing patches and NMUs for many of our packages, as well as many others not under our control.

Before starting a controlled transition like this one, we had to get an ACK from the release team, who was busy enough handling other huge transitions like Perl 5.12, so by the time we got our own slot, we were well into Summer.

With libnotify done in August, it was time to get our hands dirty with more exciting stuff, like getting Nautilus in testing. This meant bumping a soname and requiring all packages providing Nautilus extensions to migrate to GTK+3.0, or drop the extension entirely, as you can’t mix GTK+2.0 and GTK+3.0 symbols in the same process. However, in GNOME 3.0, automounting code had moved from Nautilus to gnome-settings-daemon, so in order to not break filesystem automounting in testing for an unreasonable amount of time, both Nautilus and g-s-d needed to go in at the same time. The fun thing is that g-s-d dragged glib2.0, gvfs, gnome-control-center, gdm3, gnome-media, gnome-session and gnome-panel into the equation, so this transition needed extra planning and a lot more work than initially expected: migrating all nautilus extensions, plus ensuring all Panel applets had migrated to GTK+3.0 and the new libpanel-applet-4 interface. In short, this was the monster transition we were trying to avoid.

By the time all this mess was sorted out, GNOME 3.2 had been released, and for what users said, it was a lot better than 3.0. We still had no more than a few bits and pieces of 3.0 in testing, and we were working hard to get 3.0 in wheezy. With all the excitement around 3.2, at times it was difficult to explain outsiders why we were beating a dead 3.0 horse… Going back to our huge transition, it was just a matter of time before all the packages would be built and be ready to enter, on the same run, in testing.

A few weeks later, in early November and after several rounds of mass-bug-filings, fixing unrelated FTBFS, many NMUs, package removal requests and dealing with any possible problem that could block our transition, everything seemed to be set, and our release team magicians had everything in place for the big magic to happen. However, our first clash with the rest of Debian happened a few hours before our victory, in the form of an unannounced ruby-gnome2 upload which resetted the count for everyone. It was fun to see the release team trying all sorts of black magic in an attempt to mitigate the damage. Fortunately, after a few tries they managed to fool britney (the script that handles package transitions from unstable to testing) somehow, and the hardest part of the job was done with just one day of delay.

At last, the core of GNOME 3 was in testing, and testing users found soon after. The rest of the week saw a cascade of hate posts against GNOME 3 in Planet Debian, and personally I didn’t find that especially motivating to keep on working on the rest of GNOME bits. With experimental clear of GNOME 3.0 stuff, we finally were able to focus on packaging whatever GNOME 3.2 components were not already done, and preparing for what should be a plain simple transition of GNOME 3.0 to 3.2.

After our share of wait for a transition slot, as Perl 5.14, ICU and OpenSSL were in the line before us, and after dealing with a minor tracker 0.12 transition, we were ready for our next episode: evolution-data-server.

At first sight, we thought this would be a lot easier, but it still got a bit hairy due to evo-data-server massive soname bumps. We were given our slot just before Christmas, after a few weeks of wait for others to finish their migration rounds, and most of the pack entered wheezy a few days before the new year.

No rejoicing, though, as GNOME Shell 3.2 didn’t make it. First, we discovered it was FTBFS on kFreeBSD architectures, as NetworkManager had been promoted from optional to required, for apparently no good reason, leaving the BSD world in the cold, including our exotic GNU/kFreeBSD architectures. Now, let’s clarify that I’m a supporter of the Debian kFreeBSD architectures and was really happy to see it accepted as a technology preview in squeeze. However, as you know, GNOME Shell currently requires hardware acceleration to run, a requirement hardly met in kFreeBSD, unless you’re using a DRI1 X driver. We seriously doubted anyone had ever ran a GNOME 3 session on kfreebsd-*. However, if it didn’t build, it was a blocker bug for GNOME Shell. We considered creating different meta-packages for kFreeBSD architectures, to conclude it’d be a mess, so our awesome Michael Biebl ended up cooking up a patch that restored the ability to build the Shell without NetworkManager support.

With this out of our way, we just needed to upload Michael’s fix and watch the buildds do their part of the job. Or maybe not?

Enter Iceweasel 9.

In parallel, and with incredible bad timing, Iceweasel 9.0 was uploaded to Debian the very same day it was released by Mozilla. Again, it greeted us with a nasty surprise: yet another mozjs API change, which made gjs FTBFS, which meant our kFreeBSD fixes would be unusable until someone who knew Gjs’ internals well enough bit the bullet and worked around the new API changes. Again, Michael Biebl tried to be our saviour, but unfortunately wasn’t able to fix all the problems, so we tried to focus on plan B.

Mozilla had released a fork of the mozjs that is included in Firefox, so that embedders would have a bit less of a hard time with these recurrent API changes. This was based on Firefox 4, and was already being packaged by Ubuntu. Gjs would build using this older version just fine, so we just needed to get it in Debian as soon as possible. We just needed to find a sucke^Wvolunteer that would be inclined to maintain the beast. Only after a few weeks we managed to get Chris Coulson, the Ubuntu packager, to maintain the package directly through the Debian archive via package syncs. However, his package had only been auto-compiled in the three Ubuntu architectures, that is amd64, armel and i386. It’s late January 2012, and we’ve been fighting this war for 10 months.

After getting some help from Michael to get the new package in shape for Debian standards, we were excited to sponsor it for Chris. Duh, after a few days in the NEW fridge, it was rejected by the ftp-masters. The license statement was missing quite a few details, so I went ahead and sacrificed a few hours of my copious free time to get this sorted out. A few days later, mozjs was accepted, but the result was horrible. It was very red. mozjs didn’t build on half of our targets.

Mike Hommey was quick to file a bug and point us to the most obvious fuckups. As he had dealt with this in the past as the Iceweasel maintainer, all of these issues were fixed and patches were ready to be applied verbatim or with minimal changes to our sources. With mozjs finally built successfully (although with severe problems on ia64), we were finally able to rebuild Gjs against it, upload GNOME Shell with our kFreeBSD fixes and wait until today for this mess to be over. Whew.

I can’t say I’ve enjoyed all the stages of this ride. Some bumps on the road were clearly there to test our patience, but it has helped me get back in touch with non-leaf GNOME packaging, which was all I was doing for a while due to being super-busy lately with studies. It also reminds me of the privilege of working side by side with some awesome people, not only Joss, Michael, Sjoerd, Laurent or Gustavo, to name just a few Debian GNOME team members, but also the receptive release team members like Julien or Cyril, and NEW-processing record-breaking ftp-master Luca. Without them, we might be trying to figure out the Nautilus transition since last Summer.

We really hope GNOME 3.4 will be a piece of cake compared to this. ;)

Well, as someone that's been along the ride since around march last year (in experimental), a big thank you to the whole team.

I knew it couldn't have been easy, but it's still good to read all the troubles you've faced.

Considering I was using an unholy mix of sid and experimental, I've had surprisingly little breakage. It all felt quite solid.

So, thanks.

Posted by nona at Tue Jan 31 04:38:49 2012

Let me say "Thank you!". It was huge effort. I thought that the packaging was difficult, but honestly, I've never thought that it was so complicated.

Kudos to the gnome packaging team.

Posted by Felipe Reyes at Tue Jan 31 06:07:36 2012

Your post is very revealing and inspiring, thanks a lot for this and for all of your work — which I knew it was hard, but never imagined this!

Posted by Faidon Liambotis at Tue Jan 31 06:32:33 2012

Thank you so much for your incredible work.

Posted by Luca at Tue Jan 31 07:14:37 2012

Respect to you and the team. Thanks for all your work!

Posted by Frank Hart at Tue Jan 31 09:07:45 2012

How did Ubuntu get the transition so fast? Is it just because of the number of packages and archs supported? Couldn't we just tolerate more breakage in unstable to ease these kind of transitions?

Posted by Bernat at Tue Jan 31 10:16:00 2012

Thanks for the great work!
Send my thanks to the rest of the team.  I run Gnome-shell and if my hardware support for 3D was better, it would be greate.  But that isn't your problem.

I didn't know that it was this hairy, but now when I know this I are more thankfull to the teams hard work.

And I special are impressive on the teams work to support all architectures. That will lead to better software for all architectures, even to x86.

So, thank you!

Posted by Anders Jackson at Tue Jan 31 12:06:48 2012

Well, thank you very much for so detailed explanation. Now I know it's a very very hard work.
I know it's only the first day, but I can't get extensions web page work for me. And gnome-shell-extensions common is not in the repositories. Why did you say "Now it works with the extensions web"? Am I doing something wrong?
Thank you again.

Posted by Javier Amor at Tue Jan 31 14:39:35 2012

I'm not actually a Gnome user, but I appreciate the time and effort you and others in the gnome maintenance team put into it, as that work keeps Debian relevant and featureful.

Posted by Michael Alan Dorman at Tue Jan 31 14:48:49 2012

Sorry it was so much work but trust that is very much appreciated.

Thank you

Posted by Jon at Tue Jan 31 17:27:47 2012

I'd like to join the "Thanks for the great work!" crowd. I've followed the process to some extent, and it did seem hairy, but I had no idea it was this hard.

All things considered, the job you guys did is amazing, thank you for everything, and may 3.4 be a much deserved easy ride!

Posted by Gergely Nagy at Tue Jan 31 17:29:53 2012

Thank you Debian guys for all the great work you do; it's rellay appreciated.

Posted by Matt at Tue Jan 31 17:57:30 2012

Thank you!

After Debian Testing got gnome-shell, I went to extensions.gnome.org and tried to install some extensions, but this didn't work, so I just gave up and switched to xfce4.

After I read your blog post I instantly logged out and in to gnome-shell so I could browse extensions.gnome.org and try to install some stuff. Now I finally have a gnome-shell that I can at least try to use for more than 3 minutes. I still haven't decided which desktop is the "least worse" (gnome 3, xfce, kde), but at least now gnome-shell got promoted from "useless crap" to "maybe as bad as the others".

I still see some of the extensions as "not installable" since shell 3.2 is not the most recent, so I'm really hoping to see other gnome shell versions in Debian Testing.

Here's the list of extensions I installed:
- AlternateTab
- Alternative Status Menu
- Battery Percentage Indicator
- CPU Temperature Indicator
- Frippery Applications Menu
- Frippery Move Clock
- Frippery Panel Favorites
- Places Status Indicator
- Removable Drive Menu
- Remove Accessibility
- Remove Bluetooth
- Remove User Name
- Show Desktop Button
- Window List
- Workspace Indicator

Thank you again!

Posted by annonymous at Tue Jan 31 19:06:57 2012

Forgot to tell you: it's really hard to read your comments section. Add something that visually separates each different comment (e.g., make the "posted by" line have a different font color, or add some
between posts).

Posted by annonymous at Tue Jan 31 19:08:33 2012

Thanks for amazing work, Debian-Gnome team!

Posted by Kartik Mistry at Tue Jan 31 19:52:05 2012

Thank you so much for this great work!

Posted by poinck at Tue Jan 31 23:17:25 2012

Amazing read; thank you thank you, Debian-Gnome Team!

Posted by zab at Tue Jan 31 23:31:43 2012

OK. So how much of this mess could have been avoided if the Gnome people would simply have renamed all the libraries they changed in a backwards-incompatible way?
Perhaps even allowing us to install Gnome2 and Gnome3 in parallel?

Posted by Matthias Urlichs at Thu Feb 2 12:37:17 2012

Thanks for the hard work, and for the insight in the release process of a distro that supports as many platforms and packages as Debian does.

I noticed you used the word "hairy" a few times there, and I can only imagine how hairy things can get.  There's the default DE that decides to reinvent itself radically and I also follow Raphael Herzog's updates on multiarch support.  Take your time, but I do expect to own a tablet running native Debian and Gnome 3 somewhere next week :-)

I've been using 3.2 for a couple of days now and - apart from a few glitches - I'm totally in love with it.  It's slick, clean and stable and I can see a very awesome Wheezy release coming up.

I did notice however that gnome-documents and gnome-contacts are not in the gnome meta-package.  Is that because they depend on the tracker search engine?

Posted by Stephen at Fri Feb 3 05:26:46 2012

Any word on gdm3 3.2 yet? Seems to be the last remaining part of Gnome 3.2 not up-to-date in Debian.

Posted by Ralph Aichinger at Fri Feb 3 17:40:44 2012

I hope the GNOME people will reconsider introducing this many API dependencies. Having yet another scripting language in a core part of the desktop was a bad idea right from the start.

(Just like they did a big usability testing fail by hiding the shutdown option behind the magic ALT button...)

Posted by Me at Mon Feb 6 15:05:20 2012

Wow, that sounded epic. Thanks for all the hard work!

Posted by Mjog at Tue Feb 7 00:37:51 2012