Future of Fedora Release Engineering

I have now been doing Fedora Release Engineering for nearly 5 years. My first task was to rebuild every Fedora Core package for a gcc change leading up to the release of Fedora Core 5 (hey look, a --turbo option!). I've seen us through 10 releases, the merger of Core and Extras, countless mass rebuilds, the creation of Live Media and the explosion of spins, an unfortunate security incident, many evolutionary changes in our development process, the creation and growth of a release engineering volunteer team, the creation of release criteria, the migration of source control, and the creation of a plethora of Standard Operating Procedures for release engineering. It has been a challenging and very rewarding 5 years. But I need a break.

Starting after the release of Fedora 14, I will be stepping down as the lead release engineer for Fedora. I will be taking the knowledge and lessons learned from our migration of CVS to git and applying it internally at Red Hat to migrate our internal package source control to git as well. The number of packages and contributors is smaller, but the environment is far more complex, and I am very much looking forward to the challenge. We have estimated that it could take up to a year or longer to complete the task. During that time, Dennis Gilmore will be stepping in to lead the Fedora release engineering team. Dennis has been involved with Fedora for just about as long as I have, if not longer and will be able to fill the role perhaps even better than I could.

I won't be far away from the project, and I will continue to support and improve
things such as fedpkg development and necessary changes for our compose tools such as pungi. I'm always just an email or IRC ping (with data) away.

Fedora talk at fi.muni.cz

I just finished giving a Fedora talk at Faculty of Informatics Masaryk University. That is one of the reasons why I've spent the week working here from the Brno Red Hat office.

I was told to expect about 20 students, so I was happy to see 40~50 folks sitting in the audience. I gave a talk on the history of Fedora and how we got to where we are now, the lessons we've learned along the way. I also talked a bit about the future direction and why we're doing what we are doing.

It was a short talk with a few slides (because I like talking more than clicking a button) and it only took about 30 minutes. I was worried as I was booked for 60, and I was told before hand not to expect (m)any questions. Imagine my surprise when we then had a good 20 minutes of questions and discussion about Fedora! Many of the students seemed knowledgeable about Fedora and the current happenings. I hope I answered questions well enough and gave them a good sense of what is going on and a good feeling about where we are going. Apparently that is the most discussion any RHT speaker has gotten there, ever!

Tonight is my last night in Brno. It has been a fun week, but I really miss my family and I can't wait to go back. The Brno folks have been very hospitable and helpful. I will look back on this trip fondly.

Tonight I hope to have a simple dinner and find a shop to pick up some gifts, then get to sleep early. I flee the city at 07:30 tomorrow morning for my flight home out of Vienna. Looking forward to seeing my boys!

Looking for emacs folks who use git

Do you use emacs? Do you use git? Do you use the emacs git integration system? Are you willing to do some tests on an EL-5 (RHEL 5 or CentOS 5) system? Please make yourself known at this bug.

FUDCon Zurich day one

Like a few people I'm here in Zurich for FUDCon. This is my second European event and I'm excited!

Long flight getting here only to be met with a minor migraine. I was able to sleep it off as Jarod Smith and Spot relaxed in our room and recovered from their travel. We met up with mizmo for a dinner involving lots of cheese. Then we helped a little bit setting up the "expo" part of the show after seeing a bit more of the area around the show building than originally intended :) Went to sleep early to be ready for today, after employing extreme self control and skipping drinks with other Fedora people.

Today we got to the show with little drama having been there the night before. Jared gave a quick "welcome" talk and then Adam gave a talk on Fedora QA. I sat through Marcela's talk on perl but I was working on notes for my workshop and talk.

Lunch happened and it involved curry and more Fedora people. Now I'm listening to Miroslav's talk on Spacewalk.

More talks today and my workshop on rel-eng SOPs at 1600. Then there is an afterparty with the Froscamp folks, and I'm sure more socializing at the hotel bar tonight.

Posted via LjBeetle

Eclipse, Python, and Vim

I love python. I love vim, and I'm growing to love Eclipse. Until recently I thought I could only enjoy python and vim, or python and Ecilpse, but not Eclipse and vim. At least not freely. That has changed.

Vrapper is a GPLv3 licensed vim emulator for Eclipse. It doesn't work by replacing the editor like some others, it wraps the existing editor which means you get good keybindings but keep all the other good stuff about the base editor. It does it really really well!

It's too bad that I found this after I worked on fedpkg, would have been a much more enjoyable experience.

Help Wanted!

Do you want to work on some software that will be used by every Fedora packager?

Do you like hacking in python?

Do you have a thing for git?

Well have I got a project for you!!!

The Dist Git Project is back in action and steaming ahead to try and finish before we branch for Fedora 14. Currently I'm working on fedpkg the software to replace our Make system. I could use some help. If you'd like to help, find me on IRC (Oxf13 on freenode) or email. I don't really like livejournal comments so try to use a different way of finding me.

Happy Hacking!

Switching Gears

Once again we've done a Fedora release. This happens twice a year, and every time it happens I have a sudden gear shift to work through. The three months or so after a release are relatively quiet for release engineering. There are few fires, and fewer milestones to worry about. It's during this time that we work on our longer term projects, fix up our scripts, and generally pay attention to things we were ignoring during the release sprint.

While the build up to the frantic state we're usually in at release time is gradual, the drop off is much more severe. Each time we release, it's like we drop off a cliff. There are no more fires, there are no more immediate or late deadlines. There is just a nice open few months of runway in front. It is kinda hard to switch from being almost entirely interrupt driven to working from a to do list. It's hard to keep concentration going when you're used to being interrupted frequently. My ADD tendencies start to come out.

The other hard gear shift is going from working on release related tasks to working on programming and system administration tasks. I get pretty rusty in my python programming, requiring more time to look up libraries and remember tricks, and remembering where I left various projects off and what work needs to be done.

Things that have helped include flagging emails for later follow up, and judicious use of to do lists. To help with the latter, I've started using "Getting things Gnome" ( http://gtg.fritalk.com/ ). This is an upgrade from using gnotes alone and has helped me organize and sort my tasks, as well as put notes in about each task so that now that I have time to do things I can find things to do easily and work on them. There is also a nice feeling of completion as I clear each item off the list.

So if I look a bit bewildered over the next week or so, maybe confused or waiting for that interrupt you'll know why. Please forgive me and I'll catch back up soon!

The move to git

The time has come to bite the bullet and move Fedora's package source control on from CVS. CVS has done is well, and although it is a decaying source control, it handled our needs rather well for many years. However nothing is a constant, and over time more and more cracks have shown up in our source control. The time to move on is now, and I feel pretty confident in the plan we are exploring.

First though, why git? Git has skyrocketed in popularity over the past few years. More than I ever imagined, when I first toyed with the idea of doing the jump to git, back in the Fedora Core 6 days. Most notably the Gnome project has recently made the leap from SVN over to git (amusingly they started in CVS, we'll just be skipping that SVN step). Not only is git popular, it's really a better choice. Why git is better than x does a good job at hitting some of the highlights. Fedora is also a meritocracy, in that the people doing the work are generally the people making the decisions. Code talks. Since the work to do the migration is largely going to fall on me (because I've volunteered to do it) I get to make some of these decisions. Looking at the source control landscape over the past few years I feel confident that git is the way to go.

How are we going to do it? I've been thinking on this problem for years. There are two ideals that I really want to satisfy. Firstly, I want to make sure we stay true to our principles and differ from upstream as little as possible. To facilitate that I want to make it easy to discover our modifications and pass them along to upstream and/or other consumers of a given upstream. One way to force this is to continue working with a upstream archive (tarball release, maybe a snapshot) and our modifications as patches. Easy to see our changes, easy to pass them around. The other ideal is that working directly on source code in a scm repository is far better than trying to work with a tarball + patches. Trying to reconcile both of these ideals has been hard for me to manage, until the light went on the other day...

Git helps here in a few ways. Creating new repos easy, fast, and scriptable. Since it's a DSCM everything can be done locally on any filesystem. Commits to git can be exported in a text file format that is easily transportable and contains not just the diff, but context about the change such as author, reason, further log info, etc... These exported commits can be applied to a given repo easily and quickly, even if it's a brand new repo. So while we can maintain a forward moving repository of .spec file and patch files management, we can also on the fly create "throw away" repos of the upstream source code, plus our patches applied git style, leaving the developer with a repo that they can do development/patch management with and export the results back into our package source control, throwing stuff at upstream along the way. This really is the best I could come up with for the best of both worlds.

Details? See my slides.

Time line? I'm currently throwing our CVS archives at the cvs to git conversion tools, cvs2git (part of cvs2svn) and the git cvsimport utility. I think I have a good strategy for getting our release sub directories imported as real branches of a package module, where the master or unbranched content would be the rawhide content. Lots of work needed here to verify that the import is doing the right thing. But that's as far as I've gotten. I'm still in the idea phase, attempting to proof of concept my current idea and discovering where it falls down. I hope to get through this phase in the next few weeks so that we can get to the proposal phase where we get FESCo buy in. Implementation depends on the amount of work needed, but hopefully near Fedora 13 release.

Help wanted! I would love to have your help. Your thoughts on my plan, your experience in this area, your time to write scripts, your time to verify conversion output, your time to document progress in wiki pages, etc... As I work on the raw cvs into git conversion, we'll need a fpkg utility written that will take over from the Make system. Work on this tool can be very modular and well suited for many people working on it collaboratively (hey, great idea for a Fedora Activity Day!). Once we get a good format settled on for the packages and a good start on fpkg development, we'll need koji changes to be able to build from this style of repo. Good news, that code is pretty isolated and there already exists some git code. If you're interested in helping out, find me and I'll try to put you to work.

Keeping up with progress. I plan to talk about our progress in this task at our weekly Fedora Release Engineering meetings. Those meetings are summarized on fedora-devel-list so you can keep watch there. Eventually there will be a wiki page that will track progress, as well as a trac milestone and ticket set and all sorts of fun project management stuff like that as we progress.

FUDCon Toronto 2009

I'm back in the USA, having just attended FUDCon Toronto 2009. I haven't blogged about this before due to the state of both the hotel and the campus Internet offerings.

I felt that this FUDCon was wildly successful. We had a lot of familiar faces there, plus a good many who have never been to a FUDCon before. The barcamp talks pitched were amazing both in the sheer number of them, and also in the quality of the topics. There were very few talks that I /didn't/ want to go to, which made it really hard to pick the few I could go to, particularly because I was talking twice.

The first talk I did was a joint talk about message buses, and in particular how I plan to use a message bus to tie a number of our services together and start automating certain things. This is a fun topic for me as I've wanted to do this for many years and the pieces are now starting to come together for it.

The second talk I did was a last minute entry, something I decided to do after talking about it one evening with Jarod Wilson, and then with a variety of people during the long bus ride up to Toronto. The talk was my current plans for moving away from CVS as Fedora's package source control and instead using git. Not just that but some improvements to our work flow along the way. Slides found here.

Hackfest days were spent planning out the first onslaught of messaging targets, fine tuning release criteria with QA, discussing the Fedora updates experience, fine tuning release engineering tasks around releases, fine tuning the F13 schedule, and working on the dist-cvs->git transition. By the time the bus rolled into Westford this evening I had a series of commands working pretty well to convert some package modules in CVS into git the way we want it, but much more testing is required.

I'm really looking forward to seeing more results from FUDCon and while I don't want to travel again for a while, I can't wait for the next one!

Another 6^H 5 months, another Fedora release

Fedora 12 released today, go get it. This is the 8th Fedora release I've had the pleasure of being paid to work on in a release engineering role. It has been a long and hard trip, but with lots of rewards along the way. Our community is stronger than ever, with more volunteer (both outside and inside Red Hat) people taking up leadership roles or driving features through or just helping out where help is needed.

12 is cool. 12 was really hard to get out too. 12 had a shorter development period than previous releases, because we lost a month due to an infrastructure incident during the Fedora 10 cycle. We decided to give Fedora 11 a full 6 months as we had a lot of really cool features lining up for it and we were going to do a shorter 12 cycle to polish things up. Turns out we had people doing lots of polish, but other people doing lots of features too, even in a compressed timeline. I'm deeply impressed with the amount of work that can get done, often in spite of the challenges faced by a Fedora developer/maintainer. I can't wait to see what kind of productivity we see as some of these challenges are addressed in the Fedora 13 cycle. Some big changes are coming, tune in to the upcoming FUDCon where there will be multiple talks and hack sessions regarding these challenges our developers, maintainers, testers, and users face.

A few interesting data points for Fedora 12:
- First release in a looong time (ever since I've been doing it perhaps?) that we didn't have to slip the final release date once we got past Beta.
- First release probably ever where the x86_64 DVD is seeing more torrent downloads than the i386 DVD
- Smoothest release day EVER. Seriously. The infrastructure folks really have this down and there were 0 issues.

One more thing that seriously impresses me about Fedora 12 is the launch of our spins site. This is a fabulous new site to really help showcase the different looks that Fedora as a project can offer, finally bringing to fruition something we had been talking about for many releases/years now. There is more website (re)design coming, and I am very excited about it.

Go on, give Fedora 12 a try. A Live image gives you folks with commitment issues something to try without risk, and the install DVD/CDs (and network install iso) gives you fine tuners the ability to hand select the software you get from our vast repository. I promise you we'll do the best we can to ensure a fun, safe, and free ride.