Home

Advertisement

Will Woods, Fedora Testing Guy [entries|archive|friends|userinfo]
Will Woods, Fedora Testing Guy

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

"Linux is about choice" - yes, but not how you mean it [Sep. 23rd, 2009|11:18 am]
I agree with Richard - the phrase "Linux is about choice" sets my teeth on edge. I think it's because every time I hear it, it's being waved about angrily, like a club, by someone who really means "Linux is about me getting what I want".

But here's the thing: in a way, Linux (and all Open Source) really is all about choice. Because one of the essential guarantees of Open Source is the freedom to fork, and the freedom to fork gives you the power to make any choice you could possibly want to make. And this fact makes all other arguments about choice irrelevant.

It also frees the developers of all Open Source projects from any responsibility to consider these sorts of demands - if you want it so bad, fork your own project.

Oh, you can't/don't want to fork? Well, remember that every time you add support for an optional backend or a checkbox for optional behavior, you've just added at least two moving parts and increased possible failure rates by a factor of six. Put another way: Every choice presented to the user means less testing and more bugs for all users. So you better have a really damn good justification - this means code or data, not anecdotes and slogans - before you even bring up the idea of adding a new choice. And "Linux is about choice!" is definitely not a valid justification. At all. Ever.

I think Ajax's seminal post from January 2008 may have said it best:

There is a legitimate discussion to be had about where and how we draw
the line for feature inclusion [...] But the chain of logic from "Linux is about
choice" to "ship everything and let the user choose how they want their
sound to not work" starts with fallacy and ends with disaster.
linkpost comment

Inches [Aug. 14th, 2009|04:57 pm]
Ladies and gentlemen, if I could direct your attention to the autoqa-results mailing list for a moment? What you see here are the first stuttering steps of Fedora's autoqa project.

The basic design of the system is pretty simple: We wait for certain things to happen (the repos are updated, new Rawhide boot/installer images are built, a new update is created in Bodhi, a package is built in Koji, etc.) and launch appropriate tests. The test results get sent somewhere people can see them. And we examine those test results to figure out what's broken, and how to fix it.

Obviously reality is much more complicated than that, but you get the idea. As of now we have two 'hooks' (events that can trigger tests) implemented and four tests written.

The first hook - post-repo-update - triggers repoclosure, conflicts, and rats_sanity. repoclosure and conflicts check for broken deps and file conflicts in the repo, and rats_sanity is a sanity test of the repo metadata and the integrity of the Critical Path Packages.

The second - and more exciting to me - is post-tree-compose, which runs when a new install tree is composed (usually the nightly Rawhide build). It currently runs one test: rats_install. This sanity-checks the boot images and then uses them to boot a virtual guest system. It watches the console output and log files generated by the installer, and tries to do a very simple install.

These tests run automatically when the repos get updated - including Fedora 10 and 11 updates and updates-testing.

Everything is under heavy development but we're working on documentation for writing new tests and implementing new hooks as we go along. We also have plans to set up a public instance of the test system in the Fedora Infrastructure so people can examine the full logs of failed jobs and make neat reports of test statistics and pull useful data out of test results and all that good stuff.

So - what kind of hooks / tests would you write for this system? Anyone have existing tests they'd like to run automatically? Responses in comments are welcome - or fedora-test-list if you'd like to discuss with a wider audience.
link4 comments|post comment

There's something stuck in my brain. [May. 12th, 2009|06:33 pm]
Saw this on boingboing earlier and I can't stop thinking about it:
1. No one cares about what you think, unless you do what you think.
2. No one cares what you do, unless you think about what you do.
3. No one ever really cares what you say.

Spending less time on talk - and more effort on thinking things through and getting them done - seems like some good advice for me right now.
linkpost comment

Why glxgears is slower with Kernel Modesetting (and why it doesn't matter) [Mar. 13th, 2009|04:43 pm]
or: "glxgears is not a 3d benchmark"

One interesting fact came out of yesterday's Intel KMS Test Day. Everyone noticed that glxgears is much slower under KMS/DRI2 than it was before (e.g. in Fedora 9 or 10). A typical result was ~1000FPS in F10, but only ~440FPS in Rawhide. So what's changed?

Here's the deal, as best as I understand it (thanks to krh/halfline/ajax for trying to explain it to me): glxgears is rendering an insanely simple scene - so simple that the actual 3D rendering time is basically zero. So the only thing glxgears really tests is the performance of glXSwapBuffers() - basically, how fast we can push render buffers into the card. This operation is slower with DRI2, but - roughly speaking - unless it was an order of magnitude slower (e.g. glxgears drops from 1000FPS to under 100FPS) it wouldn't make any real difference.

So if you're going to be comparing "3D performance" - please, don't bother with glxgears. ajax has suggested a couple of things that might make better benchmarks:
  • The mesa-demos package has a few useful things - teapot in particular is nicely simple.
  • sierpinski3d and glblur from xscreensaver-gl-extras also work well.
  • extremetuxracer is, in my opinion, the most fun way to benchmark 3D.

If you see major (100% or more) performance differences between KMS and non-KMS in those apps, it's probably worth investigating further.
link6 comments|post comment

.i586 packages: not a big deal [Mar. 9th, 2009|01:58 pm]
A note about .i586 packages in Fedora 11, and why you shouldn't worry about them:

Since we have a new compiler, some RPM checksum improvements, and - for x86 users - changes to the default architecture support, we've done a mass rebuild of every package in Fedora. x86 users will likely notice the following changes:

  • All .i386 packages have been rebuilt for i586.

  • Instead of building four kernel packages (i386, i586, i686, and i686-PAE) we're now building just two kernels: i586 and i686-PAE.

    • Newly-installed x86 systems that have 'pae' and 'nx' in /proc/cpuinfo will get i686-PAE.

    • Everything else will use kernel.i586.

Here's the thing: moving to i586 isn't a big deal because we're already optimizing for modern (i.e. Pentium 4 or later) processors. For the past couple years we've been using gcc's -mtune=generic flag, which optimizes for "the processors that were most common when that version of GCC was released" (according to the gcc info page). The only thing that moving to i586 really changes is the instruction set that's used. i586 adds a handful of new instructions over i386 - notably CMPXCHG8B, which is useful for efficient atomic operations - but nothing world-changing. This probably isn't going to give you more FPS in glxgears.

As for the kernel, the only difference between i586 and i686 is the CMOV instruction family, which - as Linus explains here - generally performs worse than the equivalent i586 instructions on modern CPUs. So moving the kernel from i686 to i586 should have no real effect on performance.

I'm sure there will be much discussion on the subject, but I'd like to see some good benchmarks - real data, rather than people reporting that Fedora 11 is "snappier" because it's "Pentium optimized", or "slower" because "the kernel is only i586 now".
link1 comment|post comment

PING WITH DATA [Feb. 10th, 2009|11:17 am]

If you want to ask a question on IRC, NEVER DO THIS:

<someguy> wwoods: ping

What's gonna happen if I'm not there? An hour later when I get back to IRC, all I know is - you wanted something. I have no idea what. So then all I can do is ping you back, and if you're not around, then we have to keep sending pings back and forth until we're both online at the same time.

An empty "ping" is the same as saying "Can I ask a question?" You just did. Just ask the question.

<someguy> wwoods: ping - what's that blue thing doing there?

Now when I get back to IRC, I can actually answer your question immediately. And if I'm online you'll get your answer right then - no need to spend the extra network roundtrip on a useless ping/pong.

Ping with data - ask the question, mention what you wanted to talk about, whatever. Just don't send empty pings. It's a waste of time.

This has been an Unwelcome IRC Etiquette™ lesson
© 2009 Surly Fedora QA Guy Enterprises, Inc. All wrongs reversed.

linkpost comment

python-bugzilla - give me your queries! [Aug. 19th, 2008|05:12 pm]

I've been working on a branch of python-bugzilla that supports multiple different Bugzilla variants. The current version (0.3) only supports the functions provided by Red Hat's Bugzilla 2.18.x. This branch (which will become python-bugzilla 0.4) adds support for the upstream 3.0 and 3.2 WebServices and Red Hat's shiny new 3.2 instance. (Hooray!)

Yes, I'm sure you've noticed that we've updated to Bugzilla 3.2. Besides the shiny new web frontend there's also new and shiny XMLRPC code for python-bugzilla to use. It still supports the old RHBugzilla stuff, and it now supports all the upstream Bugzilla 3.2 services, but there are some additional new-style methods that will someday make their way to upstream Bugzilla - although probably not until 4.0.

The point is this - I'm trying to get python-bugzilla using the new methods wherever possible. This includes a new and slightly different Bug.search() call, and I want to make sure I can make the switch without breaking everyone's existing code.

So here's my request. If you use python-bugzilla, send me your queries - or just a link to the code, if that's too much to ask - and I will do my best to make sure that your code will work, unchanged, with the new upstream(ish) Bugzilla methods.

I'll update soon to talk about what else is new in python-bugzilla 0.4, but having some real-world test code to work with would be a great boon. Thanks!

linkpost comment

Writing Test Plans [Jul. 22nd, 2008|01:35 pm]

Everyone knows what Fedora QA does. "They're the testers. They test stuff." Sure, that's true. But.. what, exactly, do we test? And how?

That's where Test Plans come in. A Test Plan is a document that tells the testers how to test something.

A good software Test Plan should at least answer these four questions:
  1. What special hardware / data / etc. is needed to test this (if any)?
  2. How do you prepare your system to test this? What packages need to be installed, config files edited, etc.?
  3. What specific actions do you perform to check that the software is working like it's supposed to?
  4. What are the expected results of those actions?

Your answers can be short: "get a bluetooth keyboard and yum install bluez-gnome newer than 0.25" answers question 1 and 2 just fine. But they need to be complete and explicit: "Get an appropriate keyboard and install the new packages" is not helpful.

Questions 3 and 4 are answered by Test Cases. A Test Case is a single, specific thing to test - a list of actions to perform, and the expected results of those actions. Depending on how complicated a package or feature is, you could have one Test Case or a whole bunch of them.

While writing a Test Plan, pretend that you have an intern whose job it is to test Fedora releases. This brave, brave soul has a few years' experience as a Fedora user and basic familiarity with using the commandline to configure stuff, install packages, and so on.

Your job is to tell this person how they can test your favorite package or your cool new feature, so they can either a) tell their friends about how great it is, or b) tell you (via bugzilla) if it breaks.

Here's an example Test Case for Banshee, originally written by our own Balaji Gurudass:

Description:
    Test loading a Local File in Banshee.
Actions:
  1. Start Banshee
  2. Choose "Import Media" from the "Media" menu
  3. Select "Local Files" from the import source dropdown and click the "Import Media Source" button
  4. Select the music file and click Open
  5. Find the song in the music library and play it.
Expected Results:
  1. Banshee starts successfully
  2. The "Import Media" menu item brings up the "Import Media to Library" dialog
  3. The chosen music file is imported into the Banshee library
  4. The music file can be played and the tracker moves along and shows time elapsed.
This gives us a clear, easy-to-follow way for anyone to prove that a new banshee package does (or doesn't) work like it's supposed to. Or, at least, one small part of it.

So there's an overview of how to write test plans. Currently we're keeping Test Plans in the wiki. There's a section in each Feature Page for a Test Plan, and if you wanted to write a Test Plan for a specific package (or set of packages), you could add a wiki page named "QA/[package] Test Plan". For example, here's QA/Banshee Test Plan.

Next time, I'll talk about why I think this stuff is so important.

link2 comments|post comment

preupgrade is not 1.0 [May. 19th, 2008|05:23 pm]
So, yes, preupgrade is in the F8 repos now, and that's nice. But it's not 100% complete and that's why it's not in the menus or anything yet.

The current version - 0.9.3-3 - will fail to boot the installer if your /boot directory is a) on RAID1 (#444497), or b) on your root partition (#446826). These should both be fixed in the next preupgrade release.

At the moment, though, there's a bug that prevents preupgrade from being able to finish any upgrades without a network connection. This is the bug that Greg hit during his preupgrade. Luckily, as he noticed, you can bring up the network in the installer, but you can't use anything fancy - no wireless, no VPN, etc.

This may not be fixable without a respin of Fedora 9 (or, at least, a custom stage2.img specifically for preupgrade). I'm not really happy about it.

Hopefully we can do this right for F10. Sigh.
link5 comments|post comment

Translation Halp! [Apr. 1st, 2008|06:20 pm]
So if you've used GNOME in Rawhide lately you probably noticed this new "About this Computer" thing in your System menu:



That was me! I did that!

The problem is.. those strings need to be translated into all the various languages I don't speak.

Eventually the patches involved will be in upstream GNOME or some Fedora-specific package. And then we can use the magic of Transifex to do the translation properly. But while that's being sorted out, I still need to get those strings translated for F9.

If you can help, please check out this wiki page and add translations for your language.

Thanks a million!
link3 comments|post comment

We're intercontinental when we eat french toast. [Nov. 25th, 2007|12:28 pm]
So today (as with most days) I hit http://planet.fedoraproject.org/ to see what's been going on in the Fedora Community.

The first six posts, in order, were in Spanish, Kashmiri (well, ok, English, but talking about Kashmiri), Chinese, Indonesian (I think?), French, and English.

Fedora might technically be US-based but it's pretty obvious that we are a worldwide community. And that's really cool. It makes us monolingual* Americans feel kind of silly, though.

* Well, OK, I took 6 years of French in high school and 2 years of Japanese in college, but I haven't spoken either language in years.
link1 comment|post comment

Fedora 8 - RC3 vs. Final bits [Nov. 4th, 2007|10:24 am]
Someone asked in my last post if RC3 was the final version.

The answer is: Nope - we had to do two respins after RC3.

Thursday night we found, filed, debugged, and fixed the fsck-at-boot / rhgb hang at first boot problems (see bugs #357401 and #362721 respectively), completed the fix for #357401 (dmraid+livecd) and then rebuilt all the install and Live images to create RC4 for Friday morning. Friday's rawhide report notes these changes.

During Friday we found and fixed a bug in selinux-policy-targeted where restorecon etc. would screw up at install time and leave you with SELinux non-functional. Once again Jesse rebuilt all install/Live images - creating RC5 - with this change included. This change is listed in Saturday's Rawhide Report.

Once the RC5 bits were built (mid-afternoon Friday) we desperately tried to smoketest them and get them rsync'd to the Raleigh office before 9pm so that we could hand them off to the internal IT guys. (After 9, it would be too late to get it done for Monday, and we'd have to delay the release.)

The bits landed at 8:45pm. All our testing showed that the bugs above were, indeed, fixed. Jesse made the call, got a response, and that was that.

There were heroic efforts from a lot of people in those last two days, and I raise my glass to one and all.
link8 comments|post comment

F8 is done! [Nov. 3rd, 2007|02:57 pm]
We got the final Fedora 8 bits handed off for mirroring at 9pm last night. Hooray! When I got home afterward the house was still empty - my wife was out to dinner with some friends - but I poured a celebratory glass of bourbon1 anyway.

Then the three of them appeared at the door, bearing this balloon:

and sang me a song they made up which I don't fully remember, except that it rhymed "fleece" and "Fedora release".

I have some pretty awesome friends. And a totally sweet wife.

1. Blanton's bourbon is pretty much the best ever. Just look at the bottle! It's like the Holy Hand-Grenade of Bourbons.
link3 comments|post comment

Werewolf Calls [Oct. 22nd, 2007|06:06 pm]
Just created the installer test matrix wiki page for Fedora 8 (Werewolf).

Today's rawhide wasn't installable without this updates.img (see here for instructions on using updates.img files with anaconda). Boo. But that bug is fixed, so tomorrow's rawhide should be very close to a first release candidate for Fedora 8.

Spent last week poking bugs on the blocker list - we started the week with, if memory serves, 54 blocker bugs. After a week's worth of tracking down bugs and testing fixes and pinging maintainers and reporters and such, we ended the week with.. 54 blocker bugs. Sigh.

But - good news, dear readers! Today (with help from jkeating, warren, jeremy, notting, and dozens more) we are down to 43 blocker bugs.

Some things were just dropped from the list - as much as I want the fingerprint reader on my thinkpad to work, we're not going to slip the release for bug #311361.

Quite a few things have been fixed - for instance, this bug where pirut and pup would demand the DVD before letting you update your system. Now, when pirut or pup fail to contact one of your configured repos, there's a nice "Edit Repositories" button that lets you enable and disable repositories.

Tomorrow should also bring a new NetworkManager build that restores the ability to connect to hidden wireless networks, adds dynamic WEP support, and other goodies.

Things seem to be going nicely. Hopefully we can clean it all up and hit the November 8 release date. It'd be nice to have a release go out on time for once!
link1 comment|post comment

Frozen for Fedora 8. Brace for impact! [Oct. 19th, 2007|01:53 pm]
Hoorj. So, yeah, we are definitely frozen for F8. Now comes the time when I spend every day staring at the blocker list and poking the bugs there (and the people responsible for them) and trying to make them die. The bugs, not the people.

All in all - and I realize that I'm jinxing it by saying this - the Fedora 8 release process has been much easier than F7, and I think F8 will be a good bit more stable out of the gate than its predecessor.

There's a bunch of lingering problems around the switchover to NM 0.7, but Dan Williams assures us that everything will be awesome by release time. Hopefully this will mean we can start work for making NetworkManager the default for F9.

PulseAudio is still cool, and the KDE guys have quickly whipped up a method for using PulseAudio as the backend for artsd. Things will be saner when KDE4 rolls around - it separates the sound server (like esound, pulseaudio, etc) from the file decoders (like gstreamer). For now, this should work. KDE users, please please please test that out and test it well.

I've got some things to say about Live Upgrades and Automated Testing and other stuff that's kicking around the back of my head, but.. right now I need to poke more bugs.
linkpost comment

Rawhide Report [Sep. 19th, 2007|10:44 am]
Rawhide's a bit wonky today. XULRunner has finally landed - hooray! - but unfortunately it wants to obsolete firefox < 2.1 and replace gecko-libs. This causes trouble with stuff that uses gecko-libs (like yelp and liferea), and - of course - firefox, since we're only at 2.0 in the repos.

So, skip the 'xulrunner' update until we get this worked out.
linkpost comment

Possible rpm badness upgrading from F8Test2 [Sep. 10th, 2007|03:53 pm]
Edit: take this post with a grain of salt - we're currently unable to reproduce the problem, and we've done multiple tests on i386 and x86_64 machines. It may have already been fixed. Still, it's a weird and interesting problem so I'm leaving this here for future reference.

We're tracking down a problem where a fresh F8Test2 install can, after upgrading to current rawhide, end up with no rpm-libs - and thus no way to install packages. Yikes. Luckily it's fairly tricky to hit this bug - we haven't been able to reproduce it on i386, for instance.

As far as we can tell the required sequence of events is as follows:

  1. Install F8Test2 - on a 64-bit machine - with a custom package selection that includes the Development group

    This will install rpm-devel-4.4.2.2-0.1.rc1, which (accidentally) includes all the libs that rpm-libs normally provides. The installer will not install rpm-libs, since all the files in that package are already provided by rpm-devel.

  2. Boot your fresh new F8T2 system and notice ~150 new updates.

  3. Install the new updates.

    During this upgrade, yum will upgrade rpm (and friends) to 4.4.2.2-0.4.rc1. This new rpm-devel package does not include the rpm libraries anymore, so yum should notice this and pull in the rpm-libs package, to keep rpm working. For some reason, on multilib (i.e. 64-bit) machines, it does not.

After this update, your system would be missing rpm-libs and thus unable to install any packages. So you can't install the rpm-libs package to fix the problem.

Hopefully we'll have this isolated and fixed before F8Test2 goes live. It's pretty easy to work around, if you know about it - just install rpm-libs as soon as you boot your F8Test2 system for the first time - but if your system gets messed up it's tricky to recover.
link1 comment|post comment

Rawhide Report [Sep. 9th, 2007|11:00 am]
Well, F8Test2 is complete, and the rawhide floodgates are open again. There's plenty of fun and excitement to be had in those new hundred-and-something updates.

The most obvious one is that kernel .169 reboots on startup on some machines- see Bug #283631. I've only tried one machine, but it failed there. Falling back to .164 is easy enough, so it's not that big a deal.

I've actually not seen any success reports with this kernel. If you're doing a rawhide update, you might save some time and skip that package.

Once you get the system booted, you may encounter some flakiness with dbus not starting up, and all dbus-related services thus also failing to start. I haven't tracked this one down enough yet to file a bug, but restarting messagebus seems to let ConsoleKit, NetworkManager, hal and friends work OK.

I'm not sure if this is related or not, but pulseaudio also seems to have trouble finding the ALSA devices now. Again, haven't tracked it enough to file a bug, but be on the lookout (and file a bug if you find a lead on the cause..)

I kinda hate these post-freeze updates. There's so many changes all at once and I'm never exactly sure where to start looking when something breaks. And something always breaks.
linkpost comment

Rawhide Report and python-bugzilla [Sep. 5th, 2007|07:18 pm]
Today's rawhide seems to install just fine, at least if done without a kickstart on x86_64. That's good. I didn't get around to testing ppc, and my kickstart-based install broke somewhere.

I've still got a cold so I'm sorry if I'm a bit vague on details here. I'll be sharper soon.

I've been working on a python module to talk to Red Hat's bugzilla instance - you can get a copy of my git repo like so:

git clone http://wwoods.fedorapeople.org/python-bugzilla.git

The Red Hat XMLRPC stuff is a lot more featureful than the upstream bugzilla 3.0 web services. On the other hand, the upstream stuff is much cleaner. I'd really love to get some work done to get the RH stuff cleaned up and moved upstream so everyone could benefit. I think the RH bugzilla team is planning on doing this "soon" - but that's "soon" in RHEL terms, which means, like, early next year maybe. Sigh.

Anyway. If you've got patches or suggestions for my bugzilla module (I'm sure it's not the best code ever - I'm still getting to know python) I'd love to hear 'em.
linkpost comment

Rawhide Report [Sep. 4th, 2007|06:15 pm]
Well, rawhide has installer images again today. Hooray for that. Unfortunately something about current rawhide keeps it from working right in KATE, our venerable installation test system. I'm told that manual installs work properly though.

I haven't done any manual installs myself because I'm out of the office with a nasty head cold. I can work for about two hours before I get all fuzzy and unfocused and I have to go nap for a while.

It appears that dbus has grown a dbus-libs package, and that's why upgrading to dbus-1.1.2-3 made my systems freak out - it didn't require its own libs! dbus-1.1.2-4 fixes this.

fonts-japanese-0.20061016-{8,10}.fc8 have broken %postun scripts, which prevent them from being installed. This would be merely cosmetic, except fonts-japanese...-11 requires the new sazanami-fonts packages, which conflict with fonts-japanese older than -9. So if you want to upgrade your japanese fonts, you'll need to do something like:
# force-remove the old fonts
rpm -e --noscripts $(rpm -q fonts-japanese)
# manually install new ones
yum install fonts-japanese
linkpost comment

navigation
[ viewing | most recent entries ]
[ go | earlier ]

Advertisement