||[Aug. 14th, 2009|04:57 pm]
Ladies and gentlemen, if I could direct your attention to the autoqa-results mailing list for a moment? What you see here are the first stuttering steps of Fedora's autoqa project. |
The basic design of the system is pretty simple: We wait for certain things to happen (the repos are updated, new Rawhide boot/installer images are built, a new update is created in Bodhi, a package is built in Koji, etc.) and launch appropriate tests. The test results get sent somewhere people can see them. And we examine those test results to figure out what's broken, and how to fix it.
Obviously reality is much more complicated than that, but you get the idea. As of now we have two 'hooks' (events that can trigger tests) implemented and four tests written.
The first hook -
post-repo-update - triggers
conflicts check for broken deps and file conflicts in the repo, and
rats_sanity is a sanity test of the repo metadata and the integrity of the Critical Path Packages.
The second - and more exciting to me - is
post-tree-compose, which runs when a new install tree is composed (usually the nightly Rawhide build). It currently runs one test:
rats_install. This sanity-checks the boot images and then uses them to boot a virtual guest system. It watches the console output and log files generated by the installer, and tries to do a very simple install.
These tests run automatically when the repos get updated - including Fedora 10 and 11 updates and updates-testing.
Everything is under heavy development but we're working on documentation for writing new tests and implementing new hooks as we go along. We also have plans to set up a public instance of the test system in the Fedora Infrastructure so people can examine the full logs of failed jobs and make neat reports of test statistics and pull useful data out of test results and all that good stuff.
So - what kind of hooks / tests would you write for this system? Anyone have existing tests they'd like to run automatically? Responses in comments are welcome - or fedora-test-list if you'd like to discuss with a wider audience.
I'd say yards, brother. Nice f-ing work.
Test writing seems like an excellent topic for a hack session at some point.
I need to get a better feel for how the bits of python in the git tree work together to actually fire off a particular test.
So, does that mean I can finally stop running extras-repoclosure for F11 and F10? Or is none of this in production yet? Btw, repoclosure is still missing the Obsoletes checks (yum upstream ticket #176).
Fantastic work. Let me answer a question with a question... What would be the best way for community members to help QA, by capturing details from the current pending Anaconda bug list (ASSIGNED or better, let's say) about users' pre-existing partition configurations, and turning those into a set of examples against which installation could be tested on some regular basis? It might not be possible to do all these daily right now, but with more machines avaialble... say through a virt farm *cough cough*... that gets easier. Nothing novel here, I know, just trying to say something constructive on a Saturday morning beyond "You rock."