The basic design of the system is pretty simple: We wait for certain things to happen (the repos are updated, new Rawhide boot/installer images are built, a new update is created in Bodhi, a package is built in Koji, etc.) and launch appropriate tests. The test results get sent somewhere people can see them. And we examine those test results to figure out what's broken, and how to fix it.
Obviously reality is much more complicated than that, but you get the idea. As of now we have two 'hooks' (events that can trigger tests) implemented and four tests written.
The first hook -
conflictscheck for broken deps and file conflicts in the repo, and
rats_sanityis a sanity test of the repo metadata and the integrity of the Critical Path Packages.
The second - and more exciting to me - is
post-tree-compose, which runs when a new install tree is composed (usually the nightly Rawhide build). It currently runs one test:
rats_install. This sanity-checks the boot images and then uses them to boot a virtual guest system. It watches the console output and log files generated by the installer, and tries to do a very simple install.
These tests run automatically when the repos get updated - including Fedora 10 and 11 updates and updates-testing.
Everything is under heavy development but we're working on documentation for writing new tests and implementing new hooks as we go along. We also have plans to set up a public instance of the test system in the Fedora Infrastructure so people can examine the full logs of failed jobs and make neat reports of test statistics and pull useful data out of test results and all that good stuff.
So - what kind of hooks / tests would you write for this system? Anyone have existing tests they'd like to run automatically? Responses in comments are welcome - or fedora-test-list if you'd like to discuss with a wider audience.