Will Woods, Fedora Testing Guy (qa_rockstar) wrote,
Will Woods, Fedora Testing Guy

depcheck: the why and the how (part 2)

In part 1 I discussed the general idea of the depcheck test: use yum to simulate installing proposed updates, to be sure that they don't have any unresolved dependencies that would cause yum to reject them (and thus cause everyone to be unable to update their systems and be unhappy with Fedora and the world in general.)

In this part we're going to look at two of the trickier parts of the problem - interdependent updates and multilib.

Interdependent Updates: no package is an island

This, by itself, is a pretty simple concept to understand: some packages require a certain version of another package. For example, empathy and evolution both require a certain matching version of evolution-data-server (e-d-s for short) to operate properly. So if we update e-d-s we also have to rebuild and update evolution and empathy.[1]

So - that's all fine, but what happens if we test the new empathy update before the new evolution-data-server has been tested and released?

If we test empathy by itself, depcheck will reject it because we haven't released the new e-d-s yet. And then checking e-d-s by itself would fail, because we rejected the new empathy package that works with it - switching to the new e-d-s would cause the existing empathy to break.

Obviously this is no good - these two updates are perfectly legitimate so long as they're tested together, but tested independently they both get rejected. And it's not an uncommon problem, really - there are actually 8 other packages on my system which require e-d-s, and dozens (probably hundreds) of other examples exist. So we have to handle this sensibly.

The solution isn't terribly complicated: rather than testing every new update individually, we put new updates into a holding area, test them all as a batch, and then the packages in the batch that are judged to be safe are allowed to move out of the holding area. So interdependent packages will sit in the holding area until all the required pieces are there - and then they all move along together. Easy!

This can be confusing, though. For instance: it's true that we run depcheck for every new proposed update - but remember that we aren't only testing the new update. We're testing the new update along with every previously-proposed update that hasn't passed depcheck yet. This means that a package that fails depcheck will be retested with every new update until it passes (or gets manually removed or replaced[2]).

Because of this quirk, we need to design depcheck to notify the maintainer if their package fails its initial test[3], but not send mail for every failure - after the first time, failed updates can just sit quietly in the holding area[4] until they finally have their dependencies satisfied and pass the test. At that point, the maintainer should get a followup notification to let them know that the update is OK. We might also want to notify maintainers if their packages get stuck in the holding area for a long time, but we haven't decided if (or when) this would be useful or necessary.

It's Actually Even More Complicated Than That

There's actually more subtle complications here. First, you need to know that all Fedora updates are pushed into the live repos by hand - by someone from Fedora Release Engineering (aka rel-eng). So there's going to be a delay - perhaps a few hours - between depcheck approving a package for release and the actual release of the package.

So: updates that have passed depcheck won't actually get moved out of the holding area until someone from rel-eng comes along and pushes them out.[5] But that's fine - we want to include approved (but not-yet-pushed) updates in the depcheck test. We need them there, in fact, because we need to test subsequent updates as if the approved ones are already part of the public package repos (because they will be, just as soon as someone from rel-eng hits the button).

But: if someone revokes or replaces an update, this could cause other previously-approved updates to lose their approval. For example, let's say evolution-data-server turns out to actually have some horrible security bug and needs to be fixed and rebuilt before it gets released into the wild. This would cause our previously-approved empathy update to fail depcheck! So clearly we need to retest all the proposed update - including approved ones - whenever new updates land. And rel-eng should only consider the currently-approved updates when they're pushing out new updates.[6]

Handling Multilib

Multilib is the term for the magic hack that allows you to run 32-bit code on your 64-bit system.[7] It's also the reason i686 packages show up on x86_64 systems (which annoys a lot of x86_64 users, but hey, at least you can use the Flash plugin!). Multilib support allows you to do some strange things - like have two versions of the same package installed (e.g. my system has sqlite.x86_64 and sqlite.i686). They can even both install the same file under certain circumstances (e.g. both sqlite packages install /usr/bin/sqlite3 - and this is totally allowed on multilib systems.)

You might think this would cause some strange complications with (already complicated) dependency checking - and you'd be absolutely right. Luckily, though, yum already handles all of this for us - provided we give it the right things to check.

The i686 packages are placed into the x86_64 repo by a program called mash. Its job is to take a set of builds and decide which ones are multilib - that is, which ones are required for proper functioning of 32-bit binaries on 64-bit systems. When new updates are pushed out, mash is the thing that runs behind the scenes and actually picks the required RPMs and writes out all the metadata.

This means that if we want depcheck's results to be accurate, we need to feed it the same RPMs and metadata that normal users would see once we pushed the updates. Which means that depcheck needs to run mash on the proposed updates, and use the resulting set of RPMs and metadata to run its testing. Otherwise we'll completely miss any weird problems arising from incorrect handling of multilib packages.

Since mash was designed to take a Koji tag as its input, having the -pending tags for proposed updates allows us to use mash just like the normal push would, and therefore we can be sure we're testing the right set of packages. Which means all our multilib problems are solved forever! ..right?

Unsolved Problems and Future Work

Sadly, no. The fact that we're correctly checking multilib dependencies doesn't necessarily mean we'll catch all yum problems involving multilib. For example: problems keep arising when a package (e.g. nss-softokn) accidentally stops being multilib - so then you get an update that upgrades nss-softokn.x86_64 but not nss-softokn.i686. Unfortunately, yum considers this type of update legitimate, and these dependencies to be properly resolved. But subsequent updates that want to use nss-softokn will be confused by the fact that there are two different versions of nss-softokn installed, and then yum will fail.

Another example is file conflicts. Normally it's not allowed for multiple files to install the same package - but, as mentioned above, multilib packages can (under certain circumstances) install multiple copies of the same file. But depcheck doesn't check this - mostly because yum (by itself) does not check for file conflicts. It does use the RPM libraries to check for file conflicts, but this is completely separate from yum's dependency checking code. And strictly speaking, the purpose of the depcheck test is to check dependencies, and this is.. something else.

So: there are problems that depcheck will not solve - not because of bugs in depcheck, but because they're outside of the reach of its design. But it's important to understand what those problems are and - more importantly - to plan for future AutoQA tests that will be able to catch these problems. And we also need to think about how to use the test results to enforce policy - that is, how to make the Fedora infrastructure reject obviously broken updates. Or how to flag seemingly-broken updates for review, and require signoff before releasing them. We'll talk about all that in part 3.

1 Technically not every change to evolution-data-server requires us to rebuild empathy and evolution, but let's just ignore that for now.
2 Like if the maintainer replaces it with a newer version (hopefully one with fixed dependencies!), or if the Fedora Release Engineering team decides to remove it.
3 This message will include the error output so the maintainer knows what other package(s) are causing the problem, and therefore which maintainer to talk to if they want to get the problem resolved.
4 The holding area is actually a set of Koji tags that end in -pending - package maintainers may have seen some email involving this tag. Well, that's what it's for.
5 Yes, this means approved packages will actually keep getting retested even after they get approved. This is another place where we need to avoid notifying maintainers over and over.
6 Note that there's also a small delay between when the update set changes and when the test completes - and so it could be possible for rel-eng to be looking at obsolete test results. We're still trying to figure out the best way to make sure rel-eng is only dealing with up-to-date info.
7 Or 64-bit binaries on your mostly-32-bit system, in the case of ppc64.


  • depcheck: the why and the how (part 3)

    In part 1 I talked about the general idea of the depcheck test, and part 2 got into some of the messy details. If you'd like a more detailed look…

  • depcheck: the why and how (part 1)

    From the very beginning, one of the big goals of the AutoQA project was to set up an automated test that would keep broken updates out of the repos.…

  • A helpful git config snippet

    So, if you're like me, you like to poke through the source of things from time to time but you always forget what the proper URL is for, say, GNOME…

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded