FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Infrastructure

 
 
LinkBack Thread Tools
 
Old 07-25-2008, 05:44 PM
Matt Domsch
 
Default YUM security issues...

On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote:
> On 25 July 2008, Matt Domsch wrote:
> >
> > Yes, this is a known challenge with subnet delegation in
> > MirrorManager. We're trusting package signing (and soon, repodata
> > signing) to prevent rogue mirrors from issuing unsigned data. In
> > addition, I'm working on adding in a way to prevent stale mirrors
> > (with signed content) from being used.
> >
>
> How does one get this subnet delegation though? Can I request any subnet I
> want, or do we do some sort of verification?

At present there is no verification (I'm not at all sure how one
_could_ verify except by ARIN & co delegation). However there are
limits as to how large a block can be requested. Nothing larger than
a IPv4 /16 can be automatically requested. Fedora Infrastructure
admins can add larger blocks, and request ARIN & co data when doing so.


> What happens if the client decided its mirror is bad, I presume it will go
> off and find a better one, even with delegation?

Yes, the mirrorlist returned includes quite a few mirrors, in priority order.

--
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-25-2008, 05:52 PM
Josh Bressers
 
Default YUM security issues...

On 25 July 2008, Matt Domsch wrote:
> On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote:
> > On 25 July 2008, Matt Domsch wrote:
> > >
> > > Yes, this is a known challenge with subnet delegation in
> > > MirrorManager. We're trusting package signing (and soon, repodata
> > > signing) to prevent rogue mirrors from issuing unsigned data. In
> > > addition, I'm working on adding in a way to prevent stale mirrors
> > > (with signed content) from being used.
> > >
> >
> > How does one get this subnet delegation though? Can I request any subnet I
> > want, or do we do some sort of verification?
>
> At present there is no verification (I'm not at all sure how one
> _could_ verify except by ARIN & co delegation). However there are
> limits as to how large a block can be requested. Nothing larger than
> a IPv4 /16 can be automatically requested. Fedora Infrastructure
> admins can add larger blocks, and request ARIN & co data when doing so.
>

That's a lot of IPs though. Can I request multiple /16s, or only one?

How many mirrors are doing this? Does the mirror have to be part of the
/16 to request it?

Thanks for the patience here. I'm trying to understand the risk we're
dealing with.

Thanks.

--
JB

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-25-2008, 06:36 PM
Justin Samuel
 
Default YUM security issues...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt Domsch wrote:
> On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote:
>> On 25 July 2008, Matt Domsch wrote:
>>> Yes, this is a known challenge with subnet delegation in
>>> MirrorManager. We're trusting package signing (and soon, repodata
>>> signing) to prevent rogue mirrors from issuing unsigned data. In
>>> addition, I'm working on adding in a way to prevent stale mirrors
>>> (with signed content) from being used.
>>>
>> How does one get this subnet delegation though? Can I request any subnet I
>> want, or do we do some sort of verification?
>
> At present there is no verification (I'm not at all sure how one
> _could_ verify except by ARIN & co delegation). However there are
> limits as to how large a block can be requested. Nothing larger than
> a IPv4 /16 can be automatically requested. Fedora Infrastructure
> admins can add larger blocks, and request ARIN & co data when doing so.
>
>
>> What happens if the client decided its mirror is bad, I presume it will go
>> off and find a better one, even with delegation?
>
> Yes, the mirrorlist returned includes quite a few mirrors, in priority order.

Our testing showed that when our client was in a MirrorManager-defined
CIDR block for a mirror, the returned mirrorlist included only the
single mirror. -- It's dangerous either way, of course, but I'm just
wondering if our testing was faulty, if this has changed since we
tested, or if it might be behaving differently than you expect.

Possibly you tested with a block that was already defined by other
mirrors and so multiple entries were returned in the mirrorist? That's
just a guess, we didn't test with a block that was defined by more than
one mirror (as far as we knew, at least).

- --
Justin Samuel
https://www.cs.arizona.edu/~jsamuel/
gpg: 0xDDF1F3EE [66EF 84E2 F184 B140 712B 55A7 2B96 AB8F DDF1 F3EE]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIih0cK5arj93x8+4RAklWAKC+Lewfd+pixUvL2MvbdC YxnjHBpQCdHtNd
x5BQsM6GqW5zKpJt+RH8Vco=
=w9yV
-----END PGP SIGNATURE-----

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-25-2008, 06:47 PM
 
Default YUM security issues...

Fedora 7 definitely behaves differently than Fedora 8 and 9. The
behavior I describe began with F8. For F7 and earlier, the yum policy
would chose any random mirror from the returned list, so having many
mirrors on the list, some of which are unreachable from inside an
organization, would be bad. The yum default policy was changed to treat
the mirrorlist as a priority list, so MM returns a longer list.


--
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux


-----Original Message-----
From: Justin Samuel [mailto:jsamuel.cs.arizona.edu@gmail.com] On Behalf
Of Justin Samuel
Sent: Friday, July 25, 2008 1:36 PM
To: Domsch, Matt
Cc: Josh Bressers; Mike McGrath; fedora-infrastructure-list@redhat.com;
Justin Cappos; security@fedoraproject.org
Subject: Re: YUM security issues...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt Domsch wrote:
> On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote:
>> On 25 July 2008, Matt Domsch wrote:
>>> Yes, this is a known challenge with subnet delegation in
>>> MirrorManager. We're trusting package signing (and soon, repodata
>>> signing) to prevent rogue mirrors from issuing unsigned data. In
>>> addition, I'm working on adding in a way to prevent stale mirrors
>>> (with signed content) from being used.
>>>
>> How does one get this subnet delegation though? Can I request any
subnet I
>> want, or do we do some sort of verification?
>
> At present there is no verification (I'm not at all sure how one
> _could_ verify except by ARIN & co delegation). However there are
> limits as to how large a block can be requested. Nothing larger than
> a IPv4 /16 can be automatically requested. Fedora Infrastructure
> admins can add larger blocks, and request ARIN & co data when doing
so.
>
>
>> What happens if the client decided its mirror is bad, I presume it
will go
>> off and find a better one, even with delegation?
>
> Yes, the mirrorlist returned includes quite a few mirrors, in priority
order.

Our testing showed that when our client was in a MirrorManager-defined
CIDR block for a mirror, the returned mirrorlist included only the
single mirror. -- It's dangerous either way, of course, but I'm just
wondering if our testing was faulty, if this has changed since we
tested, or if it might be behaving differently than you expect.

Possibly you tested with a block that was already defined by other
mirrors and so multiple entries were returned in the mirrorist? That's
just a guess, we didn't test with a block that was defined by more than
one mirror (as far as we knew, at least).

- --
Justin Samuel
https://www.cs.arizona.edu/~jsamuel/
gpg: 0xDDF1F3EE [66EF 84E2 F184 B140 712B 55A7 2B96 AB8F DDF1 F3EE]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIih0cK5arj93x8+4RAklWAKC+Lewfd+pixUvL2MvbdC YxnjHBpQCdHtNd
x5BQsM6GqW5zKpJt+RH8Vco=
=w9yV
-----END PGP SIGNATURE-----

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-25-2008, 07:06 PM
Matt Domsch
 
Default YUM security issues...

On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote:
> That's a lot of IPs though. Can I request multiple /16s, or only one?

As many as you like. And recall, such changes are made using your FAS
credentials.

> How many mirrors are doing this?

374 total Hosts
185 have at least 1 netblock entry
94 of these are "private" - don't serve the public


> Does the mirror have to be part of the /16 to request it?

no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US,
but the mirror IPs are located inside the 10/8 private space.


> Thanks for the patience here. I'm trying to understand the risk we're
> dealing with.


I'm happy for the comments and review. Keep it coming!

--
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-26-2008, 01:11 AM
Josh Bressers
 
Default YUM security issues...

On 25 July 2008, Matt Domsch wrote:
> On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote:
> > That's a lot of IPs though. Can I request multiple /16s, or only one?
>
> As many as you like. And recall, such changes are made using your FAS
> credentials.

Are these ever checked? Does say a mail get generated every time someone
adds one of these? My fear would be that someone could blanket quite a
large IP space without anyone noticing. Granted that would no doubt
generate a huge volume of traffic, but if they're serving up a frozen repo,
they probably won't be pushing all that much data.

>
> > How many mirrors are doing this?
>
> 374 total Hosts
> 185 have at least 1 netblock entry
> 94 of these are "private" - don't serve the public
>

wow, that's quite a few. I wasn't expecting numbers this high honestly.

>
> > Does the mirror have to be part of the /16 to request it?
>
> no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US,
> but the mirror IPs are located inside the 10/8 private space.
>

OK, so here is the problem the way I see it, signing the repository won't
fix it. I'll try to explain this clearly, Justin can yell at me if I've
gotten any of this wrong.

So let's say Mallory (the bad guy) decides that he wants to host a
malicious mirror and wait for a nasty security flaw. He sets up his mirror
and even claims some IP subnets to serve. Bob and Alice are happily
installing valid updates from him for some period of time. Since Mallory
has claimed to serve a specific subnet, he has a rather impressive view of
what Bob and Alice have installed.

Now let's say there is a horrible security bug found in a mail server.
Mallory knows for a fact that Bob and Alice both have it installed as he's
been their mirror for a while. Mallory stops updating his mirror, so none
of the users being served will get the mail server updates. Mallory also
knows the IP address of the vulnerable clients and can easily break into
their systems.

So from what I understand MirrorManager will check on the mirrors to ensure
they're not out of date. Mallory knows this and makes sure that when
MirrorManager connects to his mirror, it lies and serves up current
metadata.


So here is the problem. The repodata was valid. The packages are signed.
Even if we sign the repodata, this attack works. Being able to acquire an
IP block simply makes this attack easier to do. It's still very possible
that a bad mirror will wait for users to connect, serve up old content then
use this knowledge to break into their system.

What this problem boils down to, is we need a way for clients to ask
MirrorManager what the current valid repo data is. Ideally we want the
results to be signed in some manner so it can't be spoofed.

Some thoughts I've had are:

1) Have MirrorManager use https and return some repo verification data.
2) Sign the repo data, and if it's older than X, don't use it (I don't like
this solution, but it's probably the easiest, just push out a new
signed repo file once a day, even if nothing changes.)
3) Always get repo data from fedoraproject.org (probably not practical due
to resource issues)
4) use DNS, have the client query
<repodata sha1sum>.repo.fedoraproject.org
if the lookup fails, the repo is invalid. (this is really cheap from a
resource standpoint, but hard to do technically)
5) ???

Thanks.

--
JB

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-26-2008, 01:41 AM
Toshio Kuratomi
 
Default YUM security issues...

Josh Bressers wrote:

On 25 July 2008, Matt Domsch wrote:

On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote:

That's a lot of IPs though. Can I request multiple /16s, or only one?

As many as you like. And recall, such changes are made using your FAS
credentials.


Are these ever checked? Does say a mail get generated every time someone
adds one of these? My fear would be that someone could blanket quite a
large IP space without anyone noticing. Granted that would no doubt
generate a huge volume of traffic, but if they're serving up a frozen repo,
they probably won't be pushing all that much data.



How many mirrors are doing this?

374 total Hosts
185 have at least 1 netblock entry
94 of these are "private" - don't serve the public



wow, that's quite a few. I wasn't expecting numbers this high honestly.


Does the mirror have to be part of the /16 to request it?

no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US,
but the mirror IPs are located inside the 10/8 private space.



OK, so here is the problem the way I see it, signing the repository won't
fix it. I'll try to explain this clearly, Justin can yell at me if I've
gotten any of this wrong.

So let's say Mallory (the bad guy) decides that he wants to host a
malicious mirror and wait for a nasty security flaw. He sets up his mirror
and even claims some IP subnets to serve. Bob and Alice are happily
installing valid updates from him for some period of time. Since Mallory
has claimed to serve a specific subnet, he has a rather impressive view of
what Bob and Alice have installed.

Now let's say there is a horrible security bug found in a mail server.
Mallory knows for a fact that Bob and Alice both have it installed as he's
been their mirror for a while. Mallory stops updating his mirror, so none
of the users being served will get the mail server updates. Mallory also
knows the IP address of the vulnerable clients and can easily break into
their systems.

So from what I understand MirrorManager will check on the mirrors to ensure
they're not out of date. Mallory knows this and makes sure that when
MirrorManager connects to his mirror, it lies and serves up current
metadata.


So here is the problem. The repodata was valid. The packages are signed.
Even if we sign the repodata, this attack works. Being able to acquire an
IP block simply makes this attack easier to do. It's still very possible
that a bad mirror will wait for users to connect, serve up old content then
use this knowledge to break into their system.

What this problem boils down to, is we need a way for clients to ask
MirrorManager what the current valid repo data is. Ideally we want the
results to be signed in some manner so it can't be spoofed.

Some thoughts I've had are:

1) Have MirrorManager use https and return some repo verification data.
We'd need to push repo verification data for both the latest and
previous repo information. MirrorManager would have to be updated with
new data as part of the updates/rawhide push so that it's always up to
date with the mirror. We'll have to revise mirrormanager's caching
model so that it always has access to the latest repository verification
information.



2) Sign the repo data, and if it's older than X, don't use it (I don't like
this solution, but it's probably the easiest, just push out a new
signed repo file once a day, even if nothing changes.)
This is going to have to have some policy attached to it. Do we
continue to sign the repo data for EOL releases because people use them
even though we tell people they're EOL?



3) Always get repo data from fedoraproject.org (probably not practical due
to resource issues)
This is the easiest to implement. It means the small repomd.xml file
always comes from our server. But the rest of the metadata can come
from the individual mirrors. However, it does mean that *very* large
swaths of the mirrors will be unable to serve packages to users at any
time because their metadata won't match with ours for some period after
we have an update pushed. Maybe we could do this with versioned
repodata and the client can decide how far back in time or how many past
revisions it is willing to accept.



4) use DNS, have the client query
<repodata sha1sum>.repo.fedoraproject.org
if the lookup fails, the repo is invalid. (this is really cheap from a
resource standpoint, but hard to do technically)


I don't know enough about implementing this one to comment.

Also, all of these solutions need client-side support. That being the
case, it should be generic enough that other repomd consuming clients
and distributions will be willing to use it. Otherwise we'll be
securing our updates and mirror infrastructure with the default package
manager but doing nothing for the ecosystem as a whole.


-Toshio

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-26-2008, 02:04 AM
"Justin Cappos"
 
Default YUM security issues...

Yes, you clearly described one of the attacks we see with MirrorManager.

A few comments:

> 1) Have MirrorManager use https and return some repo verification data.

Is the verification data a signed repomd.xml? Can you expand on this a little?

By the way, before I forget it would be a good idea to avoid using a
detached signature for repomd.xml. If you have the signature and
data in separate files, you can have false positives for incorrect
signatures (for example when the files are updated).


> 2) Sign the repo data, and if it's older than X, don't use it (I don't like
> this solution, but it's probably the easiest, just push out a new
> signed repo file once a day, even if nothing changes.)

You also want to make sure clients don't accept older versions of the
metadata than what they've seen. In other words, if they'll accept
metadata up to 1 week old, and they've downloaded metadata from
yesterday, they shouldn't accept metadata that was signed 5 days ago.

> 3) Always get repo data from fedoraproject.org (probably not practical due
> to resource issues)

I think this is similar to what openSUSE / SUSE Linux Enterprise do
(they actually serve signed metadata from their Download Redirector),
so it may be practical for you to do in practice. However, you need
to worry about man-in-the-middle attackers...

> 4) use DNS, have the client query
> <repodata sha1sum>.repo.fedoraproject.org
> if the lookup fails, the repo is invalid. (this is really cheap from a
> resource standpoint, but hard to do technically)

Same comment about man-in-the-middle attacks. DNS-cache poisoning
trivially circumvents this protection...

Thanks,
Justin

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-26-2008, 03:08 AM
seth vidal
 
Default YUM security issues...

On Fri, 2008-07-25 at 18:41 -0700, Toshio Kuratomi wrote:

> > 3) Always get repo data from fedoraproject.org (probably not practical due
> > to resource issues)
> This is the easiest to implement. It means the small repomd.xml file
> always comes from our server. But the rest of the metadata can come
> from the individual mirrors. However, it does mean that *very* large
> swaths of the mirrors will be unable to serve packages to users at any
> time because their metadata won't match with ours for some period after
> we have an update pushed. Maybe we could do this with versioned
> repodata and the client can decide how far back in time or how many past
> revisions it is willing to accept.

We don't need to version the metadata, we have timestamps in them
already.

We can easily do a comparison from this timestamp to now. And we can set
this number to be different from the base repo as to the updates repo.

But as you've already mentioned we're stuck with the question of EOL'd
releases and how to deal with things deeply out of date.

I can make yum throw out warnings and alerts but at what point does it
actually STOP doing anything and does that not open us up to a different
kind of DoS?


> I don't know enough about implementing this one to comment.
>
> Also, all of these solutions need client-side support. That being the
> case, it should be generic enough that other repomd consuming clients
> and distributions will be willing to use it. Otherwise we'll be
> securing our updates and mirror infrastructure with the default package
> manager but doing nothing for the ecosystem as a whole.

We need to make sure that whatever we implement is trivially done so by
people running a local downstream branch of fedora or centos or
what-have-you. Or, as you say, we've saved ourselves and screwed
everyone else.

-sv


_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 07-26-2008, 05:06 PM
Josh Bressers
 
Default YUM security issues...

On 25 July 2008, seth vidal wrote:
>
> But as you've already mentioned we're stuck with the question of EOL'd
> releases and how to deal with things deeply out of date.
>
> I can make yum throw out warnings and alerts but at what point does it
> actually STOP doing anything and does that not open us up to a different
> kind of DoS?
>

This is of course a policy decision that can be dictated via a
configuration file. There is also the issue of what happens when the
client keeps getting back "old" data? It need to go find some new data as
in our attack scenario, they are running something that needs updating. I
would think that the client should try some number of mirrors, if they all
return the same old data, it's probably EOL or something similar. If an
attacker has enough control over you to prevent you connecting to arbitrary
mirrors, you likely have bigger issues than this.

>
> We need to make sure that whatever we implement is trivially done so by
> people running a local downstream branch of fedora or centos or
> what-have-you. Or, as you say, we've saved ourselves and screwed
> everyone else.
>

Yes, I'm starting to think we should be having this discussion with yum
upstream. Now that I have a grasp of what MirrorManager does, I don't
think there's really anything to fix there. We need to fix the client.

Seth, which list would you suggest to move this discussion so I can stop
pestering the infrastructure folks?

Thanks.

--
JB

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 

Thread Tools




All times are GMT. The time now is 11:53 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org