Fedora Torrent seeder plans and future

Greetings.

As some of you may know, we are migrating all of Fedora Infrastructure
over to RHEL6 from the old existing RHEL5 servers. This allows us to
offer newer/better services and have faster/better upstream support for
our services. As part of this migration we have come to our torrent
server ( torrent.fedoraproject.org ).

We have been able to migrate it to a new RHEL6 instance, but looking at
a number of factors, we wonder if this might be a good time to retire
this service.

Problems/Issues/Background:

1. On the release engineering side, the process of creating and signing
the torrent checksums, then syncing them to the torrent seed is
difficult and time consuming. (The torrent seed server can't be in the
same location as all our other rel-eng resources due to open external
port requirements).

2. Torrent usage has been decreasing over time.

a. median active users per day over the last 800 days has shown
a decrease from 528 at the start of the time period to 220 at the
end.
b. total seeds have remained relatively constant
we explain this in that the many clients are consistently seeds
for every release - roughly
c. the total completed downloads are decreasing.

You can see stats for the rhel6/opentracker instance at:
http://torrent.fedoraproject.org/stats/

3. There is not much in the way of open source torrent seed software.
We have cobbled together a working setup on RHEL6, but it's not
trivial. Maintenance and updating this setup will continue to be
difficult over time.

There's a number possible reasons for the decline in usage:

- many providers throttle or block torrent downloads.
- Our download infrastructure and mirrors system has gotten good enough
that direct downloads are usually faster and easier.
- many sites that made torrents popular are also discontinuing them in
favor of other things like magnet links. (see thepiratebay.org).
- More of our users are downloading RC's or the like directly instead
of waiting for release, meaning they don't seed or download the
torrents.

Solutions/Suggestions:

1. Discontinue torrent seeding before F17 alpha.

2. Discontinue torrent seeding, but generate "official" torrent files
for download so others in the community could seed if they wished to.

3. Continue providing a torrent seeder, but set some kind of threshold
for when we might want to discontinue it down the road.

4. Publish a magnet link and point to sites like openbittorrent.com as
a tracker (with direct download also listed).

Thoughts? Questions?

kevin
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On Fri, Jan 20, 2012 at 16:23:10 -0700,
Kevin Fenzi <kevin@scrye.com> wrote:
> 1. Discontinue torrent seeding before F17 alpha.

Some spins have not been advertised on the direct download pages (and to
the best of my knowledge aren't mirrored). If torrents are not provided
for these is there going to be a plan to get download links to them
and have them mirrored? Is increasing the amount of stuff being mirrored
going to cause a problem?

> 2. Discontinue torrent seeding, but generate "official" torrent files
> for download so others in the community could seed if they wished to.

I don't think we want to do this. It will reflect badly on Fedora to
have pointers to torrents where no one is seeding them. While the main
spins will probably get someone to seed them, some of the less popular
ones might not.
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 20/01/12 23:23, Kevin Fenzi wrote:


Solutions/Suggestions:

1. Discontinue torrent seeding before F17 alpha.


I prefer torrents, but if you need to free up resoursessecurity.
No problem, if 2 or 4 enabled.



2. Discontinue torrent seeding, but generate "official" torrent files
for download so others in the community could seed if they wished to.


I would prefer, if infra uploaded them,
to whatever tracker for peace of mind.
(uploaded by ruth.kitt12)



3. Continue providing a torrent seeder, but set some kind of threshold
for when we might want to discontinue it down the road.


temp solution, not the best.

> 4. Publish a magnet link and point to sites like openbittorrent.com as

a tracker (with direct download also listed).



As long as the official ones dieare removed at official EOL time

Personally, I dl all the Fedora torrents,
and reseed, until "no authorised" time.
Would have no objection to continue to do so.
F14 is still running

--
Regards,

Frank Murphy
UTF_8 Encoded
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On Sat, 21 Jan 2012 00:49:58 -0600
Bruno Wolff III <bruno@wolff.to> wrote:

> On Fri, Jan 20, 2012 at 16:23:10 -0700,
> Kevin Fenzi <kevin@scrye.com> wrote:
> > 1. Discontinue torrent seeding before F17 alpha.
>
> Some spins have not been advertised on the direct download pages (and
> to the best of my knowledge aren't mirrored). If torrents are not
> provided for these is there going to be a plan to get download links
> to them and have them mirrored? Is increasing the amount of stuff
> being mirrored going to cause a problem?

They are in fact mirrored and available for direct download.

They are not in the main Fedora sync collection, but are in the 'alt'
collection. Currently thats not as widely mirrored as our main content,
but it is mirrored. I see 8 mirrors currently up to date on that
content.

> > 2. Discontinue torrent seeding, but generate "official" torrent
> > files for download so others in the community could seed if they
> > wished to.
>
> I don't think we want to do this. It will reflect badly on Fedora to
> have pointers to torrents where no one is seeding them. While the main
> spins will probably get someone to seed them, some of the less popular
> ones might not.

True.

kevin


_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 20 January 2012 16:23, Kevin Fenzi <kevin@scrye.com> wrote:
> Greetings.
>
> As some of you may know, we are migrating all of Fedora Infrastructure
> over to RHEL6 from the old existing RHEL5 servers. This allows us to
> offer newer/better services and have faster/better upstream support for
> our services. As part of this migration we have come to our torrent
> server ( torrent.fedoraproject.org ).
>

I see there were outstanding questions about the torrents from the
board, but I am not exactly sure what they were:

I will put in the costs as I see:

1) current server/seeder is no longer maintained or opensource
2) replacement code servers are either:
a) meant to be run from a desktop and not good un-supervised server items
b) compile various bits and configs in statically.
3) Due to large amount of copyright violations torrent ports and
protocols are becoming harder to get access to in the US making
seeding harder for main servers.
4) We get more service requests for EOL seeds than we do present ones
that aren't seeded.


Benefits as I see them:

1) Slow speed downloads are "faster overall" via torrent than rsync.
[Faster being measured in continual net speed and automatic error
recovery.]
2) Clients outside of US don't have as many problems as US in closed
ports or active net blocking

Solutions:
1) Fedora uses closed sourced torrent server for ipv6 and other options.
2) We publish the torrent 'seeds' and signatures for other sites and
users to set up.

others reported.

--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Years ago my mother used to say to me,... Elwood, you must be oh
so smart or oh so pleasant. Well, for years I was smart. I
recommend pleasant. You may quote me." Â*—James Stewart as Elwood P. Dowd
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 01/27/2012 12:45 AM, Stephen John Smoogen wrote:

>
> Solutions:
> 1) Fedora uses closed sourced torrent server for ipv6 and other options.
> 2) We publish the torrent 'seeds' and signatures for other sites and
> users to set up.

What are other major distributions using?

Rahul

_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 27/01/12 17:35, Rahul Sundaram wrote:

On 01/27/2012 12:45 AM, Stephen John Smoogen wrote:



Solutions:
1) Fedora uses closed sourced torrent server for ipv6 and other options.
2) We publish the torrent 'seeds' and signatures for other sites and
users to set up.


What about azureus, it's in Fedora?



What are other major distributions using?

Rahul



Most use torrents, along with direct download.
But some use 3rd Party for to host their *torrent files
Scientific Linux:
http://www.scientificlinux.org/download/torrents

CentOS: you have to traverse the mirror sites to find the *.torrent files
http://ftp.heanet.ie/pub/centos/6.2/isos/i386/

Ubuntu: has torrents on their "alternate doownloads"
http://www.ubuntu.com/download/ubuntu/alternative-download

--
Regards,

Frank Murphy, friend of fedoraproject
UTF_8 Encoded
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On Fri, 27 Jan 2012 17:50:31 +0000
Frank Murphy <frankly3d@gmail.com> wrote:

> On 27/01/12 17:35, Rahul Sundaram wrote:
> > On 01/27/2012 12:45 AM, Stephen John Smoogen wrote:
> >
> >>
> >> Solutions:
> >> 1) Fedora uses closed sourced torrent server for ipv6 and other
> >> options. 2) We publish the torrent 'seeds' and signatures for
> >> other sites and users to set up.
>
> What about azureus, it's in Fedora?

it's not in EPEL (we use RHEL for our servers).

it's java. We have 0 other deployed java applications, so this would be
a very large ramp up for support and management.

it's a gui application, making it difficult or impossible to manage
remotely.

> > What are other major distributions using?
> >
> > Rahul
> >
>
> Most use torrents, along with direct download.
> But some use 3rd Party for to host their *torrent files
> Scientific Linux:
> http://www.scientificlinux.org/download/torrents

yeah, this was an option I listed... we could produce the torrent files
and have some other place seed/track them.

> CentOS: you have to traverse the mirror sites to find the *.torrent
> files http://ftp.heanet.ie/pub/centos/6.2/isos/i386/
>
> Ubuntu: has torrents on their "alternate doownloads"
> http://www.ubuntu.com/download/ubuntu/alternative-download

From a quick look:

ubuntu and debian both use: tracker version: T-0.3.18 (BitTornado)
It's a MIT licenced fork of bittorrent I think. It's not packaged for
Fedora/EPEL. Looks like the last update on it was 2+ years ago
upstream.

opensuse I can't really tell.

kevin
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 01/28/2012 01:11 AM, Kevin Fenzi wrote:

>
> From a quick look:
>
> ubuntu and debian both use: tracker version: T-0.3.18 (BitTornado)
> It's a MIT licenced fork of bittorrent I think. It's not packaged for
> Fedora/EPEL. Looks like the last update on it was 2+ years ago
> upstream.

Is this feasible to use? Have we tested it at all?

>
> opensuse I can't really tell.

Could drop in a mail and ask

Rahul
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On Sat, 28 Jan 2012 01:17:14 +0530
Rahul Sundaram <metherid@gmail.com> wrote:

> On 01/28/2012 01:11 AM, Kevin Fenzi wrote:
>
> >
> > From a quick look:
> >
> > ubuntu and debian both use: tracker version: T-0.3.18 (BitTornado)
> > It's a MIT licenced fork of bittorrent I think. It's not packaged
> > for Fedora/EPEL. Looks like the last update on it was 2+ years ago
> > upstream.
>
> Is this feasible to use? Have we tested it at all?

It's basically the code we were running on the old rhel5 instance.

It's a bit scary to run a internet facing service like this with
basically no upstream maint and security fixes. ;(
I don't know if it even builds on RHEL6/Anything modern.

> > opensuse I can't really tell.
>
> Could drop in a mail and ask

Please do.

> Rahul

kevin
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board