FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 04-07-2011, 11:45 AM
"tv.debian@googlemail.com"
 
Default Managing large numbers (100+) of Debian-based machines

07/04/2011 13:21, Laurence Hurst:
> Hi folks,
> I am looking at possibly deploying Debian (or a derivative thereof)
>on a large number of machines (initially 80 but I fully expect the final
>number to end up being in excess of 100) which are intermittently
>connected to the network (due to multi-boot setup). I was wondering
>if anyone else has a large(ish) installation base and what tools they
>were using to manage the machines (especially when it comes to patching)?
>On this scale manual ssh methods become unmanageable, even using the
>likes of pdsh and before I even consider the fact the machines will not
>all be in Linux at any given time. Unfortunately I have no budget for
>this project so an Ubuntu/Landscape solution is not workable. Ideally
>something usable by non-Linux (but technically competent Windows)
>administrators would be nice (e.g. a "point and shoot" web-interface
>to mass deploy updates). Does anyone have any suggestions of existing
>projects which fit this bill or am I going to have to roll my own
>solution?
> Thanks in advance
> Laurence
>
>

Hi, maybe you can have a look at cfengine[1], seems to fit your needs
and it's readily available in Debian. I don't have hands-on experience
with it though.

[1] http://www.cfengine.org/pages/what_is_cfengine


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4D9DA3F3.8060902@googlemail.com">http://lists.debian.org/4D9DA3F3.8060902@googlemail.com
 
Old 04-07-2011, 12:05 PM
Alex Mestiashvili
 
Default Managing large numbers (100+) of Debian-based machines

On 04/07/2011 01:21 PM, Laurence Hurst wrote:

Hi folks,
I am looking at possibly deploying Debian (or a derivative thereof) on a large number of machines (initially 80 but I fully expect the final number to end up being in excess of 100) which are intermittently connected to the network (due to multi-boot setup). I was wondering if anyone else has a large(ish) installation base and what tools they were using to manage the machines (especially when it comes to patching)? On this scale manual ssh methods become unmanageable, even using the likes of pdsh and before I even consider the fact the machines will not all be in Linux at any given time. Unfortunately I have no budget for this project so an Ubuntu/Landscape solution is not workable. Ideally something usable by non-Linux (but technically competent Windows) administrators would be nice (e.g. a "point and shoot" web-interface to mass deploy updates). Does anyone have any suggestions of existing projects which fit this bill or am I going to have to roll my own solution?
Thanks in advance
Laurence




have a look on FAI http://fai-project.org/ for installations .

Regards ,
Alex


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4D9DA875.3060303@biotec.tu-dresden.de">http://lists.debian.org/4D9DA875.3060303@biotec.tu-dresden.de
 
Old 04-07-2011, 12:30 PM
Axel Freyn
 
Default Managing large numbers (100+) of Debian-based machines

Hi,
On Thu, Apr 07, 2011 at 01:45:55PM +0200, tv.debian@googlemail.com wrote:
> 07/04/2011 13:21, Laurence Hurst:
> > Hi folks,
> > I am looking at possibly deploying Debian (or a derivative thereof)
> >on a large number of machines (initially 80 but I fully expect the final
> >number to end up being in excess of 100) which are intermittently
> >connected to the network (due to multi-boot setup). I was wondering
> >if anyone else has a large(ish) installation base and what tools they
> >were using to manage the machines (especially when it comes to patching)?
> >On this scale manual ssh methods become unmanageable, even using the
> >likes of pdsh and before I even consider the fact the machines will not
> >all be in Linux at any given time. Unfortunately I have no budget for
> >this project so an Ubuntu/Landscape solution is not workable. Ideally
> >something usable by non-Linux (but technically competent Windows)
> >administrators would be nice (e.g. a "point and shoot" web-interface
> >to mass deploy updates). Does anyone have any suggestions of existing
> >projects which fit this bill or am I going to have to roll my own
> >solution?
> > Thanks in advance
> > Laurence
> >
> >
>
> Hi, maybe you can have a look at cfengine[1], seems to fit your needs
> and it's readily available in Debian. I don't have hands-on experience
> with it though.
>
> [1] http://www.cfengine.org/pages/what_is_cfengine

Another approach would be "FAI" http://fai-project.org
In addition to a "update", it also contains the installation system. In short:
- you create so-called "classes" which describe the
machines/configuration/software (e.g. graphics drivers, which
software to install, what is the default printer, ...)
- for every machine you define which classes apply to this machine
- this configuration space is put on a central server
No, for a full installation of one client, you have to boot the client
from the "install" system (can be USB-Stick, CD, or you boot from the
network -- its only a minimal linux)
- you need a DHCP-Server which distributes the correct
Hostnames/IP-Adresse to the machine (identified by its MAC-adress) --
or you have to select the hostname by hand during the full
installation.
- then, FAI runs the full install (partition harddisks, create
filesystems, install debian base-system, install additional software,
apply patches,... as configured by the "classes")
- after a restart, your machine is fully installed & configured :-)

For patches: fai supports a feature "softupdate", which can be executed
on a running client, and which then reads again the full
configuration-space.
So you just need a way to start the softupdate regularly (my solution: I
have a "version-file" on the server and a local one on each client. At
every boot, the client checks the version-number in the server &
compares it to its local one -- and starts the update if necessary).

I'm happy with FAI ;-) Once you have the configuration space, everything
"just works" -- and re-installations (e.g. due to hardware failures)
are no problems.

Axel


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110407123045.GF18887@axel">http://lists.debian.org/20110407123045.GF18887@axel
 
Old 04-07-2011, 07:01 PM
Peter Beck
 
Default Managing large numbers (100+) of Debian-based machines

On Thu, 2011-04-07 at 13:45 +0200, tv.debian@googlemail.com wrote:
> Hi, maybe you can have a look at cfengine[1], seems to fit your needs
> and it's readily available in Debian. I don't have hands-on experience
> with it though.

what about puppet ? Anyone made experience with puppet on debian ?

http://projects.puppetlabs.com/projects/1/wiki/Puppet_Debian



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1302202912.4589.1.camel@peanut.datentraeger.li">ht tp://lists.debian.org/1302202912.4589.1.camel@peanut.datentraeger.li
 
Old 04-07-2011, 08:10 PM
Tom Grace
 
Default Managing large numbers (100+) of Debian-based machines

On 07/04/11 20:01, Peter Beck wrote:


what about puppet ? Anyone made experience with puppet on debian ?
I do lots of Puppet on Debian, and it's a good way to get consistent
machines. Not sure it's the right thing for the OP, I got the impression
that they were just after I way of making sure all the boxes were updated.


If it's just updates, something like cron-apt might be the best thing.
If it's managing the whole machine, Puppet/Cfengine is the way to go
(though there is a learning curve).



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: 4D9E1A2E.3000502@deathbycomputers.co.uk">http://lists.debian.org/4D9E1A2E.3000502@deathbycomputers.co.uk
 
Old 04-07-2011, 09:55 PM
Rob Owens
 
Default Managing large numbers (100+) of Debian-based machines

On Thu, Apr 07, 2011 at 12:21:30PM +0100, Laurence Hurst wrote:
> Hi folks,
> I am looking at possibly deploying Debian (or a derivative thereof) on a large number of machines (initially 80 but I fully expect the final number to end up being in excess of 100) which are intermittently connected to the network (due to multi-boot setup). I was wondering if anyone else has a large(ish) installation base and what tools they were using to manage the machines (especially when it comes to patching)? On this scale manual ssh methods become unmanageable, even using the likes of pdsh and before I even consider the fact the machines will not all be in Linux at any given time. Unfortunately I have no budget for this project so an Ubuntu/Landscape solution is not workable. Ideally something usable by non-Linux (but technically competent Windows) administrators would be nice (e.g. a "point and shoot" web-interface to mass deploy updates). Does anyone have any suggestions of existing projects which fit this bill or am I going to have to roll my own solution?
> Thanks in advance
> Laurence
>
If you want to check out updates before rolling them out to all
machines, I would set up a local apt repository. Have the machines set
to automatically update, but have sources.list point to your local repo.
That way anything you add to the repo will automatically get rolled out
to all machines.

If you decide not to go with your own apt repo, I highly recommend you
at least use something like apt-cacher-ng. It'll save a lot of download
bandwidth and time.

-Rob


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110407215512.GB13614@aurora.owens.net">http://lists.debian.org/20110407215512.GB13614@aurora.owens.net
 
Old 04-07-2011, 10:11 PM
Peter Beck
 
Default Managing large numbers (100+) of Debian-based machines

On Thu, 2011-04-07 at 17:55 -0400, Rob Owens wrote:
> If you decide not to go with your own apt repo, I highly recommend you
> at least use something like apt-cacher-ng. It'll save a lot of
> download bandwidth and time.

If you're using different distros (debian, ubuntu...) I also recommend
apt-cacher-ng because of the remapping feature. apt-cacher does not work
well with debian and ubuntu at the same time because sometimes there are
packages with the same name...

a little bit off-topic:

I am using apt-cacher, with apt-cacher-ng I had issues with changing the
repository-path to an nfs-share. Is that usually working or is it not
possible to change the path ? I couldn't find anything related in the
manual of apt-cacher-ng

--
Wer die Freiheit aufgibt um Sicherheit zu gewinnen,
der wird am Ende beides verlieren.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1302214291.4589.25.camel@peanut.datentraeger.li">h ttp://lists.debian.org/1302214291.4589.25.camel@peanut.datentraeger.li
 
Old 04-07-2011, 10:42 PM
"Dr. Ed Morbius"
 
Default Managing large numbers (100+) of Debian-based machines

on 12:21 Thu 07 Apr, Laurence Hurst (L.A.Hurst@lboro.ac.uk) wrote:
> Hi folks,
> I am looking at possibly deploying Debian (or a derivative thereof) on
> a large number of machines (initially 80 but I fully expect the final
> number to end up being in excess of 100) which are intermittently
> connected to the network (due to multi-boot setup). I was wondering if
> anyone else has a large(ish) installation base and what tools they
> were using to manage the machines (especially when it comes to
> patching)? On this scale manual ssh methods become unmanageable, even
> using the likes of pdsh and before I even consider the fact the
> machines will not all be in Linux at any given time. Unfortunately I
> have no budget for this project so an Ubuntu/Landscape solution is not
> workable. Ideally something usable by non-Linux (but technically
> competent Windows) administrators would be nice (e.g. a "point and
> shoot" web-interface to mass deploy updates). Does anyone have any
> suggestions of existing projects which fit this bill or am I going to
> have to roll my own solution?

Basics:

- Deployment: FAI + PXEboot (especially for servers).

- Management: pdsh/dsh for ad-hockery is good. You'll need a tweaked
sudoers file (to be able to run at least diagnostic commands) for
management accounts. For real management...

- Puppet, chef, or cfengine. I'd lean toward puppet or chef. These
are used by a number of organizations especially for end-user system
management (Google makes extensive use of Puppet for engineering
desktops/laptops).

- A user-accessible ticket request system for end-user support.
Backed by ...

- A bugtracking and issues database (wiki). The ticket-tracking,
bugtracking, and wiki don't have to be fully integrated, though the
first two likely are. There's some separation of roles, though in
my experience the less between the first two the better. It's just
that some organizations have a lower tolerance for seeing the
sausage-making involved in issues resolution than others, and of
course, of learning a complex interface. A good BTS will offer
appropriate end-user and engineering views on the process.

- A host management database of some sort. Unfortunately I don't have
any recommendations as most I've seen are in-house, DIY systems.
I'd be very open to discussion of methods.

- A local Debian mirror, or at the very least, apt-proxy, to minimize
bandwidth utilization.

- Proper test/staging of new releases / updates, as well as a
designated "guinea pig" group for testing updates (some things just
can't be tested in an automated fashion), essential for end-user
deployments.


A bevvy of other things I'm leaving out, but I'd like to see what others
have to say.


--
Dr. Ed Morbius, Chief Scientist / |
Robot Wrangler / Staff Psychologist | When you seek unlimited power
Krell Power Systems Unlimited | Go to Krell!


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110407224229.GA7364@altaira.krellpowersys.exo">h ttp://lists.debian.org/20110407224229.GA7364@altaira.krellpowersys.exo
 
Old 04-07-2011, 11:17 PM
Brad Alexander
 
Default Managing large numbers (100+) of Debian-based machines

Actually, this is going to be one of my projects at work in the near future. I have some experience with cfengine2 and cfengine3 from my previous job. I have a** VM set up to run puppet, on my home network, but haven't had a chance to revisit it. So I will give you my opinions.


* cfengine2 - this is the granddaddy of configuration management tools, written by Mark Burgess back in 1993 (?). Its a great tool, the language is a little long in the tooth, however there is a ton of documentation for it. It is fairly easy to get stood up. The syntax is perl-like.


* cfengine3 - a complete rewrite of the cfengine code, including the declarative language. There was not not nearly as many docs, and last I checked (about 6 months ago), was still under heavy development, so the code is something of a moving target for the docs. Now, I have a personal opinion, and that while Mark writes a very good CM engine, I have a hard time following his docs...And since his is the only game in town for CF3, I had a hard time upgrading from CF2 to CF3 in our rather complex environment. , which consisted of multiple HPC clusters which were marginally the same, plus a number of standalone boxes.


* puppet - ruby-on-rails based. From an article on oreillynet, "while cfengine focuses on managing textfiles, puppet manages semantically more powerful constructs like users, services and packages..."


There is also bcfg2 (puppet-based),,, chef, uniconf, cdist, munki, etc....

--b


On Thu, Apr 7, 2011 at 3:01 PM, Peter Beck <peter@datentraeger.li> wrote:

On Thu, 2011-04-07 at 13:45 +0200, tv.debian@googlemail.com wrote:


> Hi, maybe you can have a look at cfengine[1], seems to fit your needs

> and it's readily available in Debian. I don't have hands-on experience

> with it though.



what about puppet ? Anyone made experience with puppet on debian ?



http://projects.puppetlabs.com/projects/1/wiki/Puppet_Debian







--

To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org

with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: http://lists.debian.org/1302202912.4589.1.camel@peanut.datentraeger.li
 
Old 04-08-2011, 05:28 PM
Rob Owens
 
Default Managing large numbers (100+) of Debian-based machines

On Fri, Apr 08, 2011 at 12:11:31AM +0200, Peter Beck wrote:
> On Thu, 2011-04-07 at 17:55 -0400, Rob Owens wrote:
> > If you decide not to go with your own apt repo, I highly recommend you
> > at least use something like apt-cacher-ng. It'll save a lot of
> > download bandwidth and time.
>
> If you're using different distros (debian, ubuntu...) I also recommend
> apt-cacher-ng because of the remapping feature. apt-cacher does not work
> well with debian and ubuntu at the same time because sometimes there are
> packages with the same name...
>
I had the same experience, which is why I now use apt-cacher-ng instead
of apt-cacher.

> a little bit off-topic:
>
> I am using apt-cacher, with apt-cacher-ng I had issues with changing the
> repository-path to an nfs-share. Is that usually working or is it not
> possible to change the path ? I couldn't find anything related in the
> manual of apt-cacher-ng
>
I have moved my apt-cacher-ng files to a new partition, but kept the
same path name. That was successful. Later I moved it to a different
machine altogether, and that was also successful.

-Rob


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110408172830.GB18659@aurora.owens.net">http://lists.debian.org/20110408172830.GB18659@aurora.owens.net
 

Thread Tools




All times are GMT. The time now is 02:35 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org