FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 05-05-2011, 02:10 PM
ken
 
Default How to copy a system?

On 05/05/2011 08:01 AM Brunner, Brian T. wrote:
> centos-bounces@centos.org wrote:
>> At Thu, 05 May 2011 07:44:52 -0400 CentOS mailing list
>> <centos@centos.org> wrote:
>>
>>>
>>> On 05/05/2011 07:13 AM, Timothy Murphy wrote:
>>>> Is there a standard way of copying a working system
>>>> from one machine to another with different partitions?
>>> You could also utilize cloning software, such as the client version
>>> of drbl, clonezilla livecd.
>>>
>>> You could also do a direct copy with dd onto a connected drive.
>> Warning: dd is not a good choise if the source and desination
>> drives/partitions are *different* sizes.
>
> Different block mappings will also give you grief.
> .:. The drives must be identical manufacturer and model, down to the
> firmware revision.
> dd is not a backup tool in the general sense.

I had doubts about dd also. But last year, when I needed to upgrade to
a larger drive, I used it and it worked fine. I bought a new drive (of
course of larger size... different manufacturer too), put it into a
drive enclosure, plugged that new drive into my USB port, and ran dd to
copy the entirety of hda to hdb. Shutting down the machine, I swapped
the hard drives and booted with the new drive and-- viola!-- new bigger
drive with everything running just like on the old drive. I didn't have
to reconfigure anything; even the networking worked on the new drive
without touching anything. The only thing I did on the new drive was to
create a new partition from all the extra new hd space I had. Indeed,
this is a multi-boot machine and all OSs on it copied over just fine.
In addition, all my linux partitions are encrypted, and all that copied
over perfectly as well.

One tip: Use dd's smallest block size (BS). I did this copy using dd
several times, starting with 4k, then 2k block sizes and the new disk
had problems when I tried to use it. IIRC, I had to rachet down to 256
to get a working drive. And this took eight or ten hours to copy an 80G
drive.

Another tip: in your BIOS the parameter for the hard drive should
probably be Auto-Detect if your source and destination drives aren't
identical. That's generally the default anyway.

Final tip (I think): For me, my machine A and machine B were the same
machine... so of course the hardware was absolutely identical. Using dd
might not work if the hardware on A and B are too different from one
another.


hth,
ken


--
"Truth is the most valuable thing we have, so I try to conserve it."
--Mark Twain
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:10 PM
ken
 
Default How to copy a system?

On 05/05/2011 08:01 AM Brunner, Brian T. wrote:
> centos-bounces@centos.org wrote:
>> At Thu, 05 May 2011 07:44:52 -0400 CentOS mailing list
>> <centos@centos.org> wrote:
>>
>>>
>>> On 05/05/2011 07:13 AM, Timothy Murphy wrote:
>>>> Is there a standard way of copying a working system
>>>> from one machine to another with different partitions?
>>> You could also utilize cloning software, such as the client version
>>> of drbl, clonezilla livecd.
>>>
>>> You could also do a direct copy with dd onto a connected drive.
>> Warning: dd is not a good choise if the source and desination
>> drives/partitions are *different* sizes.
>
> Different block mappings will also give you grief.
> .:. The drives must be identical manufacturer and model, down to the
> firmware revision.
> dd is not a backup tool in the general sense.

I had doubts about dd also. But last year, when I needed to upgrade to
a larger drive, I used it and it worked fine. I bought a new drive (of
course of larger size... different manufacturer too), put it into a
drive enclosure, plugged that new drive into my USB port, and ran dd to
copy the entirety of hda to hdb. Shutting down the machine, I swapped
the hard drives and booted with the new drive and-- viola!-- new bigger
drive with everything running just like on the old drive. I didn't have
to reconfigure anything; even the networking worked on the new drive
without touching anything. The only thing I did on the new drive was to
create a new partition from all the extra new hd space I had. Indeed,
this is a multi-boot machine and all OSs on it copied over just fine.
In addition, all my linux partitions are encrypted, and all that copied
over perfectly as well.

One tip: Use dd's smallest block size (BS). I did this copy using dd
several times, starting with 4k, then 2k block sizes and the new disk
had problems when I tried to use it. IIRC, I had to rachet down to 256
to get a working drive. And this took eight or ten hours to copy an 80G
drive.

Another tip: in your BIOS the parameter for the hard drive should
probably be Auto-Detect if your source and destination drives aren't
identical. That's generally the default anyway.

Final tip (I think): For me, my machine A and machine B were the same
machine... so of course the hardware was absolutely identical. Using dd
might not work if the hardware on A and B are too different from one
another.


hth,
ken


--
"Truth is the most valuable thing we have, so I try to conserve it."
--Mark Twain
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:37 PM
Lamar Owen
 
Default How to copy a system?

On Thursday, May 05, 2011 08:01:57 AM Brunner, Brian T. wrote:
> centos-bounces@centos.org wrote:
> > At Thu, 05 May 2011 07:44:52 -0400 CentOS mailing list
> > Warning: dd is not a good choise if the source and desination
> > drives/partitions are *different* sizes.

> Different block mappings will also give you grief.
> .:. The drives must be identical manufacturer and model, down to the
> firmware revision.
> dd is not a backup tool in the general sense.

I do dd imaging quite frequently, and as long as everything is LBA48 capable and setup, I don't have problems copying partitions or whole drives between multiple drives of different sizes and manufacturers; even in instances between different interface technologies. This gets better once you're on an OS rev that treats ATA drives as SCSI, and CHS is no longer in play at all, which is the case in EL6 and Fedora revs around EL6. (At least I think that's correct; but it has been an awfully long time since I've done a CentOS 4 or 5 install on an ATA/IDE system, as all of my server systems are either SCSI or FibreChannel, physical or virtual).

Having said that, I quarterly rotate two identical drives in this laptop; each quarter, I clone the currently operating drive to the secondary and to a dated whole-disk image file, and then swap the drives, putting the previous primary back into the fire safe for storage. This both wear-levels and tests the backups drives.

I use a three-tiered approach to backups of my own laptop:
1.) Quarterly swapping drive clones as described above, using dd (which is faster than the slightly more friendly ddrescue, unless a bad sector is found) booted from rescue or live media of the OS that's installed (this provides a fast bare-metal base recovery that I can then update and restore from the rolling rsync in item 3);
2.) Three quarters of kept images along with the partition mapping (I use GPT, and thus use gdisk for this, which works better in my particular case than parted does (parted puts an inappropriate partition type on one of my partitions when recreating the partition map)) on multiple disks;
3.) Frequent rsyncs of /home and /etc to multiple drives, in rotation. This does mean an SELinux relabel might be required on a restore, but that's ok.

For servers I do the same, but with annual images and more rigorous scheduling of tarballs of important files, along with rolling rsyncs (I've looked at rsnapshot, and backing up the backup can be somewhat interesting in that case). Dump/restore has its advantages, too, however.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:37 PM
Lamar Owen
 
Default How to copy a system?

On Thursday, May 05, 2011 08:01:57 AM Brunner, Brian T. wrote:
> centos-bounces@centos.org wrote:
> > At Thu, 05 May 2011 07:44:52 -0400 CentOS mailing list
> > Warning: dd is not a good choise if the source and desination
> > drives/partitions are *different* sizes.

> Different block mappings will also give you grief.
> .:. The drives must be identical manufacturer and model, down to the
> firmware revision.
> dd is not a backup tool in the general sense.

I do dd imaging quite frequently, and as long as everything is LBA48 capable and setup, I don't have problems copying partitions or whole drives between multiple drives of different sizes and manufacturers; even in instances between different interface technologies. This gets better once you're on an OS rev that treats ATA drives as SCSI, and CHS is no longer in play at all, which is the case in EL6 and Fedora revs around EL6. (At least I think that's correct; but it has been an awfully long time since I've done a CentOS 4 or 5 install on an ATA/IDE system, as all of my server systems are either SCSI or FibreChannel, physical or virtual).

Having said that, I quarterly rotate two identical drives in this laptop; each quarter, I clone the currently operating drive to the secondary and to a dated whole-disk image file, and then swap the drives, putting the previous primary back into the fire safe for storage. This both wear-levels and tests the backups drives.

I use a three-tiered approach to backups of my own laptop:
1.) Quarterly swapping drive clones as described above, using dd (which is faster than the slightly more friendly ddrescue, unless a bad sector is found) booted from rescue or live media of the OS that's installed (this provides a fast bare-metal base recovery that I can then update and restore from the rolling rsync in item 3);
2.) Three quarters of kept images along with the partition mapping (I use GPT, and thus use gdisk for this, which works better in my particular case than parted does (parted puts an inappropriate partition type on one of my partitions when recreating the partition map)) on multiple disks;
3.) Frequent rsyncs of /home and /etc to multiple drives, in rotation. This does mean an SELinux relabel might be required on a restore, but that's ok.

For servers I do the same, but with annual images and more rigorous scheduling of tarballs of important files, along with rolling rsyncs (I've looked at rsnapshot, and backing up the backup can be somewhat interesting in that case). Dump/restore has its advantages, too, however.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:41 PM
Robert Heller
 
Default How to copy a system?

At Thu, 05 May 2011 10:10:52 -0400 CentOS mailing list <centos@centos.org> wrote:

>
> On 05/05/2011 08:01 AM Brunner, Brian T. wrote:
> > centos-bounces@centos.org wrote:
> >> At Thu, 05 May 2011 07:44:52 -0400 CentOS mailing list
> >> <centos@centos.org> wrote:
> >>
> >>>
> >>> On 05/05/2011 07:13 AM, Timothy Murphy wrote:
> >>>> Is there a standard way of copying a working system
> >>>> from one machine to another with different partitions?
> >>> You could also utilize cloning software, such as the client version
> >>> of drbl, clonezilla livecd.
> >>>
> >>> You could also do a direct copy with dd onto a connected drive.
> >> Warning: dd is not a good choise if the source and desination
> >> drives/partitions are *different* sizes.
> >
> > Different block mappings will also give you grief.
> > .:. The drives must be identical manufacturer and model, down to the
> > firmware revision.
> > dd is not a backup tool in the general sense.
>
> I had doubts about dd also. But last year, when I needed to upgrade to
> a larger drive, I used it and it worked fine. I bought a new drive (of
> course of larger size... different manufacturer too), put it into a
> drive enclosure, plugged that new drive into my USB port, and ran dd to
> copy the entirety of hda to hdb. Shutting down the machine, I swapped
> the hard drives and booted with the new drive and-- viola!-- new bigger
> drive with everything running just like on the old drive. I didn't have
> to reconfigure anything; even the networking worked on the new drive
> without touching anything. The only thing I did on the new drive was to
> create a new partition from all the extra new hd space I had. Indeed,
> this is a multi-boot machine and all OSs on it copied over just fine.
> In addition, all my linux partitions are encrypted, and all that copied
> over perfectly as well.
>
> One tip: Use dd's smallest block size (BS). I did this copy using dd
> several times, starting with 4k, then 2k block sizes and the new disk
> had problems when I tried to use it. IIRC, I had to rachet down to 256
> to get a working drive. And this took eight or ten hours to copy an 80G
> drive.

Hmmm.... Using dump & restore (or tar or rsync or cpio, etc.) would
likely be a lot faster. Esp. if the disk is not 100% full. Remember,
dd will copy even the unused free blocks (which is a total waste of
time). And dump & restore will likely use a more optimal block size,
which will copy the data faster as well...

>
> Another tip: in your BIOS the parameter for the hard drive should
> probably be Auto-Detect if your source and destination drives aren't
> identical. That's generally the default anyway.
>
> Final tip (I think): For me, my machine A and machine B were the same
> machine... so of course the hardware was absolutely identical. Using dd
> might not work if the hardware on A and B are too different from one
> another.
>
>
> hth,
> ken
>
>

--
Robert Heller -- 978-544-6933 / heller@deepsoft.com
Deepwoods Software -- http://www.deepsoft.com/
() ascii ribbon campaign -- against html e-mail
/ www.asciiribbon.org -- against proprietary attachments



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:41 PM
Robert Heller
 
Default How to copy a system?

At Thu, 05 May 2011 10:10:52 -0400 CentOS mailing list <centos@centos.org> wrote:

>
> On 05/05/2011 08:01 AM Brunner, Brian T. wrote:
> > centos-bounces@centos.org wrote:
> >> At Thu, 05 May 2011 07:44:52 -0400 CentOS mailing list
> >> <centos@centos.org> wrote:
> >>
> >>>
> >>> On 05/05/2011 07:13 AM, Timothy Murphy wrote:
> >>>> Is there a standard way of copying a working system
> >>>> from one machine to another with different partitions?
> >>> You could also utilize cloning software, such as the client version
> >>> of drbl, clonezilla livecd.
> >>>
> >>> You could also do a direct copy with dd onto a connected drive.
> >> Warning: dd is not a good choise if the source and desination
> >> drives/partitions are *different* sizes.
> >
> > Different block mappings will also give you grief.
> > .:. The drives must be identical manufacturer and model, down to the
> > firmware revision.
> > dd is not a backup tool in the general sense.
>
> I had doubts about dd also. But last year, when I needed to upgrade to
> a larger drive, I used it and it worked fine. I bought a new drive (of
> course of larger size... different manufacturer too), put it into a
> drive enclosure, plugged that new drive into my USB port, and ran dd to
> copy the entirety of hda to hdb. Shutting down the machine, I swapped
> the hard drives and booted with the new drive and-- viola!-- new bigger
> drive with everything running just like on the old drive. I didn't have
> to reconfigure anything; even the networking worked on the new drive
> without touching anything. The only thing I did on the new drive was to
> create a new partition from all the extra new hd space I had. Indeed,
> this is a multi-boot machine and all OSs on it copied over just fine.
> In addition, all my linux partitions are encrypted, and all that copied
> over perfectly as well.
>
> One tip: Use dd's smallest block size (BS). I did this copy using dd
> several times, starting with 4k, then 2k block sizes and the new disk
> had problems when I tried to use it. IIRC, I had to rachet down to 256
> to get a working drive. And this took eight or ten hours to copy an 80G
> drive.

Hmmm.... Using dump & restore (or tar or rsync or cpio, etc.) would
likely be a lot faster. Esp. if the disk is not 100% full. Remember,
dd will copy even the unused free blocks (which is a total waste of
time). And dump & restore will likely use a more optimal block size,
which will copy the data faster as well...

>
> Another tip: in your BIOS the parameter for the hard drive should
> probably be Auto-Detect if your source and destination drives aren't
> identical. That's generally the default anyway.
>
> Final tip (I think): For me, my machine A and machine B were the same
> machine... so of course the hardware was absolutely identical. Using dd
> might not work if the hardware on A and B are too different from one
> another.
>
>
> hth,
> ken
>
>

--
Robert Heller -- 978-544-6933 / heller@deepsoft.com
Deepwoods Software -- http://www.deepsoft.com/
() ascii ribbon campaign -- against html e-mail
/ www.asciiribbon.org -- against proprietary attachments



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 03:35 PM
Les Mikesell
 
Default How to copy a system?

On 5/5/2011 9:37 AM, Lamar Owen wrote:
>
>> Different block mappings will also give you grief.
>> .:. The drives must be identical manufacturer and model, down to the
>> firmware revision.
>> dd is not a backup tool in the general sense.
>
> I do dd imaging quite frequently, and as long as everything is LBA48 capable and setup, I don't have problems copying partitions or whole drives between multiple drives of different sizes and manufacturers; even in instances between different interface technologies. This gets better once you're on an OS rev that treats ATA drives as SCSI, and CHS is no longer in play at all, which is the case in EL6 and Fedora revs around EL6. (At least I think that's correct; but it has been an awfully long time since I've done a CentOS 4 or 5 install on an ATA/IDE system, as all of my server systems are either SCSI or FibreChannel, physical or virtual).

Clonezilla-live is a handy, faster way to do this. It boots from cd/usb
into a menu and generally uses partclone to do the work so on most
filesystems it only copies the blocks that are actually used. It also
has a mode to resize the partitions on the new disk but it isn't all
that useful because you can't control them individually.

> Having said that, I quarterly rotate two identical drives in this laptop; each quarter, I clone the currently operating drive to the secondary and to a dated whole-disk image file, and then swap the drives, putting the previous primary back into the fire safe for storage. This both wear-levels and tests the backups drives.

Besides disk->disk, clonezilla can save/restore compressed image copies
over the network to space mounted via nfs/samba/sshfs so if you are
making the copy as a backup or the source for future clones you can drop
it on some other filesystem instead of needing a matching disk.


> I use a three-tiered approach to backups of my own laptop:
> 1.) Quarterly swapping drive clones as described above, using dd (which is faster than the slightly more friendly ddrescue, unless a bad sector is found) booted from rescue or live media of the OS that's installed (this provides a fast bare-metal base recovery that I can then update and restore from the rolling rsync in item 3);
> 2.) Three quarters of kept images along with the partition mapping (I use GPT, and thus use gdisk for this, which works better in my particular case than parted does (parted puts an inappropriate partition type on one of my partitions when recreating the partition map)) on multiple disks;
> 3.) Frequent rsyncs of /home and /etc to multiple drives, in rotation. This does mean an SELinux relabel might be required on a restore, but that's ok.
>
> For servers I do the same, but with annual images and more rigorous scheduling of tarballs of important files, along with rolling rsyncs (I've looked at rsnapshot, and backing up the backup can be somewhat interesting in that case). Dump/restore has its advantages, too, however.

I always recommend backuppc for scheduled backups. It's pretty much
configure and forget and it compresses and pools all identical content
so you can keep much more history online than you would expect.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 03:36 PM
ken
 
Default How to copy a system?

On 05/05/2011 10:41 AM Robert Heller wrote:
> At Thu, 05 May 2011 10:10:52 -0400 CentOS mailing list <centos@centos.org> wrote:
>
>> On 05/05/2011 08:01 AM Brunner, Brian T. wrote:
>>> centos-bounces@centos.org wrote:
>>>> At Thu, 05 May 2011 07:44:52 -0400 CentOS mailing list
>>>> <centos@centos.org> wrote:
>>>>
>>>>> On 05/05/2011 07:13 AM, Timothy Murphy wrote:
>>>>>> Is there a standard way of copying a working system
>>>>>> from one machine to another with different partitions?
>>>>> You could also utilize cloning software, such as the client version
>>>>> of drbl, clonezilla livecd.
>>>>>
>>>>> You could also do a direct copy with dd onto a connected drive.
>>>> Warning: dd is not a good choise if the source and desination
>>>> drives/partitions are *different* sizes.
>>> Different block mappings will also give you grief.
>>> .:. The drives must be identical manufacturer and model, down to the
>>> firmware revision.
>>> dd is not a backup tool in the general sense.
>> ...
>
> Hmmm.... Using dump & restore (or tar or rsync or cpio, etc.) would
> likely be a lot faster. Esp. if the disk is not 100% full. Remember,
> dd will copy even the unused free blocks (which is a total waste of
> time). And dump & restore will likely use a more optimal block size,
> which will copy the data faster as well...

Speed is good sometimes. But I was probably either sleeping or watching
TV during those eight to ten hours, so the length of time to do the copy
didn't matter at all.

The most time-consuming part of the job was finding the particular
command with the correct args that actually worked-- not the command or
utility that "should" work or that "theoretically ought to" work-- but
one which in fact *did* work. So if anyone actually finds a faster way
to clone a system-- meaning they've run the command(s), and done the
testing to determine that it was successful-- I'm all ears. The other
possibilities are interesting, but given what my schedule is like,
unless success with something else is 99.9% assured, I'll probably do it
the same way again next time. Hey, what can I say...? I like success.






--
"Truth is the most valuable thing we have, so I try to conserve it."
--Mark Twain
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 04:10 PM
Les Mikesell
 
Default How to copy a system?

On 5/5/2011 10:36 AM, ken wrote:
>
> The most time-consuming part of the job was finding the particular
> command with the correct args that actually worked-- not the command or
> utility that "should" work or that "theoretically ought to" work-- but
> one which in fact *did* work. So if anyone actually finds a faster way
> to clone a system-- meaning they've run the command(s), and done the
> testing to determine that it was successful-- I'm all ears.

That's what clonezilla is all about. And it is released frequently on
both debian and ubuntu (the 'alternative' version) live bases so it has
pretty good hardware handling.

--
Les Mikesell
lesmikesell@gmail.com


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 04:11 PM
Lamar Owen
 
Default How to copy a system?

On Thursday, May 05, 2011 11:35:01 AM Les Mikesell wrote:
> On 5/5/2011 9:37 AM, Lamar Owen wrote:
> > I do dd imaging quite frequently, and as long as everything is LBA48 capable and setup, [snippage] .... using dd .... booted from rescue or live media of the OS that's installed...

> Clonezilla-live is a handy, faster way to do this.

I've recast my original message slightly, as you've missed a critical point: I use the cloning tool from the rescue or live media of the OS that's installed. There are a number of reasons for this, not the least of which is that LVM, RAID, and some other things behave differently depending upon the kernel, lvm tools, etc, that's running the clone.

I'm familiar with and have used clonezilla numerous times, but not for this purpose. The 'using dd ... booted from rescue or live media of the OS that's installed' part isn't as important during backup as it can be during restore. And I have been bit by that, using F12 (or 13) live media to do a C4 backup/restore; some metadata got farkled and the restore didn't 'take' until I did the restore with C4 media.

Also, well, there are uses for manually-marked badblocks other than drive errors.... :-)

[snip]

> I always recommend backuppc for scheduled backups. It's pretty much
> configure and forget and it compresses and pools all identical content
> so you can keep much more history online than you would expect.

I've actually thought about using DragonFly BSD and its HAMMER filesystem for the backup storage device...... quick restores rely on quickly finding what is needed, and many times I get requests like 'please restore the file that has the stuff about the instrument we built for grant so-and-so' rather than an exact filename; greppability of the backup set is a must for us. Complete, straight-dd, clones are mountable (RO, of course) and searchable, and rolling rsyncs and tarballs are searchable without a whole lot of effort. Deduplication would be nice, but it's secondary, as is the time and space spent on the backup, for our purposes.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 11:47 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org