Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora/Linux Management Tools (http://www.linux-archive.org/fedora-linux-management-tools/)
-   -   Package/unpackage an image for distribution (http://www.linux-archive.org/fedora-linux-management-tools/18570-package-unpackage-image-distribution.html)

David Lutterkort 12-13-2007 01:24 AM

Package/unpackage an image for distribution
 
# HG changeset patch
# User David Lutterkort <dlutter@redhat.com>
# Date 1197512462 28800
# Node ID e9798e85269935f0a2a92fc964d3794e94c4d32d
# Parent da46770a461b6299c409522d82eec58d45e29bf1
Package/unpackage an image for distribution

Pack all the files for the image into a tarball. The name and toplevel
directory are derived from the name and version of the image descriptor.
Scratch disks are omitted, since they can be rebuilt when the image is
deployed.

To make things more interesting, information for running the image under
VMWare is also generated. This consists of a VMX file and VMDK descriptors
for each disk in the image. For scratch disks, sparse VMDK files are
generated, since VMWare has no notion of scratch disks. The generated VMX
file is very conservative, so that it's widely usable, but there are many
things that could be improved in it if we had more information about the
image (one example: are vmware-tools installed ?)

diff -r da46770a461b -r e9798e852699 virt-pack
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/virt-pack Wed Dec 12 18:21:02 2007 -0800
@@ -0,0 +1,149 @@
+#!/usr/bin/python -tt
+#
+# Package and unpackage images for distribution
+#
+# Copyright 2007 Red Hat, Inc.
+# David Lutterkort <dlutter@redhat.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA 02110-1301 USA.
+
+import os, sys, string
+from optparse import OptionParser, OptionValueError
+import subprocess
+import logging
+import libxml2
+import urlgrabber.progress as progress
+
+import virtinst
+import virtinst.ImageParser
+import virtinst.CapabilitiesParser
+import virtinst.cli as cli
+import virtinst.util as util
+import virtinst.UnWare
+
+import tempfile
+
+import gettext
+import locale
+
+locale.setlocale(locale.LC_ALL, ')
+gettext.bindtextdomain(virtinst.gettext_app, virtinst.gettext_dir)
+gettext.install(virtinst.gettext_app, virtinst.gettext_dir)
+
+class PackageException(Exception):
+ def __init__(self, msg):
+ Exception.__init__(self, msg)
+
+class Package:
+
+ def __init__(self, image):
+ self.image = image
+ if image.name is None or image.version is None:
+ raise PackageException(
+ _("The image name and version must be present"))
+ self.vername = "%s-%s" % (image.name, image.version)
+ self.tmpdir = tempfile.mkdtemp(dir="/var/tmp", prefix="virt-pack")
+ self.stagedir = os.path.join(self.tmpdir, self.vername)
+ self.files = []
+
+ def add_image_files(self):
+ cwd = os.getcwd()
+ img = self.image
+ self.files.append(os.path.basename(img.filename))
+ try:
+ os.chdir(img.base)
+ for d in img.storage.keys():
+ disk = img.storage[d]
+ if disk.use == disk.USE_SCRATCH:
+ if disk.size is None:
+ raise PackageException(_("Scratch disk %s does not have a size attribute") % disk.id)
+ else:
+ if not os.path.exists(disk.file):
+ raise PackageException(_("Disk file %s could not be found") % disk.id)
+ self.files.append(disk.file)
+ finally:
+ os.chdir(cwd)
+
+ def make_vmx_files(self):
+ img = virtinst.UnWare.Image(self.image)
+ files = img.make(self.image.base)
+ self.files.extend(files)
+
+ def pack(self, outdir):
+ outfile = os.path.join(outdir, self.vername + ".tgz")
+ for f in set(self.files):
+ dir = os.path.join(self.stagedir, os.path.dirname(f))
+ if not os.path.exists(dir):
+ os.makedirs(dir)
+ os.symlink(os.path.join(self.image.base, f),
+ os.path.join(self.stagedir, f))
+ print "Writing %s" % outfile
+ cmd = "tar chzv -C %s -f %s %s" % (self.tmpdir,
+ outfile,
+ os.path.basename(self.vername))
+ util.system(cmd)
+ return outfile
+
+### Option parsing
+def parse_args():
+ parser = OptionParser()
+ parser.set_usage("%prog [options] image.xml")
+
+ parser.add_option("-o", "--output", type="string", dest="output",
+ action="callback", callback=cli.check_before_store,
+ help=_("Directory in which packaged file will be put"))
+ parser.add_option("-d", "--debug", action="store_true", dest="debug",
+ help=_("Print debugging information"))
+
+ (options,args) = parser.parse_args()
+ if len(args) < 1:
+ parser.error(_("You need to provide an image XML descriptor"))
+ options.image = args[0]
+
+ return options
+
+def main():
+ options = parse_args()
+
+
+ cli.setupLogging("virt-pack", options.debug)
+
+ image = virtinst.ImageParser.parse_file(options.image)
+
+ if image.name is None or image.version is None:
+ print >> sys.stderr, _("The image descriptor must contain name and version")
+ valid = False
+
+ if options.output is None:
+ options.output = os.path.join(image.base, "..")
+
+ pkg = Package(image)
+ try:
+ pkg.add_image_files()
+ except PackageException, e:
+ print >> sys.stderr, "Validation failed: %s", e
+ return 1
+
+ try:
+ pkg.make_vmx_files()
+ pkg.pack(options.output)
+ except PackageException, e:
+ print >> sys.stderr, "Packaging failed: %s" % e
+ return 1
+
+if __name__ == "__main__":
+ main()
+
diff -r da46770a461b -r e9798e852699 virtinst/UnWare.py
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/virtinst/UnWare.py Wed Dec 12 18:21:02 2007 -0800
@@ -0,0 +1,291 @@
+#
+# Processing of VMWare(tm) .vmx files
+#
+# Copyright 2007 Red Hat, Inc.
+# David Lutterkort <dlutter@redhat.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA 02110-1301 USA.
+
+import time
+import sys
+import os
+
+import ImageParser
+import util
+
+class Disk:
+ """A disk for a VMWare(tm) virtual machine"""
+
+ MONOLITHIC_FLAT = "monolithicFlat"
+ TWO_GB_MAX_EXTENT_SPARSE = "twoGbMaxExtentSparse"
+ # This seems only to be usable if the vmdk header is embedded in the
+ # data file, not when the descriptor is in a separate text file. Use
+ # TWO_GB_MAX_EXTENT_SPARSE instead.
+ # VMWare's(tm) documentation of VMDK seriously sucks. A lot.
+ MONOLITHIC_SPARSE = "monolithicSparse"
+
+ IDE_HEADS = 16
+ IDE_SECTORS = 63
+
+ def __init__(self, descriptor, extent, size, dev, format):
+ """Create a new disk. DESCRIPTOR is the name of the VMDK descriptor
+ file. EXTENT is the name of the file holding the actual data. SIZE
+ is the filesize in bytes. DEV identifies the device, for IDE (the
+ only one supported right now) it should be $bus:$master. FORMAT is
+ the format of the underlying extent, one of the formats defined in
+ virtinst.ImageParser.Disk"""
+ self.cid = 0xffffffff
+ self.createType = Disk.MONOLITHIC_FLAT
+ self.descriptor = descriptor
+ self.extent = extent
+ self.size = size
+ self.dev = dev
+ self.format = format
+
+ def make_extent(self, base):
+ """Write the descriptor file, and create the extent as a monolithic
+ sparse extent if it does not exist yet"""
+ f = os.path.join(base, self.extent)
+ print "Checking %s" % f
+ if not os.path.exists(f):
+ util.system("qemu-img create -f vmdk %s %d" % (f, self.size/1024))
+ self.createType = Disk.TWO_GB_MAX_EXTENT_SPARSE
+ else:
+ qemu = os.popen("qemu-img info %s" % f, "r")
+ for l in qemu:
+ (tag, val) = l.split(":")
+ if tag == "file format" and val.strip() == "vmdk":
+ self.createType = Disk.TWO_GB_MAX_EXTENT_SPARSE
+ qemu.close()
+ return self.extent
+
+ def to_vmdk(self):
+ """Return the VMDK descriptor for this disk"""
+
+ vmdk = """# Disk DescriptorFile
+# Generated from libvirt
+version=1
+"""
+ vmdk += "CID=%08x
parentCID=ffffffff
" % self.cid
+ vmdk += "createType="%s"

" % self.createType
+ vmdk += "# Extent description
"
+ blocks = self.size/512
+ if self.createType == Disk.MONOLITHIC_FLAT:
+ vmdk += "RW %d FLAT "%s" 0
" % (blocks, os.path.basename(self.extent))
+ else: # Disk.MONOLITHIC_SPARSE
+ vmdk += "RW %d SPARSE "%s"
" % (blocks, os.path.basename(self.extent))
+
+ vmdk += """
+# Disk Data Base
+ddb.virtualHWVersion = "4"
+ddb.adapterType = "ide"
+ddb.geometry.sectors = "%d"
+ddb.geometry.heads = "%d"
+ddb.geometry.cylinders = "%d"
+""" % (Disk.IDE_SECTORS, Disk.IDE_HEADS, blocks/(Disk.IDE_SECTORS*Disk.IDE_HEADS))
+ return vmdk
+
+ def to_vmx(self):
+ """Return the fragment for the VMX file for this disk"""
+
+ vmx = ""
+ dict = {
+ "dev" : self.dev,
+ "disk_filename" : self.descriptor
+ }
+ if self.format == ImageParser.Disk.FORMAT_ISO:
+ vmx = _VMX_ISO_TEMPLATE % dict
+ else: # FORMAT_RAW
+ vmx = _VMX_IDE_TEMPLATE % dict
+ return vmx
+
+class Image:
+ """Represent an image for generation of a VMWare(tm) description"""
+
+ def __init__(self, image = None):
+ if image is not None:
+ self._init_from_image(image)
+
+ def _init_from_image(self, image):
+ domain = image.domain
+ boot = domain.boots[0]
+
+ self.base = image.base
+ self.name = image.name
+ self.descr = image.descr
+ self.label = image.label
+ self.vcpu = domain.vcpu
+ self.memory = domain.memory
+ self.interface = domain.interface
+
+ self.disks = []
+ for d in boot.drives:
+ disk = d.disk
+ descriptor = sub_ext(disk.file, ".vmdk")
+ if disk.size is None:
+ f = os.path.join(image.base, disk.file)
+ size = os.stat(f).st_size
+ else:
+ size = long(disk.size) * 1024L * 1024L
+ ide_count = len(self.disks)
+ dev = "%d:%d" % (ide_count / 2, ide_count % 2)
+ self.disks.append(Disk(descriptor, disk.file, size, dev,
+ disk.format))
+
+ def make(self, base):
+ """Write the descriptor file and all the disk descriptors"""
+ files = []
+ out = open(os.path.join(self.base, self.name + ".vmx"), "w")
+ out.write(self.to_vmx())
+ out.close()
+ files.append(self.name + ".vmx")
+
+ for d in self.disks:
+ f = d.make_extent(self.base)
+ files.append(f)
+ out = open(os.path.join(base, d.descriptor), "w")
+ out.write(d.to_vmdk())
+ out.close()
+ files.append(d.descriptor)
+ return files
+
+ def to_vmx(self):
+ """Return the VMX description of this image"""
+ dict = {
+ "now": time.strftime("%Y-%m-%dT%H:%M:%S %Z", time.localtime()),
+ "progname": os.path.basename(sys.argv[0]),
+ "/image/name": self.name,
+ "/image/description": self.descr or "None",
+ "/image/label": self.label or self.name,
+ "/image/devices/vcpu" : self.vcpu,
+ "/image/devices/memory": long(self.memory)/1024
+ }
+
+ vmx = _VMX_MAIN_TEMPLATE % dict
+ if self.interface:
+ vmx += _VMX_ETHER_TEMPLATE
+
+ for d in self.disks:
+ vmx += d.to_vmx()
+
+ return vmx
+
+def sub_ext(filename, ext):
+ return os.path.splitext(filename)[0] + ext
+
+_VMX_MAIN_TEMPLATE = """
+#!/usr/bin/vmplayer
+
+# Generated %(now)s by %(progname)s
+# http://virt-manager.et.redhat.com/
+
+# This is a Workstation 5 or 5.5 config file
+# It can be used with Player
+config.version = "8"
+virtualHW.version = "4"
+
+# Selected operating system for your virtual machine
+guestOS = "other"
+
+# displayName is your own name for the virtual machine
+displayName = "%(/image/name)s"
+
+# These fields are free text description fields
+annotation = "%(/image/description)s"
+guestinfo.vmware.product.long = "%(/image/label)s"
+guestinfo.vmware.product.url = "http://virt-manager.et.redhat.com/"
+guestinfo.vmware.product.class = "virtual machine"
+
+# Number of virtual CPUs. Your virtual machine will not
+# work if this number is higher than the number of your physical CPUs
+numvcpus = "%(/image/devices/vcpu)s"
+
+# Memory size and other memory settings
+memsize = "%(/image/devices/memory)d"
+MemAllowAutoScaleDown = "FALSE"
+MemTrimRate = "-1"
+
+# Unique ID for the virtual machine will be created
+uuid.action = "create"
+
+## For appliances where tools are installed already, this should really
+## be false, but we don't have that ionfo in the metadata
+# Remind to install VMware Tools
+# This setting has no effect in VMware Player
+tools.remindInstall = "TRUE"
+
+# Startup hints interfers with automatic startup of a virtual machine
+# This setting has no effect in VMware Player
+hints.hideAll = "TRUE"
+
+# Enable time synchronization between computer
+# and virtual machine
+tools.syncTime = "TRUE"
+
+# First serial port, physical COM1 is not available
+serial0.present = "FALSE"
+
+# Optional second serial port, physical COM2 is not available
+serial1.present = "FALSE"
+
+# First parallell port, physical LPT1 is not available
+parallel0.present = "FALSE"
+
+# Logging
+# This config activates logging, and keeps last log
+logging = "TRUE"
+log.fileName = "%(/image/name)s.log"
+log.append = "TRUE"
+log.keepOld = "3"
+
+# These settings decides interaction between your
+# computer and the virtual machine
+isolation.tools.hgfs.disable = "FALSE"
+isolation.tools.dnd.disable = "FALSE"
+isolation.tools.copy.enable = "TRUE"
+isolation.tools.paste.enabled = "TRUE"
+
+# Settings for physical floppy drive
+floppy0.present = "FALSE"
+"""
+
+_VMX_ETHER_TEMPLATE = """
+## if /image/devices/interface is present:
+# First network interface card
+ethernet0.present = "TRUE"
+ethernet0.connectionType = "nat"
+ethernet0.addressType = "generated"
+ethernet0.generatedAddressOffset = "0"
+ethernet0.autoDetect = "TRUE"
+"""
+
+_VMX_ISO_TEMPLATE = """
+# CDROM drive
+ide%(dev)s.present = "TRUE"
+ide%(dev)s.deviceType = "cdrom-raw"
+ide%(dev)s.startConnected = "TRUE"
+ide%(dev)s.fileName = "%(disk_filename)s"
+ide%(dev)s.autodetect = "TRUE"
+"""
+
+_VMX_IDE_TEMPLATE = """
+# IDE disk
+ide%(dev)s.present = "TRUE"
+ide%(dev)s.fileName = "%(disk_filename)s"
+ide%(dev)s.mode = "persistent"
+ide%(dev)s.startConnected = "TRUE"
+ide%(dev)s.writeThrough = "TRUE"
+"""
diff -r da46770a461b -r e9798e852699 virtinst/util.py
--- a/virtinst/util.py Wed Dec 12 18:21:02 2007 -0800
+++ b/virtinst/util.py Wed Dec 12 18:21:02 2007 -0800
@@ -206,3 +206,9 @@ def get_phy_cpus(conn):
hostinfo = conn.getInfo()
pcpus = hostinfo[4] * hostinfo[5] * hostinfo[6] * hostinfo[7]
return pcpus
+
+def system(cmd):
+ st = os.system(cmd)
+ if os.WIFEXITED(st) and os.WEXITSTATUS(st) != 0:
+ raise OSError("Failed to run %s, exited with %d" %
+ (cmd, os.WEXITSTATUS(st)))

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools

Mark McLoughlin 12-13-2007 09:28 AM

Package/unpackage an image for distribution
 
On Wed, 2007-12-12 at 18:24 -0800, David Lutterkort wrote:

> Package/unpackage an image for distribution

Interesting ... one thought is that you should expose the Package[1]
class, and an UnWarePackage sub-class, from virtinst instead of UnWare.
Also, I'd really only expect a single public Package method - pack().

Basically, I'm thinking about how the likes of livecd's image-creator
could use this stuff.

Cheers,
Mark.

[1] - Also, I don't think it's usual for constructors to raise
exceptions ... I think constructors are always supposed to be safe in
python

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools

David Lutterkort 12-13-2007 04:38 PM

Package/unpackage an image for distribution
 
On Thu, 2007-12-13 at 10:28 +0000, Mark McLoughlin wrote:
> On Wed, 2007-12-12 at 18:24 -0800, David Lutterkort wrote:
>
> > Package/unpackage an image for distribution
>
> Interesting ... one thought is that you should expose the Package[1]
> class, and an UnWarePackage sub-class, from virtinst instead of UnWare.
> Also, I'd really only expect a single public Package method - pack().
>
> Basically, I'm thinking about how the likes of livecd's image-creator
> could use this stuff.

Yeah, I structured it that way because I didn't think there would be
many users for it - how would you include that with image-creator ? Are
you thinking of doing the image creation soup-to-nuts (kickstart to
tarball, really) in one big jump ?

> [1] - Also, I don't think it's usual for constructors to raise
> exceptions ... I think constructors are always supposed to be safe in
> python

Yeah, it's lame .. I'll make it a factory method

David


_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools

Mark McLoughlin 12-14-2007 11:11 AM

Package/unpackage an image for distribution
 
On Thu, 2007-12-13 at 09:38 -0800, David Lutterkort wrote:
> On Thu, 2007-12-13 at 10:28 +0000, Mark McLoughlin wrote:

> > Interesting ... one thought is that you should expose the Package[1]
> > class, and an UnWarePackage sub-class, from virtinst instead of UnWare.
> > Also, I'd really only expect a single public Package method - pack().
> >
> > Basically, I'm thinking about how the likes of livecd's image-creator
> > could use this stuff.
>
> Yeah, I structured it that way because I didn't think there would be
> many users for it - how would you include that with image-creator ? Are
> you thinking of doing the image creation soup-to-nuts (kickstart to
> tarball, really) in one big jump ?

(soup-to-nuts? wtf? :-)

But sure, why wouldn't an image creator not dump out the image in a
distributable format rather than an intermediate format? I think of the
image creator as being similar to rpmbuild in that respect.

Also, it's partly about what API we want to add to virtinst and retain
compatibility for in future versions. A simple Package base-class, a
VMWarePackage sub-class and a pack() method sounds about the extent of
the API we'd want to commit to at this point.

Cheers,
Mark.

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools


All times are GMT. The time now is 07:23 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.