FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu Server Development

 
 
LinkBack Thread Tools
 
Old 10-28-2009, 02:23 AM
David Cantrell
 
Default Find and format any unformatted DASD devices (#528386).

Before the device tree is built in the storage code, run through the
DASD devices on the system and format any with a status of
'unformatted'. We have to run dasdfmt on these devices before we can
partition them.

Under RHEL-5, this process was serial and the user had to click Yes for
each unformatted DASD found, which could take a really long time if
you had thousands of DASDs.

The idea now is that if the DASD is seen by anaconda, we want to use it
for installation. The stage 1 device initialization routines as well as
the CMS conf file provided at boot time allow the user to restrict the
range of devices we see during installation. If any of the devices we
see are unformatted, run dasdfmt before building the devicetree.

This process is implemented in a DASD class that works similar to the
iscsi and zfcp classes. That is, devices may need to be started or
brought in to a working state before the anaconda storage code can use
them.
---
storage/__init__.py | 3 +
storage/dasd.py | 173 ++++++++++++++++++++++++++++++++++++++++++++++++++ +
2 files changed, 176 insertions(+), 0 deletions(-)
create mode 100644 storage/dasd.py

diff --git a/storage/__init__.py b/storage/__init__.py
index 0a9dad0..e1569c8 100644
--- a/storage/__init__.py
+++ b/storage/__init__.py
@@ -51,6 +51,7 @@ from udev import *
import iscsi
import fcoe
import zfcp
+import dasd

import gettext
_ = lambda x: gettext.ldgettext("anaconda", x)
@@ -252,6 +253,7 @@ class Storage(object):
self.iscsi = iscsi.iscsi()
self.fcoe = fcoe.fcoe()
self.zfcp = zfcp.ZFCP()
+ self.dasd = dasd.DASD()

self._nextID = 0
self.defaultFSType = get_default_filesystem_type()
@@ -320,6 +322,7 @@ class Storage(object):
self.iscsi.startup(self.anaconda.intf)
self.fcoe.startup(self.anaconda.intf)
self.zfcp.startup()
+ self.dasd.startup(self.anaconda.intf)
if self.anaconda.id.getUpgrade():
clearPartType = CLEARPART_TYPE_NONE
else:
diff --git a/storage/dasd.py b/storage/dasd.py
new file mode 100644
index 0000000..efaed40
--- /dev/null
+++ b/storage/dasd.py
@@ -0,0 +1,173 @@
+#
+# dasd.py - DASD class
+#
+# Copyright (C) 2009 Red Hat, Inc. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+# Red Hat Author(s): David Cantrell <dcantrell@redhat.com>
+#
+
+import iutil
+import sys
+import os
+from storage.devices import deviceNameToDiskByPath
+from constants import *
+from flags import flags
+
+import logging
+log = logging.getLogger("anaconda")
+
+import gettext
+_ = lambda x: gettext.ldgettext("anaconda", x)
+P_ = lambda x, y, z: gettext.ldngettext("anaconda", x, y, z)
+
+class DASD:
+ """ Controlling class for DASD interaction before the storage code in
+ anaconda has initialized.
+
+ The DASD class can determine if any DASD devices on the system are
+ unformatted and can perform a dasdfmt on them.
+ """
+
+ def __init__(self):
+ self._dasdlist = []
+ self._totalCylinders = 0
+ self._completedCylinders = 0.0
+ self._maxFormatJobs = 0
+ self.started = False
+
+ def startup(self, intf=None):
+ """ Look for any unformatted DASDs in the system and offer the user
+ the option for format them with dasdfmt or exit the installer.
+ """
+ if self.started:
+ return
+
+ self.started = True
+
+ if not iutil.isS390():
+ return
+
+ log.info("Checking for unformatted DASD devices:")
+
+ for device in os.listdir("/sys/block"):
+ if not device.startswith("dasd"):
+ continue
+
+ statusfile = "/sys/block/%s/device/status" % (device,)
+ if not os.path.isfile(statusfile):
+ continue
+
+ f = open(statusfile, "r")
+ status = f.read().strip()
+ f.close()
+
+ if status == "unformatted":
+ log.info(" %s is an unformatted DASD" % (device,))
+ self._dasdlist.append(device)
+
+ if not len(self._dasdlist):
+ log.info(" no unformatted DASD devices found")
+ return
+
+ tmplist = map(lambda s: "/dev/" + s, self._dasdlist)
+ self._dasdlist = map(lambda s: deviceNameToDiskByPath(s), tmplist)
+
+ if intf:
+ c = len(self._dasdlist)
+
+ title = P_("Unformatted DASD Device Found",
+ "Unformatted DASD Devices Found", c)
+ msg = P_("Format uninitialized DASD device?

"
+ "There is %d uninitialized DASD device on this "
+ "system. To continue installation, the device must "
+ "be formatted. Formatting will remove any data on "
+ "this device." % c,
+ "Format uninitialized DASD devices?

"
+ "There are %d uninitialized DASD devices on this "
+ "system. To continue installation, the devices must "
+ "be formatted. Formatting will remove any data on "
+ "these devices." % c,
+ c)
+
+ devs = '
+ for dasd in self._dasdlist:
+ devs += "%s
" % (dasd,)
+
+ icon = "/usr/share/icons/gnome/32x32/status/dialog-error.png"
+ buttons = [_("_Format"), _("_Exit installer")]
+ rc = intf.detailedMessageWindow(title, msg, devs.strip(),
+ type="custom",
+ custom_icon=icon,
+ custom_buttons=buttons)
+ if rc == 1:
+ sys.exit(0)
+
+ for dasd in self._dasdlist:
+ log.info("Running dasdfmt on %s" % (dasd,))
+
+ argv = ["-y", "-P", "-d", "cdl", "-b", "4096"]
+
+ if intf:
+ title = P_("Formatting DASD Device", "Formatting DASD Devices", c)
+ msg = P_("Preparing %d DASD device for use with Linux..." % c,
+ "Preparing %d DASD devices for use with Linux..." % c, c)
+ pw = intf.progressWindow(title, msg, 1.0)
+
+ for dasd in self._dasdlist:
+ iutil.execWithCallback("/sbin/dasdfmt", argv + [dasd],
+ stdout="/dev/tty5", stderr="/dev/tty5",
+ callback=self._updateProgressWindow,
+ callback_data=pw)
+
+ pw.pop()
+ else:
+ for dasd in self._dasdlist:
+ iutil.execWithRedirect("/sbin/dasdfmt", argv + [dasd],
+ stdout="/dev/tty5", stderr="/dev/tty5")
+
+ def _updateProgressWindow(self, data, callback_data=None):
+ """ Reads progress output from dasdfmt and collects the number of
+ cylinders completed so the progress window can update.
+ """
+ if not callback_data:
+ return
+
+ if data == '
':
+ # each newline we see in this output means one more cylinder done
+ self._completedCylinders += 1.0
+ callback_data.set(self._completedCylinders / self.totalCylinders)
+
+ @property
+ def totalCylinders(self):
+ """ Total number of cylinders of all unformatted DASD devices. """
+ if self._totalCylinders:
+ return self._totalCylinders
+
+ argv = ["-t", "-v", "-y", "-d", "cdl", "-b", "4096"]
+ for dasd in self._dasdlist:
+ buf = iutil.execWithCapture("/sbin/dasdfmt", argv + [dasd],
+ stderr="/dev/tty5")
+ for line in buf.splitlines():
+ if line.startswith("Drive Geometry: "):
+ # line will look like this:
+ # Drive Geometry: 3339 Cylinders * 15 Heads = 50085 Tracks
+ cyls = long(filter(lambda s: s, line.split(' '))[2])
+ self._totalCylinders += cyls
+ break
+
+ return self._totalCylinders
+
+# vim:tw=78:ts=4:et:sw=4
--
1.6.2.5

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-28-2009, 12:25 PM
Chris Lumens
 
Default Find and format any unformatted DASD devices (#528386).

> >Would this kill all potential partitions on DASDs that were already both
> >low-level formatted as well as already having partitions?
> >That would mean whenever the user had one or more DASDs yet to be
> >low-level formatted, he would have to recreate all existing partitioning
> >on other DASDs which had been in use and thus prepared before.
> >How was this handled with RHEL5 in order not to introduce regressions?
>
> Hmm, yeah that could be a problem. I'm going to save the kickstart handling
> for another patch. I want to talk to clumens and see what, if anything,
> exists in kickstart that we could define/use for unformatted DASDs.

Actually, what I meant to suggest was the zerombr command, which
reinitializes any invalid partition tables it finds. We overload this
to do the same with LVM metadata. I suggest we further overload it to
do the same for unformatted DASDs.

I can't overstate my opposition to adding any new options or commands to
the clearpart/zerombr disaster, as that can only make things even more
complicated and introduce even more corner cases we don't test for.

- Chris

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-28-2009, 12:46 PM
Chris Lumens
 
Default Find and format any unformatted DASD devices (#528386).

> Under RHEL-5, this process was serial and the user had to click Yes for
> each unformatted DASD found, which could take a really long time if
> you had thousands of DASDs.
>
> The idea now is that if the DASD is seen by anaconda, we want to use it
> for installation. The stage 1 device initialization routines as well as
> the CMS conf file provided at boot time allow the user to restrict the
> range of devices we see during installation. If any of the devices we
> see are unformatted, run dasdfmt before building the devicetree.

I just realized that this is going to need to change when I get my
storage filtering patches in. At the least, DASD.startup is going to
have to respect storage.exclusiveDisks, or we could be formatting things
that the user explicitly said to not use.

- Chris

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-28-2009, 06:58 PM
David Cantrell
 
Default Find and format any unformatted DASD devices (#528386).

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 28 Oct 2009, Steffen Maier wrote:


Only two minor comments below.

On 10/28/2009 04:23 AM, David Cantrell wrote:

diff --git a/storage/dasd.py b/storage/dasd.py
new file mode 100644
index 0000000..efaed40
--- /dev/null
+++ b/storage/dasd.py
@@ -0,0 +1,173 @@



+class DASD:
+ """ Controlling class for DASD interaction before the storage code in
+ anaconda has initialized.
+
+ The DASD class can determine if any DASD devices on the system are
+ unformatted and can perform a dasdfmt on them.
+ """
+
+ def __init__(self):
+ self._dasdlist = []
+ self._totalCylinders = 0
+ self._completedCylinders = 0.0
+ self._maxFormatJobs = 0


Although parallel formatting is not yet implemented, initializing this
to zero almost sounds like unbounded to me. Especially with many devices
this is not desirable since it takes too much compute power. Typically,
there are many virtual machines running in parallel, even in production
mode and we should leave most of the compute power to them (and also not
just put this burden on the VM scheduler). Initializing
self._maxFormatJobs to 1 seems like a sane value to begin with.


self._maxFormatJobs is currently not referenced anywhere else in this code.
Only initialized here.


+ self.started = False
+
+ def startup(self, intf=None):
+ """ Look for any unformatted DASDs in the system and offer the user
+ the option for format them with dasdfmt or exit the installer.
+ """



+ for dasd in self._dasdlist:
+ log.info("Running dasdfmt on %s" % (dasd,))


Hm, this logs all future jobs which have not even started, so the user
or developers can hardly determine the progress from the logs, e.g.
during problem determination. Can we move this to the two places below?


Yeah, that's fine.




+
+ argv = ["-y", "-P", "-d", "cdl", "-b", "4096"]
+
+ if intf:
+ title = P_("Formatting DASD Device", "Formatting DASD Devices", c)
+ msg = P_("Preparing %d DASD device for use with Linux..." % c,
+ "Preparing %d DASD devices for use with Linux..." % c, c)
+ pw = intf.progressWindow(title, msg, 1.0)
+
+ for dasd in self._dasdlist:


here


+ iutil.execWithCallback("/sbin/dasdfmt", argv + [dasd],
+ stdout="/dev/tty5", stderr="/dev/tty5",
+ callback=self._updateProgressWindow,
+ callback_data=pw)
+
+ pw.pop()
+ else:
+ for dasd in self._dasdlist:


and here


+ iutil.execWithRedirect("/sbin/dasdfmt", argv + [dasd],
+ stdout="/dev/tty5", stderr="/dev/tty5")


Steffen

Linux on System z Development

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Erich Baier
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list



- --
David Cantrell <dcantrell@redhat.com>

Red Hat / Honolulu, HI

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkroonwACgkQ5hsjjIy1VkkZEgCg55Gk2HO/eIOsu7WsBa9wp49a
2qEAnRLyYen+PYge+ugQuNqK4qvAo2cx
=rx58
-----END PGP SIGNATURE-----_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-28-2009, 07:00 PM
David Cantrell
 
Default Find and format any unformatted DASD devices (#528386).

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 28 Oct 2009, Chris Lumens wrote:


Under RHEL-5, this process was serial and the user had to click Yes for
each unformatted DASD found, which could take a really long time if
you had thousands of DASDs.

The idea now is that if the DASD is seen by anaconda, we want to use it
for installation. The stage 1 device initialization routines as well as
the CMS conf file provided at boot time allow the user to restrict the
range of devices we see during installation. If any of the devices we
see are unformatted, run dasdfmt before building the devicetree.


I just realized that this is going to need to change when I get my
storage filtering patches in. At the least, DASD.startup is going to
have to respect storage.exclusiveDisks, or we could be formatting things
that the user explicitly said to not use.


That's fine, let me know what needs to be added and I can fix it up in a
future patch.

- --
David Cantrell <dcantrell@redhat.com>

Red Hat / Honolulu, HI

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkrootkACgkQ5hsjjIy1VkmKNACgxElsRPl+2n PkZwSAl9QrFupY
pwoAoMEObfhTQTQQOzGmsHxl5g43Z+Zk
=V9YW
-----END PGP SIGNATURE-----

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-29-2009, 12:14 PM
Chris Lumens
 
Default Find and format any unformatted DASD devices (#528386).

> >I just realized that this is going to need to change when I get my
> >storage filtering patches in. At the least, DASD.startup is going to
> >have to respect storage.exclusiveDisks, or we could be formatting things
> >that the user explicitly said to not use.
>
> That's fine, let me know what needs to be added and I can fix it up in a
> future patch.

Will do. I've got some more patches in the works before I get back to
storage filtering, but at that point I need to meet with people to talk
about multipath and s390 stuff. I'll let you know.

- Chris

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 

Thread Tools




All times are GMT. The time now is 04:20 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org