FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora/Linux Management Tools

 
 
LinkBack Thread Tools
 
Old 10-01-2008, 02:37 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

I'm done creating a sha256 hash setup should I offer more than just
sha256for now? and checksum generation is off by default


Here's a preview. Not sure how to catch the module import failure for
hashlib though




Cole Robinson wrote:

Daniel P. Berrange wrote:


On Tue, Sep 30, 2008 at 05:39:13PM -0400, Joey Boggs wrote:

Here's a sample that works, just want to verify it's alright. Is 64MB
too much/too little to read at one time?



f = open("test.raw","r")
m = sha.new()
while 1:
chunk = f.read(65536)
if not chunk:
break
m.update(chunk)
print m.hexdigest()


Both md5 and sha1 are becoming obsolete, and indeed forbidden by some
of the more paranoid organizations. I'd recommend we go straight
to using at least sha256. Also the docs recommend using hashlib module
directly, eg

import hashlib

m = hashlib.sha256()
while 1:
chunk = f.read(65536)
if not chunk:
break
m.update(chunk)
print m.hexdigest()

Daniel



Yeah, the only problem with hashlib is that it's python2.5
only. But we could just catch the import error and disable
the functionality if need be.

As far as md5 or sha1, no comment, though we probably want
to support whatever other config formats use (if any do
indeed offer hash support).

- Cole

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools



_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-01-2008, 02:42 PM
"Daniel P. Berrange"
 
Default virt-convert add disk signature into virt-image format export

On Wed, Oct 01, 2008 at 09:43:54AM -0400, Cole Robinson wrote:
> Daniel P. Berrange wrote:
> > On Tue, Sep 30, 2008 at 05:39:13PM -0400, Joey Boggs wrote:
> >> Here's a sample that works, just want to verify it's alright. Is 64MB
> >> too much/too little to read at one time?
> >>
> >>
> >> f = open("test.raw","r")
> >> m = sha.new()
> >> while 1:
> >> chunk = f.read(65536)
> >> if not chunk:
> >> break
> >> m.update(chunk)
> >> print m.hexdigest()
> >
> > Both md5 and sha1 are becoming obsolete, and indeed forbidden by some
> > of the more paranoid organizations. I'd recommend we go straight
> > to using at least sha256. Also the docs recommend using hashlib module
> > directly, eg
> >
> > import hashlib
> >
> > m = hashlib.sha256()
> > while 1:
> > chunk = f.read(65536)
> > if not chunk:
> > break
> > m.update(chunk)
> > print m.hexdigest()
> >
> > Daniel
>
> Yeah, the only problem with hashlib is that it's python2.5
> only. But we could just catch the import error and disable
> the functionality if need be.
>
> As far as md5 or sha1, no comment, though we probably want
> to support whatever other config formats use (if any do
> indeed offer hash support).

I suggest then we include multiple checksums. Either a md5 or sha1
checksum which we can do everywhere, and a second sha256 checksum
for stronger validation where available.

Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-01-2008, 02:43 PM
"Daniel P. Berrange"
 
Default virt-convert add disk signature into virt-image format export

On Wed, Oct 01, 2008 at 10:37:17AM -0400, Joey Boggs wrote:
> I'm done creating a sha256 hash setup should I offer more than just
> sha256for now? and checksum generation is off by default
>
> Here's a preview. Not sure how to catch the module import failure for
> hashlib though

If we go for doing a compulsory md5 checksum, and optional
sha256 checksum with new enough python, then something
like....

try:
import hashlib
m1 = hashlib.md5()
m2 = hashlib.sha256()
except:
import md5
m1 = md5.new()
m2 = None


Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-01-2008, 04:27 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

Here's an update, will this work best or any other suggestions?

try:
import hashlib
m1 = hashlib.md5(path)
m2 = hashlib.sha256(path)
except:
import md5
m1 = md5.new(path)
m2 = None

f = open(path,"r")
while 1:
chunk = f.read(65536)
if not chunk:
break
m1.update(chunk)
md5checksum = m1.hexdigest()

if m2:
m2.update(chunk)
shachecksum = m2.hexdigest()
storage.append(""" <checksum type="md5">%s</checksum>
"""
% md5checksum)

if shachecksum:
storage.append(""" <checksum
type="sha256">%s</checksum>
""" % shachecksum)

storage.append(""" </disk>
""")




Daniel P. Berrange wrote:

On Wed, Oct 01, 2008 at 10:37:17AM -0400, Joey Boggs wrote:

I'm done creating a sha256 hash setup should I offer more than just
sha256for now? and checksum generation is off by default


Here's a preview. Not sure how to catch the module import failure for
hashlib though



If we go for doing a compulsory md5 checksum, and optional
sha256 checksum with new enough python, then something
like....

try:
import hashlib
m1 = hashlib.md5()
m2 = hashlib.sha256()
except:
import md5
m1 = md5.new()
m2 = None


Daniel



_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-01-2008, 04:41 PM
"Daniel P. Berrange"
 
Default virt-convert add disk signature into virt-image format export

On Wed, Oct 01, 2008 at 12:27:46PM -0400, Joey Boggs wrote:
> Here's an update, will this work best or any other suggestions?
>
> try:
> import hashlib
> m1 = hashlib.md5(path)
> m2 = hashlib.sha256(path)
> except:
> import md5
> m1 = md5.new(path)
> m2 = None
>
> f = open(path,"r")
> while 1:
> chunk = f.read(65536)
> if not chunk:
> break
> m1.update(chunk)
> md5checksum = m1.hexdigest()
>
> if m2:
> m2.update(chunk)
> shachecksum = m2.hexdigest()

hexdigest() should only be called at end, outside
the loop. As it stands you're computing a checksum
for each chunk & resetting it

Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-01-2008, 04:46 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

That's one thing I forgot to move out of the loop before I sent it. I
was still working on the handling of m2 missing prior to that and put it
inside the loop for testing. Once moved should that be good to go?


Daniel P. Berrange wrote:

On Wed, Oct 01, 2008 at 12:27:46PM -0400, Joey Boggs wrote:


Here's an update, will this work best or any other suggestions?

try:
import hashlib
m1 = hashlib.md5(path)
m2 = hashlib.sha256(path)
except:
import md5
m1 = md5.new(path)
m2 = None

f = open(path,"r")
while 1:
chunk = f.read(65536)
if not chunk:
break
m1.update(chunk)
md5checksum = m1.hexdigest()

if m2:
m2.update(chunk)
shachecksum = m2.hexdigest()



hexdigest() should only be called at end, outside
the loop. As it stands you're computing a checksum
for each chunk & resetting it

Daniel



_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-01-2008, 09:15 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

Here's what I've got, moved the hexdigest() out of the loop, cleanup and
more testing to verify each scenario outputs the right xml configuration
data.


Hope this is the final revision






Joey Boggs wrote:
That's one thing I forgot to move out of the loop before I sent it. I
was still working on the handling of m2 missing prior to that and put
it inside the loop for testing. Once moved should that be good to go?


Daniel P. Berrange wrote:

On Wed, Oct 01, 2008 at 12:27:46PM -0400, Joey Boggs wrote:


Here's an update, will this work best or any other suggestions?

try:
import hashlib
m1 = hashlib.md5(path)
m2 = hashlib.sha256(path)
except:
import md5
m1 = md5.new(path)
m2 = None

f = open(path,"r")
while 1:
chunk = f.read(65536)
if not chunk:
break
m1.update(chunk)
md5checksum = m1.hexdigest()

if m2:
m2.update(chunk)
shachecksum = m2.hexdigest()



hexdigest() should only be called at end, outside
the loop. As it stands you're computing a checksum
for each chunk & resetting it

Daniel



_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools


_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-03-2008, 03:15 PM
Cole Robinson
 
Default virt-convert add disk signature into virt-image format export

Joey Boggs wrote:
> Here's what I've got, moved the hexdigest() out of the loop, cleanup and
> more testing to verify each scenario outputs the right xml configuration
> data.
>
> Hope this is the final revision
>
>
>
>

Some comments below.

> diff -r 58a909b4f71c virt-convert
> --- a/virt-convert Mon Sep 22 11:32:11 2008 -0400
> +++ b/virt-convert Wed Oct 01 17:12:45 2008 -0400
> @@ -64,6 +64,8 @@
> opts.add_option("", "--os-variant", type="string", dest="os_variant",
> action="callback", callback=cli.check_before_store,
> help=("The OS variant for fully virtualized guests, e.g. 'fedora6', 'rhel5', 'solaris10', 'win2k', 'vista'"))
> + opts.add_option("", "--checksum", action="store_true", dest="checksum",
> + help=("Generate a checksum for a virt-image guest"))
> opts.add_option("", "--noapic", action="store_true", dest="noapic",
> help=("Disables APIC for fully virtualized guest (overrides value in os-type/os-variant db)"), default=False)
> opts.add_option("", "--noacpi", action="store_true", dest="noacpi",
> @@ -184,6 +186,9 @@
>
> unixname = vmdef.name.replace(" ", "-")
>
> + if options.checksum:
> + vmdef.checksum = "yes"
> +

Rather than use a string yes/no, why not just call
the variable 'use_checksum' and have it as a bool
value?

We probably also want to add an option like
checksum_type, since it really isn't a simple
yes/no option. If no type is specified, we can
just whatever we deem is a sensible default.
This can be worked out later though.

> if not options.output_dir:
> options.output_dir = unixname
> try:
> diff -r 58a909b4f71c virtconv/parsers/virtimage.py
> --- a/virtconv/parsers/virtimage.py Mon Sep 22 11:32:11 2008 -0400
> +++ b/virtconv/parsers/virtimage.py Wed Oct 01 17:12:45 2008 -0400
> @@ -171,9 +171,41 @@
> type = "raw"
> if disk.type == diskcfg.DISK_TYPE_ISO:
> type = "iso"
> - storage.append(
> - """<disk file="%s" use="system" format="%s"/>
""" %
> - (path, type))
> +
> + if vm.checksum == "yes":
> + md5checksum = None
> + shachecksum = None
> +
> + storage.append("""<disk file="%s" use="system" format="%s">
""" % (path, type))
> +
> + try:
> + import hashlib
> + m1 = hashlib.md5(path)
> + m2 = hashlib.sha256(path)
> + except:
> + import md5
> + m1 = md5.new(path)
> + m2 = None
> +
> + f = open(path,"r")
> + while 1:
> + chunk = f.read(65536)
> + if not chunk:
> + break
> + m1.update(chunk)
> +
> + if m2:
> + m2.update(chunk)
> +

I tested the above, and it generates different checksums than
the cli utils (md5sum, sha256sum). Problem seems to be that
you initialize the hash with the file's pathname. Is that
intentional?

We should probably match the output of the cli utils, so
let's make sure the hashes match for both small and large
files to be sure we aren't missing anything.

> + md5checksum = m1.hexdigest()
> + storage.append(""" <checksum type="md5">%s</checksum>
""" % md5checksum)
> +
> + if m2:
> + shachecksum = m2.hexdigest()
> + storage.append(""" <checksum type="sha256">%s</checksum>
""" % shachecksum)
> + storage.append(""" </disk>
""")
> + else:
> + storage.append("""<disk file="%s" use="system" format="%s"/>
""" % (path, type))
>
> return storage, diskout
>
> diff -r 58a909b4f71c virtconv/vmcfg.py
> --- a/virtconv/vmcfg.py Mon Sep 22 11:32:11 2008 -0400
> +++ b/virtconv/vmcfg.py Wed Oct 01 17:12:45 2008 -0400
> @@ -59,6 +59,7 @@
> self.noapic = None
> self.os_type = None
> self.os_variant = None
> + self.checksum = None
>
> def validate(self):
> """

Thanks,
Cole

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-03-2008, 03:32 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

diff -r 58a909b4f71c virt-convert
--- a/virt-convert Mon Sep 22 11:32:11 2008 -0400
+++ b/virt-convert Wed Oct 01 17:12:45 2008 -0400
@@ -64,6 +64,8 @@
opts.add_option("", "--os-variant", type="string", dest="os_variant",
action="callback", callback=cli.check_before_store,
help=("The OS variant for fully virtualized guests, e.g. 'fedora6', 'rhel5', 'solaris10', 'win2k', 'vista'"))
+ opts.add_option("", "--checksum", action="store_true", dest="checksum",
+ help=("Generate a checksum for a virt-image guest"))
opts.add_option("", "--noapic", action="store_true", dest="noapic",
help=("Disables APIC for fully virtualized guest (overrides value in os-type/os-variant db)"), default=False)
opts.add_option("", "--noacpi", action="store_true", dest="noacpi",
@@ -184,6 +186,9 @@

unixname = vmdef.name.replace(" ", "-")

+ if options.checksum:

+ vmdef.checksum = "yes"
+



Rather than use a string yes/no, why not just call
the variable 'use_checksum' and have it as a bool
value?

We probably also want to add an option like
checksum_type, since it really isn't a simple

yes/no option. If no type is specified, we can
just whatever we deem is a sensible default.
This can be worked out later though.


I'll make that change for use_checksum. Would that mean we're only
generating 1 checksum by default?



+
+ if vm.checksum == "yes":
+ md5checksum = None
+ shachecksum = None
+
+ storage.append("""<disk file="%s" use="system" format="%s">
""" % (path, type))
+
+ try:

+ import hashlib
+ m1 = hashlib.md5(path)
+ m2 = hashlib.sha256(path)
+ except:
+ import md5
+ m1 = md5.new(path)
+ m2 = None
+
+ f = open(path,"r")
+ while 1:
+ chunk = f.read(65536)
+ if not chunk:
+ break
+ m1.update(chunk)
+
+ if m2:
+ m2.update(chunk)
+



I tested the above, and it generates different checksums than
the cli utils (md5sum, sha256sum). Problem seems to be that
you initialize the hash with the file's pathname. Is that
intentional?

We should probably match the output of the cli utils, so
let's make sure the hashes match for both small and large
files to be sure we aren't missing anything.


I just discovered that this morning when working on the
appliance-creator portions of this and just removed the filename out of
the hash.


_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-03-2008, 03:39 PM
Cole Robinson
 
Default virt-convert add disk signature into virt-image format export

Joey Boggs wrote:
>>
>>> diff -r 58a909b4f71c virt-convert
>>> --- a/virt-convert Mon Sep 22 11:32:11 2008 -0400
>>> +++ b/virt-convert Wed Oct 01 17:12:45 2008 -0400
>>> @@ -64,6 +64,8 @@
>>> opts.add_option("", "--os-variant", type="string", dest="os_variant",
>>> action="callback", callback=cli.check_before_store,
>>> help=("The OS variant for fully virtualized guests, e.g. 'fedora6', 'rhel5', 'solaris10', 'win2k', 'vista'"))
>>> + opts.add_option("", "--checksum", action="store_true", dest="checksum",
>>> + help=("Generate a checksum for a virt-image guest"))
>>> opts.add_option("", "--noapic", action="store_true", dest="noapic",
>>> help=("Disables APIC for fully virtualized guest (overrides value in os-type/os-variant db)"), default=False)
>>> opts.add_option("", "--noacpi", action="store_true", dest="noacpi",
>>> @@ -184,6 +186,9 @@
>>>
>>> unixname = vmdef.name.replace(" ", "-")
>>>
>>> + if options.checksum:
>>> + vmdef.checksum = "yes"
>>> +
>>>
>> Rather than use a string yes/no, why not just call
>> the variable 'use_checksum' and have it as a bool
>> value?
>>
>> We probably also want to add an option like
>> checksum_type, since it really isn't a simple
>> yes/no option. If no type is specified, we can
>> just whatever we deem is a sensible default.
>> This can be worked out later though.
>>
>>
> I'll make that change for use_checksum. Would that mean we're only
> generating 1 checksum by default?

Hmm, good question. Maybe just do away with use_checksum. We can
have a value checksum_types, which is a list of strings. Then
we can define constants like CSUM_MD5, CSUM_SHA256, etc. One
of these can be CSUM_DEFAULT, which is up to the individual
conversion drivers. So if the list is empty, no csum, otherwise
generate a csum for every entry in the list.


_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 

Thread Tools




All times are GMT. The time now is 04:30 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org