FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora/Linux Management Tools

 
 
LinkBack Thread Tools
 
Old 09-26-2008, 08:00 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

Adds disk signatures into virt-convert for virt-image format virtual
machines
_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 09-30-2008, 05:50 PM
Bryan Kearney
 
Default virt-convert add disk signature into virt-image format export

Cole Robinson wrote:

Joey Boggs wrote:
Adds disk signatures into virt-convert for virt-image format virtual
machines





Comments inline.


diff -r 58a909b4f71c virtconv/parsers/virtimage.py
--- a/virtconv/parsers/virtimage.py Mon Sep 22 11:32:11 2008 -0400
+++ b/virtconv/parsers/virtimage.py Fri Sep 26 15:58:29 2008 -0400
@@ -22,7 +22,7 @@
import virtconv.vmcfg as vmcfg
import virtconv.diskcfg as diskcfg
import virtinst.FullVirtGuest as fv
-
+import sha


Please keep the preceeding newline. Generally imports are
grouped in some attempt at a logical manner for the sake

of readability.


from xml.sax.saxutils import escape
from string import ascii_letters
import re
@@ -171,9 +171,11 @@
type = "raw"
if disk.type == diskcfg.DISK_TYPE_ISO:
type = "iso"
+ diskfile=open(path,'r').read()
+ checksum=sha.new(diskfile).hexdigest()


Please try to be consistent with surrounding code
format.


The code above is using: var = val
You've added: var=val

It helps code readability if spacing is reasonably
consistent throughout.

Hmm, so I just tried the above. What that is essentially
trying to do is read the entire disk into memory, then
pass it off as a giant string to the sha function. Clearly
this is not a workable solution. This also applies to
the virt-image hash changes as well. Haven't looked into
the correct way to do it though.

This is also a rather large performance drain. So at the
very least it should not be the default. That said, I
don't know the optimal way to expose this option, or
even if we should.


I think this will become important as we look to distribute the
appliances either via RHN or on public sites. Also.. this makes us a bit
more consistent with OVF.


I am fine with making it off by default, and then causing the command
line to turn it on.


-- bk

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 09-30-2008, 05:50 PM
Cole Robinson
 
Default virt-convert add disk signature into virt-image format export

Joey Boggs wrote:
> Adds disk signatures into virt-convert for virt-image format virtual
> machines
>
>

Comments inline.

>
> diff -r 58a909b4f71c virtconv/parsers/virtimage.py
> --- a/virtconv/parsers/virtimage.py Mon Sep 22 11:32:11 2008 -0400
> +++ b/virtconv/parsers/virtimage.py Fri Sep 26 15:58:29 2008 -0400
> @@ -22,7 +22,7 @@
> import virtconv.vmcfg as vmcfg
> import virtconv.diskcfg as diskcfg
> import virtinst.FullVirtGuest as fv
> -
> +import sha

Please keep the preceeding newline. Generally imports are
grouped in some attempt at a logical manner for the sake
of readability.

> from xml.sax.saxutils import escape
> from string import ascii_letters
> import re
> @@ -171,9 +171,11 @@
> type = "raw"
> if disk.type == diskcfg.DISK_TYPE_ISO:
> type = "iso"
> + diskfile=open(path,'r').read()
> + checksum=sha.new(diskfile).hexdigest()

Please try to be consistent with surrounding code
format.

The code above is using: var = val
You've added: var=val

It helps code readability if spacing is reasonably
consistent throughout.

Hmm, so I just tried the above. What that is essentially
trying to do is read the entire disk into memory, then
pass it off as a giant string to the sha function. Clearly
this is not a workable solution. This also applies to
the virt-image hash changes as well. Haven't looked into
the correct way to do it though.

This is also a rather large performance drain. So at the
very least it should not be the default. That said, I
don't know the optimal way to expose this option, or
even if we should.

Is there a way that vmx files can store checksums?

> storage.append(
> - """<disk file="%s" use="system" format="%s"/>
""" %
> - (path, type))
> + """<disk file="%s" use="system" format="%s">
"""
> + """ <checksum type="sha1">%s</checksum>
</disk>
""" % (path, type,checksum))
>
> return storage, diskout


Thanks,
Cole


_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 09-30-2008, 06:24 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

It helps code readability if spacing is reasonably
consistent throughout.

Hmm, so I just tried the above. What that is essentially
trying to do is read the entire disk into memory, then
pass it off as a giant string to the sha function. Clearly
this is not a workable solution. This also applies to
the virt-image hash changes as well. Haven't looked into
the correct way to do it though.

This is also a rather large performance drain. So at the
very least it should not be the default. That said, I
don't know the optimal way to expose this option, or
even if we should.


I think this will become important as we look to distribute the
appliances either via RHN or on public sites. Also.. this makes us a
bit more consistent with OVF.


I am fine with making it off by default, and then causing the command
line to turn it on.


-- bk
I'd rather it be right than kill performance and make it unusable at
best, should the imagefile be bigger than the machines memory. If we
outsource it to run a system command is that feasible, since we're
dealing with big files? I know Cole was against that originally. I'll
keep looking for a better way and cleanup the variables as well. Making
it off by default is probably best too.





_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 09-30-2008, 06:33 PM
Cole Robinson
 
Default virt-convert add disk signature into virt-image format export

Joey Boggs wrote:
>>>
>>> It helps code readability if spacing is reasonably
>>> consistent throughout.
>>>
>>> Hmm, so I just tried the above. What that is essentially
>>> trying to do is read the entire disk into memory, then
>>> pass it off as a giant string to the sha function. Clearly
>>> this is not a workable solution. This also applies to
>>> the virt-image hash changes as well. Haven't looked into
>>> the correct way to do it though.
>>>
>>> This is also a rather large performance drain. So at the
>>> very least it should not be the default. That said, I
>>> don't know the optimal way to expose this option, or
>>> even if we should.
>> I think this will become important as we look to distribute the
>> appliances either via RHN or on public sites. Also.. this makes us a
>> bit more consistent with OVF.
>>
>> I am fine with making it off by default, and then causing the command
>> line to turn it on.
>>
>> -- bk
> I'd rather it be right than kill performance and make it unusable at
> best, should the imagefile be bigger than the machines memory. If we
> outsource it to run a system command is that feasible, since we're
> dealing with big files? I know Cole was against that originally. I'll
> keep looking for a better way and cleanup the variables as well. Making
> it off by default is probably best too.
>

When I was talking about performance, I was addressing the
'correct' way to do, when the 'read into memory' piece is
fixed. Even then, generating a checksum is a very slow
process if you are using a large disk (I used sha1sum on
a DVD iso and it took about 3-4 minutes).

So in this case, turning it off by default is the thing to
do, we can figure out a cli switch to enable it, but this
is all dependent on determining the correct way to use the
libraries. You'll just have to use a loop and read in blocks
of data at a time, and keep passing it to the library to
update the hash.

- Cole

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 09-30-2008, 09:39 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

Here's a sample that works, just want to verify it's alright. Is 64MB
too much/too little to read at one time?



f = open("test.raw","r")
m = sha.new()
while 1:
chunk = f.read(65536)
if not chunk:
break
m.update(chunk)
print m.hexdigest()



Cole Robinson wrote:

Joey Boggs wrote:


It helps code readability if spacing is reasonably
consistent throughout.

Hmm, so I just tried the above. What that is essentially
trying to do is read the entire disk into memory, then
pass it off as a giant string to the sha function. Clearly
this is not a workable solution. This also applies to
the virt-image hash changes as well. Haven't looked into
the correct way to do it though.

This is also a rather large performance drain. So at the
very least it should not be the default. That said, I
don't know the optimal way to expose this option, or
even if we should.

I think this will become important as we look to distribute the
appliances either via RHN or on public sites. Also.. this makes us a
bit more consistent with OVF.


I am fine with making it off by default, and then causing the command
line to turn it on.


-- bk

I'd rather it be right than kill performance and make it unusable at
best, should the imagefile be bigger than the machines memory. If we
outsource it to run a system command is that feasible, since we're
dealing with big files? I know Cole was against that originally. I'll
keep looking for a better way and cleanup the variables as well. Making
it off by default is probably best too.





When I was talking about performance, I was addressing the
'correct' way to do, when the 'read into memory' piece is
fixed. Even then, generating a checksum is a very slow
process if you are using a large disk (I used sha1sum on
a DVD iso and it took about 3-4 minutes).

So in this case, turning it off by default is the thing to
do, we can figure out a cli switch to enable it, but this
is all dependent on determining the correct way to use the
libraries. You'll just have to use a loop and read in blocks
of data at a time, and keep passing it to the library to
update the hash.


- Cole




_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 09-30-2008, 09:43 PM
Cole Robinson
 
Default virt-convert add disk signature into virt-image format export

Joey Boggs wrote:
> Here's a sample that works, just want to verify it's alright. Is 64MB
> too much/too little to read at one time?
>

64MB certainly would be too much, but the below code
is using 64KB Should be fine.

>
> f = open("test.raw","r")
> m = sha.new()
> while 1:
> chunk = f.read(65536)
> if not chunk:
> break
> m.update(chunk)
> print m.hexdigest()
>

Thanks,
Cole

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 09-30-2008, 09:46 PM
Joey Boggs
 
Default virt-convert add disk signature into virt-image format export

hehe yeah i realized that a sec ago but didnt want to waste an email
correcting myself.




Cole Robinson wrote:

Joey Boggs wrote:

Here's a sample that works, just want to verify it's alright. Is 64MB
too much/too little to read at one time?





64MB certainly would be too much, but the below code
is using 64KB Should be fine.



f = open("test.raw","r")
m = sha.new()
while 1:
chunk = f.read(65536)
if not chunk:
break
m.update(chunk)
print m.hexdigest()




Thanks,
Cole



_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-01-2008, 09:54 AM
"Daniel P. Berrange"
 
Default virt-convert add disk signature into virt-image format export

On Tue, Sep 30, 2008 at 05:39:13PM -0400, Joey Boggs wrote:
> Here's a sample that works, just want to verify it's alright. Is 64MB
> too much/too little to read at one time?
>
>
> f = open("test.raw","r")
> m = sha.new()
> while 1:
> chunk = f.read(65536)
> if not chunk:
> break
> m.update(chunk)
> print m.hexdigest()

Both md5 and sha1 are becoming obsolete, and indeed forbidden by some
of the more paranoid organizations. I'd recommend we go straight
to using at least sha256. Also the docs recommend using hashlib module
directly, eg

import hashlib

m = hashlib.sha256()
while 1:
chunk = f.read(65536)
if not chunk:
break
m.update(chunk)
print m.hexdigest()

Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 
Old 10-01-2008, 01:43 PM
Cole Robinson
 
Default virt-convert add disk signature into virt-image format export

Daniel P. Berrange wrote:
> On Tue, Sep 30, 2008 at 05:39:13PM -0400, Joey Boggs wrote:
>> Here's a sample that works, just want to verify it's alright. Is 64MB
>> too much/too little to read at one time?
>>
>>
>> f = open("test.raw","r")
>> m = sha.new()
>> while 1:
>> chunk = f.read(65536)
>> if not chunk:
>> break
>> m.update(chunk)
>> print m.hexdigest()
>
> Both md5 and sha1 are becoming obsolete, and indeed forbidden by some
> of the more paranoid organizations. I'd recommend we go straight
> to using at least sha256. Also the docs recommend using hashlib module
> directly, eg
>
> import hashlib
>
> m = hashlib.sha256()
> while 1:
> chunk = f.read(65536)
> if not chunk:
> break
> m.update(chunk)
> print m.hexdigest()
>
> Daniel

Yeah, the only problem with hashlib is that it's python2.5
only. But we could just catch the import error and disable
the functionality if need be.

As far as md5 or sha1, no comment, though we probably want
to support whatever other config formats use (if any do
indeed offer hash support).

- Cole

_______________________________________________
et-mgmt-tools mailing list
et-mgmt-tools@redhat.com
https://www.redhat.com/mailman/listinfo/et-mgmt-tools
 

Thread Tools




All times are GMT. The time now is 04:06 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org