FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 08-04-2011, 10:02 AM
Olivier Sallou
 
Default Fwd: BLAST+ speed & build issues

Hi,

we have a package on DebianMed which has performance issue regarding
static build vs dynamic link build (latest gcc)

Has only impacted compile option is the static vs dynamic choice,
has anyone an idea on where we could investigate or how we could
speedup this?



For info, executable is quite large 11MB (static) vs 2M (dynamic).



Dynamic build exe is linked against:

osallou@debiansid:/tmp/ncbi-blast-2.2.25+-src/debian/ncbi-blast+/usr/bin$
ldd /usr/bin/blastx

*** linux-vdso.so.1 =>* (0x00007fffb01a8000)

*** libblastinput.so => /usr/lib/ncbi-blast+/libblastinput.so
(0x00007f06d9572000)

*** libncbi_xloader_blastdb.so =>
/usr/lib/ncbi-blast+/libncbi_xloader_blastdb.so (0x00007f06d9350000)

*** libncbi_xloader_blastdb_rmt.so =>
/usr/lib/ncbi-blast+/libncbi_xloader_blastdb_rmt.so
(0x00007f06d9127000)

*** libxblastformat.so => /usr/lib/ncbi-blast+/libxblastformat.so
(0x00007f06d8edd000)

*** libalign_format.so => /usr/lib/ncbi-blast+/libalign_format.so
(0x00007f06d8c32000)

*** libblastdb_format.so =>
/usr/lib/ncbi-blast+/libblastdb_format.so (0x00007f06d8a1a000)

*** libgene_info.so => /usr/lib/ncbi-blast+/libgene_info.so
(0x00007f06d8808000)

*** libxalnmgr.so => /usr/lib/ncbi-blast+/libxalnmgr.so
(0x00007f06d856a000)

*** libblastxml.so => /usr/lib/ncbi-blast+/libblastxml.so
(0x00007f06d8353000)

*** libxcgi.so => /usr/lib/ncbi-blast+/libxcgi.so
(0x00007f06d80c8000)

*** libxhtml.so => /usr/lib/ncbi-blast+/libxhtml.so
(0x00007f06d7e48000)

*** libxblast.so => /usr/lib/ncbi-blast+/libxblast.so
(0x00007f06d7a36000)

*** libxalgoblastdbindex.so =>
/usr/lib/ncbi-blast+/libxalgoblastdbindex.so (0x00007f06d77fa000)

*** libcomposition_adjustment.so =>
/usr/lib/ncbi-blast+/libcomposition_adjustment.so
(0x00007f06d75e6000)

*** libxalgodustmask.so =>
/usr/lib/ncbi-blast+/libxalgodustmask.so (0x00007f06d73dd000)

*** libxalgowinmask.so => /usr/lib/ncbi-blast+/libxalgowinmask.so
(0x00007f06d718a000)

*** libseqmasks_io.so => /usr/lib/ncbi-blast+/libseqmasks_io.so
(0x00007f06d6f7a000)

*** libseqdb.so => /usr/lib/ncbi-blast+/libseqdb.so
(0x00007f06d6cb2000)

*** libblast_services.so =>
/usr/lib/ncbi-blast+/libblast_services.so (0x00007f06d6aa1000)

*** libxobjutil.so => /usr/lib/ncbi-blast+/libxobjutil.so
(0x00007f06d67f5000)

*** libxobjread.so => /usr/lib/ncbi-blast+/libxobjread.so
(0x00007f06d64de000)

*** libcreaders.so => /usr/lib/ncbi-blast+/libcreaders.so
(0x00007f06d62d5000)

*** libxnetblastcli.so => /usr/lib/ncbi-blast+/libxnetblastcli.so
(0x00007f06d60c8000)

*** libxnetblast.so => /usr/lib/ncbi-blast+/libxnetblast.so
(0x00007f06d5e1b000)

*** libblastdb.so => /usr/lib/ncbi-blast+/libblastdb.so
(0x00007f06d5c06000)

*** libscoremat.so => /usr/lib/ncbi-blast+/libscoremat.so
(0x00007f06d59e2000)

*** libtables.so => /usr/lib/ncbi-blast+/libtables.so
(0x00007f06d57de000)

*** libncbi_xloader_genbank.so =>
/usr/lib/ncbi-blast+/libncbi_xloader_genbank.so (0x00007f06d55a1000)

*** libncbi_xreader_id1.so =>
/usr/lib/ncbi-blast+/libncbi_xreader_id1.so (0x00007f06d537b000)

*** libncbi_xreader_id2.so =>
/usr/lib/ncbi-blast+/libncbi_xreader_id2.so (0x00007f06d5160000)

*** libncbi_xreader_cache.so =>
/usr/lib/ncbi-blast+/libncbi_xreader_cache.so (0x00007f06d4f2c000)

*** libxconnect.so => /usr/lib/ncbi-blast+/libxconnect.so
(0x00007f06d4c94000)

*** libncbi_xreader.so => /usr/lib/ncbi-blast+/libncbi_xreader.so
(0x00007f06d49e5000)

*** libid1.so => /usr/lib/ncbi-blast+/libid1.so
(0x00007f06d47cb000)

*** libid2.so => /usr/lib/ncbi-blast+/libid2.so
(0x00007f06d457e000)

*** libseqsplit.so => /usr/lib/ncbi-blast+/libseqsplit.so
(0x00007f06d4324000)

*** libxcompress.so => /usr/lib/ncbi-blast+/libxcompress.so
(0x00007f06d40cf000)

*** libxobjmgr.so => /usr/lib/ncbi-blast+/libxobjmgr.so
(0x00007f06d3be3000)

*** libgenome_collection.so =>
/usr/lib/ncbi-blast+/libgenome_collection.so (0x00007f06d3981000)

*** libseqset.so => /usr/lib/ncbi-blast+/libseqset.so
(0x00007f06d3762000)

*** libseqedit.so => /usr/lib/ncbi-blast+/libseqedit.so
(0x00007f06d3519000)

*** libseq.so => /usr/lib/ncbi-blast+/libseq.so
(0x00007f06d2f16000)

*** libseqcode.so => /usr/lib/ncbi-blast+/libseqcode.so
(0x00007f06d2d03000)

*** libsequtil.so => /usr/lib/ncbi-blast+/libsequtil.so
(0x00007f06d2af2000)

*** libpub.so => /usr/lib/ncbi-blast+/libpub.so
(0x00007f06d28d2000)

*** libmedline.so => /usr/lib/ncbi-blast+/libmedline.so
(0x00007f06d26b2000)

*** libbiblio.so => /usr/lib/ncbi-blast+/libbiblio.so
(0x00007f06d246a000)

*** libgeneral.so => /usr/lib/ncbi-blast+/libgeneral.so
(0x00007f06d220c000)

*** libxser.so => /usr/lib/ncbi-blast+/libxser.so
(0x00007f06d1ec0000)

*** libxutil.so => /usr/lib/ncbi-blast+/libxutil.so
(0x00007f06d1bea000)

*** libxncbi.so => /usr/lib/ncbi-blast+/libxncbi.so
(0x00007f06d180e000)

*** libz.so.1 => /usr/lib/libz.so.1 (0x00007f06d15ed000)

*** libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x00007f06d13dd000)

*** libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
(0x00007f06d11d8000)

*** libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1
(0x00007f06d0fc0000)

*** librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
(0x00007f06d0db8000)

*** libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f06d0b9b000)

*** libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007f06d0891000)

*** libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
(0x00007f06d060f000)

*** libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x00007f06d03f8000)

*** libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
(0x00007f06d0074000)

*** /lib64/ld-linux-x86-64.so.2 (0x00007f06d97e6000)





For a simple "-h" call, here is a benchmark:



# Static build"

osallou@debiansid:/tmp/ncbi-blast-2.2.25+-src/debian/ncbi-blast+/usr/bin$
time bash -c 'for (( c=1; c<=50; c++ )) ; do ./blastx -h >
/dev/null ; done'



real*** 0m0.403s

user*** 0m0.292s

sys*** 0m0.124s



# Dynamic build

osallou@debiansid:/tmp/ncbi-blast-2.2.25+-src/debian/ncbi-blast+/usr/bin$
time bash -c 'for (( c=1; c<=50; c++ )) ; do blastx -h >
/dev/null ; done'



real*** 0m8.002s

user*** 0m4.200s

sys*** 0m3.856s





-------- Message original --------



Sujet:
Re: BLAST+ speed & build issues


Date de
renvoi*:
Thu, 4 Aug 2011 09:31:54 +0000 (UTC)


De
(renvoi)*:
debian-med@lists.debian.org


Date*:
Thu, 04 Aug 2011 11:31:34 +0200


De*:
Olivier Sallou <olivier.sallou@irisa.fr>


Pour*:
Tim Booth <avarus@fastmail.fm>


Copie à*:

debian-med@lists.debian.org, "Aaron M. Ucko"
<amu@alum.mit.edu>







Hi,
I quickly build the ncbi-blast+ locally with static build using the
debian build process, and indeed,
static build binaries are much faster (blastx -h) than dynamically
linked ones.

I will investigate if there is a way to speedup this. Or if anyone has
ideas....

Olivier

Le 8/3/11 10:48 AM, Tim Booth a écrit :
> time bash -c 'for (( c=1; c<=50; c++ )) ; do ~/tings/ncbi-blast-2.2.25+/bin/blastx -h > /dev/null ; done'

--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95

gpg key id: 4096R/326D8438 (pgp.mit.edu)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438



--
To UNSUBSCRIBE, email to debian-med-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/4E3A66F6.7040008@irisa.fr
 
Old 08-04-2011, 10:32 AM
Julian Taylor
 
Default Fwd: BLAST+ speed & build issues

On 08/04/2011 12:02 PM, Olivier Sallou wrote:
> Hi,
> we have a package on DebianMed which has performance issue regarding
> static build vs dynamic link build (latest gcc)
> Has only impacted compile option is the static vs dynamic choice, has
> anyone an idea on where we could investigate or how we could speedup this?
>
> For a simple "-h" call, here is a benchmark:
>
> # Static build"
> osallou@debiansid:/tmp/ncbi-blast-2.2.25+-src/debian/ncbi-blast+/usr/bin$ time
> bash -c 'for (( c=1; c<=50; c++ )) ; do ./blastx -h > /dev/null ; done'
>
> real 0m0.403s
> user 0m0.292s
> sys 0m0.124s
>
> # Dynamic build
> osallou@debiansid:/tmp/ncbi-blast-2.2.25+-src/debian/ncbi-blast+/usr/bin$ time
> bash -c 'for (( c=1; c<=50; c++ )) ; do blastx -h > /dev/null ; done'
>
> real 0m8.002s
> user 0m4.200s
> sys 0m3.856s
>
>

Hi,
unless the purpose of this program is to display help messages to stdout
this is hardly a meaningful benchmark.
Starting a dynamic linked library is slower than starting a static one,
but that does not yet mean the dynamic version will be so much slower
when its running.
You might be able to reduce startup time by only linking against the
libraries you need or lazyly dynamically loading them.
using ld --as-needed is the solution for the former, but see bug 633567
for problems with this.

Best Regards,
Julian Taylor
 
Old 08-04-2011, 11:27 AM
Samuel Thibault
 
Default Fwd: BLAST+ speed & build issues

Julian Taylor, le Thu 04 Aug 2011 12:32:27 +0200, a écrit :
> You might be able to reduce startup time by only linking against the
> libraries you need or lazyly dynamically loading them.

Or use prelink.

Samuel


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110804112744.GF4985@type.bordeaux.inria.fr">http ://lists.debian.org/20110804112744.GF4985@type.bordeaux.inria.fr
 
Old 08-04-2011, 11:50 AM
Yves-Alexis Perez
 
Default Fwd: BLAST+ speed & build issues

On jeu., 2011-08-04 at 13:27 +0200, Samuel Thibault wrote:
> Julian Taylor, le Thu 04 Aug 2011 12:32:27 +0200, a écrit :
> > You might be able to reduce startup time by only linking against the
> > libraries you need or lazyly dynamically loading them.
>
> Or use prelink.

Which as issues especially wrt. ASLR.

Regards,
--
Yves-Alexis
 
Old 08-04-2011, 11:56 AM
Mike Hommey
 
Default Fwd: BLAST+ speed & build issues

On Thu, Aug 04, 2011 at 01:50:01PM +0200, Yves-Alexis Perez wrote:
> On jeu., 2011-08-04 at 13:27 +0200, Samuel Thibault wrote:
> > Julian Taylor, le Thu 04 Aug 2011 12:32:27 +0200, a écrit :
> > > You might be able to reduce startup time by only linking against the
> > > libraries you need or lazyly dynamically loading them.
> >
> > Or use prelink.
>
> Which as issues especially wrt. ASLR.

And doesn't help with the biggest problem: I/O

Mike


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110804115646.GA10717@glandium.org">http://lists.debian.org/20110804115646.GA10717@glandium.org
 
Old 08-04-2011, 03:33 PM
olivier sallou
 
Default Fwd: BLAST+ speed & build issues

Linking against "needed" lib is quite difficult (would need to be done on upstream), package is rather complex regarding compilation.And each solution has its pros and cons....
I think we will have to discuss on validity to provide static builds ( takes a lot of space on package vs perf loss)

Thanks anyway

2011/8/4 Samuel Thibault <sthibault@debian.org>

Julian Taylor, le Thu 04 Aug 2011 12:32:27 +0200, a écrit :

> You might be able to reduce startup time by only linking against the

> libraries you need or lazyly dynamically loading them.



Or use prelink.



Samuel





--

To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org

with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: http://lists.debian.org/20110804112744.GF4985@type.bordeaux.inria.fr
 
Old 08-04-2011, 03:54 PM
Stephen Frost
 
Default Fwd: BLAST+ speed & build issues

Olivier,

Because "-h is slow" hardly seems like a good justification for having
static packages. Last I checked (which wasn't long ago..), BLAST is
typically a long-running process (at least on the stuff we're doing..).
Also, are subsequent calls (even '-h' ones) faster? I'd expect them to
be, once everything is cache'd, which would reduce the justification
that much further..

No, having the database stay in memory and being able to query against
it rather than having to start up blast every time (ala FastCGI or
something like you'd do for a webserver) might be a way to kill two
birds w/ one stone, as it was, but that'd be a fair bit of work, I
expect.

Thanks,

Stephen

* olivier sallou (olivier.sallou@gmail.com) wrote:
> Linking against "needed" lib is quite difficult (would need to be done on
> upstream), package is rather complex regarding compilation.
> And each solution has its pros and cons....
>
> I think we will have to discuss on validity to provide static builds ( takes
> a lot of space on package vs perf loss)
>
> Thanks anyway
>
> 2011/8/4 Samuel Thibault <sthibault@debian.org>
>
> > Julian Taylor, le Thu 04 Aug 2011 12:32:27 +0200, a écrit :
> > > You might be able to reduce startup time by only linking against the
> > > libraries you need or lazyly dynamically loading them.
> >
> > Or use prelink.
> >
> > Samuel
> >
> >
> > --
> > To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
> > with a subject of "unsubscribe". Trouble? Contact
> > listmaster@lists.debian.org
> > Archive:
> > http://lists.debian.org/20110804112744.GF4985@type.bordeaux.inria.fr
> >
> >
 
Old 08-06-2011, 10:37 AM
Kurt Roeckx
 
Default Fwd: BLAST+ speed & build issues

On Thu, Aug 04, 2011 at 12:02:25PM +0200, Olivier Sallou wrote:
> For info, executable is quite large 11MB (static) vs 2M (dynamic).

The 11MB probably contains a whole bunch of libraries instead,
making it larger. I don't see how this is relavant.

> Dynamic build exe is linked against:
> osallou@debiansid:/tmp/ncbi-blast-2.2.25+-src/debian/ncbi-blast+/usr/bin$ ldd
> /usr/bin/blastx
> linux-vdso.so.1 => (0x00007fffb01a8000)
> libblastinput.so => /usr/lib/ncbi-blast+/libblastinput.so
> (0x00007f06d9572000)
[...lots of blast libs...]

> For a simple "-h" call, here is a benchmark:

Like others have pointed out, this is not a useful benchmark.
You're basicly testing the startup time of your application. When
you start an applications that dynamic linker has to resolv
symbols and relocations. This takes time which for the static
version was done at link time. Having more libraries it needs to
check for the symbol means it will take longer.

What can help is that you reduce the total amount of symbols that
it needs to check be only exporting the symbols you need from the
shared libraries. There are various methods for that.

The basic 3 methods I know:
- Making things as much as possible static.
- Using symbol visibility: tell gcc which symbols should be visible.
- Using a linker map to only export those that should be visible.


Kurt


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110806103714.GA2861@roeckx.be">http://lists.debian.org/20110806103714.GA2861@roeckx.be
 

Thread Tools




All times are GMT. The time now is 06:21 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org