* Stefan Fritsch:

> I may be a bit late to this discussion, but aren't 64bit ints (and
> especially pack/unpack "Q") very useful for 64bit file pointers and
> such? IMHO, this means that they would also be very useful on
> "smaller" architectures like arm.

Yes, they are, and that's where I have run into incompatibilities. In
theory, Perl scalars degrade to doubles when the 32 bit range is
exceeded on 32 bit architectures, which gives you around 52 bits.
This should be sufficient for file sizes. But the differing internal
representation sometimes becomes visible and causes issues.

> In any case, there isn't really a strong connection with 64bit
> instructions in the processor. An executable compiled for 32bit cannot
> use those instructions anyway. So the issue is just perl's memory
> usage compared to the memory usually available on such systems. Do you
> have any experience how memory usage increases with 64bit support in
> real life usage?

On lenny-i386, pahole says this about struct sv:

struct sv {
void * sv_any; /* 0 4 */
U32 sv_refcnt; /* 4 4 */
U32 sv_flags; /* 8 4 */
union {
IV svu_iv; /* 4 */
UV svu_uv; /* 4 */
SV * svu_rv; /* 4 */
char * svu_pv; /* 4 */
SV * * svu_array; /* 4 */
HE * * svu_hash; /* 4 */
GP * svu_gp; /* 4 */
} sv_u; /* 12 4 */

/* size: 16, cachelines: 1 */
/* last cacheline: 16 bytes */
}; /* definitions: 39 */

So it boils down to malloc granularity (I don't think Perl's allocator
is used on Debian). For that, I wrote this little test program:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
unsigned size = atoi(argv[1]);
unsigned count = atoi(argv[2]);
unsigned i;

for (i = 0; i < count; ++i) {
malloc(size);
}
getc(stdin);
return 0;
}

Running it (again, on lenny-i386, albeit with a squeeze amd64 kernel),
I get:

./a.out 16 10000000 267m
./a.out 20 10000000 267m
./a.out 24 10000000 343m

And at least on i386, long long hasn't got alignment requirements, so
switching the type of IV only increases the size of struct to 20.

#include <stdio.h>

typedef unsigned int U32;
typedef long long IV;
typedef unsigned long long UV;
#define SV void
#define HE void
#define GP void

struct sv {
void * sv_any;
U32 sv_refcnt;
U32 sv_flags;
union {
IV svu_iv;
UV svu_uv;
SV * svu_rv;
char * svu_pv;
SV * * svu_array;
HE * * svu_hash;
GP * svu_gp;
} sv_u;
};

int main()
{
printf("%u
", sizeof(struct sv));
}

So if this analysis is correct, due to malloc granularity, this change
should not have a significant space overhead (the IV type is used in
other places, too, but SVs are the most sensitive place, I guess).
There might be a performance hit if the IV value caches a cache line
boundary, but in my experiment, malloc returns pointers congruent 8
modulo 32, so this should not happen in practice.

It's best to double-check all this by compiling Perl with the
appropriate flags, though.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 874oihqsyz.fsf@mid.deneb.enyo.de">http://lists.debian.org/874oihqsyz.fsf@mid.deneb.enyo.de

* Niko Tyni:

> I wasn't initially going for long doubles, but several upstream
> developers recommended that they be enabled together.
>
> http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2010-04/msg00773.html

This shows that long doubles are not backwards-compatible. 8-) The
root cause is that ** is a floating point operator, just like /. "use
integer" changes this, so I would say that this is just a user error.

> http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2010-04/msg00895.html

No matter how many bits you add, you can't change the fact that 0.1
cannot be represented exactly in binary floating point. And in
Debian's context, the argument goes in the other way: we'd have
NVs with 64, 80 and perhaps 96 and 128 bits, so we'd only see
accuracy issues on obscure architectures, making debugging much
harder.

> Given that we've already run into a dozen or so incompatibilities
> with just the CPAN modules, -Duselongdouble seems to be a pretty
> rare thing to do. I'm inclined to revert this setting.

That is, 64 bit NVs across all Debian architectures? That's good.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87hbmhpe1k.fsf@mid.deneb.enyo.de">http://lists.debian.org/87hbmhpe1k.fsf@mid.deneb.enyo.de

On Sun, May 09, 2010 at 12:17:56PM +0200, Florian Weimer wrote:

> So it boils down to malloc granularity (I don't think Perl's allocator
> is used on Debian). For that, I wrote this little test program:

While we do use the system malloc(), I think Perl allocates bigger chunks
at a time and thus avoids these granularity problems. From sv.c:

In all but the most memory-paranoid configuations (ex: PURIFY), heads
and bodies are allocated out of arenas, which by default are
approximately 4K chunks of memory parcelled up into N heads or bodies.
Sv-bodies are allocated by their sv-type, guaranteeing size
consistency needed to allocate safely from arrays.

> So if this analysis is correct, due to malloc granularity, this change
> should not have a significant space overhead (the IV type is used in
> other places, too, but SVs are the most sensitive place, I guess).

> It's best to double-check all this by compiling Perl with the
> appropriate flags, though.

Testing 5.12.0 with and without use64bitint on x86 shows an approximately
10% increase with scalars and arrays:

perl -e '${"v$i"} = $i while ($i++ < $ARGV[0]); system("ps -o rss $$")' 1000000
perl -e '$a[$i++] = $i while ($i++ < $ARGV[0]); system("ps -o rss $$")' 10000000

and a 7-8% increase for hashes:

perl -e '$h{$i++} = $i while ($i++ < $ARGV[0]); system("ps -o rss $$")' 5000000

Tracing the malloc calls indeed shows that only the allocation of 4k
areas increases significantly.

Somewhat more limited tests on agricola.d.o (5.10.1 without use64bitint vs.
5.12.0 with use64bitint) indicate similar results.

I don't think this gives any definite answer (expect "you can't please
everybody").

I'm partial to enabling use64bitint on all architectures, if only for the
sake of uniformity already mentioned in the uselongdouble discussion: bugs
that only happen on the "smaller" architectures because of differences
like this are not nice to isolate and debug.

And I expect pointers to gigabyte files are not an uncommon thing to
need on arm systems either.
--
Niko Tyni ntyni@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100509203724.GA21736@madeleine.local.invalid">ht tp://lists.debian.org/20100509203724.GA21736@madeleine.local.invalid

On Sun, May 09, 2010 at 12:25:43PM +0200, Florian Weimer wrote:
> * Niko Tyni:
>
> > Given that we've already run into a dozen or so incompatibilities
> > with just the CPAN modules, -Duselongdouble seems to be a pretty
> > rare thing to do. I'm inclined to revert this setting.
>
> That is, 64 bit NVs across all Debian architectures? That's good.

Yes, that's what I meant. Sorry for the unclear wording.

Thanks for the comments,
--
Niko Tyni ntyni@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100509203845.GA22197@madeleine.local.invalid">ht tp://lists.debian.org/20100509203845.GA22197@madeleine.local.invalid

On Tue, May 04, 2010 at 05:29:03PM +0200, Florian Weimer wrote:
> * Niko Tyni:
>
> > The benefits are obviously improved numeric range and precision. The
> > downside is presumably increased memory usage. I have no measurement
> > data on this; suggestions on suitable tests would be welcome.
>
> I have run into several incompatibilities between i386 and amd64 due
> to different Perl integer sizes, so I'm definitely in favor of 64-bit
> integers.
>
> I'm not sure if long doubles are a win. The rest of the world runs on
> 64 bit floating point numbers, so this would introduce additional
> incompatibilities.

I'd argue against a default setting where floating point numbers
are less precise than integers.

The problem here is that, running Perl code, it is hard to avoid
implicit numerical conversions, as Perl is designed to keep
those transparent. This is reasonably save when different types
are subsets of each other. However, as 64-bit floats cannot
hold 64-bit ints without loss of precision, hidden "upgrades"
will have dangerous effects, like rendering different things
equal. This in turn is bound to hurt the same code that
needed more precision in the first place.

Therefore, I'd like to see the migration to 64-bit integers and
long doubles happen simultaneously, painful as it might be, or
not at all. Keep in mind, I am talking default settings here,
which should lead to a perl interpreter with sane arithmetic.
Individual admins may always choose differently and live with
the consequences.

-Martin


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100510095023.GA17170@corcomroe.in-ulm.de">http://lists.debian.org/20100510095023.GA17170@corcomroe.in-ulm.de

On Mon, May 10, 2010 at 11:50:23AM +0200, Martin Becker wrote:

> I'd argue against a default setting where floating point numbers
> are less precise than integers.

I believe this has always been the case on our 64-bit architectures
(ia64, alpha, amd64.)

On current sid / amd64 (perl 5.10.1-12):

% perl -le 'print ((2**60 + 1) - (1 << 60))'
0
% perl -Minteger -le 'print ((2**60 + 1) - (1 << 60))'
1

(2**60 is a float in the first case and an integer in the second one.
1<<60 is an integer in both.)

> Therefore, I'd like to see the migration to 64-bit integers and
> long doubles happen simultaneously, painful as it might be, or
> not at all. Keep in mind, I am talking default settings here,
> which should lead to a perl interpreter with sane arithmetic.
> Individual admins may always choose differently and live with
> the consequences.

Given the amount of amd64 users that never had sane arithmetic in this
sense, I don't quite see why it would be so important for i386.

(Most of the discussion is on debian-devel, so please followup there.)
--
Niko Tyni ntyni@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100510185938.GA4561@madeleine.local.invalid">htt p://lists.debian.org/20100510185938.GA4561@madeleine.local.invalid

On Sun, May 09, 2010 at 11:37:24PM +0300, Niko Tyni wrote:

> I'm partial to enabling use64bitint on all architectures, if only for the
> sake of uniformity already mentioned in the uselongdouble discussion: bugs
> that only happen on the "smaller" architectures because of differences
> like this are not nice to isolate and debug.

Unfortunately it looks like sparc is a blocker for the uniformity due
to test failures with use64bitint but without uselongdouble. See #577016.
--
Niko Tyni ntyni@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100510200130.GA7031@madeleine.local.invalid">htt p://lists.debian.org/20100510200130.GA7031@madeleine.local.invalid

On Sun, May 09, 2010 at 11:37:24PM +0300, Niko Tyni wrote:

> Testing 5.12.0 with and without use64bitint on x86 shows an approximately
> 10% increase with scalars and arrays:
>
> perl -e '${"v$i"} = $i while ($i++ < $ARGV[0]); system("ps -o rss $$")' 1000000
> perl -e '$a[$i++] = $i while ($i++ < $ARGV[0]); system("ps -o rss $$")' 10000000
>
> and a 7-8% increase for hashes:
>
> perl -e '$h{$i++} = $i while ($i++ < $ARGV[0]); system("ps -o rss $$")' 5000000

To supplement these pathological cases, I compared a possibly more relevant
/usr/bin/time -f %M dpkg-reconfigure tzdata

on i386 with debconf-english and whiptail installed, and got

5.10.1/32bitint 23008
5.12.0/32bitint 23744
5.12.0/64bitint 24512

which means a 3% increase or thereabouts.

> I'm partial to enabling use64bitint on all architectures, if only for the
> sake of uniformity already mentioned in the uselongdouble discussion: bugs
> that only happen on the "smaller" architectures because of differences
> like this are not nice to isolate and debug.

To conclude, I'm going ahead with use64bitint but not uselongdouble on all
architectures. 5.12.0-2 is now in experimental with this configuration,
and I don't intend to change it for sid.

Many thanks for your comments,
--
Niko Tyni ntyni@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100512192020.GA12898@madeleine.local.invalid">ht tp://lists.debian.org/20100512192020.GA12898@madeleine.local.invalid