read() wont allow me to read files larger than 2 gig (on a 64bit)
 

Hi
Sorry if I'm asking on the wrong mailinglist,
but I've asked on various forums and I havn't found a solution.

I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
the problem is not related to the allocation itself, which works.
Attached is a sample program that illustrates this.

I've tried with various, compiler flags like
-D_FILE_OFFSET_BITS=64

According to man 2 read, the maximum is limited by SSIZE_MAX
which is defined in

/usr/include/bits/posix1_lim.h
# define SSIZE_MAX LONG_MAX

And LONG_MAX is defined in /usr/include/limits.h as
# if __WORDSIZE == 64
# define LONG_MAX 9223372036854775807L
# else
# define LONG_MAX 2147483647L
# endif
# define LONG_MIN (-LONG_MAX - 1L)

Is this the expected behaviour for the read function on a 64bit platform,
or is something broken

Thanks in advance

edit:

by the way

readelf -h ./a.out
ELF Header:
* Magic:** 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
* Class:**************************** ELF64
* Data:***************************** 2's complement, little endian
* Version:************************** 1 (current)
* OS/ABI:*************************** UNIX - System V
* ABI Version:********************** 0
* Type:***************************** EXEC (Executable file)
* Machine:************************** Advanced Micro Devices X86-64
* Version:************************** 0x1
* Entry point address:************** 0x400750
* Start of program headers:********* 64 (bytes into file)
* Start of section headers:********* 5312 (bytes into file)
* Flags:**************************** 0x0
* Size of this header:************** 64 (bytes)
* Size of program headers:********** 56 (bytes)
* Number of program headers:******** 9
* Size of section headers:********** 64 (bytes)
* Number of section headers:******** 37
* Section header string table index: 34

ldd ./a.out
******* linux-vdso.so.1 =>* (0x00007fff689ff000)
******* libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
******* libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
******* libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
******* libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
******* /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)

uname -a
Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
2009 x86_64 GNU/Linux
--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

read() wont allow me to read files larger than 2 gig (on a 64bit)
 

Dear Glide,

Does the stat() return the correct file size?
I remember that there is a 64bits version of stat(), stat64(), maybe
that's the problem?

Best regards,
AceLan Kao.

2009/12/25 glide creme <glidecreme@gmail.com>:
> Hi
> Sorry if I'm asking on the wrong mailinglist,
> but I've asked on various forums and I havn't found a solution.
>
> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
> the problem is not related to the allocation itself, which works.
> Attached is a sample program that illustrates this.
>
> I've tried with various, compiler flags like
> -D_FILE_OFFSET_BITS=64
>
> According to man 2 read, the maximum is limited by SSIZE_MAX
> which is defined in
>
> /usr/include/bits/posix1_lim.h
> # define SSIZE_MAX LONG_MAX
>
> And LONG_MAX is defined in /usr/include/limits.h as
> # if __WORDSIZE == 64
> # define LONG_MAX 9223372036854775807L
> # else
> # define LONG_MAX 2147483647L
> # endif
> # define LONG_MIN (-LONG_MAX - 1L)
>
> Is this the expected behaviour for the read function on a 64bit platform,
> or is something broken
>
> Thanks in advance
>
> edit:
>
> by the way
>
> readelf -h ./a.out
> ELF Header:
> * Magic:** 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
> * Class:**************************** ELF64
> * Data:***************************** 2's complement, little endian
> * Version:************************** 1 (current)
> * OS/ABI:*************************** UNIX - System V
> * ABI Version:********************** 0
> * Type:***************************** EXEC (Executable file)
> * Machine:************************** Advanced Micro Devices X86-64
> * Version:************************** 0x1
> * Entry point address:************** 0x400750
> * Start of program headers:********* 64 (bytes into file)
> * Start of section headers:********* 5312 (bytes into file)
> * Flags:**************************** 0x0
> * Size of this header:************** 64 (bytes)
> * Size of program headers:********** 56 (bytes)
> * Number of program headers:******** 9
> * Size of section headers:********** 64 (bytes)
> * Number of section headers:******** 37
> * Section header string table index: 34
>
> ldd ./a.out
> ******* linux-vdso.so.1 =>* (0x00007fff689ff000)
> ******* libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
> ******* libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
> ******* libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
> ******* libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
> ******* /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
>
> uname -a
> Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
> 2009 x86_64 GNU/Linux
>
> --
> kernel-team mailing list
> kernel-team@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team
>
>



--
Chia-Lin Kao(AceLan)
http://blog.acelan.idv.tw/
E-Mail: acelan.kaoATcanonical.com (s/AT/@/)

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

read() wont allow me to read files larger than 2 gig (on a 64bit)
 

Thanks for your reply

Yes stat, calculates the filesize correctly,

This is the output of the program

$ls -l bigfile.dat
-rw-r--r-- 1 tmp tmp 2163946253 2009-12-23 07:57 bigfile.dat
$ ./a.out
LONG_MAX:9223372036854775807
SSIZE_MAX:9223372036854775807
a.out: Read only 2147479552 of 2163946253 bytes: Succes

This was I find strange,
because the LONG_MAX is 64bit, and SSIZE_MAX should be defined to be LONG_MAX,
but it looks like read is only limited to 32bit.
-------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <err.h>
#include <fcntl.h>
#include <sysexits.h>
#include <unistd.h>
#include <sys/stat.h>
#include <limits.h>
#include <bits/posix1_lim.h>
#include <sys/types.h>
// get bytesize of file
size_t fsize(const char* fname){
struct stat st ;
stat(fname,&st);
return st.st_size;
}




int main() {
const char *infile = "bigfile.dat";
int fd;
size_t bytes_read, bytes_expected = fsize(infile);
char *data;
printf("LONG_MAX:%lu
",LONG_MAX);
printf("SSIZE_MAX:%lu
",SSIZE_MAX);

if ((fd = open(infile,O_RDONLY)) < 0)
err(EX_NOINPUT, "%s", infile);

if ((data =(char *) malloc(bytes_expected)) == NULL)
err(EX_OSERR, "data malloc");

bytes_read = read(fd, data, bytes_expected);

if (bytes_read != bytes_expected)
err(EX_DATAERR, "Read only %lu of %lu bytes",bytes_read, bytes_expected);

/* ... operate on data ... */

free(data);

exit(EX_OK);
}
-------------------------------------------------------------

On Fri, Dec 25, 2009 at 10:04 AM, AceLan Kao <acelan.kao@canonical.com> wrote:
> Dear Glide,
>
> Does the stat() return the correct file size?
> I remember that there is a 64bits version of stat(), stat64(), maybe
> that's the problem?
>
> Best regards,
> AceLan Kao.
>
> 2009/12/25 glide creme <glidecreme@gmail.com>:
>> Hi
>> Sorry if I'm asking on the wrong mailinglist,
>> but I've asked on various forums and I havn't found a solution.
>>
>> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
>> the problem is not related to the allocation itself, which works.
>> Attached is a sample program that illustrates this.
>>
>> I've tried with various, compiler flags like
>> -D_FILE_OFFSET_BITS=64
>>
>> According to man 2 read, the maximum is limited by SSIZE_MAX
>> which is defined in
>>
>> /usr/include/bits/posix1_lim.h
>> # define SSIZE_MAX LONG_MAX
>>
>> And LONG_MAX is defined in /usr/include/limits.h as
>> # if __WORDSIZE == 64
>> # define LONG_MAX 9223372036854775807L
>> # else
>> # define LONG_MAX 2147483647L
>> # endif
>> # define LONG_MIN (-LONG_MAX - 1L)
>>
>> Is this the expected behaviour for the read function on a 64bit platform,
>> or is something broken
>>
>> Thanks in advance
>>
>> edit:
>>
>> by the way
>>
>> readelf -h ./a.out
>> ELF Header:
>> * Magic:** 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>> * Class:**************************** ELF64
>> * Data:***************************** 2's complement, little endian
>> * Version:************************** 1 (current)
>> * OS/ABI:*************************** UNIX - System V
>> * ABI Version:********************** 0
>> * Type:***************************** EXEC (Executable file)
>> * Machine:************************** Advanced Micro Devices X86-64
>> * Version:************************** 0x1
>> * Entry point address:************** 0x400750
>> * Start of program headers:********* 64 (bytes into file)
>> * Start of section headers:********* 5312 (bytes into file)
>> * Flags:**************************** 0x0
>> * Size of this header:************** 64 (bytes)
>> * Size of program headers:********** 56 (bytes)
>> * Number of program headers:******** 9
>> * Size of section headers:********** 64 (bytes)
>> * Number of section headers:******** 37
>> * Section header string table index: 34
>>
>> ldd ./a.out
>> ******* linux-vdso.so.1 =>* (0x00007fff689ff000)
>> ******* libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
>> ******* libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
>> ******* libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
>> ******* libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
>> ******* /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
>>
>> uname -a
>> Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
>> 2009 x86_64 GNU/Linux
>>
>> --
>> kernel-team mailing list
>> kernel-team@lists.ubuntu.com
>> https://lists.ubuntu.com/mailman/listinfo/kernel-team
>>
>>
>
>
>
> --
> Chia-Lin Kao(AceLan)
> http://blog.acelan.idv.tw/
> E-Mail: acelan.kaoATcanonical.com (s/AT/@/)
>

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team