FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu Kernel Team

 
 
LinkBack Thread Tools
 
Old 12-25-2009, 09:29 AM
Stefan Bader
 
Default read() wont allow me to read files larger than 2 gig (on a 64bit)

glide creme wrote:
> Thanks for your reply
>
> Yes stat, calculates the filesize correctly,
>
> This is the output of the program
>
> $ls -l bigfile.dat
> -rw-r--r-- 1 tmp tmp 2163946253 2009-12-23 07:57 bigfile.dat
> $ ./a.out
> LONG_MAX:9223372036854775807
> SSIZE_MAX:9223372036854775807
> a.out: Read only 2147479552 of 2163946253 bytes: Succes
>
> This was I find strange,
> because the LONG_MAX is 64bit, and SSIZE_MAX should be defined to be LONG_MAX,
> but it looks like read is only limited to 32bit.
> -------------------------------------------------------------
> #include <stdio.h>
> #include <stdlib.h>
> #include <err.h>
> #include <fcntl.h>
> #include <sysexits.h>
> #include <unistd.h>
> #include <sys/stat.h>
> #include <limits.h>
> #include <bits/posix1_lim.h>
> #include <sys/types.h>
> // get bytesize of file
> size_t fsize(const char* fname){
> struct stat st ;
> stat(fname,&st);
> return st.st_size;
> }
>
>
>
>
> int main() {
> const char *infile = "bigfile.dat";
> int fd;
> size_t bytes_read, bytes_expected = fsize(infile);
> char *data;
> printf("LONG_MAX:%lu
",LONG_MAX);
> printf("SSIZE_MAX:%lu
",SSIZE_MAX);
>
> if ((fd = open(infile,O_RDONLY)) < 0)
> err(EX_NOINPUT, "%s", infile);
>
> if ((data =(char *) malloc(bytes_expected)) == NULL)
> err(EX_OSERR, "data malloc");
>
> bytes_read = read(fd, data, bytes_expected);
>
> if (bytes_read != bytes_expected)
> err(EX_DATAERR, "Read only %lu of %lu bytes",bytes_read, bytes_expected);
>
> /* ... operate on data ... */
>
> free(data);
>
> exit(EX_OK);
> }
> -------------------------------------------------------------
>
> On Fri, Dec 25, 2009 at 10:04 AM, AceLan Kao <acelan.kao@canonical.com> wrote:
>> Dear Glide,
>>
>> Does the stat() return the correct file size?
>> I remember that there is a 64bits version of stat(), stat64(), maybe
>> that's the problem?
>>
>> Best regards,
>> AceLan Kao.
>>
>> 2009/12/25 glide creme <glidecreme@gmail.com>:
>>> Hi
>>> Sorry if I'm asking on the wrong mailinglist,
>>> but I've asked on various forums and I havn't found a solution.
>>>
>>> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
>>> the problem is not related to the allocation itself, which works.
>>> Attached is a sample program that illustrates this.
>>>
>>> I've tried with various, compiler flags like
>>> -D_FILE_OFFSET_BITS=64
>>>
>>> According to man 2 read, the maximum is limited by SSIZE_MAX
>>> which is defined in
>>>
>>> /usr/include/bits/posix1_lim.h
>>> # define SSIZE_MAX LONG_MAX
>>>
>>> And LONG_MAX is defined in /usr/include/limits.h as
>>> # if __WORDSIZE == 64
>>> # define LONG_MAX 9223372036854775807L
>>> # else
>>> # define LONG_MAX 2147483647L
>>> # endif
>>> # define LONG_MIN (-LONG_MAX - 1L)
>>>
>>> Is this the expected behaviour for the read function on a 64bit platform,
>>> or is something broken
>>>
>>> Thanks in advance
>>>
>>> edit:
>>>
>>> by the way
>>>
>>> readelf -h ./a.out
>>> ELF Header:
>>> Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>>> Class: ELF64
>>> Data: 2's complement, little endian
>>> Version: 1 (current)
>>> OS/ABI: UNIX - System V
>>> ABI Version: 0
>>> Type: EXEC (Executable file)
>>> Machine: Advanced Micro Devices X86-64
>>> Version: 0x1
>>> Entry point address: 0x400750
>>> Start of program headers: 64 (bytes into file)
>>> Start of section headers: 5312 (bytes into file)
>>> Flags: 0x0
>>> Size of this header: 64 (bytes)
>>> Size of program headers: 56 (bytes)
>>> Number of program headers: 9
>>> Size of section headers: 64 (bytes)
>>> Number of section headers: 37
>>> Section header string table index: 34
>>>
>>> ldd ./a.out
>>> linux-vdso.so.1 => (0x00007fff689ff000)
>>> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
>>> libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
>>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
>>> libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
>>> /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
>>>
>>> uname -a
>>> Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
>>> 2009 x86_64 GNU/Linux
>>>
>>> --
>>> kernel-team mailing list
>>> kernel-team@lists.ubuntu.com
>>> https://lists.ubuntu.com/mailman/listinfo/kernel-team
>>>
>>>
>>
>>
>> --
>> Chia-Lin Kao(AceLan)
>> http://blog.acelan.idv.tw/
>> E-Mail: acelan.kaoATcanonical.com (s/AT/@/)
>>
>

Have you tried the following?

-Stefan

FEATURE_TEST_MACROS(7) Linux Programmer's Manual FEATURE_TEST_MACROS(7)

NAME
feature_test_macros - feature test macros

SYNOPSIS
#include <features.h>

_FILE_OFFSET_BITS
Defining this macro with the value 64 automatically converts
references to 32-bit functions and data types related to file
I/O and file system operations into references to their 64-bit
counterparts. This is useful for performing I/O on large files
(> 2 Gigabytes) on 32-bit systems. (Defining this macro permits
correctly written programs to use large files with only a recom‐
pilation being required.) 64-bit systems naturally permit file
sizes greater than 2 Gigabytes, and on those systems this macro
has no effect.


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 12-25-2009, 11:35 AM
glide creme
 
Default read() wont allow me to read files larger than 2 gig (on a 64bit)

Yes I tried that (as I also wrote in my original message).
But adding D_FILE_OFFSET shouldnt be necessary on 64bit system right?

thanks
On Fri, Dec 25, 2009 at 5:29 AM, Stefan Bader
<stefan.bader@canonical.com> wrote:
> glide creme wrote:
>> Thanks for your reply
>>
>> Yes stat, calculates the filesize correctly,
>>
>> This is the output of the program
>>
>> $ls -l bigfile.dat
>> -rw-r--r-- 1 tmp tmp 2163946253 2009-12-23 07:57 bigfile.dat
>> $ ./a.out
>> LONG_MAX:9223372036854775807
>> SSIZE_MAX:9223372036854775807
>> a.out: Read only 2147479552 of 2163946253 bytes: Succes
>>
>> This was I find strange,
>> because the LONG_MAX is 64bit, and SSIZE_MAX should be defined to be LONG_MAX,
>> but it looks like read is only limited to 32bit.
>> -------------------------------------------------------------
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <err.h>
>> #include <fcntl.h>
>> #include <sysexits.h>
>> #include <unistd.h>
>> #include <sys/stat.h>
>> #include <limits.h>
>> #include <bits/posix1_lim.h>
>> #include <sys/types.h>
>> // get bytesize of file
>> size_t fsize(const char* fname){
>> * struct stat st ;
>> * stat(fname,&st);
>> * return st.st_size;
>> }
>>
>>
>>
>>
>> int main() {
>> * const char *infile = "bigfile.dat";
>> * int fd;
>> * size_t bytes_read, bytes_expected = fsize(infile);
>> * char *data;
>> * printf("LONG_MAX:%lu
",LONG_MAX);
>> * printf("SSIZE_MAX:%lu
",SSIZE_MAX);
>>
>> * if ((fd = open(infile,O_RDONLY)) < 0)
>> * * err(EX_NOINPUT, "%s", infile);
>>
>> * if ((data =(char *) malloc(bytes_expected)) == NULL)
>> * * err(EX_OSERR, "data malloc");
>>
>> * bytes_read = read(fd, data, bytes_expected);
>>
>> * if (bytes_read != bytes_expected)
>> * * err(EX_DATAERR, "Read only %lu of %lu bytes",bytes_read, bytes_expected);
>>
>> * /* ... operate on data ... */
>>
>> * free(data);
>>
>> * exit(EX_OK);
>> }
>> -------------------------------------------------------------
>>
>> On Fri, Dec 25, 2009 at 10:04 AM, AceLan Kao <acelan.kao@canonical.com> wrote:
>>> Dear Glide,
>>>
>>> Does the stat() return the correct file size?
>>> I remember that there is a 64bits version of stat(), stat64(), maybe
>>> that's the problem?
>>>
>>> Best regards,
>>> AceLan Kao.
>>>
>>> 2009/12/25 glide creme <glidecreme@gmail.com>:
>>>> Hi
>>>> Sorry if I'm asking on the wrong mailinglist,
>>>> but I've asked on various forums and I havn't found a solution.
>>>>
>>>> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
>>>> the problem is not related to the allocation itself, which works.
>>>> Attached is a sample program that illustrates this.
>>>>
>>>> I've tried with various, compiler flags like
>>>> -D_FILE_OFFSET_BITS=64
>>>>
>>>> According to man 2 read, the maximum is limited by SSIZE_MAX
>>>> which is defined in
>>>>
>>>> /usr/include/bits/posix1_lim.h
>>>> # define SSIZE_MAX LONG_MAX
>>>>
>>>> And LONG_MAX is defined in /usr/include/limits.h as
>>>> # if __WORDSIZE == 64
>>>> # define LONG_MAX 9223372036854775807L
>>>> # else
>>>> # define LONG_MAX 2147483647L
>>>> # endif
>>>> # define LONG_MIN (-LONG_MAX - 1L)
>>>>
>>>> Is this the expected behaviour for the read function on a 64bit platform,
>>>> or is something broken
>>>>
>>>> Thanks in advance
>>>>
>>>> edit:
>>>>
>>>> by the way
>>>>
>>>> readelf -h ./a.out
>>>> ELF Header:
>>>> * Magic: * 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>>>> * Class: * * * * * * * * * * * * * * ELF64
>>>> * Data: * * * * * * * * * * * * * * *2's complement, little endian
>>>> * Version: * * * * * * * * * * * * * 1 (current)
>>>> * OS/ABI: * * * * * * * * * * * * * *UNIX - System V
>>>> * ABI Version: * * * * * * * * * * * 0
>>>> * Type: * * * * * * * * * * * * * * *EXEC (Executable file)
>>>> * Machine: * * * * * * * * * * * * * Advanced Micro Devices X86-64
>>>> * Version: * * * * * * * * * * * * * 0x1
>>>> * Entry point address: * * * * * * * 0x400750
>>>> * Start of program headers: * * * * *64 (bytes into file)
>>>> * Start of section headers: * * * * *5312 (bytes into file)
>>>> * Flags: * * * * * * * * * * * * * * 0x0
>>>> * Size of this header: * * * * * * * 64 (bytes)
>>>> * Size of program headers: * * * * * 56 (bytes)
>>>> * Number of program headers: * * * * 9
>>>> * Size of section headers: * * * * * 64 (bytes)
>>>> * Number of section headers: * * * * 37
>>>> * Section header string table index: 34
>>>>
>>>> ldd ./a.out
>>>> * * * * linux-vdso.so.1 => *(0x00007fff689ff000)
>>>> * * * * libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
>>>> * * * * libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
>>>> * * * * libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
>>>> * * * * libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
>>>> * * * * /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
>>>>
>>>> uname -a
>>>> Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
>>>> 2009 x86_64 GNU/Linux
>>>>
>>>> --
>>>> kernel-team mailing list
>>>> kernel-team@lists.ubuntu.com
>>>> https://lists.ubuntu.com/mailman/listinfo/kernel-team
>>>>
>>>>
>>>
>>>
>>> --
>>> Chia-Lin Kao(AceLan)
>>> http://blog.acelan.idv.tw/
>>> E-Mail: acelan.kaoATcanonical.com (s/AT/@/)
>>>
>>
>
> Have you tried the following?
>
> -Stefan
>
> FEATURE_TEST_MACROS(7) * * Linux Programmer's Manual * *FEATURE_TEST_MACROS(7)
>
> NAME
> * * * feature_test_macros - feature test macros
>
> SYNOPSIS
> * * * #include <features.h>
>
> * * * _FILE_OFFSET_BITS
> * * * * * * *Defining this macro with the *value *64 *automatically *converts
> * * * * * * *references *to *32-bit *functions and data types related to file
> * * * * * * *I/O and file system operations into references to *their *64-bit
> * * * * * * *counterparts. * This is useful for performing I/O on large files
> * * * * * * *(> 2 Gigabytes) on 32-bit systems. *(Defining this macro permits
> * * * * * * *correctly written programs to use large files with only a recom‐
> * * * * * * *pilation being required.) *64-bit systems naturally permit *file
> * * * * * * *sizes *greater than 2 Gigabytes, and on those systems this macro
> * * * * * * *has no effect.
>
>

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 12-25-2009, 01:02 PM
Bastian Blank
 
Default read() wont allow me to read files larger than 2 gig (on a 64bit)

On Fri, Dec 25, 2009 at 09:29:35AM +0100, glide creme wrote:
> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
> the problem is not related to the allocation itself, which works.
> Attached is a sample program that illustrates this.

This is a misinterpretation of the spec by you. Please reread the
manpage of read(2)[1]. The second parameter of read defines the maximum
amount that it may read. It is no bug if the actual read returned less
than this amount.

Bastian

[1]: man 2 read
--
Suffocating together ... would create heroic camaraderie.
-- Khan Noonian Singh, "Space Seed", stardate 3142.8

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 12-27-2009, 08:32 PM
Ben Hutchings
 
Default read() wont allow me to read files larger than 2 gig (on a 64bit)

On Fri, 2009-12-25 at 09:29 +0100, glide creme wrote:
> Hi
> Sorry if I'm asking on the wrong mailinglist,
> but I've asked on various forums and I havn't found a solution.
>
> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
> the problem is not related to the allocation itself, which works.
> Attached is a sample program that illustrates this.
>
> I've tried with various, compiler flags like
> -D_FILE_OFFSET_BITS=64
[...]

Since you are building a 64-bit program, you don't need to do anything
special. If you were building a 32-bit program, you would need to use
-D_FILE_OFFSET_BITS=64.

I think your mistake is to assume that read() only returns a smaller
length than you requested if it encounters an error or end-of-file. In
fact it can return a smaller length for any reason (though you will
rarely see this behaviour for regular files). Here's a wrapper I have
used to provide the behaviour you want:

static ssize_t read_retry(int fd, void * buf, size_t count)
{
ssize_t chunk, total = 0;

do
{
chunk = read(fd, buf, count);
if (chunk <= 0)
{
if (total == 0)
return chunk;
break;
}
total += chunk;
buf = (char *)buf + chunk;
count -= chunk;
}
while (count);

return total;
}

Ben.

--
Ben Hutchings
Always try to do things in chronological order;
it's less confusing that way.
--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 12-28-2009, 09:19 AM
glide creme
 
Default read() wont allow me to read files larger than 2 gig (on a 64bit)

Thanks for all your replies,
but it seems that its not possible to use posix read to read file
chunks larger than 2.1 gig.
This is according to the linus
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e28cc715

At somepoint I guess it was possible but the kernel has regressed to
only allow smaller chunks being read.

kernelsourcetree/fs/read_write.c
---------------------------------------------------------
/*
* rw_verify_area doesn't like huge counts. We limit
* them to something that fits in "int" so that others
* won't have to do range checks all the time.
*/
#define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)
int rw_verify_area(int read_write, struct file *file, loff_t *ppos,
size_t count);
---------------------------------------------------------
This was rather annoying, it would have been nice if this had been
documented outside of the kernelsource and kernel git.

I'm aware that the returnvalue from the 'read' function can be smaller
than the number of bytes expected.

But I think it should be clarified in the documentation, that the
function can't return a value larger than 2147479552.

Instead of the strange " If count is greater than SSIZE_MAX, the
result is unspecified.", which implies that a value larger than SIGNED
and smaller than
SSIZE_MAX is actually supported.

It would be sensible to write that the posix read is still limited to
a 32bit value.

This is the sort of not incorrect documentation, that is misleading
and makes the transition to 64bit worse than it has to be.

I don't think many are aware of this 32bit limitation on a 64bit platform.

Thanks again for your replies
On Fri, Dec 25, 2009 at 9:29 AM, glide creme <glidecreme@gmail.com> wrote:
> Hi
> Sorry if I'm asking on the wrong mailinglist,
> but I've asked on various forums and I havn't found a solution.
>
> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
> the problem is not related to the allocation itself, which works.
> Attached is a sample program that illustrates this.
>
> I've tried with various, compiler flags like
> -D_FILE_OFFSET_BITS=64
>
> According to man 2 read, the maximum is limited by SSIZE_MAX
> which is defined in
>
> /usr/include/bits/posix1_lim.h
> # define SSIZE_MAX LONG_MAX
>
> And LONG_MAX is defined in /usr/include/limits.h as
> # if __WORDSIZE == 64
> # define LONG_MAX 9223372036854775807L
> # else
> # define LONG_MAX 2147483647L
> # endif
> # define LONG_MIN (-LONG_MAX - 1L)
>
> Is this the expected behaviour for the read function on a 64bit platform,
> or is something broken
>
> Thanks in advance
>
> edit:
>
> by the way
>
> readelf -h ./a.out
> ELF Header:
> * Magic:** 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
> * Class:**************************** ELF64
> * Data:***************************** 2's complement, little endian
> * Version:************************** 1 (current)
> * OS/ABI:*************************** UNIX - System V
> * ABI Version:********************** 0
> * Type:***************************** EXEC (Executable file)
> * Machine:************************** Advanced Micro Devices X86-64
> * Version:************************** 0x1
> * Entry point address:************** 0x400750
> * Start of program headers:********* 64 (bytes into file)
> * Start of section headers:********* 5312 (bytes into file)
> * Flags:**************************** 0x0
> * Size of this header:************** 64 (bytes)
> * Size of program headers:********** 56 (bytes)
> * Number of program headers:******** 9
> * Size of section headers:********** 64 (bytes)
> * Number of section headers:******** 37
> * Section header string table index: 34
>
> ldd ./a.out
> ******* linux-vdso.so.1 =>* (0x00007fff689ff000)
> ******* libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
> ******* libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
> ******* libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
> ******* libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
> ******* /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
>
> uname -a
> Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
> 2009 x86_64 GNU/Linux
>

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 12-30-2009, 08:39 AM
Andy Whitcroft
 
Default read() wont allow me to read files larger than 2 gig (on a 64bit)

On Mon, Dec 28, 2009 at 11:19:08AM +0100, glide creme wrote:
> Thanks for all your replies,
> but it seems that its not possible to use posix read to read file
> chunks larger than 2.1 gig.
> This is according to the linus
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e28cc715
>
> At somepoint I guess it was possible but the kernel has regressed to
> only allow smaller chunks being read.
>
> kernelsourcetree/fs/read_write.c
> ---------------------------------------------------------
> /*
> * rw_verify_area doesn't like huge counts. We limit
> * them to something that fits in "int" so that others
> * won't have to do range checks all the time.
> */
> #define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)
> int rw_verify_area(int read_write, struct file *file, loff_t *ppos,
> size_t count);
> ---------------------------------------------------------
> This was rather annoying, it would have been nice if this had been
> documented outside of the kernelsource and kernel git.
>
> I'm aware that the returnvalue from the 'read' function can be smaller
> than the number of bytes expected.
>
> But I think it should be clarified in the documentation, that the
> function can't return a value larger than 2147479552.

The problem with that is that the standard says those values are valid.
The standard says that read will return between 0 and the number of bytes
you asked for. That means that in the future it could return more than
2GB, to document that it could not would allow you to make assumptions
about the return value which may not be correct in the future. Bad.

-apw

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 

Thread Tools




All times are GMT. The time now is 04:02 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org