string: introduce memweight
 

memweight() is the function that counts the total number of bits set
in memory area. The memory area doesn't need to be aligned to
long-word boundary unlike bitmap_weight().

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Anders Larsen <al@alarsen.net>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: dm-devel@redhat.com
Cc: linux-fsdevel@vger.kernel.org
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: linux-media@vger.kernel.org
Cc: Mark Fasheh <mfasheh@suse.com>
Cc: Joel Becker <jlbec@evilplan.org>
Cc: ocfs2-devel@oss.oracle.com
Cc: Jan Kara <jack@suse.cz>
Cc: linux-ext4@vger.kernel.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andreas Dilger <adilger.kernel@dilger.ca>
Cc: "Theodore Ts'o" <tytso@mit.edu>
---
include/linux/string.h | 3 +++
lib/string.c | 37 +++++++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+), 0 deletions(-)

diff --git a/include/linux/string.h b/include/linux/string.h
index e033564..ffe0442 100644
--- a/include/linux/string.h
+++ b/include/linux/string.h
@@ -145,4 +145,7 @@ static inline bool strstarts(const char *str, const char *prefix)
return strncmp(str, prefix, strlen(prefix)) == 0;
}
#endif
+
+extern size_t memweight(const void *ptr, size_t bytes);
+
#endif /* _LINUX_STRING_H_ */
diff --git a/lib/string.c b/lib/string.c
index e5878de..c8b92a0 100644
--- a/lib/string.c
+++ b/lib/string.c
@@ -26,6 +26,7 @@
#include <linux/export.h>
#include <linux/bug.h>
#include <linux/errno.h>
+#include <linux/bitmap.h>

#ifndef __HAVE_ARCH_STRNICMP
/**
@@ -824,3 +825,39 @@ void *memchr_inv(const void *start, int c, size_t bytes)
return check_bytes8(start, value, bytes % 8);
}
EXPORT_SYMBOL(memchr_inv);
+
+/**
+ * memweight - count the total number of bits set in memory area
+ * @ptr: pointer to the start of the area
+ * @bytes: the size of the area
+ */
+size_t memweight(const void *ptr, size_t bytes)
+{
+ size_t w = 0;
+ size_t longs;
+ union {
+ const void *ptr;
+ const unsigned char *b;
+ unsigned long address;
+ } bitmap;
+
+ for (bitmap.ptr = ptr; bytes > 0 && bitmap.address % sizeof(long);
+ bytes--, bitmap.address++)
+ w += hweight8(*bitmap.b);
+
+ for (longs = bytes / sizeof(long); longs > 0; ) {
+ size_t bits = min_t(size_t, INT_MAX & ~(BITS_PER_LONG - 1),
+ longs * BITS_PER_LONG);
+
+ w += bitmap_weight(bitmap.ptr, bits);
+ bytes -= bits / BITS_PER_BYTE;
+ bitmap.address += bits / BITS_PER_BYTE;
+ longs -= bits / BITS_PER_LONG;
+ }
+
+ for (; bytes > 0; bytes--, bitmap.address++)
+ w += hweight8(*bitmap.b);
+
+ return w;
+}
+EXPORT_SYMBOL(memweight);
--
1.7.7.6

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

string: introduce memweight
 

On Sun 20-05-12 22:23:14, Akinobu Mita wrote:
> memweight() is the function that counts the total number of bits set
> in memory area. The memory area doesn't need to be aligned to
> long-word boundary unlike bitmap_weight().
Thanks for the patch. I have some comments below.

> Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
> Cc: Anders Larsen <al@alarsen.net>
> Cc: Alasdair Kergon <agk@redhat.com>
> Cc: dm-devel@redhat.com
> Cc: linux-fsdevel@vger.kernel.org
> Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Cc: linux-media@vger.kernel.org
> Cc: Mark Fasheh <mfasheh@suse.com>
> Cc: Joel Becker <jlbec@evilplan.org>
> Cc: ocfs2-devel@oss.oracle.com
> Cc: Jan Kara <jack@suse.cz>
> Cc: linux-ext4@vger.kernel.org
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Andreas Dilger <adilger.kernel@dilger.ca>
> Cc: "Theodore Ts'o" <tytso@mit.edu>
> ---
> include/linux/string.h | 3 +++
> lib/string.c | 37 +++++++++++++++++++++++++++++++++++++
> 2 files changed, 40 insertions(+), 0 deletions(-)
>
> diff --git a/include/linux/string.h b/include/linux/string.h
> index e033564..ffe0442 100644
> --- a/include/linux/string.h
> +++ b/include/linux/string.h
> @@ -145,4 +145,7 @@ static inline bool strstarts(const char *str, const char *prefix)
> return strncmp(str, prefix, strlen(prefix)) == 0;
> }
> #endif
> +
> +extern size_t memweight(const void *ptr, size_t bytes);
> +
> #endif /* _LINUX_STRING_H_ */
> diff --git a/lib/string.c b/lib/string.c
> index e5878de..c8b92a0 100644
> --- a/lib/string.c
> +++ b/lib/string.c
> @@ -26,6 +26,7 @@
> #include <linux/export.h>
> #include <linux/bug.h>
> #include <linux/errno.h>
> +#include <linux/bitmap.h>
>
> #ifndef __HAVE_ARCH_STRNICMP
> /**
> @@ -824,3 +825,39 @@ void *memchr_inv(const void *start, int c, size_t bytes)
> return check_bytes8(start, value, bytes % 8);
> }
> EXPORT_SYMBOL(memchr_inv);
> +
> +/**
> + * memweight - count the total number of bits set in memory area
> + * @ptr: pointer to the start of the area
> + * @bytes: the size of the area
> + */
> +size_t memweight(const void *ptr, size_t bytes)
> +{
> + size_t w = 0;
> + size_t longs;
> + union {
> + const void *ptr;
> + const unsigned char *b;
> + unsigned long address;
> + } bitmap;
Ugh, this is ugly and mostly unnecessary. Just use "const unsigned char
*bitmap".

> +
> + for (bitmap.ptr = ptr; bytes > 0 && bitmap.address % sizeof(long);
> + bytes--, bitmap.address++)
> + w += hweight8(*bitmap.b);
This can be:
count = ((unsigned long)bitmap) % sizeof(long);
while (count--) {
w += hweight(*bitmap);
bitmap++;
bytes--;
}
> +
> + for (longs = bytes / sizeof(long); longs > 0; ) {
> + size_t bits = min_t(size_t, INT_MAX & ~(BITS_PER_LONG - 1),
> + longs * BITS_PER_LONG);
I find it highly unlikely that someone would have such a large bitmap
(256 MB or more on 32-bit). Also the condition as you wrote it can just
overflow so it won't have the desired effect. Just do
BUG_ON(longs >= ULONG_MAX / BITS_PER_LONG);
and remove the loop completely. If someone comes with such a huge bitmap,
the code can be modified easily (after really closely inspecting whether
such a huge bitmap is really well justified).

> +
> + w += bitmap_weight(bitmap.ptr, bits);
> + bytes -= bits / BITS_PER_BYTE;
> + bitmap.address += bits / BITS_PER_BYTE;
> + longs -= bits / BITS_PER_LONG;
> + }
> +
> + for (; bytes > 0; bytes--, bitmap.address++)
> + w += hweight8(*bitmap.b);
> +
> + return w;
> +}
> +EXPORT_SYMBOL(memweight);

Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

string: introduce memweight
 

2012/5/23 Jan Kara <jack@suse.cz>:
> On Sun 20-05-12 22:23:14, Akinobu Mita wrote:
>> memweight() is the function that counts the total number of bits set
>> in memory area. *The memory area doesn't need to be aligned to
>> long-word boundary unlike bitmap_weight().
> *Thanks for the patch. I have some comments below.

Thanks for the review.

>> @@ -824,3 +825,39 @@ void *memchr_inv(const void *start, int c, size_t bytes)
>> * * * return check_bytes8(start, value, bytes % 8);
>> *}
>> *EXPORT_SYMBOL(memchr_inv);
>> +
>> +/**
>> + * memweight - count the total number of bits set in memory area
>> + * @ptr: pointer to the start of the area
>> + * @bytes: the size of the area
>> + */
>> +size_t memweight(const void *ptr, size_t bytes)
>> +{
>> + * * size_t w = 0;
>> + * * size_t longs;
>> + * * union {
>> + * * * * * * const void *ptr;
>> + * * * * * * const unsigned char *b;
>> + * * * * * * unsigned long address;
>> + * * } bitmap;
> *Ugh, this is ugly and mostly unnecessary. Just use "const unsigned char
> *bitmap".
>
>> +
>> + * * for (bitmap.ptr = ptr; bytes > 0 && bitmap.address % sizeof(long);
>> + * * * * * * * * * * bytes--, bitmap.address++)
>> + * * * * * * w += hweight8(*bitmap.b);
> *This can be:
> * * * *count = ((unsigned long)bitmap) % sizeof(long);

The count should be the size of unaligned area and it can be greater than
bytes. So

count = min(bytes,
sizeof(long) - ((unsigned long)bitmap) % sizeof(long));

> * * * *while (count--) {
> * * * * * * * *w += hweight(*bitmap);
> * * * * * * * *bitmap++;
> * * * * * * * *bytes--;
> * * * *}
>> +
>> + * * for (longs = bytes / sizeof(long); longs > 0; ) {
>> + * * * * * * size_t bits = min_t(size_t, INT_MAX & ~(BITS_PER_LONG - 1),
>> + * * * * * * * * * * * * * * * * * * longs * BITS_PER_LONG);
> *I find it highly unlikely that someone would have such a large bitmap
> (256 MB or more on 32-bit). Also the condition as you wrote it can just
> overflow so it won't have the desired effect. Just do
> * * * *BUG_ON(longs >= ULONG_MAX / BITS_PER_LONG);

The bits argument of bitmap_weight() is int type. So this should be

BUG_ON(longs >= INT_MAX / BITS_PER_LONG);

> and remove the loop completely. If someone comes with such a huge bitmap,
> the code can be modified easily (after really closely inspecting whether
> such a huge bitmap is really well justified).

size_t memweight(const void *ptr, size_t bytes)
{
size_t w = 0;
size_t longs;
const unsigned char *bitmap = ptr;

for (; bytes > 0 && ((unsigned long)bitmap) % sizeof(long);
bytes--, bitmap++)
w += hweight8(*bitmap);

longs = bytes / sizeof(long);
BUG_ON(longs >= INT_MAX / BITS_PER_LONG);
w += bitmap_weight((unsigned long *)bitmap, longs * BITS_PER_LONG);
bytes -= longs * sizeof(long);
bitmap += longs * sizeof(long);

for (; bytes > 0; bytes--, bitmap++)
w += hweight8(*bitmap);

return w;
}

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

string: introduce memweight
 

On Wed 23-05-12 21:12:18, Akinobu Mita wrote:
> 2012/5/23 Jan Kara <jack@suse.cz>:
> > On Sun 20-05-12 22:23:14, Akinobu Mita wrote:
> >> memweight() is the function that counts the total number of bits set
> >> in memory area. *The memory area doesn't need to be aligned to
> >> long-word boundary unlike bitmap_weight().
> > *Thanks for the patch. I have some comments below.
>
> Thanks for the review.
>
> >> @@ -824,3 +825,39 @@ void *memchr_inv(const void *start, int c, size_t bytes)
> >> * * * return check_bytes8(start, value, bytes % 8);
> >> *}
> >> *EXPORT_SYMBOL(memchr_inv);
> >> +
> >> +/**
> >> + * memweight - count the total number of bits set in memory area
> >> + * @ptr: pointer to the start of the area
> >> + * @bytes: the size of the area
> >> + */
> >> +size_t memweight(const void *ptr, size_t bytes)
> >> +{
> >> + * * size_t w = 0;
> >> + * * size_t longs;
> >> + * * union {
> >> + * * * * * * const void *ptr;
> >> + * * * * * * const unsigned char *b;
> >> + * * * * * * unsigned long address;
> >> + * * } bitmap;
> > *Ugh, this is ugly and mostly unnecessary. Just use "const unsigned char
> > *bitmap".
> >
> >> +
> >> + * * for (bitmap.ptr = ptr; bytes > 0 && bitmap.address % sizeof(long);
> >> + * * * * * * * * * * bytes--, bitmap.address++)
> >> + * * * * * * w += hweight8(*bitmap.b);
> > *This can be:
> > * * * *count = ((unsigned long)bitmap) % sizeof(long);
>
> The count should be the size of unaligned area and it can be greater than
> bytes. So
>
> count = min(bytes,
> sizeof(long) - ((unsigned long)bitmap) % sizeof(long));
You are right, I didn't quite think this through.

> > * * * *while (count--) {
> > * * * * * * * *w += hweight(*bitmap);
> > * * * * * * * *bitmap++;
> > * * * * * * * *bytes--;
> > * * * *}
> >> +
> >> + * * for (longs = bytes / sizeof(long); longs > 0; ) {
> >> + * * * * * * size_t bits = min_t(size_t, INT_MAX & ~(BITS_PER_LONG - 1),
> >> + * * * * * * * * * * * * * * * * * * longs * BITS_PER_LONG);
> > *I find it highly unlikely that someone would have such a large bitmap
> > (256 MB or more on 32-bit). Also the condition as you wrote it can just
> > overflow so it won't have the desired effect. Just do
> > * * * *BUG_ON(longs >= ULONG_MAX / BITS_PER_LONG);
>
> The bits argument of bitmap_weight() is int type. So this should be
>
> BUG_ON(longs >= INT_MAX / BITS_PER_LONG);
OK, I didn't check and thought it's size_t.

> > and remove the loop completely. If someone comes with such a huge bitmap,
> > the code can be modified easily (after really closely inspecting whether
> > such a huge bitmap is really well justified).
>
> size_t memweight(const void *ptr, size_t bytes)
> {
> size_t w = 0;
> size_t longs;
> const unsigned char *bitmap = ptr;
>
> for (; bytes > 0 && ((unsigned long)bitmap) % sizeof(long);
> bytes--, bitmap++)
> w += hweight8(*bitmap);
>
> longs = bytes / sizeof(long);
> BUG_ON(longs >= INT_MAX / BITS_PER_LONG);
> w += bitmap_weight((unsigned long *)bitmap, longs * BITS_PER_LONG);
> bytes -= longs * sizeof(long);
> bitmap += longs * sizeof(long);
>
> for (; bytes > 0; bytes--, bitmap++)
> w += hweight8(*bitmap);
>
> return w;
> }
Yup, this looks much more readable. Thanks!

Honza

--
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

string: introduce memweight
 

On Wed, May 23, 2012 at 09:12:18PM +0900, Akinobu Mita wrote:
> size_t memweight(const void *ptr, size_t bytes)

Why should this return size_t instead of unsigned long?

> {
> size_t w = 0;
> size_t longs;
> const unsigned char *bitmap = ptr;
>
> for (; bytes > 0 && ((unsigned long)bitmap) % sizeof(long);
> bytes--, bitmap++)
> w += hweight8(*bitmap);
>
> longs = bytes / sizeof(long);
> BUG_ON(longs >= INT_MAX / BITS_PER_LONG);
> w += bitmap_weight((unsigned long *)bitmap, longs * BITS_PER_LONG);
> bytes -= longs * sizeof(long);
> bitmap += longs * sizeof(long);
>
> for (; bytes > 0; bytes--, bitmap++)
> w += hweight8(*bitmap);
>
> return w;
> }

bitmap_weight copes with a bitmask that isn't a multiple of BITS_PER_LONG
in size already. So I think this can be done as:

unsigned long memweight(const void *s, size_t n)
{
const unsigned char *ptr = s;
unsigned long r = 0;

while (n > 0 && (unsigned long)ptr % sizeof(long)) {
r += hweight8(*ptr);
n--;
ptr++;
}

BUG_ON(n >= INT_MAX / 8)

return r + bitmap_weight((unsigned long *)ptr, n * 8);
}

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

string: introduce memweight
 

2012/5/23 Matthew Wilcox <matthew@wil.cx>:
> On Wed, May 23, 2012 at 09:12:18PM +0900, Akinobu Mita wrote:
>> size_t memweight(const void *ptr, size_t bytes)
>
> Why should this return size_t instead of unsigned long?

I just use the same type as the bytes argument without mature
consideration. If unsigned long is better than size_t, I'll
change the return type.

>> {
>> * * * size_t w = 0;
>> * * * size_t longs;
>> * * * const unsigned char *bitmap = ptr;
>>
>> * * * for (; bytes > 0 && ((unsigned long)bitmap) % sizeof(long);
>> * * * * * * * * * * * bytes--, bitmap++)
>> * * * * * * * w += hweight8(*bitmap);
>>
>> * * * longs = bytes / sizeof(long);
>> * * * BUG_ON(longs >= INT_MAX / BITS_PER_LONG);
>> * * * w += bitmap_weight((unsigned long *)bitmap, longs * BITS_PER_LONG);
>> * * * bytes -= longs * sizeof(long);
>> * * * bitmap += longs * sizeof(long);
>>
>> * * * for (; bytes > 0; bytes--, bitmap++)
>> * * * * * * * w += hweight8(*bitmap);
>>
>> * * * return w;
>> }
>
> bitmap_weight copes with a bitmask that isn't a multiple of BITS_PER_LONG
> in size already. *So I think this can be done as:
>
> unsigned long memweight(const void *s, size_t n)
> {
> * * * *const unsigned char *ptr = s;
> * * * *unsigned long r = 0;
>
> * * * *while (n > 0 && (unsigned long)ptr % sizeof(long)) {
> * * * * * * * *r += hweight8(*ptr);
> * * * * * * * *n--;
> * * * * * * * *ptr++;
> * * * *}
>
> * * * *BUG_ON(n >= INT_MAX / 8)
>
> * * * *return r + bitmap_weight((unsigned long *)ptr, n * 8);
> }

This works perfectly on little-endian machines. But it doesn't work
on big-endian machines, if the bottom edge of memory area is not
aligned on long word boundary.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

string: introduce memweight
 

On Fri, Jun 8, 2012 at 5:50 PM, Akinobu Mita <akinobu.mita@gmail.com> wrote:
> *lib/string.c * * * * * | * 36 ++++++++++++++++++++++++++++++++++++

Is lib/string.c the right place for this? I get a build error on the
ia64 sim_defconfig:

LD arch/ia64/hp/sim/boot/bootloader

It fails because it pulls in lib/lib.a(string.o) to get some
innocuous function like strcpy() ... but it also gets
given memweight() which relies on __bitmap_weight()
which it doesn't have, because it doesn't include lib/built-in.o
(which is where bitmap.o, the definer of __bitmap_weight(), has
been linked).

Moving memweight() to lib/bitmap.c fixes the problem. But it
isn't really clear that it belongs there either. Perhaps it should
be its own file lib/memweight.c that gets included in lib/lib.a?

-Tony

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

string: introduce memweight
 

2012/6/21 Tony Luck <tony.luck@gmail.com>:
> On Fri, Jun 8, 2012 at 5:50 PM, Akinobu Mita <akinobu.mita@gmail.com> wrote:
>> *lib/string.c * * * * * | * 36 ++++++++++++++++++++++++++++++++++++
>
> Is lib/string.c the right place for this? *I get a build error on the
> ia64 sim_defconfig:
>
> *LD * * *arch/ia64/hp/sim/boot/bootloader
>
> It fails because it pulls in lib/lib.a(string.o) to get some
> innocuous function like strcpy() ... but it also gets
> given memweight() which relies on __bitmap_weight()
> which it doesn't have, because it doesn't include lib/built-in.o
> (which is where bitmap.o, the definer of __bitmap_weight(), has
> been linked).
>
> Moving memweight() to lib/bitmap.c fixes the problem. But it
> isn't really clear that it belongs there either. *Perhaps it should
> be its own file lib/memweight.c that gets included in lib/lib.a?

I'll fix it by making lib/memweight.c as you suggested.
Thanks for your report and suggestion.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel