Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Build System (http://www.linux-archive.org/fedora-build-system/)
-   -   Mock 'Directory not empty' error during --clean (http://www.linux-archive.org/fedora-build-system/498140-mock-directory-not-empty-error-during-clean.html)

"Greg Trahair" 03-07-2011 08:17 AM

Mock 'Directory not empty' error during --clean
 
Hi,

I've recently setup a build machine for staging builds to go into our
in-house koji. I installed mock and setup the build scratch space in
/data/mock (perms set to 2775, root:mock).

I setup an alias for all users for mock so that everybody gets there own
buildroot:

alias mock='mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME'

Creating a buildroot is fine:

$ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --init
INFO: mock.py version 1.0.7 starting...
State Changed: init plugins
State Changed: start
State Changed: lock buildroot
State Changed: clean
State Changed: init
State Changed: lock buildroot
Mock Version: 1.0.7
INFO: Mock Version: 1.0.7
INFO: enabled root cache
State Changed: unpacking root cache
INFO: enabled yum cache
State Changed: cleaning yum metadata
INFO: enabled ccache
State Changed: running yum


But when I attempt to clean it, this happens:

$ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
INFO: mock.py version 1.0.7 starting...
State Changed: init plugins
State Changed: start
State Changed: lock buildroot
State Changed: clean
Traceback (most recent call last):
File "/usr/sbin/mock", line 693, in ?
main(retParams)
File "/usr/sbin/mock", line 567, in main
chroot.clean()
File "<peak.util.decorators.rewrap wrapping mock.backend.clean at
0x0A926848>", line 3, in clean
File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line
70, in trace
result = func(*args, **kw)
File "/usr/lib/python2.4/site-packages/mock/backend.py", line 134, in
clean
mock.util.rmtree(self.basedir)
File "<peak.util.decorators.rewrap wrapping mock.util.rmtree at
0x0A91A2A8>", line 3, in rmtree
File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line
70, in trace
result = func(*args, **kw)
File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in
rmtree
shutil.rmtree(path, *args, **kargs)
File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/data/mock/dist-rhel5-greg'


My mock config looks like this for dist-rhel5-x86_64:
config_opts['chroothome'] = '/builddir'
config_opts['use_host_resolv'] = False
config_opts['basedir'] = '/data/mock'
config_opts['rpmbuild_timeout'] = 86400
config_opts['yum.conf'] =
'[main]
cachedir=/var/cache/yum
debuglevel=1
logfile=/var/log/yum.log

reposdir=/dev/null
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1


#
repos

[build]
name=build
baseurl=http://my-koji-server/repos/dist-r
hel5-build/latest/$basearch
'
config_opts['chroot_setup_cmd'] = 'groupinstall build'
config_opts['target_arch'] = 'x86_64'
config_opts['root'] = 'dist-rhel5'

Has anyone else encountered/fixed this?


Thanks in advance,
Greg


This e-mail message contains information which is confidential and may be privileged. It is intended for use by the addressee only. If you are not the intended addressee, we request that you notify the sender immediately and delete or destroy this e-mail message and any attachment(s), without copying, saving, forwarding, disclosing or using its contents in any other way. TomTom N.V., TomTom International BV or any other company belonging to the TomTom group of companies will not be liable for damage relating to the communication by e-mail of data, documents or any other information.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

Mike McLean 03-07-2011 05:54 PM

Mock 'Directory not empty' error during --clean
 
On 03/07/2011 04:17 AM, Greg Trahair wrote:
> But when I attempt to clean it, this happens:
>
> $ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
...
> File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in
> rmtree
> shutil.rmtree(path, *args, **kargs)
> File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree
> onerror(os.rmdir, path, sys.exc_info())
> File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree
> os.rmdir(path)
> OSError: [Errno 39] Directory not empty: '/data/mock/dist-rhel5-greg'

It's hard to know without more data. The error is happening within
os.rmtree.

Have a look at the remains that were not deleted. Is there anything
unusual about them? Is it possible another process wrote files to
directory during the clean operation? Does the problem repeat?
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

"Greg Trahair" 03-08-2011 07:45 AM

Mock 'Directory not empty' error during --clean
 
Hi Mike,

The directory is completely empty, and mock --init and mock --clean give
the same error. Even a mock --no-clean --init gives the error.

The strange thing is that I can (as my unprivileged user) just run rmdir
/data/mock/dist-rhel5-greg with no problems, so permissions are not the
culprit. SELinux is disabled.

I can recreate the problem every time I run mock --init, I have to rmdir
the buildroot before I can continue testing builds.

After a little testing, it seems that /data/mock being NFS is an issue,
as I just changed the buildroot back to /var/lib/mock and all seems ok.
Both /data/mock and /var/lib/mock are set 2775 root:mock.

I can't find any limitations of os.rmtree an nfs.

Regards,
Greg


-----Original Message-----
From: buildsys-bounces@lists.fedoraproject.org
[mailto:buildsys-bounces@lists.fedoraproject.org] On Behalf Of Mike
McLean
Sent: Monday, March 07, 2011 7:55 PM
To: buildsys@lists.fedoraproject.org
Subject: Re: Mock 'Directory not empty' error during --clean

On 03/07/2011 04:17 AM, Greg Trahair wrote:
> But when I attempt to clean it, this happens:
>
> $ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
...
> File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in
> rmtree
> shutil.rmtree(path, *args, **kargs)
> File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree
> onerror(os.rmdir, path, sys.exc_info())
> File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree
> os.rmdir(path)
> OSError: [Errno 39] Directory not empty: '/data/mock/dist-rhel5-greg'

It's hard to know without more data. The error is happening within
os.rmtree.

Have a look at the remains that were not deleted. Is there anything
unusual about them? Is it possible another process wrote files to
directory during the clean operation? Does the problem repeat?
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

Clark Williams 03-08-2011 02:55 PM

Mock 'Directory not empty' error during --clean
 
Ah, you're being bitten by root-squash. This means that any thread
running as root gets mapped to the user nobody on the NFS server and in
general gets denied access to regular user files.

Do you have the ability to control how the NFS partition is exported or
is it controlled by your IT group? If you have control over the NFS
server you can export it with no_root_squash and things should work
properly.

I don't have a good solution for running mock on a root-squashed NFS
partition at the moment.

Clark

On Tue, 8 Mar 2011 09:45:00 +0100
"Greg Trahair" <Greg.Trahair@tomtom.com> wrote:

> Hi Mike,
>
> The directory is completely empty, and mock --init and mock --clean give
> the same error. Even a mock --no-clean --init gives the error.
>
> The strange thing is that I can (as my unprivileged user) just run rmdir
> /data/mock/dist-rhel5-greg with no problems, so permissions are not the
> culprit. SELinux is disabled.
>
> I can recreate the problem every time I run mock --init, I have to rmdir
> the buildroot before I can continue testing builds.
>
> After a little testing, it seems that /data/mock being NFS is an issue,
> as I just changed the buildroot back to /var/lib/mock and all seems ok.
> Both /data/mock and /var/lib/mock are set 2775 root:mock.
>
> I can't find any limitations of os.rmtree an nfs.
>
> Regards,
> Greg
>
>
> -----Original Message-----
> From: buildsys-bounces@lists.fedoraproject.org
> [mailto:buildsys-bounces@lists.fedoraproject.org] On Behalf Of Mike
> McLean
> Sent: Monday, March 07, 2011 7:55 PM
> To: buildsys@lists.fedoraproject.org
> Subject: Re: Mock 'Directory not empty' error during --clean
>
> On 03/07/2011 04:17 AM, Greg Trahair wrote:
> > But when I attempt to clean it, this happens:
> >
> > $ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
> ...
> > File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in
> > rmtree
> > shutil.rmtree(path, *args, **kargs)
> > File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree
> > onerror(os.rmdir, path, sys.exc_info())
> > File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree
> > os.rmdir(path)
> > OSError: [Errno 39] Directory not empty: '/data/mock/dist-rhel5-greg'
>
> It's hard to know without more data. The error is happening within
> os.rmtree.
>
> Have a look at the remains that were not deleted. Is there anything
> unusual about them? Is it possible another process wrote files to
> directory during the clean operation? Does the problem repeat?
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

"Greg Trahair" 03-08-2011 03:06 PM

Mock 'Directory not empty' error during --clean
 
I thought this originally, but the NFS server is setup with
no_root_squash already (netapp equiv - root=<host>).

I've worked-around the problem by just using a local disk, but does
anyone else use mock buildroots on NFS?


-----Original Message-----
From: Clark Williams [mailto:clark.williams@gmail.com]
Sent: Tuesday, March 08, 2011 4:55 PM
To: Discussion of Fedora build system
Cc: Greg Trahair
Subject: Re: Mock 'Directory not empty' error during --clean

Ah, you're being bitten by root-squash. This means that any thread
running as root gets mapped to the user nobody on the NFS server and in
general gets denied access to regular user files.

Do you have the ability to control how the NFS partition is exported or
is it controlled by your IT group? If you have control over the NFS
server you can export it with no_root_squash and things should work
properly.

I don't have a good solution for running mock on a root-squashed NFS
partition at the moment.

Clark

On Tue, 8 Mar 2011 09:45:00 +0100
"Greg Trahair" <Greg.Trahair@tomtom.com> wrote:

> Hi Mike,
>
> The directory is completely empty, and mock --init and mock --clean
give
> the same error. Even a mock --no-clean --init gives the error.
>
> The strange thing is that I can (as my unprivileged user) just run
rmdir
> /data/mock/dist-rhel5-greg with no problems, so permissions are not
the
> culprit. SELinux is disabled.
>
> I can recreate the problem every time I run mock --init, I have to
rmdir
> the buildroot before I can continue testing builds.
>
> After a little testing, it seems that /data/mock being NFS is an
issue,
> as I just changed the buildroot back to /var/lib/mock and all seems
ok.
> Both /data/mock and /var/lib/mock are set 2775 root:mock.
>
> I can't find any limitations of os.rmtree an nfs.
>
> Regards,
> Greg
>
>
> -----Original Message-----
> From: buildsys-bounces@lists.fedoraproject.org
> [mailto:buildsys-bounces@lists.fedoraproject.org] On Behalf Of Mike
> McLean
> Sent: Monday, March 07, 2011 7:55 PM
> To: buildsys@lists.fedoraproject.org
> Subject: Re: Mock 'Directory not empty' error during --clean
>
> On 03/07/2011 04:17 AM, Greg Trahair wrote:
> > But when I attempt to clean it, this happens:
> >
> > $ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
> ...
> > File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in
> > rmtree
> > shutil.rmtree(path, *args, **kargs)
> > File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree
> > onerror(os.rmdir, path, sys.exc_info())
> > File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree
> > os.rmdir(path)
> > OSError: [Errno 39] Directory not empty:
'/data/mock/dist-rhel5-greg'
>
> It's hard to know without more data. The error is happening within
> os.rmtree.
>
> Have a look at the remains that were not deleted. Is there anything
> unusual about them? Is it possible another process wrote files to
> directory during the clean operation? Does the problem repeat?
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

Mike McLean 03-09-2011 07:11 PM

Mock 'Directory not empty' error during --clean
 
On 03/08/2011 11:06 AM, Greg Trahair wrote:
> I thought this originally, but the NFS server is setup with
> no_root_squash already (netapp equiv - root=<host>).
>
> I've worked-around the problem by just using a local disk, but does
> anyone else use mock buildroots on NFS?

I've used nfs for a lot of things, but I'm having trouble seeing the
value for buildroots. They're not really all that big and local drives
are generally faster, so why bother?

Nfs has always been flaky and finicky. Could be that some race, or
alternately caching is causing the rmdir call to incorrectly fail
because something still thinks those deleted files are in there. If so
you might try tweaking the mount options (noac perhaps?) or try a
different kernel.

Is it 100% repeatable? Is it always the same directory that fails to
remove? Is the remainder always the same.

When it does happen, does a second mock --clean continue to fail repeatedly?
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys


All times are GMT. The time now is 12:37 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.