FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Edubuntu User

 
 
LinkBack Thread Tools
 
Old 09-12-2008, 05:55 PM
Scott Balneaves
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

On Fri, Sep 12, 2008 at 10:46:56AM -0700, john wrote:
> Hi all,
>
> I migrated one of our schools to Hardy on a 64 bit 4 core machine, 8
> gigs ram, running a 32 bit image for ntavo 6030 series clients.
>
>
> About 20% of my 40 workstations are going down on any given day. It
> looks like they are losing there connection to the LTSP server
> /var/log/messages has lots of stuff like:

Could you give us a few more specs on the thin clients? CPU/RAM
specifically? Do you have NBD_SWAP turned on? Is it all the same clients, or
random ones? All on the same switch? etc.

Scott

--
Scott L. Balneaves | "There are many causes I am prepared to die for,
Systems Department | but no causes I am prepared to kill for."
Legal Aid Manitoba | -- Mohandas Karamchand Gandhi

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-12-2008, 07:38 PM
john
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

Thanks for the reply Scott,


> Could you give us a few more specs on the thin clients? CPU/RAM
> specifically?

All thin clients are Ntavo 6030's with 512 MB ram. CPU is a via C3
Processor running at 800 Mhz

> Do you have NBD_SWAP turned on?

I haven't made any changes re: NBD_SWAP so it would be whatever the
images uses by default. As I mentioned earlier the server is
configured with 2 Dual-Core AMD Opteron Processor 2218. So I rebuilt
my thin client images with the 32 bit option
"ltsp-build-client --arch i386"

Is it all the same clients, or
> random ones?

random clients so far

> All on the same switch? etc.

I have a the gig nic on server going to a switch with gig uplinks that
fans out to other switches across the building. The switches mostly
have gigE copper ports although there are still a few 100mb switches
out there. No errors on the server interface facing the thin clients.

I appreciate the feed back.

John

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-12-2008, 07:46 PM
Scott Balneaves
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

On Fri, Sep 12, 2008 at 12:38:40PM -0700, john wrote:
> Thanks for the reply Scott,
>
>
> > Could you give us a few more specs on the thin clients? CPU/RAM
> > specifically?
>
> All thin clients are Ntavo 6030's with 512 MB ram. CPU is a via C3
> Processor running at 800 Mhz
>
> > Do you have NBD_SWAP turned on?
>
> I haven't made any changes re: NBD_SWAP so it would be whatever the
> images uses by default. As I mentioned earlier the server is
> configured with 2 Dual-Core AMD Opteron Processor 2218. So I rebuilt
> my thin client images with the 32 bit option
> "ltsp-build-client --arch i386"

OK, first thing I'd do is add in say 256 megs of NBD swap (assuming you've got
the disk space) with something like:

NBD_SWAP=True

and edit /etc/ltsp/nbdswapd.conf

SIZE=256

Also, within the thin client chroot, when they freeze up, can you ctrl-alt-f1
and get the login prompt? If so, here might be an interesting test:


Set a root passwd in the chroot:

chroot /opt/ltsp/i386
passwd

Rebuild your image, then if you get a latch-up, c-a-f1 to the text login, go in
as root, and use the free command to see if you've run out of ram, and run
dmesg to see what the client thinks is going on.

Let me know.

Scott

--
Scott L. Balneaves | "There are many causes I am prepared to die for,
Systems Department | but no causes I am prepared to kill for."
Legal Aid Manitoba | -- Mohandas Karamchand Gandhi

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-12-2008, 08:45 PM
john
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

<snip>
>
> OK, first thing I'd do is add in say 256 megs of NBD swap (assuming you've got
> the disk space) with something like:
>
> NBD_SWAP=True


Ok, I assume the proper place for this is /var/lib/tftpboot/ltsp/i386/lts.conf

I added the line you indicated. Now that file reads:


LDM_DIRECTX=True
NBD_SWAP=True


>
> and edit /etc/ltsp/nbdswapd.conf
>
> SIZE=256

Ok, there wasn't already an /etc/ltsp/nbdswapd.conf so I created one
and added the line you mentioned.

my /etc/ltsp directory now has the following files in it.

dhcpd.conf nbdswapd.conf syslogd


BTW, Where is the disk space being used? I assume this is on the
server? Could you explain this a little more fully?

>
> Also, within the thin client chroot, when they freeze up, can you ctrl-alt-f1
> and get the login prompt?

The clients are currently completely frozen. I can't get an alternate
tty, however I am game to try this out if I get one that
responds.

If so, here might be an interesting test:
>
>
> Set a root passwd in the chroot:
>
> chroot /opt/ltsp/i386
> passwd
>
> Rebuild your image, then if you get a latch-up, c-a-f1 to the text login, go in
> as root, and use the free command to see if you've run out of ram, and run
> dmesg to see what the client thinks is going on.
>


Thanks!

John

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-15-2008, 06:41 PM
john
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

Hm,

That seemed to make it worse. Scott or anybody, do we have any FAQ's
about troubleshooting nbd problems.

Are other folks having trouble with 8.04/NBD ?

Thanks,

John

On Fri, Sep 12, 2008 at 1:45 PM, john <lists.john@gmail.com> wrote:
> <snip>
>>
>> OK, first thing I'd do is add in say 256 megs of NBD swap (assuming you've got
>> the disk space) with something like:
>>
>> NBD_SWAP=True
>
>
> Ok, I assume the proper place for this is /var/lib/tftpboot/ltsp/i386/lts.conf
>
> I added the line you indicated. Now that file reads:
>
>
> LDM_DIRECTX=True
> NBD_SWAP=True
>
>
>>
>> and edit /etc/ltsp/nbdswapd.conf
>>
>> SIZE=256
>
> Ok, there wasn't already an /etc/ltsp/nbdswapd.conf so I created one
> and added the line you mentioned.
>
> my /etc/ltsp directory now has the following files in it.
>
> dhcpd.conf nbdswapd.conf syslogd
>
>
> BTW, Where is the disk space being used? I assume this is on the
> server? Could you explain this a little more fully?
>
>>
>> Also, within the thin client chroot, when they freeze up, can you ctrl-alt-f1
>> and get the login prompt?
>
> The clients are currently completely frozen. I can't get an alternate
> tty, however I am game to try this out if I get one that
> responds.
>
> If so, here might be an interesting test:
>>
>>
>> Set a root passwd in the chroot:
>>
>> chroot /opt/ltsp/i386
>> passwd
>>
>> Rebuild your image, then if you get a latch-up, c-a-f1 to the text login, go in
>> as root, and use the free command to see if you've run out of ram, and run
>> dmesg to see what the client thinks is going on.
>>
>
>
> Thanks!
>
> John
>

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-15-2008, 06:54 PM
john
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

Scott et al.

When I set the swap options you mentioned It hung worse than ever. I
wonder if the problem lies with swap. I have
512 Mb ram on the clients. Maybe I shouldn't be swapping at all?


Well I found two bug reports that might be relevant:

https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.20/+bug/136410

Oli works on this one last year.

https://bugs.launchpad.net/ubuntu/+source/ltsp/+bug/172459

John


On Mon, Sep 15, 2008 at 11:41 AM, john <lists.john@gmail.com> wrote:
> Hm,
>
> That seemed to make it worse. Scott or anybody, do we have any FAQ's
> about troubleshooting nbd problems.
>
> Are other folks having trouble with 8.04/NBD ?
>
> Thanks,
>
> John
>
> On Fri, Sep 12, 2008 at 1:45 PM, john <lists.john@gmail.com> wrote:
>> <snip>
>>>
>>> OK, first thing I'd do is add in say 256 megs of NBD swap (assuming you've got
>>> the disk space) with something like:
>>>
>>> NBD_SWAP=True
>>
>>
>> Ok, I assume the proper place for this is /var/lib/tftpboot/ltsp/i386/lts.conf
>>
>> I added the line you indicated. Now that file reads:
>>
>>
>> LDM_DIRECTX=True
>> NBD_SWAP=True
>>
>>
>>>
>>> and edit /etc/ltsp/nbdswapd.conf
>>>
>>> SIZE=256
>>
>> Ok, there wasn't already an /etc/ltsp/nbdswapd.conf so I created one
>> and added the line you mentioned.
>>
>> my /etc/ltsp directory now has the following files in it.
>>
>> dhcpd.conf nbdswapd.conf syslogd
>>
>>
>> BTW, Where is the disk space being used? I assume this is on the
>> server? Could you explain this a little more fully?
>>
>>>
>>> Also, within the thin client chroot, when they freeze up, can you ctrl-alt-f1
>>> and get the login prompt?
>>
>> The clients are currently completely frozen. I can't get an alternate
>> tty, however I am game to try this out if I get one that
>> responds.
>>
>> If so, here might be an interesting test:
>>>
>>>
>>> Set a root passwd in the chroot:
>>>
>>> chroot /opt/ltsp/i386
>>> passwd
>>>
>>> Rebuild your image, then if you get a latch-up, c-a-f1 to the text login, go in
>>> as root, and use the free command to see if you've run out of ram, and run
>>> dmesg to see what the client thinks is going on.
>>>
>>
>>
>> Thanks!
>>
>> John
>>
>

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-15-2008, 07:07 PM
Scott Balneaves
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

On Mon, Sep 15, 2008 at 11:54:28AM -0700, john wrote:
> Scott et al.
>
> When I set the swap options you mentioned It hung worse than ever. I
> wonder if the problem lies with swap. I have
> 512 Mb ram on the clients. Maybe I shouldn't be swapping at all?

Well, if you're not doing a huge amount of stuff with graphics in firefox or
Gimp, you probably won't need the swap, so I'd turn it off if it's making
things worse.

Here's the specs of a machine that's running as I type this:

2 Dual core Xeons 5140 @ 2.33GHz
8 gigs of ram
2 intel ethernet cards. It's an IBM x3550 server box.

Currently, I've got 41 people logged in:

sbalneav@sputnik:~$ ps -ef | grep [Mm]etacity | wc
41 449 4090

sbalneav@sputnik:~$ uptime
14:05:32 up 12 days, 16:49, 44 users, load average: 0.38, 0.51, 0.57


All of them are using NBD. I'm using Via E10000 mobos with 1 gig of ram

So, I know the NBD works, at least in my configuration. Now, I've got "dumb"
switches: i.e. non-managed regualar D-Link switches. I wonder if something's
going on with the networking side of things. Do you have Managed switches,
and if so, can you get some stats out of them?

Cheers,
Scott

--
Scott L. Balneaves | "There are many causes I am prepared to die for,
Systems Department | but no causes I am prepared to kill for."
Legal Aid Manitoba | -- Mohandas Karamchand Gandhi

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-15-2008, 10:04 PM
john
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

Hi Scott,
>
> Well, if you're not doing a huge amount of stuff with graphics in firefox or
> Gimp, you probably won't need the swap, so I'd turn it off if it's making
> things worse.

Ok, thanks. Is there a place I can go to get a feel for how ltsp
currently works? Is there a doc/web page etc that
sort of says here's how we implement LTSP since 7.10 and here's some
options you can work with e.g
you can edit "/etc/ltsp/nbdswap.conf" etc?

Sorry I am having a hard time getting a handle on the move to NBD etc.

> Here's the specs of a machine that's running as I type this:
>
> 2 Dual core Xeons 5140 @ 2.33GHz
> 8 gigs of ram
> 2 intel ethernet cards. It's an IBM x3550 server box.

Seems like we have pretty similar installations...

>
> All of them are using NBD. I'm using Via E10000 mobos with 1 gig of ram
>
> So, I know the NBD works, at least in my configuration. Now, I've got "dumb"
> switches: i.e. non-managed regualar D-Link switches. I wonder if something's
> going on with the networking side of things. Do you have Managed switches,
> and if so, can you get some stats out of them?


Sadly most of them are dumb as well. I believe one may be able to do
logging remotely. I'll look into turning that on.

One intersting note:

When I showed a *nix guru I know the log messages :


Sep 12 10:34:44 192.168.0.248 kernel: [679871.912346] end_request: I/O
error, dev nbd0, sector 200140
Sep 12 10:34:44 192.168.0.248 kernel: [679871.912363] end_request: I/O
error, dev nbd0, sector 200142

He pointed out "those sector numbers are just a hair above 50 megs. Have you
looked at the block files which are being served for the clients (I believe
they're in /tmp)? I wonder if the files aren't being sized correctly for
the swap space they need to serve."

Dunno if this rings a bell with you, but I"ll be look into that next.

Thanks for any ideas you may have!

John

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-15-2008, 11:23 PM
Scott Balneaves
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

On Mon, Sep 15, 2008 at 03:04:19PM -0700, john wrote:
> Hi Scott,
> >
> > Well, if you're not doing a huge amount of stuff with graphics in firefox or
> > Gimp, you probably won't need the swap, so I'd turn it off if it's making
> > things worse.
>
> Ok, thanks. Is there a place I can go to get a feel for how ltsp
> currently works? Is there a doc/web page etc that
> sort of says here's how we implement LTSP since 7.10 and here's some
> options you can work with e.g
> you can edit "/etc/ltsp/nbdswap.conf" etc?

The Edubuntu Handbooks got a lot of stuff. It's *slightly* out of date, we're
working on moving the docs into the LTSP project itself, and updating them, but
just about all the info in it's good.

http://doc.ubuntu.com/edubuntu/edubuntu/handbook/C/


> Sadly most of them are dumb as well. I believe one may be able to do
> logging remotely. I'll look into turning that on.
>
> One intersting note:
>
> When I showed a *nix guru I know the log messages :
>
>
> Sep 12 10:34:44 192.168.0.248 kernel: [679871.912346] end_request: I/O
> error, dev nbd0, sector 200140
> Sep 12 10:34:44 192.168.0.248 kernel: [679871.912363] end_request: I/O
> error, dev nbd0, sector 200142
>
> He pointed out "those sector numbers are just a hair above 50 megs. Have you
> looked at the block files which are being served for the clients (I believe
> they're in /tmp)? I wonder if the files aren't being sized correctly for
> the swap space they need to serve."
>
> Dunno if this rings a bell with you, but I"ll be look into that next.


Hmmm. Well, we use mksquashfs, which figures out the size of the image itself,
we never touch it, so unless it's guessing wrong.

I know this is going to sound like a completely stupid question, but have you
rebuilt the image with ltsp-update-image? Chroot into the /opt/ltsp/i386,
upgrade the image, rebuild it, and re-run ltsp-update-kernel.


Just for a laugh.

Cheers,
Scott


--
Scott L. Balneaves | "There are many causes I am prepared to die for,
Systems Department | but no causes I am prepared to kill for."
Legal Aid Manitoba | -- Mohandas Karamchand Gandhi

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 
Old 09-16-2008, 12:07 AM
john
 
Default show stopper for us: dev nbd0 clients locking up losing connection with server

> The Edubuntu Handbooks got a lot of stuff. It's *slightly* out of date, we're
> working on moving the docs into the LTSP project itself, and updating them, but
> just about all the info in it's good.
>
> http://doc.ubuntu.com/edubuntu/edubuntu/handbook/C/

Thanks for that.



>
> Hmmm. Well, we use mksquashfs, which figures out the size of the image itself,
> we never touch it, so unless it's guessing wrong.
>
> I know this is going to sound like a completely stupid question, but have you
> rebuilt the image with ltsp-update-image? Chroot into the /opt/ltsp/i386,
> upgrade the image, rebuild it, and re-run ltsp-update-kernel.
>
>
> Just for a laugh.
>
> Cheers,
> Scott

I'll give it a go. This is going to sound pedantic, but I just want to
make sure I do what you suggest. You are saying I should:

1) chroot /etc/ltsp/i386
2) apt-get update && apt-get upgrade
3) exit chroot
4) Since my server is AMD 64 based and my clients are x86 based
'ltsp-update-image -a i386'
5) ltsp-update-kernels

Do I have that right?

Thanks!

John

--
edubuntu-users mailing list
edubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
 

Thread Tools




All times are GMT. The time now is 07:15 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org