FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Crash Utility

 
 
LinkBack Thread Tools
 
Old 11-29-2010, 06:50 AM
Yuming Cheng
 
Default crash "bt" and "dmesg" show different messages

Hi all,

When using kdump, I find crash "bt" and "dmesg" give different info.
which one is more reliable ?

Thanks,
---cym


dmesg
/****************************************/
Unable to handle kernel NULL pointer dereference at 0000000000000001 RIP:
[<0000000000000001>]
PGD 323c6f067 PUD 323f13067 PMD 0
Oops: 0010 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 6
Modules linked in: igb(U) bonding ipv6 xfrm_nalgo crypto_api autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg ixgbe pcspkr i2c_i801 serio_raw i2c_core 8021q dca dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache ahci libata shpchp mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 8894, comm: ifconfig Tainted: G 2.6.18-164.el5debug #1
RIP: 0010:[<0000000000000001>] [<0000000000000001>]
RSP: 0018:ffff810323dd9cf0 EFLAGS: 00010202
RAX: 0000000000000001 RBX: ffff81032a8e5b68 RCX: 0000000000000000
RDX: 0000000000000006 RSI: 0000000000000001 RDI: ffff81032a8e5b68
RBP: ffff81033aabc850 R08: 0000000000000002 R09: 0000000000000001
R10: ffff81032a8e5c30 R11: ffffffff80049ee3 R12: ffff81032a8e5ba8
R13: 0000000000000006 R14: ffff8103238be000 R15: ffffffff8846ad00
FS: 00002ba7032083f0(0000) GS:ffff810113a9e4c8(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000001 CR3: 00000003242b4000 CR4: 00000000000006e0
Process ifconfig (pid: 8894, threadinfo ffff810323dd8000, task ffff810323f461c0)
Stack: ffffffff8023de3b ffff81032a8e5b68 ffffffff8023e0d6 ffffffff8023e122
ffffffff88468eb0 ffff8103238be000 ffffffff8846ad00 ffff8103238be000
0000000000000000 ffffffff8846ae98 ffffffff8023e12d 0000000000000002
Call Trace:
[<ffffffff8023de3b>] neigh_cleanup_and_release+0x13/0x2c
[<ffffffff8023e0d6>] neigh_flush_dev+0x9d/0xc3
[<ffffffff88439acb>] :ipv6:ndisc_netdev_event+0x30/0x3d
[<ffffffff8006ae76>] notifier_call_chain+0x20/0x32
[<ffffffff80238c52>] dev_close+0x6e/0x72
[<ffffffff80237d24>] dev_change_flags+0x5a/0x119
[<ffffffff8026cb77>] devinet_ioctl+0x235/0x59c
[<ffffffff8022f0e3>] sock_ioctl+0x1c7/0x1eb
[<ffffffff8004465d>] do_ioctl+0x21/0x6b
[<ffffffff80031f07>] vfs_ioctl+0x45d/0x4bf
[<ffffffff800c0b9d>] audit_syscall_entry+0x180/0x1b3
[<ffffffff8004ef9e>] sys_ioctl+0x59/0x78
[<ffffffff800602a6>] tracesys+0xd5/0xdf

/****************************************/
crash btcrash> bt
PID: 8894 TASK: ffff810323f461c0 CPU: 6 COMMAND: "ifconfig"
#0 [ffff810323dd9a50] crash_kexec at ffffffff800b6eae
#1 [ffff810323dd9b10] __die at ffffffff80069087
#2 [ffff810323dd9b50] do_page_fault at ffffffff8006ad73
#3 [ffff810323dd9c40] error_exit at ffffffff80060e9d
#4 [ffff810323dd9c78] skb_dequeue at ffffffff80049ee3
#5 [ffff810323dd9cf0] neigh_cleanup_and_release at ffffffff8023de3b
#6 [ffff810323dd9d00] neigh_flush_dev at ffffffff8023e0d6
#7 [ffff810323dd9d40] neigh_ifdown at ffffffff8023e12d
#8 [ffff810323dd9d80] ndisc_netdev_event at ffffffff88439acb
#9 [ffff810323dd9d90] notifier_call_chain at ffffffff8006ae76
#10 [ffff810323dd9db0] dev_close at ffffffff80238c52
#11 [ffff810323dd9dc0] dev_change_flags at ffffffff80237d24
#12 [ffff810323dd9df0] devinet_ioctl at ffffffff8026cb77
#13 [ffff810323dd9e90] sock_ioctl at ffffffff8022f0e3
#14 [ffff810323dd9eb0] do_ioctl at ffffffff8004465d
#15 [ffff810323dd9ed0] vfs_ioctl at ffffffff80031f07
#16 [ffff810323dd9f40] sys_ioctl at ffffffff8004ef9e
#17 [ffff810323dd9f80] tracesys at ffffffff800602a6 (via system_call)
RIP: 0000003749ccc557 RSP: 00007fff470ea238 RFLAGS: 00000206
RAX: ffffffffffffffda RBX: ffffffff800602a6 RCX: ffffffffffffffff
RDX: 00007fff470ea240 RSI: 0000000000008914 RDI: 0000000000000004
RBP: 0000000000000000 R8: 00007fff470ea244 R9: 0000000000000002
R10: 0000000000000001 R11: 0000000000000206 R12: 00007fff470ea360
R13: 00000000fffffffe R14: 00007fff470ea530 R15: 0000000000000004
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b







--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 11-29-2010, 01:03 PM
Dave Anderson
 
Default crash "bt" and "dmesg" show different messages

----- "Yuming Cheng" <chengyuming_ah@yahoo.com.cn> wrote:

> Hi all,
>
> When using kdump, I find crash "bt" and "dmesg" give different info.
> which one is more reliable ?
>
> Thanks,
> ---cym

In this case, the dmesg output is more helpful because it contains the
exception frame. It's pretty clear that the neigh_cleanup_and_release()
has called a destructor function, but the address stored in
neigh->parms->neigh_destructor (as stored in RAX) contains a bogus
address of 0000000000000001:

static void neigh_cleanup_and_release(struct neighbour *neigh)
{
if (neigh->parms->neigh_destructor)
neigh->parms->neigh_destructor(neigh);

__neigh_notify(neigh, RTM_DELNEIGH, 0);
neigh_release(neigh);
}

crash> dis -r neigh_cleanup_and_release+0x13
0xffffffff8022108d <neigh_cleanup_and_release>: push %rbx
0xffffffff8022108e <neigh_cleanup_and_release+0x1>: mov 0x10(%rdi),%rax
0xffffffff80221092 <neigh_cleanup_and_release+0x5>: mov %rdi,%rbx
0xffffffff80221095 <neigh_cleanup_and_release+0x8>: mov 0x18(%rax),%rax
0xffffffff80221099 <neigh_cleanup_and_release+0xc>: test %rax,%rax
0xffffffff8022109c <neigh_cleanup_and_release+0xf>: je 0xffffffff802210a0 <neigh_cleanup_and_release+0x13>
0xffffffff8022109e <neigh_cleanup_and_release+0x11>: callq *%rax
0xffffffff802210a0 <neigh_cleanup_and_release+0x13>: lock decl 0x70(%rbx)


If you do a "bt -e" I would guess that the exception frame would be
found and displayed, but it *should* have been displayed in-line by
the "bt" command.

I can't tell you why it was not displayed by "bt" unless I have the
dumpfile. You also didn't mention what version of crash you were
running -- there have been a few fixes for "missing" exception frames.
If you want to make the dumpfile available to me, I can take a look
at it.

Dave


>
> dmesg
> /****************************************/
> Unable to handle kernel NULL pointer dereference at 0000000000000001
> RIP: [<0000000000000001>]
> PGD 323c6f067 PUD 323f13067 PMD 0
> Oops: 0010 [1] SMP
> last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
> CPU 6
> Modules linked in: igb(U) bonding ipv6 xfrm_nalgo crypto_api autofs4
> hidp rfcomm l2cap bluetooth lockd sunrpc dm_mirror dm_multipath
> scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi
> acpi_memhotplug ac parport_pc lp parport sg ixgbe pcspkr i2c_i801
> serio_raw i2c_core 8021q dca dm_raid45 dm_message dm_region_hash
> dm_log dm_mod dm_mem_cache ahci libata shpchp mptsas mptscsih mptbase
> scsi_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd
> ehci_hcd
> Pid: 8894, comm: ifconfig Tainted: G 2.6.18-164.el5debug #1
> RIP: 0010:[<0000000000000001>] [<0000000000000001>]
> RSP: 0018:ffff810323dd9cf0 EFLAGS: 00010202
> RAX: 0000000000000001 RBX: ffff81032a8e5b68 RCX: 0000000000000000
> RDX: 0000000000000006 RSI: 0000000000000001 RDI: ffff81032a8e5b68
> RBP: ffff81033aabc850 R08: 0000000000000002 R09: 0000000000000001
> R10: ffff81032a8e5c30 R11: ffffffff80049ee3 R12: ffff81032a8e5ba8
> R13: 0000000000000006 R14: ffff8103238be000 R15: ffffffff8846ad00
> FS: 00002ba7032083f0(0000) GS:ffff810113a9e4c8(0000)
> knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000001 CR3: 00000003242b4000 CR4: 00000000000006e0
> Process ifconfig (pid: 8894, threadinfo ffff810323dd8000, task
> ffff810323f461c0)
> Stack: ffffffff8023de3b ffff81032a8e5b68 ffffffff8023e0d6
> ffffffff8023e122
> ffffffff88468eb0 ffff8103238be000 ffffffff8846ad00 ffff8103238be000
> 0000000000000000 ffffffff8846ae98 ffffffff8023e12d 0000000000000002
> Call Trace:
> [<ffffffff8023de3b>] neigh_cleanup_and_release+0x13/0x2c
> [<ffffffff8023e0d6>] neigh_flush_dev+0x9d/0xc3
> [<ffffffff88439acb>] :ipv6:ndisc_netdev_event+0x30/0x3d
> [<ffffffff8006ae76>] notifier_call_chain+0x20/0x32
> [<ffffffff80238c52>] dev_close+0x6e/0x72
> [<ffffffff80237d24>] dev_change_flags+0x5a/0x119
> [<ffffffff8026cb77>] devinet_ioctl+0x235/0x59c
> [<ffffffff8022f0e3>] sock_ioctl+0x1c7/0x1eb
> [<ffffffff8004465d>] do_ioctl+0x21/0x6b
> [<ffffffff80031f07>] vfs_ioctl+0x45d/0x4bf
> [<ffffffff800c0b9d>] audit_syscall_entry+0x180/0x1b3
> [<ffffffff8004ef9e>] sys_ioctl+0x59/0x78
> [<ffffffff800602a6>] tracesys+0xd5/0xdf
>
> /****************************************/
> crash btcrash> bt
> PID: 8894 TASK: ffff810323f461c0 CPU: 6 COMMAND: "ifconfig"
> #0 [ffff810323dd9a50] crash_kexec at ffffffff800b6eae
> #1 [ffff810323dd9b10] __die at ffffffff80069087
> #2 [ffff810323dd9b50] do_page_fault at ffffffff8006ad73
> #3 [ffff810323dd9c40] error_exit at ffffffff80060e9d
> #4 [ffff810323dd9c78] skb_dequeue at ffffffff80049ee3
> #5 [ffff810323dd9cf0] neigh_cleanup_and_release at ffffffff8023de3b
> #6 [ffff810323dd9d00] neigh_flush_dev at ffffffff8023e0d6
> #7 [ffff810323dd9d40] neigh_ifdown at ffffffff8023e12d
> #8 [ffff810323dd9d80] ndisc_netdev_event at ffffffff88439acb
> #9 [ffff810323dd9d90] notifier_call_chain at ffffffff8006ae76
> #10 [ffff810323dd9db0] dev_close at ffffffff80238c52
> #11 [ffff810323dd9dc0] dev_change_flags at ffffffff80237d24
> #12 [ffff810323dd9df0] devinet_ioctl at ffffffff8026cb77
> #13 [ffff810323dd9e90] sock_ioctl at ffffffff8022f0e3
> #14 [ffff810323dd9eb0] do_ioctl at ffffffff8004465d
> #15 [ffff810323dd9ed0] vfs_ioctl at ffffffff80031f07
> #16 [ffff810323dd9f40] sys_ioctl at ffffffff8004ef9e
> #17 [ffff810323dd9f80] tracesys at ffffffff800602a6 (via system_call)
> RIP: 0000003749ccc557 RSP: 00007fff470ea238 RFLAGS: 00000206
> RAX: ffffffffffffffda RBX: ffffffff800602a6 RCX:
> ffffffffffffffff
> RDX: 00007fff470ea240 RSI: 0000000000008914 RDI:
> 0000000000000004
> RBP: 0000000000000000 R8: 00007fff470ea244 R9:
> 0000000000000002
> R10: 0000000000000001 R11: 0000000000000206 R12:
> 00007fff470ea360
> R13: 00000000fffffffe R14: 00007fff470ea530 R15:
> 0000000000000004
> ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
>
>
>
>
>
>
>
> --
> Crash-utility mailing list
> Crash-utility@redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 11-30-2010, 01:42 AM
Yuming Cheng
 
Default crash "bt" and "dmesg" show different messages

Hi Dave,

Thank you very much,
I use crash in RHEL5.4.

crash version: 4.0-8.9.1.el5 gdb version: 6.1

The vmcore file is very huge (12GB). I will check it carefully.

Yuming


--- On Mon, 11/29/10, Dave Anderson <anderson@redhat.com> wrote:

> From: Dave Anderson <anderson@redhat.com>
> Subject: Re: [Crash-utility] crash "bt" and "dmesg" show different messages
> To: "Discussion list for crash utility usage, maintenance and development" <crash-utility@redhat.com>
> Date: Monday, November 29, 2010, 10:03 PM
>
> ----- "Yuming Cheng" <chengyuming_ah@yahoo.com.cn>
> wrote:
>
> > Hi all,
> >
> > When using kdump, I find crash "bt" and "dmesg" give
> different info.
> > which one is more reliable ?
> >
> > Thanks,
> > ---cym
>
> In this case, the dmesg output is more helpful because it
> contains the
> exception frame.* It's pretty clear that the
> neigh_cleanup_and_release()
> has called a destructor function, but the address stored in
>
> neigh->parms->neigh_destructor (as stored in RAX)
> contains a bogus
> address of 0000000000000001:
>
> static void neigh_cleanup_and_release(struct neighbour
> *neigh)
> {
> * * * * if
> (neigh->parms->neigh_destructor)
> * * * * * * * *
> neigh->parms->neigh_destructor(neigh);
>
> * * * * __neigh_notify(neigh,
> RTM_DELNEIGH, 0);
> * * * * neigh_release(neigh);
> }
>
> crash> dis -r neigh_cleanup_and_release+0x13
> 0xffffffff8022108d <neigh_cleanup_and_release>:
> push***%rbx
> 0xffffffff8022108e
> <neigh_cleanup_and_release+0x1>:*
> ***mov* * 0x10(%rdi),%rax
> 0xffffffff80221092
> <neigh_cleanup_and_release+0x5>:*
> ***mov* * %rdi,%rbx
> 0xffffffff80221095
> <neigh_cleanup_and_release+0x8>:*
> ***mov* * 0x18(%rax),%rax
> 0xffffffff80221099
> <neigh_cleanup_and_release+0xc>:*
> ***test***%rax,%rax
> 0xffffffff8022109c
> <neigh_cleanup_and_release+0xf>:*
> ***je*
> ***0xffffffff802210a0
> <neigh_cleanup_and_release+0x13>
> 0xffffffff8022109e
> <neigh_cleanup_and_release+0x11>:* *
> callq* *%rax
> 0xffffffff802210a0
> <neigh_cleanup_and_release+0x13>:* * lock
> decl 0x70(%rbx)
>
>
> If you do a "bt -e" I would guess that the exception frame
> would be
> found and displayed, but it *should* have been displayed
> in-line by
> the "bt" command.
>
> I can't tell you why it was not displayed by "bt" unless I
> have the
> dumpfile.* You also didn't mention what version of
> crash you were
> running -- there have been a few fixes for "missing"
> exception frames.
> If you want to make the dumpfile available to me, I can
> take a look
> at it.
>
> Dave
>
>
> >
> > dmesg
> > /****************************************/
> > Unable to handle kernel NULL pointer dereference at
> 0000000000000001
> > RIP: [<0000000000000001>]
> > PGD 323c6f067 PUD 323f13067 PMD 0
> > Oops: 0010 [1] SMP
> > last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
> > CPU 6
> > Modules linked in: igb(U) bonding ipv6 xfrm_nalgo
> crypto_api autofs4
> > hidp rfcomm l2cap bluetooth lockd sunrpc dm_mirror
> dm_multipath
> > scsi_dh video hwmon backlight sbs i2c_ec button
> battery asus_acpi
> > acpi_memhotplug ac parport_pc lp parport sg ixgbe
> pcspkr i2c_i801
> > serio_raw i2c_core 8021q dca dm_raid45 dm_message
> dm_region_hash
> > dm_log dm_mod dm_mem_cache ahci libata shpchp mptsas
> mptscsih mptbase
> > scsi_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd
> ohci_hcd
> > ehci_hcd
> > Pid: 8894, comm: ifconfig Tainted: G* *
> * 2.6.18-164.el5debug #1
> > RIP: 0010:[<0000000000000001>]*
> [<0000000000000001>]
> > RSP: 0018:ffff810323dd9cf0* EFLAGS: 00010202
> > RAX: 0000000000000001 RBX: ffff81032a8e5b68 RCX:
> 0000000000000000
> > RDX: 0000000000000006 RSI: 0000000000000001 RDI:
> ffff81032a8e5b68
> > RBP: ffff81033aabc850 R08: 0000000000000002 R09:
> 0000000000000001
> > R10: ffff81032a8e5c30 R11: ffffffff80049ee3 R12:
> ffff81032a8e5ba8
> > R13: 0000000000000006 R14: ffff8103238be000 R15:
> ffffffff8846ad00
> > FS:* 00002ba7032083f0(0000)
> GS:ffff810113a9e4c8(0000)
> > knlGS:0000000000000000
> > CS:* 0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> > CR2: 0000000000000001 CR3: 00000003242b4000 CR4:
> 00000000000006e0
> > Process ifconfig (pid: 8894, threadinfo
> ffff810323dd8000, task
> > ffff810323f461c0)
> > Stack:* ffffffff8023de3b ffff81032a8e5b68
> ffffffff8023e0d6
> > ffffffff8023e122
> >* ffffffff88468eb0 ffff8103238be000
> ffffffff8846ad00 ffff8103238be000
> >* 0000000000000000 ffffffff8846ae98
> ffffffff8023e12d 0000000000000002
> > Call Trace:
> >* [<ffffffff8023de3b>]
> neigh_cleanup_and_release+0x13/0x2c
> >* [<ffffffff8023e0d6>]
> neigh_flush_dev+0x9d/0xc3
> >* [<ffffffff88439acb>]
> :ipv6:ndisc_netdev_event+0x30/0x3d
> >* [<ffffffff8006ae76>]
> notifier_call_chain+0x20/0x32
> >* [<ffffffff80238c52>] dev_close+0x6e/0x72
> >* [<ffffffff80237d24>]
> dev_change_flags+0x5a/0x119
> >* [<ffffffff8026cb77>]
> devinet_ioctl+0x235/0x59c
> >* [<ffffffff8022f0e3>]
> sock_ioctl+0x1c7/0x1eb
> >* [<ffffffff8004465d>] do_ioctl+0x21/0x6b
> >* [<ffffffff80031f07>]
> vfs_ioctl+0x45d/0x4bf
> >* [<ffffffff800c0b9d>]
> audit_syscall_entry+0x180/0x1b3
> >* [<ffffffff8004ef9e>] sys_ioctl+0x59/0x78
> >* [<ffffffff800602a6>] tracesys+0xd5/0xdf
> >
> > /****************************************/
> > crash btcrash> bt
> > PID: 8894***TASK:
> ffff810323f461c0* CPU: 6***COMMAND:
> "ifconfig"
> >* #0 [ffff810323dd9a50] crash_kexec at
> ffffffff800b6eae
> >* #1 [ffff810323dd9b10] __die at ffffffff80069087
> >* #2 [ffff810323dd9b50] do_page_fault at
> ffffffff8006ad73
> >* #3 [ffff810323dd9c40] error_exit at
> ffffffff80060e9d
> >* #4 [ffff810323dd9c78] skb_dequeue at
> ffffffff80049ee3
> >* #5 [ffff810323dd9cf0] neigh_cleanup_and_release
> at ffffffff8023de3b
> >* #6 [ffff810323dd9d00] neigh_flush_dev at
> ffffffff8023e0d6
> >* #7 [ffff810323dd9d40] neigh_ifdown at
> ffffffff8023e12d
> >* #8 [ffff810323dd9d80] ndisc_netdev_event at
> ffffffff88439acb
> >* #9 [ffff810323dd9d90] notifier_call_chain at
> ffffffff8006ae76
> > #10 [ffff810323dd9db0] dev_close at ffffffff80238c52
> > #11 [ffff810323dd9dc0] dev_change_flags at
> ffffffff80237d24
> > #12 [ffff810323dd9df0] devinet_ioctl at
> ffffffff8026cb77
> > #13 [ffff810323dd9e90] sock_ioctl at ffffffff8022f0e3
> > #14 [ffff810323dd9eb0] do_ioctl at ffffffff8004465d
> > #15 [ffff810323dd9ed0] vfs_ioctl at ffffffff80031f07
> > #16 [ffff810323dd9f40] sys_ioctl at ffffffff8004ef9e
> > #17 [ffff810323dd9f80] tracesys at ffffffff800602a6
> (via system_call)
> >* ***RIP: 0000003749ccc557*
> RSP: 00007fff470ea238* RFLAGS: 00000206
> >* ***RAX: ffffffffffffffda*
> RBX: ffffffff800602a6* RCX:
> > ffffffffffffffff
> >* ***RDX: 00007fff470ea240*
> RSI: 0000000000008914* RDI:
> > 0000000000000004
> >* ***RBP:
> 0000000000000000***R8:
> 00007fff470ea244***R9:
> > 0000000000000002
> >* ***R10: 0000000000000001*
> R11: 0000000000000206* R12:
> > 00007fff470ea360
> >* ***R13: 00000000fffffffe*
> R14: 00007fff470ea530* R15:
> > 0000000000000004
> >* ***ORIG_RAX:
> 0000000000000010* CS: 0033* SS: 002b
> >
> >
> >
> >
> >
> >* * ***
> >
> > --
> > Crash-utility mailing list
> > Crash-utility@redhat.com
> > https://www.redhat.com/mailman/listinfo/crash-utility
>
> --
> Crash-utility mailing list
> Crash-utility@redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>





--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 

Thread Tools




All times are GMT. The time now is 02:47 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org