Just an FYI -- I'm delaying a new release that I had hoped to do today
because backtraces for blocked x86_64 tasks no longer work with recent
kernels because this commit removed the "thread_return" label:

commit c12a229bc5971534537a7d0e49e44f9f1f5d0336
Author: Masami Hiramatsu <mhiramat@redhat.com>
Date: Thu Nov 5 11:03:59 2009 -0500

x86: Remove unused thread_return label from switch_to()

Remove unused thread_return label from switch_to() macro on
x86-64. Since this symbol cuts into schedule(), backtrace at the
latter half of schedule() was always shown as thread_return().

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: systemtap <systemtap@sources.redhat.com>
Cc: DLE <dle-develop@lists.sourceforge.net>
LKML-Reference: <20091105160359.5181.26225.stgit@harusame>
Signed-off-by: Ingo Molnar <mingo@elte.hu>

diff --git a/arch/x86/include/asm/system.h b/arch/x86/include/asm/system.h
index f08f973..1a953e2 100644
--- a/arch/x86/include/asm/system.h
+++ b/arch/x86/include/asm/system.h
@@ -128,8 +128,6 @@ do {
"movq %%rsp,%P[threadrsp](%[prev])
" /* save RSP */
"movq %P[threadrsp](%[next]),%%rsp
" /* restore RSP */
"call __switch_to
"
- ".globl thread_return
"
- "thread_return:
"
"movq "__percpu_arg([current_task])",%%rsi
"
__switch_canary
"movq %P[thread_info](%%rsi),%%r8
"


I've got a fix for x86_64 -- which have always depended on the existence of
the "thread_return" label. But I note that x86 backtraces also are not working,
which I'll take a look at today.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

----- "Dave Anderson" <anderson@redhat.com> wrote:

> I've got a fix for x86_64 -- which have always depended on the existence of
> the "thread_return" label. But I note that x86 backtraces also are not working,
> which I'll take a look at today.

As it turns out, the x86 backtrace failures in 2.6.33/34 are caused by a different
kprobes-related commit, which moved the system_call assembly function to the
.kprobes.text section:

commit a00e817f42663941ea0aa5f85a9d1c4f8b212839
Author: Masami Hiramatsu <mhiramat@redhat.com>
Date: Tue Sep 8 12:47:55 2009 -0400

kprobes/x86-32: Move irq-exit functions to kprobes section

Move irq-exit functions to .kprobes.text section to protect against
kprobes recursion.

When I ran kprobe stress test on x86-32, I found below symbols
cause unrecoverable recursive probing:

ret_from_exception
ret_from_intr
check_userspace
restore_all
restore_all_notrace
restore_nocheck
irq_return

And also, I found some interrupt/exception entry points that
cause similar problems.

This patch moves those symbols (including their container functions)
to .kprobes.text section to prevent any kprobes probing.

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: Jim Keniston <jkenisto@us.ibm.com>
Cc: Ingo Molnar <mingo@elte.hu>
LKML-Reference: <20090908164755.24050.81182.stgit@dhcp-100-2-132.bos.redhat.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>

... [ snip ] ...

@@ -513,6 +521,10 @@ sysexit_audit:
PTGS_TO_GS_EX
ENDPROC(ia32_sysenter_target)

+/*
+ * syscall stub including irq exit should be protected against kprobes
+ */
+ .pushsection .kprobes.text, "ax"
# system call handler stub
ENTRY(system_call)
RING0_INT_FRAME # can't unwind into user space anyway
@@ -705,6 +717,10 @@ syscall_badsys:
jmp resume_userspace
END(syscall_badsys)
CFI_ENDPROC
+/*
+ * End of kprobes section
+ */
+ .popsection

I should have a fix tomorrow (if that's the only issue)...

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Hi Dave,

Are these issues only for crash tools? or it occurs in kernel func-backtrace too?
And how would you fix it?

Thank you,

Dave Anderson wrote:
>
> ----- "Dave Anderson" <anderson@redhat.com> wrote:
>
>> I've got a fix for x86_64 -- which have always depended on the existence of
>> the "thread_return" label. But I note that x86 backtraces also are not working,
>> which I'll take a look at today.
>
> As it turns out, the x86 backtrace failures in 2.6.33/34 are caused by a different
> kprobes-related commit, which moved the system_call assembly function to the
> .kprobes.text section:
>
> commit a00e817f42663941ea0aa5f85a9d1c4f8b212839
> Author: Masami Hiramatsu <mhiramat@redhat.com>
> Date: Tue Sep 8 12:47:55 2009 -0400
>
> kprobes/x86-32: Move irq-exit functions to kprobes section
>
> Move irq-exit functions to .kprobes.text section to protect against
> kprobes recursion.
>
> When I ran kprobe stress test on x86-32, I found below symbols
> cause unrecoverable recursive probing:
>
> ret_from_exception
> ret_from_intr
> check_userspace
> restore_all
> restore_all_notrace
> restore_nocheck
> irq_return
>
> And also, I found some interrupt/exception entry points that
> cause similar problems.
>
> This patch moves those symbols (including their container functions)
> to .kprobes.text section to prevent any kprobes probing.
>
> Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
> Cc: Jim Keniston <jkenisto@us.ibm.com>
> Cc: Ingo Molnar <mingo@elte.hu>
> LKML-Reference: <20090908164755.24050.81182.stgit@dhcp-100-2-132.bos.redhat.com>
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
>
> ... [ snip ] ...
>
> @@ -513,6 +521,10 @@ sysexit_audit:
> PTGS_TO_GS_EX
> ENDPROC(ia32_sysenter_target)
>
> +/*
> + * syscall stub including irq exit should be protected against kprobes
> + */
> + .pushsection .kprobes.text, "ax"
> # system call handler stub
> ENTRY(system_call)
> RING0_INT_FRAME # can't unwind into user space anyway
> @@ -705,6 +717,10 @@ syscall_badsys:
> jmp resume_userspace
> END(syscall_badsys)
> CFI_ENDPROC
> +/*
> + * End of kprobes section
> + */
> + .popsection
>
> I should have a fix tomorrow (if that's the only issue)...
>
> Dave

--
Masami Hiramatsu
e-mail: mhiramat@redhat.com

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

----- "Masami Hiramatsu" <mhiramat@redhat.com> wrote:

> Hi Dave,
>
> Are these issues only for crash tools? or it occurs in kernel func-backtrace too?
> And how would you fix it?

They are crash issues only, in having to deal with the shifting sands of
the underlying kernel.

In both cases, the problem has always been that assembly-code labels are
stored as text symbols, which is confusing to the backtrace code. And in
both cases, the new kernel changes interfered with the work-arounds put in
place by the crash utility to handle them.

In any case, it's not a big deal as it's fixable in the crash utility.

Thanks,
Dave


--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility