Embedded Linux S3C2440 - Multi-thread Scheduling

Summary

Test the debugging of the Multi-thread Scheduling on embedded Linux S3C2440, both in All-stop mode and Non-stop mode.

GDB supports multi-thread application debugging, and there are two modes of controlling the execution of the program with GDB dubugger. All-stop mode is the default mode, when the thread of the program stops (at a break point or being in step debugging), all other threads in the program are also stopped by the GDB. In contrast, in Non-stop mode, when the thread of the program stops, all other threads can continue to run freely when you are examining the stopped thread in the GDB, not all targets support Non-stop mode.

All-stop Mode and Non-Stop Mode debugging

All-stop mode debugging

In All-stop mode whenever your program stops under GDB, all threads of the program stop. It allows you to examine the overall state of the program, including switching between threads, without worrying things may change underfoot. And when you restart your program, all threads start executing, it is also true in the case of step or next commands of GDB. But GDB cannot single step all threads in lockstep. And you might even find your program stopped in another thread after continuing or even single-stepping.

When GDB stops due to a break point or a signal, it automatically selects the thread where that break point or signal happened. GDB will display the context switch with the message as ‘[Switcing to Thread n]’ to identify the thread.

Non-stop mode debugging

Non-stop mode supports that the operation of the program thread stopped while other program threads continue to execute freely. This minimizes intrusion when debugging the live system, such as programs where some threads have real-time constraints or must continue to respond to the external events.

In Non-stop mode, the execution commands such as continue and step apply by default only to the current thread in Non-stop mode, rather than all threads in All-stop mode. This feature enables user to control threads explicitly in ways that are not possible in All-stop mode, for example: stepping one thread whole allowing other threads to run freely, stepping one thread while holding all other thread stopped, and stepping several threads independently and simultaneously.

Use below commands to enter Non-stop mode,

#Enable the async interface.
(gdb) set target-async 1
#If using the CLI, pagination breaks non-stop.
(gdb) set pagination off
#Finally, turn it on.
(gdb) set non-stop on

Use below commands to change the Non-stop mode setting,

#Enable selection of Non-stop mode
(gdb) set non-stop on
#Disable selection of Non-stop mode
(gdb) set non-stop off
#Show current non-stop enablement setting
(gdb) show non-stop

The set non-stop on is only consulted when GDB starts or connects to the target program, and it is generally not possible to switch modes once debugging has started. And not all target support Non-stop mode, GDB may fall back to All-stop mode if the target cannot support Non-stop mode.

In Non-stop mode, the GDB commands continue -a or c -a only apply to the current thread by default. And in Non-stop mode, when a thread stops, GDB doesn’t automatically make that thread current, it is because the thread stop notifications are asynchronous with respect to GDB’s command interpreter, so the GDB may changed to a different thread just as you entered a command to operate on previously current thread.

Thread-specific Breakpoint

You can set the break points on all threads, or on a particular thread.

break linespec thread threadno.
break linespec thread threadno if...

linespec specifies source lines, thread threadno with a break point command is to specify that you only want GDB to stop the program when a particular thread reaches this break point. The threadno is thread identifier assigned by GDB, and can be obtained by GDB command info threads.
If you do not specify thread threadno when you set a break point, the break point applies to all threads of your program.

You can also use thread qualifier on conditional break points as well, like below,

(gdb) break thread_demo_RR_max_min.c:15 thread 2 if times < 500

Multi-Thread Application

The multi-thread application source code is as below,

#include <stdio.h>
#include <pthread.h>
#include <stdlib.h>
#include <errno.h>
#include <assert.h>
#include <sched.h>#define DEMO_POLICY SCHED_RR //SCHED_FIFOint count1 = 0;
int count2 = 0;int times = 10000;void *thread_func1(void)
{
    while(times > 0){
    int i = 0;for(i=0; i< 20000; i++)i++;count1 ++;times --;}return (void*) NULL;
}void *thread_func2(void)
{
    while(times > 0){
    int i = 0;for(i=0; i< 20000; i++)i++;count2 ++;times --;}return (void*) NULL;
}int main()
{
    int res;pthread_t thread_1;pthread_t thread_2;pthread_attr_t thread_1_attr;pthread_attr_t thread_2_attr;int min_prio;int max_prio; struct sched_param param1;struct sched_param param2;struct timespec tp;/* get max, min prop for policy FIFO/RR */max_prio = sched_get_priority_max(DEMO_POLICY);min_prio = sched_get_priority_min(DEMO_POLICY);printf("min/max prio (%d, %d)\n", min_prio, max_prio);res = pthread_attr_init(&thread_1_attr);assert(res == 0);res = pthread_attr_init(&thread_2_attr);assert(res == 0);/* Set sched policy */res = pthread_attr_setinheritsched(&thread_1_attr, PTHREAD_EXPLICIT_SCHED);assert(res == 0);res = pthread_attr_setinheritsched(&thread_2_attr, PTHREAD_EXPLICIT_SCHED);assert(res == 0);res = pthread_attr_setschedpolicy(&thread_1_attr, DEMO_POLICY);assert(res == 0);res = pthread_attr_setschedpolicy(&thread_2_attr, DEMO_POLICY);assert(res == 0);/* Set thread priority */param1.sched_priority = max_prio;  // min_prioparam2.sched_priority = min_prio;  // max_priores = pthread_attr_setschedparam(&thread_1_attr,&param1);assert(res == 0);res = pthread_attr_setschedparam(&thread_2_attr,&param2);assert(res == 0);/* Create threads */res = pthread_create(&thread_1, &thread_1_attr, (void *)thread_func1, NULL);assert(res == 0);res = pthread_create(&thread_2, &thread_2_attr, (void *)thread_func2, NULL);assert(res == 0);pthread_join(thread_1, NULL);pthread_join(thread_2, NULL);printf("count1 = %d, count2 = %d\n", count1, count2);exit(0);}

Compile the application as below,

[root@localhost]# arm-linux-gcc thread_demo_RR_max_min.c -g -lpthread -o thread_demo_RR_max_min

Change below line 98 and line 99 with different combination and compile again, there are 6 combinations as below table shows,

Item	Schedule Policy	Thread 1 priority	Thread 2 priority
1	SCHED_FIFO	max_prio	min_prio
2	SCHED_FIFO	max_prio	max_prio
3	SCHED_FIFO	min_prio	max_prio
4	SCHED_RR	max_prio	min_prio
5	SCHED_RR	max_prio	max_prio
6	SCHED_RR	min_prio	max_prio

Transfer the executable files to Embedded Linux S3C2440 board by TFTP,

[root@mini2440 /root]# tftp -g -r thread_demo_RR_max_min 192.168.0.1
thread_demo_RR_max_m 100% |********************************************************************************************************************************************| 10564   0:00:00 ETA
[root@mini2440 /root]# tftp -g -r thread_demo_RR_max_max 192.168.0.1
thread_demo_RR_max_m 100% |********************************************************************************************************************************************| 10564   0:00:00 ETA
[root@mini2440 /root]# tftp -g -r thread_demo_RR_min_max 192.168.0.1
thread_demo_RR_min_m 100% |********************************************************************************************************************************************| 10564   0:00:00 ETA
[root@mini2440 /root]# tftp -g -r thread_demo_FIFO_min_max 192.168.0.1
thread_demo_FIFO_min 100% |********************************************************************************************************************************************| 10574   0:00:00 ETA
[root@mini2440 /root]# tftp -g -r thread_demo_FIFO_max_max 192.168.0.1
thread_demo_FIFO_max 100% |********************************************************************************************************************************************| 10574   0:00:00 ETA
[root@mini2440 /root]# tftp -g -r thread_demo_FIFO_max_min 192.168.0.1
thread_demo_FIFO_max 100% |********************************************************************************************************************************************| 10574   0:00:00 ETA
[root@mini2440 /root]# chmod +x thread_demo*
[root@mini2440 /root]#

The test result on each executable is as below,

[root@mini2440 /root]# ./thread_demo_FIFO_max_min
min/max prio (1, 99)
count1 = 10000, count2 = 0
[root@mini2440 /root]# ./thread_demo_FIFO_max_max
min/max prio (1, 99)
count1 = 10000, count2 = 0
[root@mini2440 /root]# ./thread_demo_FIFO_min_max
min/max prio (1, 99)
count1 = 4423, count2 = 5578
[root@mini2440 /root]# ./thread_demo_RR_max_max
min/max prio (1, 99)
count1 = 6885, count2 = 3116
[root@mini2440 /root]# ./thread_demo_RR_max_min
min/max prio (1, 99)
count1 = 10000, count2 = 0
[root@mini2440 /root]# ./thread_demo_RR_min_max
min/max prio (1, 99)
count1 = 4433, count2 = 5568
[root@mini2440 /root]#

Multi-Thread Application debugging test

Tested both Non-stop mode and All-stop mode for debugging.

Multi-thread Non-stop mode debugging

Start gdbserver with the application on the Embedded Linux S3C2440 board,

[root@mini2440 /root]# gdbserver :1234 thread_demo_RR_max_min
Process thread_demo_RR_max_min created; pid = 515
Listening on port 1234

On the host, start the gdb,

[root@localhost]# arm-linux-gdb thread_demo_RR_max_min
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-buildroot-linux-uclibcgnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/iot/mini2440/ready-built/lab7/thread_demo_RR_max_min...done.
(gdb) help set non-stop
Set whether gdb controls the inferior in non-stop mode.
When debugging a multi-threaded program and this setting is
off (the default, also called all-stop mode), when one thread stops
(for a breakpoint, watchpoint, exception, or similar events), GDB stops
all other threads in the program while you interact with the thread of
interest.  When you continue or step a thread, you can allow the other
threads to run, or have them remain stopped, but while you inspect any
thread's state, all threads stop.In non-stop mode, when one thread stops, other threads can continue to run freely. You'll be able to step each thread independently,
leave it stopped or free to run as needed.

Set GDB to Non-stop mode,

(gdb) set target-async 1
(gdb) set pagination off
(gdb) set non-stop on
(gdb) show non-stop
Controlling the inferior in non-stop mode is on.

Set sysroot and attach to the Embedded Linux S3C2440 board,

(gdb) set sysroot /home/iot/mini2440/buildroot-2013.02/output/target/
(gdb) target remote 192.168.0.11:1234
Remote debugging using 192.168.0.11:1234
[New Thread 515]
(gdb)
[Thread 515] #1 stopped.
0xb6ff1e40 in ?? ()

The [Thread 515] is the main thread, set one break point at the function void *thread_func1(void) of thread 1.

(gdb) break thread_demo_RR_max_min.c:15
Breakpoint 1 at 0x8808: file thread_demo_RR_max_min.c, line 15.
(gdb) c
Continuing.
[New Thread 516]Breakpoint 1, thread_func1 () at thread_demo_RR_max_min.c:18
18		while(times > 0)

New thread 516 [New Thread 516] is thread 2.

(gdb) c
Continuing.
Cannot execute this command while the selected thread is running.

The main thread thread 1 is the first thread to run and it is the thread under debugging,
Below command is to list all thread,

(gdb) info threadsId   Target Id         Frame2    Thread 516        thread_func1 () at thread_demo_RR_max_min.c:18
* 1    Thread 515        (running)

The thread with * is the thread under debug and it is running, and thread 1 stops at the break point.
Switch to thread 2 and list the information again.

(gdb) thread 2
[Switching to thread 2 (Thread 516)]
#0 thread_func1 () at thread_demo_RR_max_min.c:18
18		while(times > 0)
(gdb) info threadsId   Target Id         Frame
* 2    Thread 516        thread_func1 () at thread_demo_RR_max_min.c:181    Thread 515        (running)

Since count2 is 10000, it indicates thread 2 has exited. The main thread is waiting for Thread 1 to finish.

(gdb) print count1
$1 = 0
(gdb) print count2
$2 = 10000
(gdb) step
31		return (void*) NULL;
(gdb) c
Continuing.
[Inferior 1 (process 515) exited normally]
(gdb) step
The program is not being run.
(gdb) info thread
No threads.
(gdb) q
[root@localhost]#

The S3C2440 board displays as,

[root@mini2440 /root]# gdbserver :1234 thread_demo_RR_max_min
Process thread_demo_RR_max_min created; pid = 515
Listening on port 1234
Remote debugging from host 192.168.0.1
min/max prio (1, 99)
count1 = 0, count2 = 10000Child exited with status 0
GDBserver exiting
[root@mini2440 /root]#

Multi-thread All-stop mode debugging

If use All-stop mode for debugging, the procedure is as below, set the break point at the same location, we can see when step debugging, all threads stopped and continue at the same time, when stepping the thread under debugging, all other threads have chance to run. GDB automatically switch to the thread which stops.

[root@localhost]# arm-linux-gdb thread_demo_RR_max_min
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-buildroot-linux-uclibcgnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/iot/mini2440/ready-built/lab7/thread_demo_RR_max_min...done.
(gdb) target remote 192.168.0.11
192.168.0.11: No such file or directory.
(gdb) target remote 192.168.0.11:1234
Remote debugging using 192.168.0.11:1234
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0xb6ff1e40 in ?? ()
(gdb) set sysroot /home/iot/mini2440/buildroot-2013.02/output/target/
Reading symbols from /home/iot/mini2440/buildroot-2013.02/output/target/lib/ld-uClibc.so.0...(no debugging symbols found)...done.
Loaded symbols for /home/iot/mini2440/buildroot-2013.02/output/target/lib/ld-uClibc.so.0
(gdb) break thread_demo_RR_max_min.c:15
Breakpoint 1 at 0x8808: file thread_demo_RR_max_min.c, line 15.
(gdb) c
Continuing.
[New Thread 522]
[Switching to Thread 522]Breakpoint 1, thread_func1 () at thread_demo_RR_max_min.c:18
18		while(times > 0)
(gdb) info threadsId   Target Id         Frame
* 2    Thread 522        thread_func1 () at thread_demo_RR_max_min.c:181    Thread 521        0xb6fe3fa8 in pthread_create () from /home/iot/mini2440/buildroot-2013.02/output/target/lib/libpthread.so.0
(gdb) step
20			int i = 0;
(gdb) info threads
[New Thread 523]Id   Target Id         Frame3    Thread 523        0x000088d8 in thread_func2 () at thread_demo_RR_max_min.c:41
* 2    Thread 522        thread_func1 () at thread_demo_RR_max_min.c:201    Thread 521        0xb6fe4da4 in pthread_join () from /home/iot/mini2440/buildroot-2013.02/output/target/lib/libpthread.so.0
(gdb) c
Continuing.
[Inferior 1 (process 521) exited normally]
(gdb) info threads
No threads.
(gdb) q
[root@localhost]#

The display on the Embedded Linux S3C2440 board is as below,

[root@mini2440 /root]# gdbserver :1234 thread_demo_RR_max_min
Process thread_demo_RR_max_min created; pid = 521
Listening on port 1234
Remote debugging from host 192.168.0.1
min/max prio (1, 99)
count1 = 2609, count2 = 7392Child exited with status 0
GDBserver exiting
[root@mini2440 /root]#

Reference

Embedded Linux S3C2440 environment setup
Embedded Linux S3C2440 Environment Startup
Embedded Linux S3C2440 Build and Boot an Image
Embedded Linux S3C2440 Application Development and Debugging
Embedded Linux S3C2440 Networking
Embedded Linux S3C2440 – Kernel Module
Embedded Linux S3C2440 – Kernel Debugging
Linux SCHED_OTHER, SCHED_FIFO and SCHED_RR - differences