Native Kernel Debugging with Acid
Tad Hunt
tad@plan9.bell-labs.com
Lucent Technologies Inc
(Revised 22 May 2000 by Vita Nuova)
Introduction
This tutorial provides an introduction to the Acid debugger. It assumes that you are familiar with the features of a typical source-level debugger. The Acid debugger is built round a command language with a syntax similar to C. This tutorial is not an introduction to Acid as a whole, but offers a brief tour of the basic built in and standard library functions, especially those needed for debugging native Inferno kernels on a target board.
Acid was originally developed by Phil Winterbottom to help debug multi-threaded programs in the concurrent language Alef, and provide more sophisticated debugging for C programs. In the paper Acid: A Debugger Built From a Language, Winterbottom discusses Acid’s design, including some worked examples of unusual applications of Acid to find memory leaks and assist code coverage analysis. Following that is the Acid Reference Manual, also by Phil Winterbottom, which gives a more precise specification of the Acid debugging language and its libraries.
Preliminaries -- the environment
Acid runs under the host operating system used for cross-development, in the same way as the Inferno compilers. Before running either compilers or Acid, the following environment variables must be set appropriately:
They might be set by a login shell profile (eg, Unix .profile, or Plan 9 lib/profile). Also ensure that the directory
$ROOT/$SYSHOST/$OBJTYPE/bin
is on your search path. For example, on a Solaris sparc, one might use:
ROOT=inferno_root
SYSHOST=Solaris
OBJTYPE=sparc
ACIDLIB=$ROOT/lib/acid
PATH=$ROOT/$SYSHOST/$OBJTYPE/bin:$PATH
export ROOT ACIDLIB PATH OBJTYPE SYSHOST
where inferno_root is the directory in which Inferno lives (eg, /usr/inferno).
An Example Program
The first example is not kernel code, but a small program that will be compiled but not run, to demonstrate basic Acid commands for source and object file inspection. The code is shown below:
int
factorial(int n)
{
if (n == 1)
return 1;
return n * factorial(n-1);
}
int f;
void
main(void)
{
f = factorial(5);
}
void
_main(void)
{
main();
}
Compiling and Linking
The first step is to create an executable. The example shows the process for creating ARM executables. Substitute the appropriate compiler and linker for other cpu types.
% 5c factorial.c
% 5l -o factorial factorial.5
% ls
factorial
factorial.5
factorial.c
Starting Acid
Even without the target machine on which to run the program, many Acid features are available. The following command starts debugging the factorial executable. Note that, upon startup, Acid will attempt to load some libaries from the directory specified in the ACIDLIB environment variable (defaults to /usr/inferno/lib/acid). It will also attempt to load the file $HOME/lib/acid, in which you can place commands to be executed during startup.
% acid factorial
factorial:Arm plan 9 executable
$ROOT/lib/acid/port
$ROOT/lib/acid/arm
acid:
Exploring the Executable
To find out what symbols are in the program:
acid: symbols("")
etext T 0x00001068
f D 0x00002000
setR12 D 0x00002ffc
end B 0x00002008
bdata D 0x00002000
edata D 0x00002008
factorial T 0x00001020
main T 0x00001048
_main T 0x0000105c
acid:
The output from the symbols() function is similar to the output from the nm(10.1) command. The first column is the symbol name, the second column gives the section the symbol is in, and the third column is the address of the symbol.
There is also a symbols global variable. Variables and functions can have the same names. It holds the list of symbol information that the symbols function uses to generate the table:
acid: symbols
{{"etext", T, 0x00001068}, {"f", D, 0x00002000}, {"setR12", D, 0x00002ffc},
{"end", B, 0x00002008}, {"bdata", D, 0x00002000}, {"edata", D, 0x00002008},
{"factorial", T, 0x00001020}, {"main", T, 0x00001048}, {"_main", T, 0x00001
05c}}
acid:
In large programs, finding the symbol you are interested in from a list that may be thousands of lines long would be difficult. The string argument of symbols() is a regular expression against which to match symbols. All symbols that contain the pattern will be displayed. For example:
acid: symbols("main")
main T 0x00001048
_main T 0x0000105c
acid: symbols("^main")
main T 0x00001048
acid:
The symbols function is written in the acid command language and lives in the port library ($ACIDLIB/port).
defn symbols(pattern)
{
local l, s;
l = symbols;
while l do {
s = head l;
if regexp(pattern, s[0]) then
print(s[0], "", s[1], "", s[2], "0);
l = tail l;
}
}
Acid retrieves the list of symbols from the executable and turns each one into a global variable whose value is the address of the symbol. If the symbol clashes with a builtin name or keyword or a previously defined function, enough $ characters are prepended to the name to make it unique. The list of such renamings is printed at startup.
Most acid functions operate on addresses. For example, to view the source code for a given address, use the src function:
acid: src(main)
/usr/jrf/factorial.c:10
5 return n * factorial(n-1);
6 }
7
8 int f;
9 void
>10 main(void)
11 {
12 f = factorial(5);
13 }
14
15 void
The src(addr) function displays a section of source code, with the line containing the address passed as an argument in the middle of the display. To print the assembly code beginning at a given address, use the asm() function.
acid: asm(factorial)
factorial 0x00001020 MOVW.W R14,#-0x8(R13)
factorial+0x4 0x00001024 CMP.S $#0x1,R0
factorial+0x8 0x00001028 MOVW.EQ $#0x1,R0
factorial+0xc 0x0000102c RET.EQ.P #0x8(R13)
factorial+0x10 0x00001030 MOVW R0,n+0(FP)
factorial+0x14 0x00001034 SUB $#0x1,R0,R0
factorial+0x18 0x00001038 BL factorial
factorial+0x1c 0x0000103c MOVW n+0(FP),R2
factorial+0x20 0x00001040 MUL R2,R0,R0
factorial+0x24 0x00001044 RET.P #0x8(R13)
main 0x00001048 MOVW.W R14,#-0x8(R13)
acid:
The output contains the symbolic address (symbol name+offset, where symbol name is the name of the enclosing function) followed by the absolute address, followed by the disassembled code. The asm(addr) function prints the assembly beginning at addr and ending after either 30 lines have been printed, or the end of the function has been reached. The casm() function continues the assembly listing from where it left off, even past the end of the function and into the next one.
acid: casm()
main+0x4 0x0000104c MOVW $#0x5,R0
main+0x8 0x00001050 BL factorial
main+0xc 0x00001054 MOVW R0,$f-SB(SB)
main+0x10 0x00001058 RET.P #0x8(R13)
_main 0x0000105c MOVW.W R14,#-0x4(R13)
acid:
All the functions presented so far are written in the acid command language. To see the source of a comand written in the acid command language, use the builtin command whatis [name ]. It prints the definition of the optional argument name. If name is an Acid builtin, whatis prints builtin function.
acid: whatis casm
defn casm() {
asm(lasmaddr);
}
acid:
acid: whatis atof
builtin function
acid:
If name is a variable, it prints the type of variable, and for the integer type, gives the format code used to print the value:
acid: whatis pid
integer variable format D
acid:
With no arguments, whatis lists all available functions:
acid: whatis
Bsrc bpmask follow new sh
_bpconddel bpneq func newproc source
_bpcondset bpor gpr next spr
_stk bpprint include notestk spsrch
access bppush interpret params src
acidinit bpset itoa pcfile start
addsrcdir bptab kill pcline startstop
asm casm kstk pfl status
atof cont labstk print stk
atoi debug line printto stmnt
bpaddr dump linkreg procs stop
bpand error lkstk rc stopped
bpconddel file locals readfile strace
bpcondset filepc lstk reason symbols
bpdel findsrc map regexp waitstop
bpderef fmt match regs
bpeq fnbound mem setproc
acid:
The Bsrc(addr) function brings up an editor on the line containing addr. It simply invokes a shell script named B that takes two arguments, -line and file The shell script invokes $EDITOR + line file. If unset, EDITOR defaults to vi. The shell script, or the Bsrc function can be easily rewritten to work with your favorite editor.
Entering a symbol name by itself will print the address of the symbol. Prefixing the symbol name with a * will print the value at the address in the variable. Continuing to use our factorial example:
acid: f
0x00002000
acid: *f
0x00000000
acid:
Remote Debugging
Now that you have a basic understanding of how to explore the executable, it is time to examine a real remote debugging session.
We’ll use the SA1100 keyboard driver as an example. Examining the kernel configuration file, you’ll see the following:
dev
keyboard
link driver/keyboard port
scanfujn860 kbd.h keycodes.h
link ./../driver plat
kbdfujitsu ./../common/ssp.h \
/driver/keyboard/kbd.h \
/driver/keyboard/keycodes.h
port
const char *defaultkeyboard = "fujitsu";
const char *defaultkeytable = "scanfujn860";
int debugkeys = 1; /* 1 = enabled, 0 = disabled */
This describes the pieces of the keyboard driver which are linked into the kernel. The source code lives in two places, $ROOT/os/driver/keyboard, and $ROOT/os/plat/sa1100/driver.
The next step is to build a kernel. Use the mk target acid to ensure that the Acid symbolic debugging data is produced. For example:
% mk ’CONF=sword’ acid isword.p9.gz
This creates the Acid file isword.acid, containing Acid declarations describing kernel structures, the kernel executable isword.p9; and finally gzips a copy of the kernel in isword.p9.gz to load onto the device. Next, copy the gzipped image onto the device and then boot it. Follow the directions found elsewhere for details of this process.
From a shell prompt on the target device, start the remote debugger by writing the letter r (for run) to #b/dbgctl. Next, start Acid in remote debug mode, specifying the serial port it is connected to with the -R option. $CONF is the name of the configuration file used, for example sword.
% acid -R /dev/cua/b -l i$CONF.acid i$CONF
isword:Arm plan 9 executable
$ROOT/lib/acid/port
i$CONF.acid
$ROOT/lib/acid/arm
/usr/jrf/lib/acid
acid:
You are now debugging the kernel that is running on the target device. All of the previously listed commands will work as described before, in addition, there are many more commands available.
Kernel Process Listing
To get a list of kernel processes, use the ps() function:
acid: ps()
PID PC PRI STATE NAME
1 0x00054684 5 Queueing interp
2 0x00000000 1 Wakeme consdbg
3 0x00000000 5 Wakeme tcpack
4 0x00000000 5 Wakeme Fs.sync
5 0x00000000 4 Wakeme touchscreen
6 0x00054684 5 Queueing dis
7 0x00059788 5 Wakeme dis
8 0x00054684 5 Queueing dis
9 0x00054684 5 Queueing dis
10 0x00054684 5 Wakeme dis
11 0x0004c26c 1 Running dbg
acid:
The PC column shows the address the process was executing at when the ps command retrieved statistics on it. The PRI column lists process priorities. The smaller the number the higher the process priority. Notice that the kernel process (kproc) running the debugger is the highest priority process in the system. The only process you will ever see in the Running state while executing the ps command will be the debugger, since it is gathering information about the other processes.
Breakpoints
Breakpoints in Inferno, unlike most traditional kernel debuggers, are conditional breakpoints. There are minimally two conditions which must be met. These conditions are address and process id. A breakpoint will only be taken when execution for a specific kernel process reaches the specified address. The user can create additional conditions that are evaluated if the address and process id match. If evaluation of these conditions result in a nonzero value, the breakpoint is taken, otherwise it is ignored, and execution continues.
Again, the best way to proceed is with an example:
acid: setproc(7)
The setproc(pid) function selects a kproc to which later commands will be applied; the one with process ID (pid) in this case.
acid: bpset(keyboardread)
Waiting...
7: stopped flush8to4+0x18c MOVW (R3<<#4),R3
After selecting a kproc, we set a breakpoint at the address referred to by the keyboardread symbol. As described before, the value of a global variable created from a symbol in the executable is the address of the symbol. In this case the address is the first instruction in the keyboardread() function. Notice that setting a breakpoint stops the kproc from executing. A bit later, we’ll see how to get it to continue execution.
Next, display the list of breakpoints using bptab():
acid: bptab()
ID PID ADDR CONDITIONS
0 7 keyboardread 0x0003c804 { }
The first column is a unique number that identifies the breakpoint. The second column is the process ID in which the breakpoint will be taken. The third and fourth columns are the address of the breakpoint, first in symbolic form, then in numeric form. Finally, the last column is a list of conditions to evaluate whenever the kproc specified in the PID column hits the the address specified in the ADDR column. When they match, the list of conditions is evaluated. If the result is nonzero, the breakpoint is taken. Since we used the simplified breakpoint creation function, bpset() , there are no additional conditions. Later on, we’ll see how to set conditional breakpoints.
Start the selected kproc executing again, and wait for it to hit the breakpoint.
acid: cont()
The cont() function will not return until a breakpoint has been hit, and there is no way to interrupt it. This means you should only set breakpoints that will be hit, otherwise you’ll have to reboot the target device and restart your debugging session.
To continue our example, repeatedly hit new line (return, enter) on the keyboard on the target device, until the breakpoint occurs:
break 0: pid 7: stopped keyboardread SUB $#0xa4,R13,R13
acid:
This message, followed by the interactive prompt returning tells you that a breakpoint was hit. It gives the breakpoint id, the kernel process id, then the symbolic address at which execution halted, followed by the disassembly of the instruction at that address.
The kstk() function prints a kernel stack trace, beginning with the current frame, all the way back to the call that started the kproc. For each function, it gives the name name, arguments, source file, and line number, followed by the symbolic address, source file, and line number of the caller.
acid: kstk()
At pc:247812:keyboardread /usr/inferno/os/driver/keyboard/devkey
board.c:350
keyboardread(offset=0x0000009d,buf=0x001267f8,n=0x00000001) /usr
/inferno/os/driver/keyboard/devkeyboard.c:350
called from kchanio+0x9c /usr/inferno/os/port/sysfile.c:
75
kchanio(buf=0x001267f8,n=0x00000001,mode=0x00000000) /usr/infern
o/os/port/sysfile.c:64
called from consread+0x144 /usr/inferno/os/driver/port/d
evcons
consread(offset=0x0000009d,buf=0x0043d4fc,n=0x00000400,c=0x0044e
c38) /
usr/inferno/os/driver/port/devcons.c:357
called from kread+0x164 /usr/inferno/os/port/sysfile.c:2
97
kread(fd=0x00000006,n=0x00000400,va=0x0043d4fc) /usr/inferno/os/
port/sysfile.c:272
called from Sys_read+0x84 /usr/inferno/os/port/inferno.c
:244
Sys_read() /usr/inferno/os/port/inferno.c:229
called from mcall+0x98 /usr/inferno/interp/xec.c:590
mcall() /usr/inferno/interp/xec.c:569
called from xec+0x128 /usr/inferno/interp/xec.c:1098
xec(p=0x0044edd8) /usr/inferno/interp/xec.c:1077
called from vmachine+0xbc /usr/inferno/os/port/dis.c:706
vmachine() /usr/inferno/os/port/dis.c:677
called from _main+0x50 /usr/inferno/os/plat/sa1100/infern
o/main.c:237
acid:
There is another kernel stack dump function, lkstk() which shows the same information as kstk() plus the names and values of local variables. Notice that in addition to the ‘called from’ information, each local variable and its value is listed on a line by itself.
acid: lkstk()
At pc:247812:keyboardread /usr/inferno/os/driver/keyboard/devkeyboard.
c:350
keyboardread(offset=0x00000018,buf=0x001267f9,n=0x00000001) /usr/inferno
/os/driver/keyboard/devkeyboard.c:350
called from kchanio+0x9c /usr/inferno/os/port/sysfile.c:75
tmp=0x00000000
kchanio(buf=0x001267f9,n=0x00000001,mode=0x00000000) /usr/inferno/os/por
t/sysfile.c:64
called from consread+0x144 /usr/inferno/os/driver/port/devcons
c=0x0045a858
r=0x00000001
consread(offset=0x00000015,buf=0x0043d4fc,n=0x00000400,c=0x0044ec38) /us
r/inferno/os/driver/port/devcons.c:357
called from kread+0x164 /usr/inferno/os/port/sysfile.c:297
r=0x00000001
ch=0x0000006c
eol=0x00000000
i=0x00000000
mt=0x60000053
tmp=0x0007317c
l=0x0044ec38
p=0x00049754
kread(fd=0x00000006,n=0x00000400,va=0x0043d4fc) /usr/inferno/os/port/sys
file.c:272
called from Sys_read+0x84 /usr/inferno/os/port/inferno.c:244
c=0x0044ec38
dir=0x00000000
Sys_read() /usr/inferno/os/port/inferno.c:229
called from mcall+0x98 /usr/inferno/interp/xec.c:590
f=0x0044eff0
n=0x00000400
mcall() /usr/inferno/interp/xec.c:569
called from xec+0x128 /usr/inferno/interp/xec.c:1098
ml=0x0043d92c
f=0x0044eff0
xec(p=0x0044edd8) /usr/inferno/interp/xec.c:1077
called from vmachine+0xbc /usr/inferno/os/port/dis.c:706
vmachine() /usr/inferno/os/port/dis.c:677
called from _main+0x50 /usr/inferno/os/plat/sa1100/inferno/main.
c:237
r=0x0044edd8
o=0x0044ee50
The step() function allows the currently selected process to execute a single instruction, and then stop.
acid: step()
break 1: pid 7: stopped keyboardread+0x4 MOVW R14,#0x0(R13)
acid:
The bpdel( id) command deletes the breakpoint identified by id:
acid: bpdel(0)
The start() command places the kproc back into the state it was in when it was stopped.
acid: start(7)
acid:
Now lets look at how to set conditional breakpoints.
acid: bpcondset(7, keyboardread, {bppush(_startup), bpderef()})
Waiting...
7: stopped sched+0x20 MOVW #0xffffff70(R12),R6
acid: bptab()
ID PID ADDR CONDITIONS
0 7 keyboardread 0x0003c804 {
{"p", 0x00008020}
{"*", 0x00000000} }
acid: *_startup = 0
acid: cont()
Conditional breakpoints are set with bpcondset() bptab() function shows the list of conditions to apply. The list is a bit confusing to read, but the p"" means push and the *"" means dereference.
No matter how much you type on the keyboard, this particular breakpoint will never be taken. That’s because before continuing, we set the value at the address _startup to zero, so whenever execution reaches keyboardread in kproc number 7, it pushes the address _startup, then pops it and pushes the word at that address. Since the top of the stack is zero, the breakpoint is ignored.
This contrived example may not be all that useful, but you can use a similar method in your driver to examine some state before making the decision to take the breakpoint.
Examining Registers
There are three commands to dump registers: gpr(), spr() and regs(). The gpr() function dumps the general purpose registers, spr() dumps special purpose registers (such as the PC and LINK registers), and regs() dumps both:
acid: regs()
PC 0x0004a3b0 sched+0x20 /home/tad/inf2.1/os/port/proc.c:82
LINK 0x0004b8e8 kchanio+0xa4 /home/tad/inf2.1/os/port/sysfile.c:75
SP 0x00453c4c
R0 0x00458798 R1 0x000fdf9c R2 0x0003c804 R3 0x00000000
R4 0xffffffff R5 0x00000001 R6 0x00458798 R7 0x00000001
R8 0x001267f8 R9 0x00000000 R10 0x0044ee50 R11 0x00029f9c
R12 0x000fc854
acid:
Complex Types
When reading in the symbol table, Acid treats all of the symbols in the executable as pointers to integers. This is fine for global integer variables, but it makes examining more complex types difficult. Luckily there is a solution. Acid allows you to create a description for more complex types, and a function which will automatically be called for these complex types. In fact, the compiler can automatically generate the acid code to describe these complex types. For example, if we wanted to print out the devtab structure for the keyboard driver, we can just give its name:
acid: whatis keyboarddevtab
integer variable format a complex Dev
acid: keyboarddevtab
dc 107
name 0x0010e0ea
reset 0x0003c3fc
init 0x0003c438
attach 0x0003c5dc
clone 0x000480d0
walk 0x0003c600
stat 0x0003c640
open 0x0003c680
create 0x0004881c
close 0x0003c768
read 0x0003c804
bread 0x0004883c
write 0x0003c968
bwrite 0x00048900
remove 0x00048978
wstat 0x00048998
acid:
Acid knows the keyboarddevtab variable is of type Dev, and it prints it by invoking the function Dev(keyboarddevtab).
acid: whatis Dev
complex Dev {
’D’ 0 dc;
’X’ 4 name;
’X’ 8 reset;
’X’ 12 init;
’X’ 16 attach;
’X’ 20 clone;
’X’ 24 walk;
’X’ 28 stat;
’X’ 32 open;
’X’ 36 create;
’X’ 40 close;
’X’ 44 read;
’X’ 48 bread;
’X’ 52 write;
’X’ 56 bwrite;
’X’ 60 remove;
’X’ 64 wstat;
};
defn Dev(addr) {
complex Dev addr;
print("\tdct",addr.dc,"\n");
print("\tnamet",addr.nameX,"\n");
print("\tresett",addr.resetX,"\n");
print("\tinitt",addr.initX,"\n");
print("\tattacht",addr.attachX,"\n");
print("\tclonet",addr.cloneX,"\n");
print("\twalkt",addr.walkX,"\n");
print("\tstatt",addr.statX,"\n");
print("\topent",addr.openX,"\n");
print("\tcreatet",addr.createX,"\n");
print("\tcloset",addr.closeX,"\n");
print("\treadt",addr.readX,"\n");
print("\tbreadt",addr.breadX,"\n");
print("\twritet",addr.writeX,"\n");
print("\tbwritet",addr.bwriteX,"\n");
print("\tremovet",addr.removeX,"\n");
print("\twstatt",addr.wstatX,"\n");
}
Notice the complex type definition and the function to print the type both have the same name. If we know that an address is the address of a complex type, even though acid may not (say we’re storing multiple types of data in a void pointer), we can print the complex type by calling the type printing function ourselves.
acid: print(fmt(keyboarddevtab, ’X’))
0x00106d50
acid: Dev(0x00106d50)
dc 107
name 0x0010e0ea
reset 0x0003c3fc