Dtrace on OEL6.6 is working!

I was close to giving up. Because of the following things:

  • I could not find much on the Dtrace error I got.
  • I tried the commands I used last time on my VirtualBox with OEL6.6 and Oracle 12c and got a different error (could not even run dtrace -l).
  • The documentation send me the wrong way. (Spoke too soon. Missed it in the documentation. It’s in there after all, all the way down.)
  • I get Linux anxiety when things don’t work as they should.

Anyway. So previously I had Dtrace installed and I had an executable I could work with and the result of almost any run was: entry does not match any probes.

When you run dtrace -l you will see this:

[root@sequoia ~]# dtrace -l
1      dtrace                                     BEGIN
2      dtrace                                     END
3      dtrace                                     ERROR

And that’s not a lot. You can only dtrace dtrace. In TF manual it says that you need to run this command:

modprobe sdt

After running this, you add a Linux module to what you can see with Dtrace and you get a lot more so called providers. On dtrace -l I now get 34 lines:

[root@sequoia ~]# dtrace -l
1      dtrace                                     BEGIN
2      dtrace                                     END
3      dtrace                                     ERROR
4       sched    vmlinux               __schedule remain-cpu
5       sched    vmlinux               __schedule sleep
6       sched    vmlinux               __schedule preempt
7       sched    vmlinux               __schedule off-cpu
8        proc    vmlinux            __send_signal signal-discard
9        proc    vmlinux            __send_signal signal-send
10         io    vmlinux         __wait_on_buffer wait-done
11         io    vmlinux         __wait_on_buffer wait-start
12         io    vmlinux               _submit_bh start
13      sched    vmlinux             dequeue_task dequeue
14       proc    vmlinux         do_execve_common exec-success
15       proc    vmlinux         do_execve_common exec
16       proc    vmlinux         do_execve_common exec-failure
17       proc    vmlinux                  do_exit exit
18       proc    vmlinux                  do_exit lwp-exit
19       proc    vmlinux                  do_fork create
20       proc    vmlinux                  do_fork lwp-create
21       proc    vmlinux          do_sigtimedwait signal-clear
22        sdt    vmlinux          dtrace_sdt_perf measure
23         io    vmlinux       end_bio_bh_io_sync done
24      sched    vmlinux             enqueue_task enqueue
25      sched    vmlinux       finish_task_switch on-cpu
26       proc    vmlinux    get_signal_to_deliver signal-handle
27        sdt    vmlinux           psinfo_cleaner test
28       proc    vmlinux            schedule_tail lwp-start
29       proc    vmlinux            schedule_tail start
30      sched    vmlinux            set_user_nice change-pri
31      sched    vmlinux          sys_sched_yield surrender
32      sched    vmlinux           try_to_wake_up wakeup
33      sched    vmlinux     update_process_times tick
34      sched    vmlinux                 yield_to surrender

But syscall is not one of the available providers. And therefor I got those “does not match any probes” errors when I tried to use it (makes sense now). So I googled some more and read on other sites you can also run this:

modprobe systrace

And now all of a sudden, dtrace -l gives 634 lines and if you grep on “syscall” you see it’s amongst the providers now. And I can run something a little more complicated, like this:

dtrace -n 'syscall::write:entry {@[execname] = count();}'
dtrace: description 'syscall::write:entry ' matched 1 probe

At first it doesn’t do anything. But then, when you break it off with Ctrl-C, you get your results. This is the result during the startup of an Oracle database:

Timer                                            1
df                                               1
dtrace                                           1
oradism                                          2
rs:main Q:Reg                                    2
init                                             6
rtkit-daemon                                     7
bash                                            22
sqlplus                                         58
gnome-terminal                                  95
oracle                                        1220

What you see here is the count of executables running during the run of dtrace. I call that succes. I can finally put a tick in front of “Create a working version of dtrace” on my todo list.

And we’ve learned that during 11g database startup aparently oradism runs, as well as sqlplus (that much? that wasn’t me) and apparently the oracle executable runs a lot.

One more dtrace oneliner. Here is CPU usage per thread during the startup of an Oracle 11g database:

[root@sequoia ~]# dtrace -n 'sched:::on-cpu {@[execname] = count();}'
dtrace: description 'sched:::on-cpu ' matched 1 probe

hald                                                   1
khungtaskd                                             1
sh                                                     1
timesync                                               2
flush-252:0                                            4
gnome-power-man                                        4
memballoon                                             4
watchdog/0                                             5
oradism                                                6
df                                                     7
vmstats                                                7
init                                                   9
jbd2/dm-0-8                                            9
udisks-daemon                                         10
console-kit-dae                                       12
rtkit-daemon                                          12
JS Watchdog                                           13
gvfs-afc-volume                                       20
gnome-screensav                                       25
vminfo                                                25
dbus-daemon                                           28
automount                                             40
sqlplus                                               40
gnome-panel                                           44
MONITOR                                               56
kworker/0:2                                           90
Timer                                                 98
nautilus                                             105
gnome-settings-                                      167
gnome-terminal                                       230
kworker/0:1H                                         251
hald-addon-inpu                                      256
metacity                                             284
firefox                                              364
wnck-applet                                          375
rcu_sched                                            807
Xorg                                                1673
ksoftirqd/0                                         2361
dtrace                                              2395
swapper/0                                           3263
oracle                                              4575

A time of experimentation is at hand!


About Marcel-Jan Krijgsman

Ever since I started working with Oracle, I had an interest in Oracle database performance tuning. This led, eventually, to a four day training I made and gave for customers of Transfer Solutions. Since 2012 I work for Rabobank Nederland. A few years ago I also became interested in Oracle database security. All technology aside, it is my experience that security usually plays out on a political level. I'm a Oracle certified professional for the 8i, 9i, 10g and 11g databases and Oracle Database 11g Performance Tuning Certified Expert.
This entry was posted in Linux and tagged , , , , , , , . Bookmark the permalink.

One Response to Dtrace on OEL6.6 is working!

  1. Pingback: Fun with Dtrace: files Oracle opens on database startup | Marcel-Jan's Oracle Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s