Training > Linux Kernel Development > Linux Kernel Debugging and Security (LFD440)
INSTRUCTOR-LED COURSE

Linux Kernel Debugging and Security (LFD440)

This instructor-led course focuses on the important tools used for debugging and monitoring the kernel, and how security features are implemented and controlled.

Who Is It For

This course is for experienced developers who need to understand the methods and internal infrastructure of the Linux kernel.
read less read more
What You’ll Learn

This four day course includes extensive hands-on exercises and demonstrations designed to give you the necessary tools to develop and debug Linux kernel code.
read less read more
What It Prepares You For

You will walk away from this course with a solid understanding of Linux kernel. debugging techniques and tools.
read less read more
Course Outline
Introduction
Objectives
Who You Are
The Linux Foundation
Linux Foundation Training
Certification Programs and Digital Badging
Linux Distributions
Platforms
Preparing Your System
Using and Downloading a Virtual Machine
Things change in Linux
Documentation and Links
Preliminaries
Procedures
Kernel Versions
Kernel Sources and Use of git
How to Work in OSS Projects **
Overview on How to Contribute Properly
Stay Close to Mainline for Security and Quality
Study and Understand the Project DNA
Figure Out What Itch You Want to Scratch
Identify Maintainers and Their Work Flows and Methods
Get Early Input and Work in the Open
Contribute Incremental Bits, Not Large Code Dumps
Leave Your Ego at the Door: Don’t Be Thin-Skinned
Be Patient, Develop Long Term Relationships, Be Helpful
Kernel Features
Components of the Kernel
User-Space vs. Kernel-Space
What are System Calls?
Available System Calls
Scheduling Algorithms and Task Structures
Process Context
Labs
Monitoring and Debugging
Debuginfo Packages
Tracing and Profiling
sysctl
SysRq Key
oops Messages
Kernel Debuggers
debugfs
Labs
The proc Filesystem **
What is the proc Filesystem?
Creating and Removing Entries
Reading and Writing Entries
The seq file Interface **
Labs
kprobes
kprobes
kretprobes
SystemTap **
Labs
Ftrace
What is ftrace?
ftrace, trace-cmd and kernelshark
Available Tracers
Using ftrace
Files in the Tracing Directory
Tracing Options
Printing with trace printk()
Trace Markers
Dumping the Buffer
trace-cmd
Labs
Perf
What is perf?
perf stat
perf list
perf record
perf report
perf annotate
perf top
Labs
eBPF
BFP
eBPF
Installation
bcc Tools
bpftrace
Labs
Crash
Crash
Main Commands
Labs
Kernel Core Dumps
Generating Kernel Core Dumps
kexec
Setting Up Kernel Core Dumps
Labs
Virtualization**
What is Virtualization?
Rings of Virtualization
Hypervisors
QEMU
What is QEMU?
Emulated Architectures
Image Formats
Third Party Hypervisor Integration
Labs
Linux Kernel Debugging Tools
Linux Kernel (built-in) tools and helpers
kdb
qemu+gdb
kgdb: hardware+serial+gdb
Labs
Embedded Linux**
Embedded and Real Time Operating Systems
Why Use Linux?
Making a Small Linux Environment
Real Time Linuxes
Notifiers**
What are Notifiers?
Data Structures
Callbacks and Notifications
Creating Notifier Chains
Labs
CPU Frequency Scaling**
What is Frequency and Voltage Scaling?
Notifiers
Drivers
Governors
Labs
Netlink Sockets**
What are netlink Sockets?
Opening a netlink Socket
netlink Messages
Labs
Introduction to Linux Kernel Security
Linux Kernel Security Basics
Discretionary Access Control (DAC)
POSIX ACLs
POSIX Capabilities
Namespaces
Linux Security Modules (LSM)
Netfilter
Cryptographic Methods
The Kernel Self Protection Project
Linux Security Modules (LSM)
What are Linux Security Modules?
LSM Basics
LSM Choices
How LSM Works
An LSM Example: Tomoyo
SELinux
SELinux
SELinux Overview
SELinux Modes
SELinux Policies
Context Utilities
SELinux and Standard Command Line Tools
SELinux Context Inheritance and Preservation**
restorecon**
semanage fcontext**
Using SELinux Booleans**
getsebool and setsebool**
Troubleshooting Tools
Labs
AppArmor
What is AppArmor?
Checking Status
Modes and Profiles
Profiles
Utilities
Netfilter
What is netfilter?
Netfilter Hooks
Netfilter Implementation
Hooking into Netfilter
Iptables
Labs
The Virtual File System
What is the Virtual File System?
Available Filesystems
Special Filesystems
The tmpfs Filesystem
The ext2/ext3 Filesystem
The ext4 Filesystem
The btrfs Filesystem
Common File Model
VFS System Calls
Files and Processes
Mounting Filesystems
Filesystems in User-Space (FUSE)**
What is FUSE?
Writing a Filesystem
Labs
Journaling Filesystems**
What are Journaling Filesystems?
Available Journaling Filesystems
Contrasting Features
Labs
Closing and Evaluation Survey
Evaluation Survey
Appendices
Kernel Architecture I
UNIX and Linux **
Monolithic and Micro Kernels
Object-Oriented Methods
Main Kernel Tasks
User-Space and Kernel-Space
Kernel Programming Preview
Error Numbers and Getting Kernel Output
Task Structure
Memory Allocation
Transferring Data between User and Kernel Spaces
Linked Lists
Jiffies
Labs
Modules
What are Modules?
A Trivial Example
Compiling Modules
Modules vs Built-in
Module Utilities
Automatic Loading/Unloading of Modules
Module Usage Count
Module Licensing
Exporting Symbols
Resolving Symbols **
Labs
Kernel Architecture II
Processes, Threads, and Tasks
Kernel Preemption
Real Time Preemption Patch
Labs
Kernel Configuration and Compilation
Installation and Layout of the Kernel Source
Kernel Browsers
Kernel Configuration Files
Kernel Building and Makefiles
initrd and initramfs
Labs
Kernel Style and General Considerations
Coding Style
Using Generic Kernel Routines and Methods
Making a Kernel Patch
sparse
Using likely() and unlikely()
Writing Portable Code, CPU, 32/64-bit, Endianness
Writing for SMP
Writing for High Memory Systems
Power Management
Keeping Security in Mind
Labs
Race Conditions and Synchronization Methods
Concurrency and Synchronization Methods
Atomic Operations
Bit Operations
Spinlocks
Seqlocks
Disabling Preemption
Mutexes
Semaphores
Completion Functions
Read-Copy-Update (RCU)
Reference Counts
Labs
Memory Addressing
Virtual Memory Management
Systems With and Without MMU and the TLB
Memory Addresses
High and Low Memory
Memory Zones
Special Device Nodes
NUMA
Paging
Page Tables
page structure
Labs
Memory Allocation
Requesting and Releasing Pages
Buddy System
Slabs and Cache Allocations
Memory Pools
kmalloc()
vmalloc()
Early Allocations and bootmem()
Memory Defragmentation
Labs

**
These sections may be considered in part or in whole as optional. They contain either background reference material, specialized topics, or advanced subjects. The instructor may choose to cover or not cover them depending on classroom experience and time constraints.
Learning Path
To make the most of this course, you should:

  • Be proficient in the C programming language.
  • Be familiar with basic Linux (UNIX) utilities such as ls, grep and tar.
  • Be comfortable using any of the available text editors (e.g. emacs, vi, etc.).
  • Experience with any major Linux distribution is helpful but not strictly required.
  • Have experience equivalent to having taken LFD420: Linux Kernel Internals and Development.

Pre-class preparation material will be provided before class.

Reviews
Aug 2020
The instructor is very knowledgeable.
Aug 2020
It introduced me to all the components, now I know what I don't know well. It was a good way to be introduced to all components.