Training > Linux Kernel Development > Linux Kernel Debugging and Security (LFD440)
INSTRUCTOR-LED COURSE

Linux Kernel Debugging and Security (LFD440)

This instructor-led course focuses on the important tools used for debugging and monitoring the kernel, and how security features are implemented and controlled.

Who Is It For

This course is for experienced developers who need to understand the methods and internal infrastructure of the Linux kernel.
read less read more
What You’ll Learn

This four day course includes extensive hands-on exercises and demonstrations designed to give you the necessary tools to develop and debug Linux kernel code.
read less read more
What It Prepares You For

You will walk away from this course with a solid understanding of Linux kernel. debugging techniques and tools.
read less read more
Course Outline
Expand All
Collapse All
Introduction
- Objectives
- Who You Are
- The Linux Foundation
- Copyright and No Confidential Information
- Linux Foundation Training
- Certification Programs and Digital Badging
- Linux Distributions
- Platforms
- Preparing Your System
- Using and Downloading a Virtual Machine
- Things Change in Linux and Open Source Projects
- Documentation and Links
Preliminaries
- Procedures
- Kernel Versions
- Kernel Sources and Use of git
- Labs
How to Work in OSS Projects **
- Overview on How to Contribute Properly
- Know Where the Code is Coming From: DCO and CLA
- Stay Close to Mainline for Security and Quality
- Study and Understand the Project DNA
- Figure Out What Itch You Want to Scratch
- Identify Maintainers and Their Work Flows and Methods
- Get Early Input and Work in the Open
- Contribute Incremental Bits, Not Large Code Dumps
- Leave Your Ego at the Door: Don’t Be Thin-Skinned
- Be Patient, Develop Long Term Relationships, Be Helpful
Kernel Features
- Components of the Kernel
- User-Space vs. Kernel-Space
- What are System Calls?
- Available System Calls
- Scheduling Algorithms and Task Structures
- Process Context
- Labs
Monitoring and Debugging
- Debuginfo Packages
- Tracing and Profiling
- sysctl
- SysRq Key
- oops Messages
- Kernel Debuggers
- debugfs
- Labs
Printk
- Debugging with printk
- Format Specifiers in printk
- no hash pointers Command Line Option
- Using early printk
- Labs
The proc Filesystem **
- What is the proc Filesystem?
- Creating and Removing Entries
- Reading and Writing Entries
- The seq file Interface **
- Labs
kprobes
- kprobes
- kretprobes
- SystemTap **
- Labs
Ftrace
- What is ftrace?
- ftrace, trace-cmd and kernelshark
- Available Tracers
- Using ftrace
- Files in the Tracing Directory
- Tracing Options
- Printing with trace printk()
- Trace Markers
- Dumping the Buffer
- trace-cmd
- Labs
Perf
- What is perf?
- perf stat
- perf list
- perf record
- perf report
- perf annotate
- perf top
- Labs
eBPF
- BPF
- eBPF
- Installation
- bcc Tools
- bpftrace
- Labs
Crash
- Crash
- Main Commands
- Labs
kexec
- kexec
- Kernel Configuration
- kexec-tools
- Using kexec
- Labs
Kernel Core Dumps
- Producing and Analyzing Kernel Core Dumps
- Labs
Virtualization**
- What is Virtualization?
- Rings of Virtualization
- Hypervisors
QEMU
- What is QEMU?
- Emulated Architectures
- Image Formats
- Third Party Hypervisor Integration
- Labs
Linux Kernel Debugging Tools
- Linux Kernel (built-in) tools and helpers
- kdb
- qemu+gdb
- kgdb: hardware+serial+gdb
- Labs
Embedded Linux**
- Embedded and Real Time Operating Systems
- Why Use Linux?
- Making a Small Linux Environment
- Real Time Linuxes
Notifiers**
- What are Notifiers?
- Data Structures
- Callbacks and Notifications
- Creating Notifier Chains
- Labs
CPU Frequency Scaling**
- What is Frequency and Voltage Scaling?
- Notifiers
- Drivers
- Governors
- Labs
Netlink Sockets**
- What are netlink Sockets?
- Opening a netlink Socket
- netlink Messages
- Labs
Kernel Deprecated Interfaces
- Why Deprecated
- deprecated
- BUG() and BUG ON()
- Computed Sizes for kmalloc()
- simple strtol() Family of Routines
- strcpy(), strncpy(), strlcpy()
- printk() %p Format Specifier
- Variable Length Arrays
- Switch Case Fall-Through
- Zero-Length and One-Element Arrays in Structs
Kernel Structure Layout Randomization
- Benefits
- How Structure Randomization Works
- Structure Initialization
- Opt-in vs Opt-out
- Partial Randomization
- Enabling Structure Randomization
- Building Out-of-tree Modules with Structure Randomization
Secure Boot VM Setup
- Labs
Module Signing
- What is Module Signing?
- Basics of Signatures
- Module Signing Keys
- Enabling Module Signature Verification
- How It Works
- Signing Modules
- Labs
Secure Boot
- Why Secure Boot?
- Secure Boot x86
- Embedded Systems Secure Boot
- Labs
Integrity Measurement Architecture (IMA)
- Why IMA?
- Conceptual Operations
- Modes of Operation
- Collect Mode (Collect and Store)
- Logging Mode (Appraise and Audit)
- Enforcing Mode (Appraise and Protect)
- Extended Verification Module (EVM)
- Labs
Introduction to Linux Kernel Security
- Linux Kernel Security Basics
- Discretionary Access Control (DAC)
- POSIX ACLs
- POSIX Capabilities
- Namespaces
- Linux Security Modules (LSM)
- Netfilter
- Cryptographic Methods
- The Kernel Self Protection Project
Linux Security Modules (LSM)
- What are Linux Security Modules?
- LSM Basics
- LSM Choices
- How LSM Works
- An LSM Example: Tomoyo
SELinux
- SELinux
- SELinux Overview
- SELinux Modes
- SELinux Policies
- Context Utilities
- SELinux and Standard Command Line Tools
- SELinux Context Inheritance and Preservation**
- restorecon**
- semanage fcontext**
- Using SELinux Booleans**
- getsebool and setsebool**
- Troubleshooting Tools
- Labs
AppArmor
- What is AppArmor?
- Checking Status
- Modes and Profiles
- Profiles
- Utilities
Netfilter
- What is netfilter?
- Netfilter Hooks
- Netfilter Implementation
- Hooking into Netfilter
- Iptables
- nftables
- Labs
The Virtual File System
- What is the Virtual File System?
- Available Filesystems
- Special Filesystems
- The tmpfs Filesystem
- The ext2/ext3 Filesystem
- The ext4 Filesystem
- The btrfs Filesystem
- Common File Model
- VFS System Calls
- Files and Processes
- Mounting Filesystems
Filesystems in User-Space (FUSE)**
- What is FUSE?
- Writing a Filesystem
- Labs
Journaling Filesystems**
- What are Journaling Filesystems?
- Available Journaling Filesystems
- Contrasting Features
- Labs
Closing and Evaluation Survey
- Evaluation Survey
Kernel Architecture I
- UNIX and Linux **
- Monolithic and Micro Kernels
- Object-Oriented Methods
- Main Kernel Components
- User-Space and Kernel-Space
Kernel Programming Preview
- Error Numbers and Getting Kernel Output
- Task Structure
- Memory Allocation
- Transferring Data between User and Kernel Spaces
- Object-Oriented Inheritance - Sort Of
- Linked Lists
- Jiffies
- Labs
Modules
- What are Modules?
- A Trivial Example
- Compiling Modules
- Modules vs Built-in
- Module Utilities
- Automatic Module Loading
- Module Usage Count
- Module Licensing
- Exporting Symbols
- Resolving Symbols **
- Labs
Kernel Architecture II
- Processes, Threads, and Tasks
- Kernel Preemption
- Real Time Preemption Patch
- Labs
Kernel Configuration and Compilation
- Installation and Layout of the Kernel Source
- Kernel Browsers
- Kernel Configuration Files
- Kernel Building and Makefiles
- initrd and initramfs
- Labs
Kernel Style and General Considerations
- Coding Style
- Using Generic Kernel Routines and Methods
- Making a Kernel Patch
- sparse
- Using likely() and unlikely()
- Writing Portable Code, CPU, 32/64-bit, Endianness
- Writing for SMP
- Writing for High Memory Systems
- Power Management
- Keeping Security in Mind
- Labs
Race Conditions and Synchronization Methods
- Concurrency and Synchronization Methods
- Atomic Operations
- Bit Operations
- Spinlocks
- Seqlocks
- Disabling Preemption
- Mutexes
- Semaphores
- Completion Functions
- Read-Copy-Update (RCU)
- Reference Counts
- Labs
Memory Addressing
- Virtual Memory Management
- Systems With and Without MMU and the TLB
- Memory Addresses
- High and Low Memory
- Memory Zones
- Special Device Nodes
- NUMA
- Paging
- Page Tables
- page structure
- Labs
Memory Allocation
- Requesting and Releasing Pages
- Buddy System
- Slabs and Cache Allocations
- Memory Pools
- kmalloc()
- vmalloc()
- Early Allocations and bootmem()
- Memory Defragmentation
- Labs

**
These sections may be considered in part or in whole as optional. They contain either background reference material, specialized topics, or advanced subjects. The instructor may choose to cover or not cover them depending on classroom experience and time constraints.
Prerequisites
To make the most of this course, you should:

  • Be proficient in the C programming language.
  • Be familiar with basic Linux (UNIX) utilities such as ls, grep and tar.
  • Be comfortable using any of the available text editors (e.g. emacs, vi, etc.).
  • Experience with any major Linux distribution is helpful but not strictly required.
  • Have experience equivalent to having taken LFD420: Linux Kernel Internals and Development.

Pre-class preparation material will be provided before class.

Reviews
Jun 2022
Overall, I think the layout of the content was great, the labs showed specifics of the topic in question.
Jun 2022
John was very nice, and a good course instructor.
Jun 2022
The instructor covered module sections which helped gear up for the rest. The overall breadth of coverage is good. The instructions for exercises were pretty clear and up to date.
Jun 2022
I really enjoyed the discussed topics. The instructor explained the topics properly, and communicated well with the audience. The diagram presented at the beginning of the kernel security helped me understand how things work under the hood.
Jun 2022
The material was well designed, and fairly complete.
May 2022
It was nice to have training where the instructor seemed to know what he was talking about, and not just reading some slides!
May 2022
What I really liked was having someone drive me through all the chapters by importance, providing thoughts about the topics, and sharing personal experiences. I liked the approach of "go ahead and try by yourself." With solutions available, I was able to try on my own, then check further improvements on the solutions, and always had John available for further comments, and to unlock blockers we had. I prefer the POV approach rather than having somebody sharing a screen, and writing the code himself.
May 2022
The instructors are very well informed, professional and experienced. The class touches on some interesting concepts, like the modules, qemu, and SELinux.
May 2022
I think the exercises from the labs were really well thought out, they were in the sweet spot where we could get a grasp of each module, without them being too overwhelming.
May 2022
Great instructor. I have always been impressed with the trainers that LFD finds for these courses.
May 2022
Compared to other LFD courses I have taken, there was actually more time for labs, which was great.
May 2022
The toughest part about learning this is figuring out where to begin, but this course presented the material in a logical and methodical manner.
May 2022
The course gave me a great overview of the debugging techniques.
May 2022
It was good to understand how the industry is moving to eBPF.
May 2022
The instructor shines when explaining architecture and whiteboarding concepts.
May 2022
Having a teacher that guides us along, and taking the time to look at some concepts in depth.