Training > Linux Kernel Development > Linux Kernel Debugging and Security (LFD440)
INSTRUCTOR-LED COURSE

Linux Kernel Debugging and Security (LFD440)

This instructor-led course focuses on the important tools used for debugging and monitoring the kernel, and how security features are implemented and controlled.

Who Is It For

This course is for experienced developers who need to understand the methods and internal infrastructure of the Linux kernel.
read less read more
What You’ll Learn

This four day course includes extensive hands-on exercises and demonstrations designed to give you the necessary tools to develop and debug Linux kernel code.
read less read more
What It Prepares You For

You will walk away from this course with a solid understanding of Linux kernel. debugging techniques and tools.
read less read more
Course Outline
Expand All
Collapse All
Introduction
- Objectives
- Who You Are
- The Linux Foundation
- Copyright and No Confidential Information
- Linux Foundation Training
- Certification Programs and Digital Badging
- Linux Distributions
- Platforms
- Preparing Your System
- Using and Downloading a Virtual Machine
- Things Change in Linux and Open Source Projects
- Documentation and Links
Preliminaries
- Procedures
- Kernel Versions
- Kernel Sources and Use of git
- Labs
How to Work in OSS Projects **
- Overview on How to Contribute Properly
- Know Where the Code is Coming From: DCO and CLA
- Stay Close to Mainline for Security and Quality
- Study and Understand the Project DNA
- Figure Out What Itch You Want to Scratch
- Identify Maintainers and Their Work Flows and Methods
- Get Early Input and Work in the Open
- Contribute Incremental Bits, Not Large Code Dumps
- Leave Your Ego at the Door: Don’t Be Thin-Skinned
- Be Patient, Develop Long Term Relationships, Be Helpful
Kernel Features
- Components of the Kernel
- User-Space vs. Kernel-Space
- What are System Calls?
- Available System Calls
- Scheduling Algorithms and Task Structures
- Process Context
- Labs
Reducing Attack Surfaces
- Why Security?
- Types of Security
- Vulnerabilities
- Layers of Protection
- Software Exploits
- Labs
Kernel Deprecated Interfaces
- Why Deprecated
- deprecated
- BUG() and BUG ON()
- Computed Sizes for kmalloc()
- simple strtol() Family of Routines
- strcpy(), strncpy(), strlcpy()
- printk() %p Format Specifier
- Variable Length Arrays
- Switch Case Fall-Through
- Zero-Length and One-Element Arrays in Structs
Kernel Structure Layout Randomization
- Benefits
- How Structure Randomization Works
- Structure Initialization
- Opt-in vs Opt-out
- Partial Randomization
- Enabling Structure Randomization
- Building Out-of-tree Modules with Structure Randomization
Introduction to Linux Kernel Security
- Linux Kernel Security Basics
- Discretionary Access Control (DAC)
- POSIX ACLs
- POSIX Capabilities
- Namespaces
- Linux Security Modules (LSM)
- Netfilter
- Cryptographic Methods
- The Kernel Self Protection Project
Secure Boot VM Setup
- Labs
Secure Boot
- Why Secure Boot?
- Secure Boot x86
- Embedded Systems Secure Boot
- Labs
Module Signing
- What is Module Signing?
- Basics of Signatures
- Module Signing Keys
- Enabling Module Signature Verification
- How It Works
- Signing Modules
- Labs
Integrity Measurement Architecture (IMA)
- Why IMA?
- Conceptual Operations
- Modes of Operation
- Collect Mode (Collect and Store)
- Logging Mode (Appraise and Audit)
- Enforcing Mode (Appraise and Protect)
- Extended Verification Module (EVM)
- Labs
Linux Security Modules (LSM)
- What are Linux Security Modules?
- LSM Basics
- LSM Choices
- How LSM Works
- An LSM Example: yama
SELinux
- SELinux
- SELinux Overview
- SELinux Modes
- SELinux Policies
- Context Utilities
- SELinux and Standard Command Line Tools
- SELinux Context Inheritance and Preservation**
- restorecon**
- semanage fcontext**
- Using SELinux Booleans**
- getsebool and setsebool**
- Troubleshooting Tools
- Labs
AppArmor
- What is AppArmor?
- Checking Status
- Modes and Profiles
- Profiles
- Utilities
Lockdown
- Why Lockdown?
- Lockdown Modes
- What Things are Locked Down?
- How It Works
- A Few Notes
- Labs
Netfilter
- What is netfilter?
- Netfilter Hooks
- Netfilter Implementation
- Hooking into Netfilter
- Iptables
- nftables
- Labs
Netlink Sockets**
- What are netlink Sockets?
- Opening a netlink Socket
- netlink Messages
- Labs
Monitoring and Debugging
- Debuginfo Packages
- Tracing and Profiling
- sysctl
- SysRq Key
- oops Messages
- Kernel Debuggers
- debugfs
- Labs
Printk
- Debugging with printk
- Format Specifiers in printk
- no hash pointers Command Line Option
- Using early printk
- Labs
The proc Filesystem **
- What is the proc Filesystem?
- Creating and Removing Entries
- Reading and Writing Entries
- The seq file Interface **
- Labs
kprobes
- kprobes
- kretprobes
- SystemTap **
- Labs
Ftrace
- What is ftrace?
- ftrace, trace-cmd and kernelshark
- Available Tracers
- Using ftrace
- Files in the Tracing Directory
- Tracing Options
- Printing with trace printk()
- Trace Markers
- Dumping the Buffer
- trace-cmd
- Labs
Perf
- What is perf?
- perf stat
- perf list
- perf record
- perf report
- perf annotate
- perf top
- Labs
eBPF
- BPF
- eBPF
- Installation
- bcc Tools
- bpftrace
- Labs
Crash
- Crash
- Main Commands
- Labs
kexec
- kexec
- Kernel Configuration
- kexec-tools
- Using kexec
- Labs
Kernel Core Dumps
- Producing and Analyzing Kernel Core Dumps
- Labs
QEMU
- What is QEMU?
- Emulated Architectures
- Image Formats
- Third Party Hypervisor Integration
- Labs
Linux Kernel Debugging Tools
- Linux Kernel (built-in) tools and helpers
- kdb
- qemu+gdb
- kgdb: hardware+serial+gdb
- Labs
Closing and Evaluation Survey
- Evaluation Survey
Kernel Architecture I
- UNIX and Linux **
- Monolithic and Micro Kernels
- Object-Oriented Methods
- Main Kernel Components
- User-Space and Kernel-Space
Kernel Programming Preview
- Error Numbers and Getting Kernel Output
- Task Structure
- Memory Allocation
- Transferring Data between User and Kernel Spaces
- Object-Oriented Inheritance - Sort Of
- Linked Lists
- Jiffies
- Labs
Modules
- What are Modules?
- A Trivial Example
- Compiling Modules
- Modules vs Built-in
- Module Utilities
- Automatic Module Loading
- Module Usage Count
- Module Licensing
- Exporting Symbols
- Resolving Symbols **
- Labs
Kernel Architecture II
- Processes, Threads, and Tasks
- Kernel Preemption
- Real Time Preemption Patch
- Labs
Kernel Configuration and Compilation
- Installation and Layout of the Kernel Source
- Kernel Browsers
- Kernel Configuration Files
- Kernel Building and Makefiles
- initrd and initramfs
- Labs
Kernel Style and General Considerations
- Coding Style
- Using Generic Kernel Routines and Methods
- Making a Kernel Patch
- sparse
- Using likely() and unlikely()
- Writing Portable Code, CPU, 32/64-bit, Endianness
- Writing for SMP
- Writing for High Memory Systems
- Power Management
- Keeping Security in Mind
- Labs
Race Conditions and Synchronization Methods
- Concurrency and Synchronization Methods
- Atomic Operations
- Bit Operations
- Spinlocks
- Seqlocks
- Disabling Preemption
- Mutexes
- Semaphores
- Completion Functions
- Read-Copy-Update (RCU)
- Reference Counts
- Labs
Memory Addressing
- Virtual Memory Management
- Systems With and Without MMU and the TLB
- Memory Addresses
- High and Low Memory
- Memory Zones
- Special Device Nodes
- NUMA
- Paging
- Page Tables
- page structure
- Labs
Memory Allocation
- Requesting and Releasing Pages
- Buddy System
- Slabs and Cache Allocations
- Memory Pools
- kmalloc()
- vmalloc()
- Early Allocations and bootmem()
- Memory Defragmentation
- Labs

**
These sections may be considered in part or in whole as optional. They contain either background reference material, specialized topics, or advanced subjects. The instructor may choose to cover or not cover them depending on classroom experience and time constraints.
Prerequisites
To make the most of this course, you should:

  • Be proficient in the C programming language.
  • Be familiar with basic Linux (UNIX) utilities such as ls, grep and tar.
  • Be comfortable using any of the available text editors (e.g. emacs, vi, etc.).
  • Experience with any major Linux distribution is helpful but not strictly required.
  • Have experience equivalent to having taken LFD420: Linux Kernel Internals and Development.

Pre-class preparation material will be provided before class.

Reviews
Nov 2022
Very interesting training, John knows his topic for sure, and explains very well and clearly.
Nov 2022
I have learned many useful things, thanks John.
Nov 2022
The content was real, heady and deep, and John had an outstanding way of explaining things.
Nov 2022
Well prepared, I liked the labs where we needed to write code for the kernel, and where we did debugging exercises.
Sep 2022
Getting to learn a lot about Linux Kernel Internals, and many topics related to OS/Networking in general.
Sep 2022
The well worked-out labs/exercises, and the in-depth knowledge of the trainer, where you could basically ask anything about Linux.
Sep 2022
I get to learn about Linux details, especially because the instructor has broad knowledge, even outside the scope of the course.