Training > Linux Kernel Development > Linux Kernel Debugging and Security (LFD440)

Linux Kernel Debugging and Security (LFD440)

This instructor-led course focuses on the important tools used for debugging and monitoring the kernel, and how security features are implemented and controlled.

Who Is It For

This course is for experienced developers who need to understand the methods and internal infrastructure of the Linux kernel.
read less read more
What You’ll Learn

This four day course includes extensive hands-on exercises and demonstrations designed to give you the necessary tools to develop and debug Linux kernel code.
read less read more
What It Prepares You For

You will walk away from this course with a solid understanding of Linux kernel. debugging techniques and tools.
read less read more
Course Outline
Expand All
Collapse All
- Objectives
- Who You Are
- The Linux Foundation
- Copyright and No Confidential Information
- Linux Foundation Training
- Certification Programs and Digital Badging
- Linux Distributions
- Platforms
- Preparing Your System
- Using and Downloading a Virtual Machine
- Things change in Linux
- Documentation and Links
- Procedures
- Kernel Versions
- Kernel Sources and Use of git
- Labs
How to Work in OSS Projects **
- Overview on How to Contribute Properly
- Stay Close to Mainline for Security and Quality
- Study and Understand the Project DNA
- Figure Out What Itch You Want to Scratch
- Identify Maintainers and Their Work Flows and Methods
- Get Early Input and Work in the Open
- Contribute Incremental Bits, Not Large Code Dumps
- Leave Your Ego at the Door: Don’t Be Thin-Skinned
- Be Patient, Develop Long Term Relationships, Be Helpful
Kernel Features
- Components of the Kernel
- User-Space vs. Kernel-Space
- What are System Calls?
- Available System Calls
- Scheduling Algorithms and Task Structures
- Process Context
- Labs
Monitoring and Debugging
- Debuginfo Packages
- Tracing and Profiling
- sysctl
- SysRq Key
- oops Messages
- Kernel Debuggers
- debugfs
- Labs
- Debugging with printk
- Format Specifiers in printk
- no hash pointers Command Line Option
- Using early printk
- Labs
The proc Filesystem **
- What is the proc Filesystem?
- Creating and Removing Entries
- Reading and Writing Entries
- The seq file Interface **
- Labs
- kprobes
- kretprobes
- SystemTap **
- Labs
- What is ftrace?
- ftrace, trace-cmd and kernelshark
- Available Tracers
- Using ftrace
- Files in the Tracing Directory
- Tracing Options
- Printing with trace printk()
- Trace Markers
- Dumping the Buffer
- trace-cmd
- Labs
- What is perf?
- perf stat
- perf list
- perf record
- perf report
- perf annotate
- perf top
- Labs
- eBPF
- Installation
- bcc Tools
- bpftrace
- Labs
- Crash
- Main Commands
- Labs
- kexec
- Kernel Configuration
- kexec-tools
- Using kexec
- Labs
Kernel Core Dumps
- Producing and Analyzing Kernel Core Dumps
- Labs
- What is Virtualization?
- Rings of Virtualization
- Hypervisors
- What is QEMU?
- Emulated Architectures
- Image Formats
- Third Party Hypervisor Integration
- Labs
Linux Kernel Debugging Tools
- Linux Kernel (built-in) tools and helpers
- kdb
- qemu+gdb
- kgdb: hardware+serial+gdb
- Labs
Embedded Linux**
- Embedded and Real Time Operating Systems
- Why Use Linux?
- Making a Small Linux Environment
- Real Time Linuxes
- What are Notifiers?
- Data Structures
- Callbacks and Notifications
- Creating Notifier Chains
- Labs
CPU Frequency Scaling**
- What is Frequency and Voltage Scaling?
- Notifiers
- Drivers
- Governors
- Labs
Netlink Sockets**
- What are netlink Sockets?
- Opening a netlink Socket
- netlink Messages
- Labs
Kernel Deprecated Interfaces
- Why Deprecated
- deprecated
- BUG() and BUG ON()
- Computed Sizes for kmalloc()
- simple strtol() Family of Routines
- strcpy(), strncpy(), strlcpy()
- printk() %p Format Specifier
- Variable Length Arrays
- Switch Case Fall-Through
- Zero-Length and One-Element Arrays in Structs
Introduction to Linux Kernel Security
- Linux Kernel Security Basics
- Discretionary Access Control (DAC)
- POSIX Capabilities
- Namespaces
- Linux Security Modules (LSM)
- Netfilter
- Cryptographic Methods
- The Kernel Self Protection Project
Linux Security Modules (LSM)
- What are Linux Security Modules?
- LSM Basics
- LSM Choices
- How LSM Works
- An LSM Example: Tomoyo
- SELinux
- SELinux Overview
- SELinux Modes
- SELinux Policies
- Context Utilities
- SELinux and Standard Command Line Tools
- SELinux Context Inheritance and Preservation**
- restorecon**
- semanage fcontext**
- Using SELinux Booleans**
- getsebool and setsebool**
- Troubleshooting Tools
- Labs
- What is AppArmor?
- Checking Status
- Modes and Profiles
- Profiles
- Utilities
- What is netfilter?
- Netfilter Hooks
- Netfilter Implementation
- Hooking into Netfilter
- Iptables
- Labs
The Virtual File System
- What is the Virtual File System?
- Available Filesystems
- Special Filesystems
- The tmpfs Filesystem
- The ext2/ext3 Filesystem
- The ext4 Filesystem
- The btrfs Filesystem
- Common File Model
- VFS System Calls
- Files and Processes
- Mounting Filesystems
Filesystems in User-Space (FUSE)**
- What is FUSE?
- Writing a Filesystem
- Labs
Journaling Filesystems**
- What are Journaling Filesystems?
- Available Journaling Filesystems
- Contrasting Features
- Labs
Closing and Evaluation Survey
- Evaluation Survey
Kernel Architecture I
- UNIX and Linux **
- Monolithic and Micro Kernels
- Object-Oriented Methods
- Main Kernel Components
- User-Space and Kernel-Space
Kernel Programming Preview
- Error Numbers and Getting Kernel Output
- Task Structure
- Memory Allocation
- Transferring Data between User and Kernel Spaces
- Object-Oriented Inheritance - Sort Of
- Linked Lists
- Jiffies
- Labs
- What are Modules?
- A Trivial Example
- Compiling Modules
- Modules vs Built-in
- Module Utilities
- Automatic Module Loading
- Module Usage Count
- Module Licensing
- Exporting Symbols
- Resolving Symbols **
- Labs
Kernel Architecture II
- Processes, Threads, and Tasks
- Kernel Preemption
- Real Time Preemption Patch
- Labs
Kernel Configuration and Compilation
- Installation and Layout of the Kernel Source
- Kernel Browsers
- Kernel Configuration Files
- Kernel Building and Makefiles
- initrd and initramfs
- Labs
Kernel Style and General Considerations
- Coding Style
- Using Generic Kernel Routines and Methods
- Making a Kernel Patch
- sparse
- Using likely() and unlikely()
- Writing Portable Code, CPU, 32/64-bit, Endianness
- Writing for SMP
- Writing for High Memory Systems
- Power Management
- Keeping Security in Mind
- Labs
Race Conditions and Synchronization Methods
- Concurrency and Synchronization Methods
- Atomic Operations
- Bit Operations
- Spinlocks
- Seqlocks
- Disabling Preemption
- Mutexes
- Semaphores
- Completion Functions
- Read-Copy-Update (RCU)
- Reference Counts
- Labs
Memory Addressing
- Virtual Memory Management
- Systems With and Without MMU and the TLB
- Memory Addresses
- High and Low Memory
- Memory Zones
- Special Device Nodes
- Paging
- Page Tables
- page structure
- Labs
Memory Allocation
- Requesting and Releasing Pages
- Buddy System
- Slabs and Cache Allocations
- Memory Pools
- kmalloc()
- vmalloc()
- Early Allocations and bootmem()
- Memory Defragmentation
- Labs

These sections may be considered in part or in whole as optional. They contain either background reference material, specialized topics, or advanced subjects. The instructor may choose to cover or not cover them depending on classroom experience and time constraints.
To make the most of this course, you should:

  • Be proficient in the C programming language.
  • Be familiar with basic Linux (UNIX) utilities such as ls, grep and tar.
  • Be comfortable using any of the available text editors (e.g. emacs, vi, etc.).
  • Experience with any major Linux distribution is helpful but not strictly required.
  • Have experience equivalent to having taken LFD420: Linux Kernel Internals and Development.

Pre-class preparation material will be provided before class.

Feb 2022
Very knowledgeable instructor.
Feb 2022
Instructor was great, with lots of stories and background knowledge in the field He put context into technologies, which helped me to understand historical reasons and decision making.
Feb 2022
Really interesting to see the thought behind the design of the kernel.
Feb 2022
I felt Behan did a great job presenting the material.
Dec 2021
The debugging chapters, especially the ftrace and perf.
Dec 2021
Relevant exercises by loading and running modules.
Dec 2021
This course was very interesting, and different from some of the core Kernel Internals and Device Drivers classes, in that there seemed to be a lot more smaller and straightforward labs, that were easily achievable. Also, each lab highlighted an area of the kernel debug or security that I hadn't used before, which was great.
Nov 2021
Loved the course, and will highly recommend it to my colleagues.
Nov 2021
I really liked the breadth of topics covered. I wasn't expecting the debugging and introspection facilities to be so fascinating.
Nov 2021
Understanding how kprobes+debugfs fit together opens up a world of possibilities for debugging. I feel like I am prepared now to go and experiment, and write up blogs on side projects!
Oct 2021
Behan is fantastic. Give him a raise.
Oct 2021
Behan is simply amazing.
Oct 2021
Behan! He was awesome. Really made the course top tier, with great background context, history, where things are heading, and tying each topic into others.
Oct 2021
The instructor delivered the content extremely well.
Oct 2021
Behan was an incredible instructor, he really made the course a great experience.
Sep 2021
There were diagrams to explain concepts.
Sep 2021
The labs, as they went into some of the more complex topics.
Sep 2021
The material was in depth, and provided great skills building.
Sep 2021
Teacher seemed very confident, talked in a calm and friendly voice, and was not afraid of explaining related questions, or diving into editing files real time.
Jul 2021
Karl is a really good instructor!
Jul 2021
I'm now able to debug and analyze low level problems in the kernel, in ways I was only able to do for user space applications before.
Jul 2021
It helped me analyze and debug problems that I wasn't able to catch before.
Jul 2021
The content covered in the course is awesome.
May 2021
The material is good.
Mar 2021
I like the Labs.
Mar 2021
I've now taken LFD420 and LFD440, and I really like how these courses provide a nice overview of the topic, and then use labs for further exploration. This provides a nice balance, and an avenue for further self-directed exploration.
Feb 2021
I got a really wide range of knowledge on Linux, and was introduced into unknown technology.
Feb 2021
Besides the material, I loved the insights shared about the maintainers, and how things came to be historically etc. Instructor very often came up with excellent, concise, and more pragmatic explanations, which I thought was really cool, and benefitted from it.
Feb 2021
The training aids (virtual machines etc.) were spot on as usual - they really help to get the most out of the training. The instructor was knowledgeable, and helped reinforce topics with real world experience. A pleasure to have had Behan take the training.
Feb 2021
The instructor was great, very nice and thoughtful.
Feb 2021
Solving the labs together, and live demos.
Feb 2021
Content, labs and trainer expertise.