rfsd: ReiserDriver

1
rfsd ReiserDriver
Mark Piper

• Porting Linuxs ReiserFSfile system to Microsoft
  Windows

Advisor Tony Lattanze Carnegie Mellon
University June 30th, 2005
2
Agenda

• Proposal Why would anyone want to do this?
• The Problem
• The Solution
• Planning
• Pragmatics Can anyone actually do this?
• Mitigating Technical Risk
• Development Strategy
• Quality Attributes and Analysis
• Progress 49 days down. 35 days to go.
• ACDM Experiment Schedule
• Progress So Far
• Project Logistics
• Looking Ahead

3
Proposal

• Why would anyone want to do this?

I find it hard to imagine why anyone would want
to subject themselves to more than 700 pages of
excruciatingly detailed information about the I/O
subsystem and its associated components. Rajeev
Nagar, The Only Thing Ever Written On Windows
File System Drivers (from 1997)
4
The Problem

• Background ReiserFS is the most widely used
  file system on Linux
• Fastest file system in existence (especially for
  small files)
• Transactional journaling (cannot become corrupt)
• Highly scalable (unrestricted file sizes and
  numbers of files)
• ProblemDual-boot Linux/Windows users cannot
  access ReiserFS partitions from Windows
• Transparently
• from any application, as if files were on a
  native NTFS or VFAT disk
• Efficiently
• without needless file copying.

5
The Problem, Restated

• Anyone know of any good free Reiser FS drivers
  for windows... I have found EXT2 drivers but as
  you probably know EXT2 sucks balls compared to
  Resier FS and i want to share data between
  windows and linux.Fat32 is out of the questing
  since the partion is over 100gig and i would use
  EXT2 over it any day. I would use something like
  NTFS but linux ntfs support sucks big time.
• sic Sirikan, MSFN.org forum user

• Trying to use Windows partitions to run Linux is
  fruitless
• FAT is, at best, wasteful at worst, impossible
• NTFS is proprietary writing from Linux is
  unsupported
• Windows drivers for Linuxs older ext2 FS exist
  (ext2fsd)
• but ReiserFS is technically superior to other
  Linux solutions

6
The Solution

• SolutionReiserDriver will provide an
  Installable File System kernel driver (IFSD)
  allowing the Windows operating system to natively
  access ReiserFS partitions.
• BenefitUsers will have seamless access to Linux
  data when using Windows, easing the migration
  path to Linux, and increasing the adoption of
  ReiserFS.

• ReiserDriver will be an Installable File System
  Driver (IFSD), used to natively read ReiserFS
  disk partitions under Microsoft Windows
  2K/XP/2003. ReiserFS partitions will appear as
  additional disks to the Windows operating system,
  allowing files on ReiserFS partitions to be
  transparently accessed from any Windows
  application.

Files are created by the user while booted into
Linux
The user wants to access those files while booted
into Windows
ReiserDriver allows Windows to read the Linux
ReiserFS partition
7
What is an IFSD?

• Modern operating systems separate the
  implementation of a file system from its
  interface to the OS
• Linux VFS (Virtual File System interface)
• WinNT IFS (Installable File System interface)

• Several ports of VFS file systems to IFS
  interface exist
• Bo Brantens romfs
• Ext2fsd, for the older ext2 file system
• ufsd, for the BSD UFS file system
• Ill be writing a driver that is inserted here.
• The driver converts incoming file-based IO
  requests into outgoing block-based IO requests.
• Reiser4 is slowly being adopted, but it changes
  the VFS.
• Therefore, ReiserDriver will initially support 3.x

The Windows IO Stack
8
Summary of Proposal

• Requirements
• Read only access to Reiser 3.x partitions via the
  IFS interface
• End users
• Dual boot Linux/Windows users, who need to access
  their Linux data from Windows
• Clients
• This is an open source project the end users are
  the clients.
• Bedros Hanounik has volunteered as a
  representative user.
• Advisors
• Tony Lattanze project mentor
• IFS and ReiserFS mailing lists used for technical
  mentoring
• Methodology
• ACDM for development
• Crystal Clear for project management and
  scheduling

9
Initial Planning

• Estimating
• Total project effort estimated according to the
  number of practicum course hours
• First half 12 hours / week
• Second half 36 hours / week
• Tracking
• Use KArm tool to track hierarchical task
  estimates and actual times
• Export to Excel
• Macro magic to obtain burn chart, task
  progress,and statistics

10
Pragmatics

• Can anyone actually do this?

File systems development for Windows
NT/2000/XP/Server 2003 remains a very challenging
task. It can be surprisingly challenging. If you
haven't tried it yourself, take our word for
it. OSR.com (original emphasis)
11
Pragmatics

• Can anyone actually do this?

File systems development for Windows
NT/2000/XP/Server 2003 remains a very challenging
task. It can be surprisingly challenging. If you
haven't tried it yourself, take our word for
it. OSR.com (original emphasis)
12
Venturing into the Unknown

• Kernel programming (especially on Windows) is
  fraught with unknown perils
• I had no basis for an estimate
• except that somebody else said it took 3 months
  to make an initial release
• Extremely complex, and highly platform-dependent
  development
• Unfamiliar environment
• Non-standard development setup and debugging
  techniques (it requires two machines!)
• Non-intuitive patchwork kernel architecture
• Resource-constrained environment
• Non-standard C dialect
• Non-standard APIs (and no access to a standard
  libc)
• Many specialized kernel programming concerns
• Synchronization issues
• Paging issues
• Interrupt levels
• Privilege levels
• Non-existent memory management
• Largely undocumented!

It could legitimately be argued that the entire
fast I/O interface was a last minute hack or
addition to the I/O subsystem in response to some
serious performance problems - Rajeev Nagar
13
Resources to Leverage

• Before proceeding, I wanted to identify what
  resources I could use to
• Gain understanding
• Base my implementation off of
• Related tools for reading ReiserFS
• ReiserFS VFS
• The original ReiserFS source code is difficult to
  understand because of the complexity of the
  journaling and writing code.
• GRUB
• Instead, the GRUB bootloader provides extremely
  simple read-only functionality.
• Rfstool
• Is a standalone (inefficient and non-transparent)
  C application that can read ReiserFS from
  Windows. It can be used to verify my results.
• Related tools for creating IFSDs
• Romfs
• Is an simple read-only IFSD, but it is
  undocumented and has some bugs
• Nagar Skeleton
• Is a very well-documented IFSD skeleton, but it
  contains no FS-specific code, and is out of date
  in some places
• Ext2fsd
• Based off a hybrid of the Romfs and Nagar code,
  it is undocumented, very complex, and bug-free

14
Strategy

• For reading ReiserFS
• GRUB
• Study its extremely simple read-only
  functionality and reuse whatever code fragments
  possible for reading the ReiserFS disk
  structures!
• For creating the IFSD Infrastructure
• Romfs
• Experiment with this simple read-only IFSD to
  discover the causes of its bugs
• Nagar Skeleton
• Base my implementation off this well-documented
  IFSD skeleton, but use the FS-specific code from
  Romfs
• Ext2fsd
• When encountering bugs in my implementation,
  study this bug-free source code relative to my
  own implementation

15
An Architectural Basis

• ACDM Priority One identify quality attributes!
• High Importance
• Correctness file retrieved is identical to the
  file originally stored
• Safety the driver will not kernel panic from an
  uncaught exception
• Medium Importance
• Portability architecture can be used to port
  other VFS file systems
• Low Importance
• Performance ReiserDriver will perform at least
  as fast as rfstool
• Footprint the driver will not exhaust the
  systems kernel memory

16
Building for Quality Attributes

• With these attributes, I wanted to come up with
  ways to guarantee their presence.
• High Importance
• Correctness
• Create a test harness that compares the file as
  read using my tool, to the file as read using
  rfstool
• Safety
• Create an exception handling skeleton, which
  causes panics and catches them
• Medium Importance
• Portability
• Coming up next
• Low Importance do nothing

17
The Impossibility of Portability

• At first I thought VFS portability would be
  incredible!
• but OSR has spent years on this problem!
• Portability was not a quality attribute of the
  IFS design
• Platform-specific development issues and
  techniques
• Tightly coupled interactions with the virtual
  memory subsystem (for memory mapped files), cache
  manager, and I/O request manager
• Less need for architectural understanding
• ? less architectural reconstruction
• Instead it has been more important to study
  implementations rather than interfaces
• Study Nagars skeleton driver
• Study and experiment with RomFS as a simple
  read-only driver
• Gain an in-depth knowledge of how to read
  ReiserFS disk structures from the GRUB ReiserFS
  code

18
Progress

• 49 days down.35 days to go.
• Eeep!

19
ACDM Experiment Schedule

• Ive created an experiment plan to mitigate
  technical risks and deliver functionality early
• I keep a continuously updated risk log as I
  research and perform experiments
• These risks are explained fully in the ACDM plan

20
Looking Behind

• I am currently on schedule with my experiment
  plan
• Experiments used to setup the development and
  debug environments are complete
• Experiments used to determine the development
  strategy are complete
• Experiments for how to create a minimal-feature
  FSD are complete
• Substantial amount of research complete
• However, I missed last week, and am 36 hours
  behind my projected time allotment

21
Details on Test Setup
Host Operating System (Linux)
Legend
Virtual Machine Software
ReiserFS VFS
Virtual Disk
Read connection via ReiserDriver
Existing Program

• Minimal Linux
• Tools to modifyReiserFS partitions

Test WinXP
Created Program
Shared ReiserFS Virtual Disks(Duplicated)
Test Harness
Virtual Machine
FSmon
Physical Machine
Dev. WinXP
DebugView
ReiserDriver
Visual Studio
Rfstool
Object View
rfstool
IFS Kit / DDK
ReiserFS VFS
WinDbg
WinDbg
Virtual Serial Port
22
Project Logistics

• The project website is deployed
• Important to establish a user base
• Acts as an information radiator
• The SourceForge project has been set up
• CVS database for code configuration management
• Archived mailing list for project discussion
• File release system for documentation resources

23
Looking Ahead

• The experiment plan now shifts from risk
  assessment and research, to development
• Development is incremental, delivering
  independent sections of code that will be merged
  into one code base
• It will be critical to monitor my progress and
  efficiency through these early experiments
• Technical risk remains very high!
• However, ACDM has helped structure mitigation
  through experimentation
• I have a higher confidence in creating the
  baseline functionality
• but a very low confidence in creating an
  industrial-strength IFSD
• Even in ext2fsd, bugs have taken several months
  to surface and be understood!

24
Questions?