DIFFSYNC A file synchronization program

1
DIFFSYNCA file synchronization program

• CLASS PROJECT EEL6883
• BY
• ADAM RADWAN
• GARGI CHIPALKATTI

2
FILE SYNCHRONIZATION

• Process of making sure that two or more locations
  contain the same up-to-date files
• One-way (Mirroring) or two-way file
  synchronization
• Useful for mobile workers, multiple computer
  users used for backup purposes
• Version control tools and File synchronization
  tools
• Various existing solutions, none of which were
  found by OpenPublic Networks, LLC to be robust
  enough for production use in a distributed
  environment

3
GOALS

• Develop a file synchronization program which is
  capable of working over a diverse range of
  geographical placement
• Must be capable of running securely over common
  public network backbone, such as the Internet
• Capable of two-way synchronization of files for
  changes to served web pages
• Ability for a non-functional server to gracefully
  degrade while other servers detect this and
  recover appropriately to keep services running

4
RSYNC

• Open Source software for Unix systems
• Mirroring takes place with only one transmission
  in each direction
• Used to synchronize local directories or
  directories via a remote shell such as RSH or SSH
• Two checksums a weak rolling 32-bit checksum
  and a strong 128-bit MD4 checksum

5
UNISON

• Runs on Unix-like operating systems (Linux, Mac
  OS X, and Solaris), and on Windows
• Can synchronize files across platforms
• Detects 'conflicts' where a file has been
  modified on both sources, and displays these to
  the user
• Uses the rsync algorithm
• Designed to be robust in the event of a program
  or system crash or a communication failure

6
DIFFSYNC

• Uses open source software rsync, inotify
• Uses PHP scripting language
• Built on top of a MySQL 5.1 triple-master
  replicated database system running in chrooted
  jail for security
• Custom SSH-based VPN for three independent
  point-to-point TUN links between the servers
• Apache-based web server for verification running
  in chrooted jail for security
• phpMyAdmin used for database management

7
DIFFSYNC FOR THREE SERVERS
Podice Location Fayetteville, NCIP
216.117.160.235CPU Pentium 4 2.8 GhzStorage
80GB, 512 MBOS Gentoo Linux
Akimbo Location Orlando, FLIP
72.189.195.130CPU Pentium III 800 MhzStorage
20GB, 128 MBOS Gentoo Linux
Philomena Location Ft. Lauderdale, FLIP
65.111.166.49CPU AMD 2000Storage 40GB, 256
MBOS Gentoo Linux
8
CREATION OF DIRECTORY FILE
9
MOVE DIRECTORY FILE
10
ATTRIB DIRECTORY FILE
11
DELETE DIRECTORY
12
DELETE FILE
13
EDIT FILE
14
COPY FILE
15
DIFFICULTIES ENCOUNTERED

• Must have a kernel module installed on the host
  server for inotify events to be recorded
• Problems detecting changes made to newly-created
  directories within the specified inotify watch
  point
• MySQL replication is sensitive to binary log
  changes when one node is disconnected
• Accurate prediction and parsing of inotify events
  for complete analysis of changes made to
  filesystem
• Not enough time in the semester to complete
  integration of all features into program

16
CONCLUSION

• Fundamentals of DiffSync program prove that file
  synchronization is possible for servers in remote
  locations
• Additional work will be completed to make this
  program ready for a production environment
• Any questions?

17
THE END

• THANK YOU!