The Roadmap to New Releases

1
The Roadmap to New Releases
Todd Tannenbaum Department of Computer
Sciences University of Wisconsin-Madison http//ww
w.cs.wisc.edu/condor tannenba_at_cs.wisc.edu
2
Stable vs. Development Series

• Much like the Linux kernel, Condor provides two
  different releases at any time
• Stable series
• Development series
• Allows Condor to be both a research project and a
  production-ready system

3
Stable series

• Series number in version is even (e.g. 6.2.0)
• Releases are heavily tested
• Only bug fixes and ports to new platforms are
  added on a stable series

4
Stable series (cont.)

• A given stable release is always compatible with
  other releases from the same series
• Recommended for production pools

5
Development Series

• Series number in the version is odd (e.g. 6.1.17,
  6.3.0)
• New features and new technology are added
  frequently
• Versions from the same development series are not
  always compatible with each other

6
Development Series (cont.)

• Releases are not as heavily tested
• Generally not recommended for production pools
• unless new features are required
• unless we recommend otherwise )

7
Where is Condor Today?

• New stable version 6.2.0 recently released
• Previous development series cycle, ver 6.1., was
  far too long
• This was our first stable/development series
  we're hopefully learning...

8
New Ports in 6.2.0

• Full support (with checkpointing and remote
  system calls)
• All current versions of Linux (x86)
• Kernel 2.2. and 2.0.
• C Library glibc-2.01 and libc-5
• NOTE libc5 support dropped
• Solaris 2.7 and 2.8(Sparc and x86)
• Irix 6.5

9
New Ports in 6.2.0 (cont.)

• Clipped" support (no checkpointing or remote
  system calls, but all other functionality is
  available)
• Windows NT
• Alpha Linux

10
What is New in 6.2.0?

• Personal Condor and Grid Support
• Flocking
• Proof-of-concept support for
• Globus Job Universe
• Globus Glide-In
• Full, integrated support for Symmetric
  Multi-Processor (SMP) machines

11
What's New in 6.2.0? (cont.)

• PVM support
• DAGMan (for managing inter-job dependencies)
• Use of machine ClassAd attributes in submit file
• Example
• Executable foo.(ARCH).(OPSYS)

12
What's New in 6.2.0? (cont.)

• Greatly expanded I/O support for STANDARD
  universe jobs
• Condor can automatically buffer I/O requests from
  jobs
• Compression/decompression on-the-fly!
• Users get much more information about the kinds
  of I/O their jobs are performing
• Users can "remap" files to alternate locations

13
What's New in 6.2.0? (cont.)

• CondorVersion and CondorPlatform strings included
  in all binaries and libraries
• Helps identify and avoid problems with having the
  wrong version installed
• Different parts of the Condor protocol
  automatically check for version incompatibilities

14
What's New in 6.2.0? (cont.)

• Better accounting
• Historical usage logging in collector
• Use condor_stats
• web interface called CondorView
• Accountant stores usage information per user
• Better control over user priorities
• "Priority factors"

15
What's New in 6.2.0? (cont.)

• More powerful administration tools
• Setting configuration values remotely
• Querying daemons directly for status
• Other misc new tools
• condor_findhost, condor_hold/release,
  condor_qedit, condor_userlog, condor_run,
• Lots of performance and bug fixes
• A complete list is online in the manual
  (www.cs.wisc.edu/condor/manual)

16
The 6.3 Development Series

• Major focus areas for ver 6.3.x Series
• Future-aware scheduling for dedicated resources
• MPI support
• Security enhancements
• Secure network communication
• Kerberos, PKI (GSI/OpenSSL)
• Limitations on executable, remote syscall sandbox
• Condor-G production version
• Integrate Bypass technology
• Enhanced Windows support

17
What else will be added in 6.3.?

• And also several easy-to-add, user-visible
  features
• Master agents" - helper programs spawned by the
  condor_master to aid in administration
• Retrieving remote log, history and/or
  configuration files
• Remote "top", "ps" and other monitoring functions

18
What will be added in 6.3.? (cont.)

• Solution for Condor clusters on
• Private network
• Behind a firewall
• New condor_starter and condor_shadow for the
  VANILLA universe on Unix
• "NT version" will be used for Unix, too
• Lots of new features, like transferring files
  automatically for "vanilla" jobs (no need for a
  shared filesystem)

19
What will be added in 6.3.? (cont.)

• condor_startd will enforce resource limits
  dynamically (like RAM usage)
• More logic put into the condor_shadow to
• Detect temporary problems with a job's execution,
  put the job on hold, and notify the user
• Enforce job submitting users policy
• Better propagation of error messages

20
The 6.5. Development Series

• 6.5. will be for adding fundamentally new
  technology to Condor
• Being developed in parallel with 6.3.
• Will ultimately be v7.0
• Will be incompatible with previous versions of
  Condor

21
New Technology in 6.5.

• New version of ClassAds
• Conditionals !!
• if/then/else
• Aggregates (lists, nested classads)
• Built-in functions
• String operations, pattern matching, time
  operators, unit conversions
• Clean implementations in C and Java
• ClassAd collections

22
Other changes for 6.5.

• Re-write of the condor_schedd
• Performance enhancements and lowered resource
  requirements (particularly RAM)
• Re-write of the checkpoint server
• NEST technology infusion
• Enhanced support for multiple servers
• Will meta-data along with checkpoint files

23
Planned Future Ports

• Full support
• Intel Linux 2.4 kernel (RedHat 7.x)
• Clipped support
• Windows 2000 / Whistler

24
Possible Future Ports

• Full Support (I.e. checkpointing, etc)
• Windows 2000
• Alpha Linux
• DUX 5.x
• Clipped-support
• free,open,netBSD
• MacOS X
• PowerPC Linux
• HPUX 11.0
• AIX 4.2

25
Thank you for coming to Paradyn/Condor Week!