Integrating Linux Technology with Condor

1
Integrating Linux Technology with Condor Kim van
der Riet Principal Software Engineer

2
Background

• Red Hat is investing in a set of defining
  technologies, actively being developed in the
  emerging technologies group and in conjunction
  with partners. These key projects include among
  others...
• Real-time (provides pre-emption and latency
  determinism)
• Virtulization (provides scale, VM, and ability to
  provide environment with Job)
• Messaging (built on AMQP)
• Stateless (ability to execute from read only
  image/device)
• ....and will work with the Condor team to provide
  native scheduling support in Linux.
• The combination of these technologies in a
  standard Linux distribution will provide a new
  generation of compute platforms.

2
3
Creating a stronger Linux compute platform

• Many large IT shops have requirement to schedule
  work to a compute platform.
• The work on Grids range from financial services
  grids, build / test environments, scientific
  analysis, batch processing, data mining, to name
  a few.
• Red Hat believes that having the key facilities
  included with and integrated into Linux will
  provide economy of scale and the ability to more
  efficiently manage compute workloads. Examples
  include
• Condor scheduler support for libvirt can provide
• Seamless delivery of an environment with a Job
• Stop, start, location transparency for execution
• Condor integration with messaging (AMQP) in the
  OS can provide
• Management of jobs together with the OS resources
• Standard interfaces for scheduling/reaping data
  on the grid
• Condor integration with d-cache and messaging
  allows parallel scheduled job execution (for
  things like rate-quote in FS).
• Stateless provides the ability to remotely host
  execution environments.
• Working together we (UW-Madison, RH, community,
  3rd party vendors) can provide a stronger, more
  effective compute platform.

3
4
What will Red Hat be doing?

• Red Hat will be investing into the Condor project
  locally in Madison WI, in addition to driving
  work required in upstream and related projects.
  This work will include
• Engineering on Condor features infrastructure
• Should result in tighter integration with related
  technologies
• Tighter kernel integration
• Information transfer between the Condor team and
  Red Hat engineers working on things like
  Messaging, Virtualization, etc.
• Creating and packaging Condor components for
  Linux distributions
• Support for Condor packaged in RH distributions
• All work goes back to upstream communities, so
  this partnership will benefit all.
• Shameless plug If you want to be involved, Red
  Hat is hiring...

4
5
Some questions answered

• Will Condor on RHEL / Fedora / Linux have an
  advantage?
• Yes, tighter integration with the OS and working
  with both Red Hat and other upstream communities
  should provide benefits.
• Will Condor become Linux only?
• No, Condor will remain a cross-platform project.
  However, there may be unique advantages to
  running it on Linux
• Is Red Hat going to provide technical support for
  Condor? At what level? At what cost?
• Yes, as a result of this work, support will be
  provided. No specifics on the cost can be
  provided at this time, however.
• Is Condor going to be packaged as part of the Red
  Hat distribution?
• Yes, it will become part of the distributions
  much the same way that related projects do.

5
6
Some questions answered (2)

• What is going to be the relationship with regards
  to the source code? Is there going to be a fork
  in the code, aka a UW Condor and a Red Hat
  Condor?
• Red Hat will co-develop with the Condor team.
  Work will be performed upstream, and all efforts
  will be made NOT to fork any code.
• What kind of users/customers is Red Hat targeting
  with this relationship with Condor? High
  Performance/Throughput market? Data centers?
• Finance, Scientific, Large-scale build/test, but
  not limited to specific markets.
• Can I expect to see Red Hat engineers responding
  to questions/posts on Condor mailing
  lists/newsgroups? What will be the relationship
  the existing Condor user community?
• Yes. Red Hat strives to play an active roll
  within the communities with which it works.

6
7
Some questions answered (3)

• Why is Red Hat doing this? What is the benefit
  for Red Hat? Conversely, what would be the
  benefit to the existing Condor community?
• Red Hat sees compute as a strategic technology.
• Red Hat customers will see a more powerful
  platform when running Linux.
• The Condor team will have access to Red Hat's
  deep technical expertise.
• What is the time frame of the commitment? Is it
  for X years? Or indefinite?
• The relationship is not limited by time, but we
  anticipate extensive development for at least 3
  years.
• How much effort (manpower) is Red Hat targeting
  for this effort? How much for support activities
  vs. development activities?
• Red Hat expects to bring at least 12 full time
  engineering resources onto the project, mostly
  located in Madison.
• Red Hat will not disclose resource effort for
  support/productization.

7
8
Some questions answered (4)

• What will Red Hat's development activities focus
  on? Why did you decide on these activities?
• The initial focus of the work will be
• Creating a distribution of Condor specifically
  for Linux.
• Integration with Linux Virtualization.
• Integration with Messaging (RHM / AMQP).
• Development of d-cache.
• These are areas where users are spending a lot of
  effort to create compute platforms today.
• What does this relationship mean for Condor on
  Windows? on Mac OS X?
• Nothing, this work will not directly benefit
  these platforms, but neither will it hurt them.
• What are the implications of this relationship
  upon the Linux kernel?
• It creates information flow between the teams, it
  also makes it possible for to Condor to utilize
  specialized kernel features. Perhaps most
  interesting is that it opens the door for
  VM-specific or virtualization models designed
  specifically for Condor and/or Grid computing in
  general.

8
9
Resources

• Contact information
• Carl Trieloff, Red Hat Emerging Technologies -
  cctrieloff_at_redhat.com
• Kim van der Riet, Red Hat Emerging Technologies -
  kim.vdriet_at_redhat.com
• If you want to join the team
• Go to www.redhat.com/careers and post a copy of
  your resume.
• E-mail one of the above contacts.
• Related Projects
• AMQP - www.amqp.org
• RHM / Qpid - incubator.apache.org/qpid
• Libvirt - www.libvirt.org
• REHL - www.redhat.com
• Fedora - fedora.redhat.com
• Red Hat emerging technologies (ET) projects -
  et.redhat.com

9
10
(No Transcript)