Windows System Programming using Python

1
Windows System Programming using Python

• Mark Hammond
• mhammond_at_skippinet.com.au
• OReilly Open Source Python ConferenceAugust
  1999, Monterey, CA

2
About this Presentation

• Most content taken directly from upcoming book
  for OReillyPython Programming on Win32By Mark
  Hammond and Andy Robinson
• http//www.ora.com/catalog/pythonwin32/

3
Who this talk is for?

• Existing Python programmers
• Even those without any Windows experience should
  follow this without problem.
• Although there is not enough time to explain the
  relevant Windows APIs
• Existing Windows Programmers
• Even without Python experience, you should
  immediately see the similarities between your
  existing language.

4
What is Python

• Interpreted, dynamic high-level language
• Obviously open source
• Hence we are here.
• Often used in a similar problem domain to
  Perl/Tcl
• Proponents consider readability and
  maintainability a big strong-point
• http//www.python.org

5
Python and Windows

• Each Python version comes with an installer
  package for Windows.
• Standard Python port contains all cross-platform
  Python features, but very few Windows specific
  features.
• Python for Windows extensions contains many
  useful Windows extensions for Python.
• http//www.python.org/windows

6
Python Windows Extensions

• Includes
• Pythonwin MFC based GUI environment and
  IDE/Debugger
• win32com Interfaces Python and COM
• win32 Extensions Interfaces to native Win32 API.
• Official releases can be found at
  http//www.python.org/windows
• Extensions home is at http//starship.python.net/c
  rew/mhammond

7
System Level Programming?

• For this talk, we define system level programming
  as working with low-level features of Windows
  Files, Pipes, Processes, Threads, Services, Event
  Log and so forth.
• Python and similar languages really not suitable
  for device-driver type development, and other
  more system-like Systems Programming!

8
Why not just use Python?

• Python has excellent native support for files,
  processes, threads etc.
• These features are typically limited to those
  defined by ANSI C.
• Many advanced Windows features are not exposed
  using these interfaces.
• Standard implementation of some of the standard
  library functions leaves a little to be desired
  in a Windows environment.

9
Portable Process Control (1 of 3)

• Standard Python functions all work
• Just often not quite how we would like!
• os.system()
• import os
• os.system(notepad C\\autoexec.bat)
• Problems
• Creates a new console window when run from a GUI.
• Waits for process to terminate.

10
Portable Process Control (2 of 3)

• os.execv family
• Doesnt search system path, and doesnt parse
  command lines
• os.execv("c\\Winnt\\notepad.exe", \
• ("c\\autoexec.bat",) )
• Does clobber your existing process - the call to
  os.execv() never returns!

11
Portable Process Control (3 of 3)

• os.popen()
• gtgtgt file os.popen("echo Hello")
• gtgtgt file.read()
• 'Hello\012'
• Works fine from Windows NT console programs, but
  fails miserably from a GUI!
• win32pipe module in the Win32 extensions provides
  a working replacement.

12
Better Process Control (1 of )

• win32api module provides some high-level, Windows
  specific functions.
• win32api.WinExec()
• Very similar to os.system(), but overcomes
  limitations.
• gtgtgt import win32api
• gtgtgt win32api.WinExec("notepad")
• Optional parameter allows you to specify the
  Windows initial state (eg, minimized)

13
Better Process Control (2 of 2)

• win32api.ShellExecute()
• Typically opens documents - eg, execute
  foo.doc, and (typically) Word will open.
• Finer control over the new process.
• Can also execute arbitrary executables - not
  limited to documents.
• For example, to print a specific document
• win32api.ShellExecute(0, "print", \
• "MyDocument.doc", None, "", 1)

14
Ultimate Process Control (1 of 2)

• win32process module exposes the low level Win32
  API.
• Full support for CreateProcess,
  CreateProcessAsUser, CreateThread etc.
• Full support for Windows Handles
• Files can be passed as stdin/out/err
• Process and thread handles are waitable using the
  win32event module.

15
Ultimate Process Control (2 of 2)

• Able to set thread and process priority and
  affinity levels
• Very handy for bugs that only appear in
  multi-processor machines.
• Able to do all this for both existing and new
  processes.
• Process usage samples included in distribution.

16
Introduction to our Sample

• Full sample Windows NT Service
• Provides event log and performance monitor
  information. Clients connect using Named Pipes
• Less than 75 lines of code for all this
  functionality.
• Still too big to present in one hit
• Selected excerpts included in slides
• Full code on CD, and athttp//starship.python.net
  /crew/mhammond/conferences/ora99

17
Portable Files and Pipes

• Python has excellent built-in file support
• Inherits platform stdio support - wont bother
  discussing them here.
• Native Windows files only useful for
  highly-advanced features, such as
• Overlapped IO
• IO Completion Ports
• Named Pipes
• NT Security requirements

18
Native Files (1 of 4)

• win32file.CreateFile() used for most file
  operations
• Create and open regular files, memory mapped
  files, etc.
• Takes seven parameters - c.f. open()s two!
• Returns a PyHANDLE object
• Can be passed to any Python API wrapper expecting
  a handle.
• Auto-closed when last reference removed.

19
Native Files (2 of 4)

• Overlapped IO for asynchronous operations
• File opened for overlapped IO requires a Windows
  event object.
• All IO operations return immediately.
• Event object signalled when IO complete
• Great for high-performance servers
• Simple to support multiple concurrent file
  operations per thread.

20
Native Files (3 of 4)

• NT Completion ports for even better asynchronous
  control
• Windows manages associating the completed IO
  operation with a connection
• More complex to use.
• Often requires a state-machine implementation.
• Offers excellent performance - Microsofts
  recommended architecture for scalable,
  high-performance servers.

21
Native Files (4 of 4)

• Full support for NT Security
• Default security can be used by passing None
• Many real-world applications require explicit
  security configuration
• Full support for Windows Impersonation
• Processes can automatically impersonate the
  remote pipe client.
• Impersonate any user given their password.

22
File Sample (1 of 2)

• Overlapped IO is used
• self.overlapped \
• pywintypes.OVERLAPPED() create the event
  to be used.self.overlapped.hEvent \
  win32event.CreateEvent(None,0,0,None)
• Special security for pipes is needed for
  services
• sa win32security.SECURITY_ATTRIBUTES()
  Allow full access!sa.SetSecurityDescriptorDacl (
  1, None, 0 )

23
File Sample (2 of 2)

• Create pipe and connect to client pipeHandle
  \ win32pipe.CreateNamedPipe(pipeName,
  openMode, pipeMode,
  win32pipe.PIPE_UNLIMITED_INSTANCES, 0,
  0, 6000, 6 second timeout. sa)...hr
  win32pipe.ConnectNamedPipe(pipeHandle,\
  self.overlapped)

24
Windows NT Services

• Similar concept to a Unix daemon
• A few special requirements
• Must respond to asynchronous commands from NT to
  (e.g.) Shutdown
• Must be capable of reporting status to NT
• NT has built-in UI for configuring and
  controlling services.

25
Python controlling Services

• Full exposure of the Windows NT Service Control
  Manager API
• Start, Stop, Pause Services, Install or Remove
  services, etc
• win32serviceutil module makes it simple
• gtgtgt win32serviceutil.StopService("Messenger")(3
  2, 3, 0, 0, 0, 6, 20000)gtgtgt win32serviceutil.Star
  tService(...)gtgtgt

26
Implementing Services

• Simple to implement Services in Python
• Smallest service is around 16 lines of code!
• Includes debug support, self-installation, and
  fully controllable from Windows NT
• Few more lines needed to make it something useful
  -)
• Simply sub-class Service base-class, and
  implement control features
• Minimum required is StopService
• Trivial to implement most controls

27
Windows NT Event Log

• Central place for programs to log information.
• Ideal for Services - can not present effective
  GUIs
• Benefits to programmer
• Built in transaction and thread safety, maximum
  size ability, etc.
• Benefits to Administrator
• Central log of messages, and third party tools to
  help analysis.

28
Python reading the Event Log

• Complex native API from win32evtlog
• Simpler interface from win32evtlogutil
• Define Feeder function
• def DumpRecord(record) print Got event ID,
  record.EventID
• And feed it!
• win32evtlogutil.FeedEventLogRecords( \
  DumpRecord)Got Event ID
  -2147483645Got Event ID -2147483645

29
Python writing the Event Log

• More complex than reading - event sources must be
  registered.
• win32evtlog and win32evtlogutil used here too
• Python Service framework also supports simple
  event log writing

30
Writing the Event Log Sample

• Our sample uses the Service Framework functions
  which makes it trivial
• Events are logged by ID, rather than explicit
  text.
• Our sample uses a built-in ID to log a service
  starting message
• import servicemanagerservicemanager.LogMsg(
  servicemanager.EVENTLOG_INFORMATION_TYPE,
  servicemanager.PYS_SERVICE_STARTED,
  (self._svc_name_, ''))

31
NT Performance Monitor

• Built-in NT tool for monitoring an applications
  performance.
• Application must be written to provide this data.
• API designed for smallest impact on program
  supplying data
• Moves the burden to the collecting application.
• Not trivial to work with

32
Python reading from PerfMon

• Useful for overcoming limitations in built-in
  tool
• For example, sample the data hourly and log to a
  longer-term database.
• Useful for reading standard information about
  another process.
• Process ID, Memory usage, etc.
• win32pdhutil module for reading data
• Good sample code in the source file

33
Python supplying PerfMon

• Complex installation and setup procedure.
• C .h file and custom .ini file necessary at
  application installation
• Supported by the Python Service framework
• Quite trivial to use once running
• Simply increment a counter - Performance Monitor
  handles conversions to required units.

34
PerfMon Sample

• Installation more than we can cover here
• See InitPerfMon() method
• Working with counters trivial
• Increment our connections counter each connection
• self.counterConnections.Increment()
• Perfmon manages converting to connections-per-seco
  nd automatically.

35
Our Sample in Detail (1 of 8)

• PipeService2.py
• The service implementation.
• PipeService2_install.hPipeService2_install.ini
• Required for performance monitor installation
• Most services dont need this!
• PipeServiceClient.py
• Sample client to connect to our service.

36
Our Sample in Detail (2 of 8)

• Service functionality in PipeService class
• Base class handles most of the grunt
• We simply supply service name and other optional
  attributes
• class PipeService(\
• win32serviceutil.ServiceFramework)
• _svc_name_ "PythonPipeService"
• _svc_display_name_ "A sample Python ... "

37
Our Sample in Detail (3 of 8)

• We respond to an NT ServiceStop request by
  telling NT we are stopping, and setting a Windows
  Event
• def SvcStop(self) Before we do anything,
  tell the SCM we are starting the stop
  process. self.ReportServiceStatus( \
  win32service.SERVICE_STOP_PENDING) And set my
  event. win32event.SetEvent(self.hWaitStop)

38
Our Sample in Detail (4 of 8)

• Overlapped IO means we can wait for either a
  connection, or our Stop request
• win32pipe.ConnectNamedPipe(pipeHandle,\
  self.overlapped)
• ...
• Wait for either a connection, or
• a service stop request.
• waitHandles self.hWaitStop, \
  self.overlapped.hEvent
• rc win32event.WaitForMultipleObjects(\
• waitHandles, 0, timeout)

39
Our Sample in Detail (5 of 8)

• Install our service simply by executing the
  service script
• If not for Performance Monitor, the command-line
  would be simple
• C\Scriptsgt PipeService2.py install
• But PerfMon needs an extra arg
• C\Scriptsgt PipeService2.py \--perfmoniniPipeSer
  vice2_install.ini installInstalling service
  PythonPipeService to ...Service installed

40
Our Sample in Detail (6 of 8)

• Once installed, we can start the service
• Start from Control Panel, or using the script
  itself
• C\Scriptsgt python.exe PipeService2.py start
• And start a client test session
• C\Scriptsgt python.exe PipeServiceClient.py \
  Hi thereThe service
  sent backYou sent meHi there

41
Our Sample in Detail (7 of 8)

• We will have Event Log records...

42
Our Sample in Detail (8 of 8)

• And Performance Monitor Data.

43
Summary

• Python library flexible, rich and portable enough
  for many tasks
• Low-level Windows programming achieved by using
  extensions that expose the raw Windows API
• This talk should have shown
• An overview of how this programming is done in
  Python.
• How simple it is to do complex Windows tasks.

44
More Information

• Python and Python Documentation
• http//www.python.org
• http//www.python.org/doc
• Python for Windows Extensions
• http//starship.python.net/crew/mhammond
• http//www.python.org/windows
• Reference Manuals and Samples included with the
  distributions

45
Thanks for coming

• Mark Hammond
• mhammond_at_skippinet.com.au
• http//starship.python.net/crew/mhammond