Extending Python with C (Part I

1
Extending Python with C(Part I the
Basics)June 2002

• Brian Quinlan
• brian_at_sweetapp.com

2
Why Mess with C?

• There is a preexisting library, available in C,
  that you would like to access from Python (e.g.
  pyexpat, math, zlib)
• Solving the problem in Python would be too CPU or
  memory inefficient (e.g. Numeric, PIL)
• Only way to create new fundament types (doesnt
  seem too important to me save it for another
  talk)

3
Jumpin In

• Create our own extension module containing an
  eclectic set of functions
• Well call it Vanpy
• Basic steps
• Do module initialization
• Write functions
• Write build script
• Testing

4
A module is just a dictionary

• foo.py
• magic_number 5
• def divide(x,y)
• return x / y
• gtgtgt import foo
• gtgtgt foo.__dict__
• '__doc__' None, 'magic_number' 5,
• 'divide' ltfunction divide at 007B1484gt,
• gtgtgt foo.__dict__'divide'(8,4)
• 2

5
Module Initialization

• void initVanpy(void)
• PyObject module
• module Py_InitModule3("Vanpy",
  module_functions,
• 
  module_doc)
• PyModule_AddStringConstant(module,
  "__version__",
• "0.0.1")

6
A VERY Simple Function

• static PyObject
• is_even(PyObject self, PyObject args)
• long number
• if (!PyArg_ParseTuple(args, "i", number))
• return NULL
• return PyBuildValue("i", number 2 0)
• static PyMethodDef module_functions
• "is_even", is_even, METH_VARARGS,
  is_even_doc,
• NULL, NULL

7
A Simple Function

• static PyObject
• classify_characters(PyObject self, PyObject
  args)
• char string
• long length
• long alpha, other 0
• if (!PyArg_ParseTuple(args, "s", string,
  length))
• return NULL
• for (int i 0 i lt length i)
• isalnum(stringi) ? alpha other
• return PyBuildValue("(iif)", alpha, other,
  ((float) alpha) / length)

8
Objects and Types

• typedef struct
• struct _typeobject ob_type
• int ob_refcnt
• / The rest depends on the type /
• PyObject
• typedef struct _typeobject
• char tp_name
• / Stuff to discuss in another talk /

9
Reference Counting

• gtgtgt a Hello
• PyObject a Py_BuildValue(s, Hello)

10
Reference Counting

• gtgtgt a Hello
• PyObject a Py_BuildValue(s, Hello)
• gtgtgt b a
• PyObject b a
• Py_INCREF(b)

a
b
11
Reference Counting

• gtgtgt a Hello
• PyObject a Py_BuildValue(s, Hello)
• gtgtgt b a
• PyObject b a
• Py_INCREF(b)
• gtgtgt del a
• Py_DECREF(a)

a
b
b
12
Reference Counting

• gtgtgt a Hello
• PyObject a Py_BuildValue(s, Hello)
• gtgtgt b a
• PyObject b a
• Py_INCREF(b)
• gtgtgt del a
• Py_DECREF(a)
• gtgtgt del b
• Py_DECREF(b)

a
b
b
13
Reference Counting Rules

• When returning a PyObject from an extension
  function or method, always increment the objects
  reference count
• USUALLY, when calling a Python API function, you
  need not increment the reference counts of
  PyObject arguments (only 2 important exceptions)
• USUALLY, when a PyObject is returned from a
  Python API function, you must decrement the
  objects reference count when you are done with
  it (gt 4 important exceptions)

14
A Realistic Example

• static PyObject password NULL
• static PyObject set_password(PyObject self,
  PyObject args)
• PyObject new_password
• if (!PyArg_ParseTuple(args, "O",
  new_password))
• return NULL
• if (password ! NULL) / or
  Py_XDECREF(password) /
• Py_DECREF(password)
• password new_password
• Py_INCREF(password)
• Py_INCREF(Py_None)
• return Py_None

15
A Realistic Example

• static PyObject
• get_password(PyObject self, PyObject args)
• if (!PyArg_ParseTuple(args, ""))
• return NULL
• if (password NULL)
• Py_INCREF(Py_None)
• return Py_None
• else
• Py_INCREF(password)
• return password

16
Exceptions

• Each Python thread has 3 exception variables
  associated with it
• The type of the exception (e.g. ValueError)
• The value of the exception (e.g. 0 denominator)
• A traceback object (e.g. file foo.py, line 1,
  in divide)
• def divide(x,y) raise ValueError, 0
  denominator
• static PyObject
• divide(PyObject self, PyObject args)
• PyErr_SetString(PyExc_ValueError, 0
  denominitor)
• return NULL

17
Passwords Revised

• static PyObject
• set_password(PyObject self, PyObject args)
• PyObject new_password
• if (!PyArg_ParseTuple(args, "O",
  new_password))
• return NULL
• if (!PyString_Check(new_password)
• !PyUnicode_Check(new_password))
• PyErr_Format(PyExc_TypeError, "expected
  string or "
• "Unicode , 80s found",
• new_password-gtob_type-gttp_name
  )
• return NULL
• ...

18
More Exceptions

• If you want to ignore an error, use PyErr_Clear
  to clear the current exception
• You can check to see if an exception has been set
  using PyErr_Occurred (this is not usually need
  since the API return value indicates failure)
• It is an error to return NULL without setting an
  exception

19
Custom Exceptions I

• static PyObject WeatherError
• void initVanpy(void)
• PyObject module
• PyObject module_dict
• module Py_InitModule3("Vanpy",
  module_methods,
• 
  module_doc)
• WeatherError PyErr_NewException("Vanpy.Weath
  erError",
• 
  NULL, NULL)
• module_dict PyModule_GetDict(module)
• PyDict_SetItemString( module_dict,
  "WeatherError",
• 
  WeatherError)

20
Custom Exceptions II

• static PyObject
• goto_beach(PyObject self, PyObject args)
• if (!PyArg_ParseTuple(args, ""))
• return NULL
• if (rand() 2)
• PyObject e Py_BuildValue("(si)", "too
  cold", 5)
• if (e NULL)
• return NULL
• PyErr_SetObject(WeatherError, e)
• Py_DECREF(e)
• else
• PyErr_SetString(WeatherError, "too
  rainy")
• return NULL

21
Building

• setup.py
• !/usr/bin/env python
• from distutils.core import setup, Extension
• setup( name "Vanpy",
• version "0.0.1",
• description "An eclectic set of
  functions",
• author "Brian Quinlan",
• author_email "brian_at_sweetapp.com",
• ext_modules
• Extension("Vanpy", "Vanpy.c")
• )

22
Python API organization

• Exception handling
• Memory Management
• Threads
• Utilities
• OS Utilities
• Process Control
• Importing Modules
• Data Marshalling
• Parsing Arguments Building Values
• Abstract Object Support
• Concrete Object Support

23
Abstract Object Support

• Applies to classes of objects, not specific types
• Divided into protocols
• Object
• Number
• Sequence
• Mapping
• Iterator
• Buffer

24
Object Protocol

• A sampling of functions in the object protocol
• int PyObject_HasAttrString(PyObject o, char
  attr_name)
• int PyObject_Cmp(PyObject o1, PyObject o2, int
  result)
• PyObject PyObject_Repr(PyObject o)
• PyObject PyObject_Str(PyObject o)
• int PyObject_IsInstance(PyObject inst, PyObject
  cls)
• PyObject PyObject_CallFunction(PyObject
  callable, char format, ...)
• PyObject PyObject_CallMethod(PyObject o, char
  method, char format, ...)
• int PyObject_IsTrue(PyObject o)

25
Number Protocol

• A sampling of functions in the number protocol
• int PyNumber_Check(PyObject o)
• PyObject PyNumber_Add(PyObject o1, PyObject
  o2)
• PyObject PyNumber_Subtract(PyObject o1,
  PyObject o2)
• PyObject PyNumber_Multiply(PyObject o1,
  PyObject o2)
• PyObject PyNumber_Negative(PyObject o)
• PyObject PyNumber_Absolute(PyObject o)
• PyObject PyNumber_Lshift(PyObject o1, PyObject
  o2)
• PyObject PyNumber_And(PyObject o1, PyObject
  o2)
• PyObject PyNumber_Int(PyObject o)
• PyObject PyNumber_Long(PyObject o)
• PyObject PyNumber_Float(PyObject o)

26
Sequence Protocol

• A sampling of functions in the sequence protocol
• int PySequence_Check(PyObject o)
• int PySequence_Length(PyObject o)
• PyObject PySequence_Repeat(PyObject o, int
  count)
• PyObject PySequence_GetItem(PyObject o, int i)
• PyObject PySequence_GetSlice(PyObject o, int
  i1, int i2)
• int PySequence_SetItem(PyObject o, int i,
  PyObject v)
• int PySequence_DelItem(PyObject o, int i)
• PyObject PySequence_Tuple(PyObject o)
• int PySequence_Contains(PyObject o, PyObject
  value)
• int PySequence_Count(PyObject o, PyObject
  value)

27
Concrete Object Support

• Applies to specific object types
• Different set of functions for every built-in
  Python type
• There any many Python object types plain int,
  long int, float, complex, string, unicode,
  buffer, tuple, list, dictionary, file, instance,
  method, module, iterator, slice, weak reference,
  etc.
• Reference counting rules are less intuitive

28
Int Objects

• This is the complete list of int functions
• int PyInt_Check(PyObject o)
• int PyInt_CheckExact(PyObject o)
• PyObject PyInt_FromLong(long ival)
• long PyInt_AsLong(PyObject io)
• long PyInt_AS_LONG(PyObject io)
• long PyInt_GetMax()

29
String Objects

• A sampling of string functions
• int PyString_Check(PyObject o)
• int PyString_CheckExact(PyObject o)
• PyObject PyString_FromString(const char v)
• PyObject PyString_FromStringAndSize(const char
  v, int len)
• PyObject PyString_FromFormat(const char format,
  ...)
• PyObject PyString_FromFormatV(const char
  format, va_list vargs)
• int PyString_Size(PyObject string)
• int PyString_GET_SIZE(PyObject string)
• char PyString_AsString(PyObject string)
• char PyString_AS_STRING(PyObject string)
• void PyString_Concat(PyObject string, PyObject
  newpart)

30
List Objects

• This is the complete list of list functions
• int PyList_Check(PyObject p)
• PyObject PyList_New(int len)
• int PyList_Size(PyObject list)
• int PyList_GET_SIZE(PyObject list)
• PyObject PyList_GetItem(PyObject list, int
  index)
• PyObject PyList_GET_ITEM(PyObject list, int i)
• int PyList_SetItem(PyObject list, int index,
  PyObject item)
• void PyList_SET_ITEM(PyObject list, int i,
  PyObject o)
• int PyList_Insert(PyObject list, int index,
  PyObject item)
• int PyList_Append(PyObject list, PyObject item)
  
• PyObject PyList_GetSlice(PyObject list, int
  low, int high)
• int PyList_SetSlice(PyObject list, int low, int
  high, PyObject itemlist)
• int PyList_Sort(PyObject list)

31
Helpful Technologies

• SWIG/SIP - generate C/C interface code for
  Python
• Boost Python - an interface generator for
  wrapping C classes with a Python interface
• CXX - C wrapper around the Python API
• Pyrex - a language for writing Python extension
  modules
• Pyfort - a wrapper generator for interfacing
  Fortran with Python
• PyInline/Weave embed C/C/Perl code inside
  your Python script