LuaSocket%20behind%20the%20scenes

1
LuaSocket behind the scenes

• Diego Nehab

2
Short Bio

• Graduated from PUC in CS E, 1999
• Worked in Tecgraf 1995-2002
• MSc in PL with Roberto, 2001
• 3rd year PhD candidate at Princeton
• Computer Graphics.

3
Outline of talk

• A few historical notes
• Case study SMTP support
• Protocol abstraction
• Message abstraction
• Implementation highlights
• Conclusions

4
Historical notes

• 1.0, 1999, 1.5k C, 200 man
• 1.1, 2000, 1.5k C, 1.3k Lua, 500 man
• added protocol support for HTTP, SMTP, FTP
• 1.2, 2001, 2k C, 1.3k Lua, 900 man
• buffered input and non-blocking I/O
• UDP support
• object oriented syntax

5
Historical notes

• 1.3, 2001, 2.3k C, 1.6k Lua, 1.2k man
• streaming with callbacks
• added select function
• 1.4, 2001-2, 2.2k C, 2.2k Lua, 1.9k man
• LTN7
• added URL module
• named parameters

6
Current version

• 2.0, 2005, 4.6k C, 2.5k Lua, 4.7k man
• Extensible C architecture, split in modules
• LTN12 (sources, sinks and filters)
• MIME support (partial but honest)
• Multipart messages support
• LTN13 (finalized exceptions)
• Package proposal
• Improved non-blocking code, robust to signals...

7
Outline of talk

• A few historical notes
• Case study SMTP support
• Protocol abstraction
• Message abstraction
• Implementation highlights
• Conclusions

8
SMTP (RFC2821)

• luaroberto telnet mail.tecgraf.puc-rio.br 25
• 220 tecgraf.puc-rio.br ESMTP Sendmail 8.9.3/8.9.3
• helo lua
• 250 tecgraf.puc-rio.br Hello lua, pleased to meet
  you
• mail from ltroberto_at_inf.puc-rio.brgt
• 250 ltroberto_at_inf.puc-rio.brgt... Sender ok
• rcpt to ltdiego_at_tecgraf.puc-rio.brgt
• 250 ltdiego_at_tecgraf.puc-rio.brgt... Recipient ok
• data
• 354 Enter mail, end with "." on a line by itself
• Subject World domination instructions.
• Commence stage two.
• .
• 250 RAA10452 Message accepted for delivery
• quit
• 221 tecgraf.puc-rio.br closing connection

9
Protocol abstraction

• status, error smtp.send
• from "ltroberto_at_inf.puc-rio.brgt",
• rcpt "ltdiego_at_tecgraf.puc-rio.brgt",
• body "Subject World domination
  instructions.\r\n\r\n" ..
• "Comence stage two."

• What if body is large?

10
LTN12 sources

• Use callback function that produces data
• Returns one chunk each time called
• Signals termination returning nil.

• function ltn12.source.file(handle)
• return function()
• local chunk handleread(BLOCKSIZE)
• if not chunk then handleclose() end
• return chunk
• end
• end

11
Using sources

• status, message smtp.send
• from "ltroberto_at_inf.puc-rio.brgt",
• rcpt "ltdiego_at_tecgraf.puc-rio.brgt",
• body ltn12.source.file(io.open("/mail/body",
  "r"))

• What if body is complicated?

12
Message Format (RFC2822)

• From Roberto Ierusalimschy ltroberto_at_inf.puc-ri
  o.brgt
• To Diego Nehab ltdiego_at_tecgraf.puc-rio.brgt
• Subject World domination roadmap.
• Content-Type multipart/mixed boundarypart
• This message contains attachments
• --part
• Content-Type text/plain
• Please see attached roadmap.
• --part
• Content-Type text/html name"roadmap.html"
• ...
• --part--

headers
headers
part 1
body
body
part 2
13
Message abstraction

• declaration
• headers
• subject "World domination",
• from "Roberto ltroberto_at_inf.puc-rio.brgt",
  
• to "Diego ltdiego_at_tecgraf.puc-rio.brgt"
• ,
• preamble "This message contains
  attachments.",
• 1
• headers ... ,
• body "Please see attatched roadmap."
• ,
• 2
• headers ... ,
• body ltn12.source.file(io.open("/plans/r
  oadmap.html", "r"))

14
Our message API

• status, message smtp.send
• from "ltroberto_at_inf.puc-rio.brgt",
• rcpt "ltdiego_at_tecgraf.puc-rio.brgt",
• body smtp.message(declaration)

• Transform declaration into an LTN12 source
• Pass source as body to sending function.

15
How hard is it?

• Message structure is recursive
• Need to return chunks but mantain context
• Nightmare to write in C!
• Use coroutines
• Write function recursively, naturally
• Call yield with each chunk
• Next call resumes wherever we left.

16
Zoom in on attachments

• 2
• headers
• "content-type" 'text/html
  name"roadmap.html"',
• "content-disposition" 'attachment
  filename "roadmap.html"'
• ,
• body ltn12.source.file(io.open("/plans/roadm
  ap.html", "r"))

• Would like to send PDF
• Binary data has to be encoded (Base64)
• Want to encode on-the-fly.

17
LTN12 filters and chains

• Filters process data one chunk at a time
• MIME module provides common filters
• base64, quoted-printable, stuffing, line-wrap...
• Can chain two filters together factory
• Produce a filter with the composite effect
• Can chain a filter with a source factory
• Produce a source that returns filtered data.

18
Zoom in on attachments
2 headers "content-type"
'application/pdf name"roadmap.pdf"',
"content-disposition" 'attachment filename
"roadmap.pdf"', "content-description"
'Detailed world domination plan',
"content-transfer-encoding" 'BASE64' ,
body ltn12.source.chain(
ltn12.source.file(io.open("/plans/roadmap.pdf",
"r")), ltn12.filter.chain(
mime.encode("base64"),
mime.wrap("base64") ) )
19
Creating filters high-level

• Chunks can be broken arbitrarily
• Filters have to keep context between calls

function ltn12.filter.cycle(low, ctx, extra)
return function(chunk) local ret
ret, ctx low(ctx, chunk, extra) return
ret end end function mime.normalize(marker)
return ltn12.filter.cycle(mime.eol, 0,
marker) end
20
Creating filters low-level
int eol(lua_State L) int ctx
luaL_checkint(L, 1) size_t isize 0
const char input luaL_optlstring(L, 2, NULL,
isize) const char last input isize
const char marker luaL_optstring(L, 3,
CRLF) luaL_Buffer buffer
luaL_buffinit(L, buffer) while (input lt
last) ctx translate(input, ctx,
marker, buffer) luaL_pushresult(buffer)
lua_pushnumber(L, ctx) return 2
21
Creating filters low-level
define candidate(c) (c CR c LF) int
translate(int c, int last, const char mark,
luaL_Buffer buffer) if (candidate(c))
if (candidate(last)) if (c
last) luaL_addstring(buffer, mark)
return 0 else
luaL_addstring(buffer, mark) return
c else
luaL_putchar(buffer, c) return 0
22
SMTP dependencies
socket
tp
smtp
ltn12
mime
23
Error checking

• Function return convention
• Return nil, followed by message on error

function metat.__indexgreet(domain) local r,
e self.tpcheck("2..") if not r then return
nil, e end r, e self.tpcommand("HELO",
domain) if not r then return nil, e end
return self.tpcheck("2..") end

• Tedious, error prone, virotic, not finalized.

24
LTN13 exceptions

• try newtry(finalizer) factory
• On success, try returns all arguments
• On failure, throws the second argument
• Calls finalizer before raising the exception.
• foo protect(bar) factory
• foo executes bar in a protected environment
• Returns nil followed by any thrown error.

25
No 'if' statements
function metat.__indexgreet(domain)
self.try(self.tpcheck("2.."))
self.try(self.tpcommand("HELO", domain))
return self.try(self.tpcheck("2..")) end

• Internal functions throw exceptions
• try calls tp.close() on error
• External functions can be protected.

26
Conclusions

• Hope you like our API, we do
• It is easy to implement
• Function factories clusures, coroutines
• It is fast
• Time critical in C, management in Lua
• Questions?