Linux Programming Interprocess Communication

1
Linux Programming Interprocess Communication

• CS 230
• ???

2
Overview

• IPC is a mechanism to transfer data between
  processes
• methods overview
• shared memory
• uncontrolled accesses to memory
• mapped file
• similar to shared memory, but with file
  operations
• pipes
• sequential communication
• FIFO
• unnamed pipe unrelated processes can communicate
• socket
• communication between different computers

3
Shared Memory
addr space of process I
addr space of process J
DRAM
main() attach()
main() attach()
page tbl
page tbl

• the fastest method
• write by a process can be seen by another process
  instantly
• no syscall intervention
• no data copying
• no synchronization is provided
• it is up to programmers responsibility

4
Programming Shared Memory

• segment_id shmget (key, size, IPC_CREAT
  IPC_EXCL S_IRUSR S_IWUSR)
• a brand new segement is created if
• key is IPC_PRIVATE or
• IPC_CREAT is asserted
• otherwise, segment_id of existing segment is
  returned
• you have to know the key
• size is rounded up to multiple of the page size
• S_Ixxx
• for read/write
• xxx for USR(owner) OTH(others) GRP(group)
• IPC_EXCL
• for segment creation(IPC_CREAT option), it
  guarantees the key is unique (not existing one)

5
Program Example
define KEY ((key_t)(1234)) define SEGSIZE
sizeof(struct some_data_structure)) struct
some_data_structure ap int id shmget(KEY,
SEGSIZE, IPC_CREAT 0666) if (id lt 0)
error_rtn(id) ap (struct some_data_structure
) shmat(id, 0, 0)
let the system choose the location of the segment
define KEY ((key_t)(1234)) define SEGSIZE
sizeof(struct some_data_structure)) struct
some_data_structure ap2 int id shmget(KEY,
SEGSIZE, 0) if (id lt 0) error_rtn(id) ap2
(struct some_data_structure ) shmat(id, 0, 0)
0 for R/W SHM_RDONLY
6
Shared Memory Control

• shmctl()
• provides information about the segment
• delete the segment when the last process finishes
• every segment should be deleted when it is done
• ipcs
• provides information about IPC resources
• -m for shared memory
• -s for semaphore
• -a for all
• ipcrm
• removes IPC resource
• discussions
• fastest way to communicate
• key should be known to processes
• synchronization should be arranged

7
Mapped File

• ptr mmap( / map the file to ptr /
• addr, / suggest ptr valur. 0 means let system
  choose /
• length, / size of mapped region /
• prot, / access PROT_READ PROT_WRITE /
• flags, / mapping type MAP_SHARED /
• fd, / file /
• offset) / location within the file /
• fd can be a device
• a device can be accessed without full blown
  device driver
• operations are more familiar than shared memory
• open, close, chmod, unlink, .

8
Network Communication Primer

• Protocol Stacks
• The data pass down the protocol stack at the
  sending end, and back up the stack at the
  receiving end.
• Entities at the same level in the protocol stack
  are called peers.
• The communication between peers is called peer to
  peer communication.

Peer-to-peer communication
Application protocol
User code
Application layer
Application layer
UDP/TCP protocol
Transport layer
Transport layer
Kernel code
IP protocol
IP layer
IP layer
Ethernet frames
Ethernet layer
Ethernet layer
Hardware
Bits
9
Data Encapsulation

• Addition of header information to a packet of
  data by a layer.
• The header contains just the information needed
  for that layer to do its job.
• Each layer of a protocol stack
• not aware of the header added by the lower layer.
• not attempt to interpret the data that have been
  handed down from the layer above.
• Process of Encapsulation

Application data
Application data
UDP header
Application data
UDP header
IP header
Application data
UDP header
IP header
Ethernet header
10
What is a Socket?

• An abstraction for connecting to a network
• Different flavors TCP and UDP
• TCP is a connection-oriented byte-stream protocol
  with reliable transmission (an end to end
  protocol)
• UDP is a connectionless best-effort datagram
  service (not much beyond IP itself)
• Both require specification of an IP address and
  port number
• When a socket is created, it has an associated
  protocol (TCP or UDP) but no address or port
  number
• The socket must be bound to an
  address/portthere are several ways more later

11
Sockets (cont.)

• Create a socket with the socket() call
• socket(int protocolFamily, int type, int proto)
• protocolFamily PF_INET for IP
• type is SOCK_STREAM (TCP) or SOCK_DGRAM (UDP)
• proto is end-to-end protocol IPPROTO_TCP or
  IPPROTO_UDP should be used (0 gives default,
  which is these anyway at the moment)

12
Sockets (cont.)

• The socket() call returns a descriptor
• A descriptor here is an integer which gives a
  handle to the socket
• Just like file handles returned from open()
• Its a reference that you store away and then use
  whenever you need to reference that socket in any
  future systems calls
• It should be nonnegative if socket() returns 1
  this indicates an error creating the socket
• close(int socket) when youre done

13
Socket Addresses

• Special datatype called sockaddr
• struct sockaddr
• unsigned short sa_family / AF_INET /
• char sa_data14 /
  Protocol-specific address formation /
• This is the generic datatype containing only
  the address family (AF_INET for IP) and the
  address specific to the protocol
• Of the 14 bytes, well use 4 bytes for the IP
  address and 2 for the port number

14
Socket Addresses (cont.)

• IP Specific version of the sockaddr struct
• struct sockaddr_in
• unsigned short sin_family / AF_INET /
• unsigned short sin_port / Port
  (16-bits) /
• struct in_addr sin_addr / Internet
  address (32-bits) /
• char sin_zero8 / Not used /
• The sockaddr_in is the internet version of the
  sockaddr struct. It can be cast to a generic
  sockaddr
• Its the right size and has a compatible first
  field

15
Connection-Oriented Client Server

• Asymmetric Client-Server Relationship
• Server
• passively sit waiting for work.
• not know where that work will come from.
• Client
• actively go out and connect to the server it
  requires.

16
Connection-Oriented Client Server

• Connection-Oriented Client and Server Operations.

Server operations
create SOCKET
Client operations
BIND a well-known port number to the socket
Establish a LISTEN queue for connections
create SOCKET
ACCEPT a connection
CONNECT to servers port
READ from connection
WRITE to connection
As required by application
As required by application
WRITE to connection
READ from connection
END-OF-FILE
CLOSE connection
17
Connection Oriented Client

• We know the steps
• 1) Create a socket
• 2) Establish a connection with connect()
• 3) Communicate with send() and recv()
• 4) Use close() to destroy the socket
• Calling connect
• int connect(int socket, struct sockaddr
  destAddress, unsigned int addressLen)
• socket is a socket descriptor,
• the sockaddr is generic,
• len is sizeof(struct sockaddr_in)

18
Calling send/recv

• int send(int socket, const void msg, unsigned
  int msgLen, int flags)
• int recv(int socket, void rcvBuffer, unsigned
  int bufferLen, int flags)
• msg is a pointer to the message, and msgLen is
  its length
• rcvBuffer captures the incoming bytes up to a max
  of bufferLen
• flags0 gives default behavior (more later)
• send blocks until ALL bytes transferred
• recv blocks until at least SOME bytes received
• send and recv return the number of bytes sent or
  received if 1 we have an error
• If recv returns 0, it means connection was closed
  at other end

19
Connection-Oriented Client (contd)

• Obtaining the IP address and the port number of
  the server
• server machine can be changed
• hard-wiring the IP address is clearly
  unacceptable
• typically, the client read a host name from its
  command line arguments
• gethostbyname() provide a mapping from machine
  names to IP address by using /etc/hosts (or NIS
  hosts map)
• port numbers rarely change
• hard-wiring the port number into the code is
  often satisfactory
• Alternatively, getservbyname() can be used to
  obtain the port number of a given named service
  by using /etc/services (or NIS servicess map)

20
A Client Program
main() int sockid struct sockaddr_in
serv_addr bzero((char ) serv_addr,
sizeof(serv_addr)) serv_addr.sin-family
AF_INET serv_addr.sin_addr.s_addr
inet_addr(SERVER_ADDR) serv_addr.sin_port
htons(SERVER_PORT) sockid socket(IF_INET,
SOCK_STREAM, 0) connect(sockid, serv_addr,
sizeof(serv_addr)) send(sockid, msg,
sizeof(msg),0) recv(sockid, buffer,
sizeof(buffer), 0) close(sockid) exit(0)
21
Constructing the Server

• Lets do TCP for now (UDP later)
• We know the method here as well
• 1) Create a TCP/IP socket
• 2) Assign a port number with bind()
• 3) Allow incoming connections with listen()
• 4) Repeat the following
• a) Call accept() to get a new socket for each
  connection
• b) Use send() and recv() to communicate via each
  socket
• c) Close the client connection when finished with
  close()

22
The bind() Call

• int bind(int socket, struct sockaddr localAddr,
  unsigned int addressLen)
• assigns a port number to the socket
• Once bind() succeeds, all connect() requests to
  this host for this address/port will go to this
  socket
• If the address is set to INADDR_ANY then all
  connections to this port will go to this socket,
  regardless of the Internet address used (useful
  if host happens to have multiple Internet
  addresses)

23
The listen() Call

• Once socket is created and bound, we need to
  begin listening for connection requests
• int listen(int socket, int queueLimit)
• socket is the socket descriptor
• queueLimit is the largest number of incoming
  connections which can be waiting (or queued)
• this socket is for setting up a connection only
• We never use send() or recv() on this socket once
  it is used for listen() calls instead we use
  this socket as a queue for incoming connections,
  and use accept() to dequeue the requests

24
The accept() Call

• accept() dequeues the next connection request
  from the listen sockets queue
• int accept(socket, struct sockaddr
  clientAddr, unsigned int addrLen)
• If queue is empty, accept() blocks until
  connection request is made
• socket is the listen socket
• clientAddr is filled in with clients address
• addrLen is filled in by you initially with max
  size of clientAddr, and changed by accept() to
  actual size of clientAddr
• accept returns new socket descriptor on success,
  -1 on failure

25
Server Interaction

• Once we have a connection (via accept()) we may
  interact with this client via send() and recv()
• Same as before for client-side
• Use the socket returned by accept() not the
  listen socket
• If we have multiple clients, we interact
  separately by referencing the appropriate socket
• When a particular client interaction is complete,
  use close() on the appropriate socket
• When server no longer wishes to accept incoming
  connections, call close on the listen socket

26
A Server Program

• main()
• int sockid, newsockid, childpid
• struct sockaddr_in cli_addr, serv_addr
• sockid socket(AF_INET, SOCK_STREAM, 0)
• bzero((char ) serv_addr, sizeof(serv_addr))
• serv_addr.sin-family AF_INET
• serv_addr.sin_addr.s_addr htonl(INADDR_ANY)
• serv_addr.sin_port htons(SERVER_PORT)
• bind(sockid, serv_addr, sizeof(serv_addr))
• listen(sockid, 5)
• for ( )
• / wait for a client connects /
• newsockid accept(sockid, cli_addr,
  sizeof(cli_addr))
• if ((childpid fork()) 0) / child
  process /
• close(sockid)

27
Interoperability

• All network info is big endian
• htons() host to network short
• htonl() host to network long
• ntohl() network to host long
• ntohs() network to host short
• Use these to send a multibyte binary value from
  one machine to another
• Any time you send an externally significant
  value (like an IP address or port number) the
  API expects network order
• Apply these funcs as the last operation before
  sending data, and as the first when receiving data

28
Alignment Issues

• Different architectures do alignment in various
  ways
• Ex Suppose I have a struct
• a (4 byte int), b (2 byte int), c (4 byte int)
• This may take 10 bytes on one machine, then I
  send (as a struct) to another machine which
  declares the same struct but pads b. We get
• a (4 byte int), b (2 byte int), 2 bytes of pad, c
  (4 byte int)
• This occurs because the machine may want c at an
  address which is a multiple of its size (4)
• Solutions
• Manually rearrange things so this doesnt happen
  (put 2 byte quantities at the end of the struct)
• Pad manually when sending to other machines

29
Framing and Parsing

• With TCP (and not UDP) one send() call does not
  necessary translate to one recv() call
• This means that the data sent must have framing
  info embedded within it (eg, spaces, nulls, etc)
• The receiving end must know how to retrieve the
  relevant info based on the framing data

30
Example Iterative Server

• char inbuf1000 / Data from client /
• char outbuf1000 / data to client /
• while(1)
• fd accept (sock, client, client_len)
• while( ( read (fd, inbuf, 1000) gt 0)
• / Process the data in inbuf, placing the
  result in outbuf /
• ...
• write (fd, outbuf, 1000)
• / When client close its end of the connection,
  we close our end.
• Otherwise we would eventually run out of
  file descriptors.
• Then we loop round to pick up another
  connection. /
• close (fd)

31
Concurrent Server

• Problem of Iterative Servers
• When Iterative servers maintain long-lived
  connections to their clients, new clients may
  wait for an arbitrarily long time
• Concurrent Server
• Server that can support multiple clients at the
  same time
• Two Approaches for Concurrent Server
• one child server process per client
• use the fork() system call.
• take care of zombies!
• single server process
• use the select() system call.

32
Concurrent Server - one child per client

• sock socket ( ... )
• bind (sock, ... )
• listen (sock, 2)
• while (1)
• fd accept ( sock, ... )
• if (fork() 0)
• / Child - process the request /
• ... / use fd to say to the client /
• exit(0) / Child is done /
• else
• close (fd) / Parent does not use the
  connection /

33
Single Process Server

• Single Process Server with select( )
• the server multiplex between the messages being
  received on each client connection
• The server return to the main processing loop
  after every interaction with client
• Advantage
• uses less system resources (memory and process
  table slots) since there is a single process.
• Disadvantage
• the server may be required to keep state
  information of each client
• the concurrent server keeps this state
  information in each child process

34
select() syscall

• select(int nfds, readfds, writefds,
  exceptfds, struct timeval timeout)
• readfds, writefds and exceptfds point to
  descriptor sets
• nfds is the highest fd number plus one
• return if any descriptor in the readfds writefds
  exceptfds set is ready for reading (or is
  ready for writing, or has an exceptional
  condition pending)
• else blocks until timeout
• if timeout 0, return immediately
• if timeoutNULL, block indefinitely
• Macros for clearing, setting, testing individual
  descriptors within a set
• FD_ZERO(fdset) / Remove all descriptor
  from set fdset /
• FD_CLR(fd, fdset) / Remove descriptor fd from
  set fdset /
• FD_SET(fd, fdset) / Add descriptor fd to set
  fdset /
• FD_ISSET(fd, fdset) / True if fd is present in
  the set fdset /

35
select() example

• fd_set myset
• FD_ZERO(myset) / Put file descriptors 3 and 4
  into myset /
• FD_SET(3, myset)
• FD_SET(4, myset)
• select (5, myset, NULL, NULL, NULL)
• if (FD_ISSET(3, myset))
• / read from descriptor 3 ... /
• else if (FD_ISSET(4, myset))
• / read from descriptor 4 ... /
• else
• printf(This should never happen)

36
select() syscall (contd)

• Because your descriptor sets get overwritten by
  the call, you usually have to keep a separate
  copy of them
• (ex) after the call, only file descriptors ready
  for reading remains in the descriptor set
• use memcpy(set1, set2, sizeof set2)
• When the call returns, there is no immediate
  indication of which file descriptor has woken you
  up. There will probably only be one, but to find
  it you have to poll each in turn

37
Connectionless Client and Server

• The server has a socket on which it receives
  datagrams
• Many clients may send datagrams to this socket,
  and in general the server will retrieve datagrams
  from several clients in an arbitrary, interleaved
  order
• Server that does not maintain state (stateless
  server)
• server simply reads the datagram, formulates a
  reply, sends it back to the client and forgets
  about it
• (ex) daytime, echo server
• Server that maintain state (stateful server)
• One approach is to parcel the per client state
  information into a structure
• Another approach is for the server to create a
  child process for each client
• easy to keep track of the state information of
  each server
• (ex) TFTP server

38
Connectionless Client and Server (2)

• Unreliability of connectionless client server
  (e.g., UDP)
• means that there is no mechanism built into the
  protocol to detect (and correct) problems such as
  dropped or misordered packet deliveries
• Assume that the underlying delivery system is
  reliable enough, and accept that the application
  will fail if the network fails.
• a timeout/retransmit mechanism may be built in at
  the application level.
• Syntactic difference between client and server is
  not much
• Both client and server simply create a socket,
  bind a port, and then send and receive datagrams
• just a matter of who goes first
• the server does not call listen() and accept()
• the client does not call connect()

39
Connectionless Client and Server (3)

• Connectionless Client and Server Operations

Server
Client 1
Client 2
create SOCKET
create SOCKET
BIND well-known port number
BIND any port number
SEND datagram
RECEIVE datagram
create SOCKET
SEND datagram
BIND any port number
RECEIVE datagram
SEND datagram
RECEIVE datagram
SEND datagram
RECEIVE datagram
40
Connectionless Client and Server (4)

• Creating a datagram socket
• int sock
• sock socket (AF_INET, SOCKDGRAM, 0)
• server bind its well-known port number to this
  socket
• client does not need to make a call to bind().
  The system will automatically bind one.
• Sending and Receiving a Datagram
• read() and write() calls cannot be used with
  connectionless sockets
• use
• sendto(sock, buff, count, flags, addr,
  addrlen)
• recvfrom(sock, buff, count, flags, addr,
  addrlen)
• first three arguments are as for write() and
  read()
• flags argument can be used to specify special
  delivery options, but is usually zero
• preserves message boundaries each recvfrom()
  corresponds to one sendto() no breaking messages
  into pieces allowed

41
Connectionless Client and Server

• send() and recv()
• send(sock, data_addr, data_len, flags)
• recv(sock, data_addr, data_len, flags)
• do not supply a peer address in their argument
  list.
• only be used on datagram sockets to which a
  connect() call has been applied to specify the
  peer address in advance.
• sendmsg() and recvmsg()
• sendmsg(sock, msg_info, flags)
• recvmsg(sock, msg_info, flags)
• msg_info points to a structure that supports
  gather-write and scatter-read operation
• gather-write collection of separate data
  buffers, at different places in memory, can be
  gathered together into a single datagram
• scatter-read single datagram may be split up and
  delivered in to several separate buffers

42
UDP details

• sendto() causes a port to be bound to the socket
  used
• client never knows (nor needs to know) what the
  port number is, but the socket becomes bound
• recvfrom() uses same socket, it is therefore
  monitoring the same port used for the sendto()
• Since datagrams could be lost, the call to
  recvfrom() might never complete and our program
  would just hang better to use timeouts
• Server blocks on call to recvfrom() instead of to
  accept()
• Only one socket is used by UDP server (unlike TCP
  server which had a listen socket and a new
  per-client socket for each client)
• Single send and receive, unlike TCP server where
  we had to continue receiving until the client
  closed the connection

43
UDP Details (cont.)

• Two calls to recvfrom() will never return data
  from the same single sendto()
• UDP does not buffer for retransmission like TCP
  (because there is no error recovery)
• Once a call to sendto() returns, the msg is out
  the door
• When data arrive via TCP or UDP, they are queued
  in a FIFO structure awaiting a call to recv() or
  recvfrom()
• For a TCP connection, we accumulate all data into
  an ever-growing queue until recv() retrieves them
• For UDP we might be getting several msgs from
  different source addresses we cant concatenate
  these!
• If you call recvfrom() with size parameter n, and
  first chunk in FIFO buffer is larger than n, it
  returns first n bytes and quietly discards the
  rest!

44
TCP and UDP Ports

• These are disjoint!
• Can have a TCP port 8888 and a UDP port 8888
• They are different and so do not conflict
• If you try to connect() to port 1234 on machine
  134.197.40.62 and there is only a UDP port 1234
  on this machine, the connection request will fail
• Ie, You must use TCP to talk to TCP ports and UDP
  to talk to UDP ports