Statistics Function Implementation

1
Statistics Function Implementation

• Traditional Programming Approach

2
minimum.pro

• function minimum, x
• n n_elements(x)
• answer x(0)
• for i1L, n-1 do begin
• if( answer gt x(i) ) then
• answer x(i)
• endfor
• return, answer
• end

3
sum.pro

• function sum, x
• n n_elements( x )
• answer 0
• for i 0, n-1 do begin
• answer answer x(i)
• endfor
• return, answer
• end

4
Overflow Problems

• What if the array is an integer array?
• Sums will have a maximum limit based on the
  number of bits specified by the integer type.

5
Corrected sum.pro

• function sum, x
• n n_elements( x )
• answer 0.0D
• for i 0L, n-1 do begin
• answer answer x(i)
• endfor
• return, answer
• end

6
mean.pro

• function mean, x
• n n_elements(x)
• answer sum(x) / n
• return, answer
• end

7
Integer Problems

• IDLgt b9/5
• IDLgt print,b
• 1

8
Integer Problems

• IDLgt b9/double(5)
• IDLgt print,b
• 1.8000000

9
Corrected mean.pro

• function mean, x
• n n_elements(x)
• answer sum(x) / double(n)
• return, answer
• end

10
sum_squares.pro

• function sum_squares, x
• squares x2.0
• answer sum( squares )
• return, answer
• end

11
variance.pro

• function variance, x
• n double(n_elements(x))
• answer(sum_squares(x)-(sum(x)2.0/n))/ (n-1)
• return, answer
• end

12
sd.pro

• function sd, x
• answer sqrt(variance(x))
• return, answer
• end

13
Shell Programming
14
UNIX Tip of the Day

• Directory maneuvering commands pushd, popd, and,
  dirs
• cd /usr/tmp
• pwd
• /usr/tmp
• pushd rvrpci
• pwd
• /cis/staff/rvrpci

15
UNIX Tip of the Day

• dirs
• /cis/staff/rvrpci /usr/tmp
• pushd
• pwd
• /usr/tmp
• dirs
• /usr/tmp /cis/staff/rvrpci
• pushd /usr/local/bin

16
UNIX Tip of the Day

• dirs
• /usr/local/bin /usr/tmp
• /cis/staff/rvrpci
• pwd
• /usr/local/bin
• pushd
• dirs
• /usr/tmp /usr/local/bin
• /cis/staff/rvrpci

17
UNIX Tip of the Day

• dirs
• /usr/tmp /usr/local/bin
• /cis/staff/rvrpci
• pwd
• /usr/tmp
• pushd 2
• pwd
• /cis/staff/rvrpci

18
UNIX Tip of the Day

• dirs
• /cis/staff/rvrpci /usr/tmp
• /usr/local/bin
• popd
• dirs
• /usr/tmp /usr/local/bin

19
IMPORTANT UNIX Concepts

• Environment and Shell Variables
• These allow you to customize your UNIX
  environment
• They are different in terms of their SCOPE
• SCOPE determines the visibility of a variable

20
Other IMPORTANT UNIX Concepts

• Environment Variable
• Examples are TERM and DISPLAY
• Set a particular variable to a value by using the
  setenv command
• You can print the value of a particular variable
  or all the environment variable using the
  printenv command

21
Environment Variables

• Examples
• To set environment variables
• setenv TERM vt100
• setenv DOG Goofy
• print out the terminal type
• printenv TERM
• vt100
• print out all environment variables
• printenv

22
Shell Variables

• Shell variables are similar to Environment
  variables except they have a limited scope, i.e.,
  they exist only in the shell which they are
  defined.
• Environment variables on the other hand, exist in
  all its children shells
• To illustrate this concept, let us look at the
  following example

23
Environment vs. Shell Variables

• set prompt "Parent Shell gt "
• Parent Shell gt setenv DOG Goofy
• Parent Shell gt set mouseMickey
• Parent Shell gt printenv DOG
• Goofy
• Parent Shell gt echo mouse
• Mickey
• Parent Shell gt xterm
• (YOU SHOULD NOW HAVE A NEW xterm WINDOW)
• THIS IS KNOWN AS
• SPAWNING A NEW (OR CHILD) PROCESS

24
Environment vs. Shell Variables

• (IN THE NEW xterm WINDOW, DO THE FOLLOWING)
• set prompt "Child Shell gt "
• Child Shell gt printenv DOG
• Goofy
• Child Shell gt echo mouse
• mouse Undefined variable.

25
Environment vs. Shell Variables

• Child Shell gt setenv DOG Pluto
• Child Shell gt set mouseMinnie
• Child Shell gt printenv DOG
• Pluto
• Child Shell gt echo mouse
• Minnie
• Child Shell gt exit
• (THE xterm WINDOW SHOULD NOW GO AWAY - THIS
  PROCESS HAS NOW BEEN KILLED)

26
Environment vs. Shell Variables

• Parent Shell gt
• Parent Shell gt printenv DOG
• Goofy
• Parent Shell gt echo mouse
• Mickey
• Parent Shell gt

27
Environment Shell Variables

• Why is this important?
• UNIX uses Environment and Shell Variables control
  a number of processes
• Customizes your working environment
• Variables used for UNIX Scripts
• They are typically defined and initialized in
  your .login and .cshrc files

28
Useful Shell Variables

• filec Allows file completion
• path List of command
• directories
• cdpath List of candidate
• directories to cd into
• history Number of commands to
• remember

29
What is shell programming?

• Shell programming
• automate a set of UNIX commands.
• Just like any programming language
• wrappers
• black box a customized collection of UNIX
  commands.
• Example of shell programs
• .login
• .cshrc

30
.login file

• set path(HOME/bin /usr/local/bin \
• /usr/ucb /usr/sbin /bin /usr/bin \
• /usr/bin/X11 .)
• stty dec new
• tset -I -Q
• set mail/usr/spool/mail/USER
• set editmode emacs
• umask 077
• biff n
• date

31
.cshrc file

• if (?prompt) then
• set notify
• set history 100
• set savehist 100
• alias pd pushd
• alias pop popd
• alias vt100 "set term vt100"
• endif

32
When these files are executed?

• .cshrc
• is automatically executed when you start a new
  shell
• .login
• only gets executed once when you first login
• Can be re-executed by giving the source command
• source .cshrc

33
Other useful .login and .cshrc entries

• set filec
• set cdpath( rvrpci/pub /mythesis)
• Other common entries
• set path( path /usr/local/bin)
• set path(/usr/local/bin path)

34
User defined shell program

• Determine name of command
• Determine input, output, and option arguments
• Determine UNIX commands to execute
• Establish error trapping
• Make shell program executable

35
A simple shell program

• dd command to swap bytes
• dd ifinput.dat ofoutput.dat bs2 convswab
• Very difficult to remember
• Very little utility to non-UNIX geeks (normal
  people)

36
We would rather see...

• swap_bytes input.dat output.dat

37
Special Shell Variables Set

• swap_bytes input.dat output.dat

0
1
2
argv1
argv2
command
38
Another Special Shell Variables

• swap_bytes input.dat output.dat

argv
Indicates how many arguments are present In this
case, 2
39
shell program swap_bytes

• !/bin/csh -f
• dd if1 of2 bs2 convswab

40
Making swap_bytes shell script executable

• ls -l swap_bytes
• -rw------- ... swap_bytes
• chmod ux swap_bytes
• ls -l swap_bytes
• -rwx------ ... swap_bytes

41
To run swap_bytes

• swap_bytes becomes just another unix command!
• swap_bytes input.dat output.dat

42
Limitation of swap_bytes

• No error trapping
• Should give usage when typing command
• swap_bytes
• usage swap_bytes input_file output_file

43
Improvement to swap_bytes

• !/bin/csh -f
• if ( argv ! 2 ) then
• echo "usage 0 input_file output_file"
• exit 1
• endif
• dd if1 of2 bs2 convswab

44
Commad exit status

• By convention
• exit 0
• Indicates successful command completion
• exit 1 (or non-zero)
• Indicates some error condition

45
Informational message from swap_bytes

• UNIX style informational message
• swap_bytes
• usage swap_bytes input_file output_file

46
Interactive swap_bytes

• If you want a friendlier shell program
• Have it query the user for the inputs
• Another special shell variable can be used
• lt

47
Interactive swap_bytes

• !/bin/csh -f
• if ( argv ! 2 ) then
• echo -n "Please enter the input filegt "
• set inputlt
• echo -n "Please enter the output filegt "
• set outputlt
• endif
• dd ifinput ofoutput bs2 convswab

48
Interactive swap_bytes example

• User simply types the command
• swap_bytes
• Please enter the input filegt input.dat
• Please enter the output filegt output.dat

49
UNIX Quotes
50
A note about quotes in UNIX

• set als
• echo a
• echo a
• set ba
• echo b
• set ba
• echo b
• set ba
• echo b

51
A note about shell variables

• Shell variables can also double up as arrays
• Using the previous example,
• echo b
• echo b1
• echo b
• echo bb

52
A more complex shell program

• In pbmplus utilities,
• rawtopgm conversion exists
• pgmtoraw conversion does not
• A version of pgmtoraw in a programming language
  like C
• Time consuming
• Program will likely be used infrequently
• Solution shell program

53
pgmtoraw shell script design

• Define input and output files
• Figure out dimensions of input image
• Determine number of bytes for input image
• Determine number of bytes for header
• Need to strip out the header bytes
• Write out headerless image data

54
Define input and output files pgmtoraw

• !/bin/csh -f
• set command_name0
• set number_argsargv
• if( number_args ! 1 ) then
• echo usagecommand_name input_file_name
• exit 1
• endif
• .
• .
• .

55
Dimensions of input image ( pnmfile)

• more test_data.pgm
• P2
• 3 3
• 255
• 1 2 3
• 4 5 6
• 7 8 9
• pnmfile test_data.pgm
• test_data.pgm PGM plain, 3 by 3 maxval 255

56
pgmtoraw (continued)

• set input_file1
• set pnm_info pnmfile input_file
• set image_type pnm_info2
• set data_type pnm_info3
• set width pnm_info4
• set height pnm_info6
• set maxval pnm_info8
• set pixel_bytes 1
• _at_ image_bytes width height

57
pgmtoraw (continued)

• set file_infowc -c input_file
• set bytes_in_file file_info1
• _at_ header bytes_in_file - image_bytes
• dd ifinput_file bspixel_bytes skipheader

58
Resulting pgmtoraw utility

• Uses existing routines to obtain information
• pnmfile
• wc
• dd
• Functional tool written in 20 command lines

59
Current Limitations of Resulting pgmtoraw utility

• No check between ASCII vs. RAW pgm
• if( data_type plain,) ...
• No provisions for multibyte per pixel case
• Use pnmfile results to check for above cases
• endian case needs to be addressed for multibyte
  case
• Above conditions can be addressed by suite of
  UNIX commands

60
Shell Scripts Wrappers and IDL

• Another utility formerly missing in pbmplus
• jpegtopnm or pnmtojpeg
• IDL has jpeg read/write capability
• Create a wrapper that makes an idl program
  pbmplus-like

61
pnmtojpeg.pro

• pro pnmtojpeg, input_file, output_file
• read_ppm, input_file, image
• write_jpeg, output_file, image
• end

62
Usage of pnmtojpeg.pro in IDL

• IDLgt pnmtojpeg,image.pnm,image.jpg

63
Usage of pnmtojpeg.pro in IDL

• IDLgt pnmtojpeg,image.pnm,image.jpg
• For IDL to automatically find pnmtojpeg.pro
• It must be in the current working directory
• Directory containing pnmtojpeg.pro must be
  defined in the ENVIRONMENT VARIABLE
• IDL_PATH

64
IDL_PATH environment variable

• setenv IDL_DIR /cis/common/rsi/idl_5
• setenv IDL_PATH \IDL_DIR/lib\IDL_DIR/examples
  
• \/dirs/common/rsi/idl_5\/dirs/common/lib/idl\
  /lib/idl

65
pnmtojpeg.csh

• !/bin/csh -f
• echo pnmtojpeg , 1 , 2 idl

66
Usage of pnmtojpeg.csh

• pnmtojpeg.csh image.pnm image.jpg

67
Limitation of pnmtojpeg.csh

• Does not conform to pbmplus piping, i.e.,
• tifftopnm file.tif pnmscale 2.0 gt
  new_file.pnm
• No error trapping

68
Usage cases of pnmtojpeg(no command line
arguments)

• tifftopnm file.tif pnmscale 2.0 pnmtojpeg gt
  new_file.jpg

69
Usage cases of pnmtojpeg(1 command line argument)

• pnmtojpeg image.pnm gt image.jpg

70
Usage cases of pnmtojpeg(2 command line
arguments)

• pnmtojpeg image.pnm image.jpg

71
Yet another wrapper pnmtojpeg

• !/bin/csh -f
• If user interrupts process, jump to stop
• onintr stop
• 0 is the command name
• argv is the number of arguments
• is the process id

72
Code for no argument case

• if(argv 0) then
• set input_file /usr/tmp/0_input_
• set output_file /usr/tmp/0_output_
• cat gt input_file
• pnmtojpeg.csh input_file output_file
• cat output_file
• .
• .
• .

73
Code for 1 argument case

• else if(argv 1) then
• set input_file 1
• set output_file /usr/tmp/0_output_
• pnmtojpeg.csh input_file output_file
• cat output_file
• .
• .
• .

74
Code for 2 argument case

• else
• set input_file 1
• set output_file 2
• pnmtojpeg.csh input_file output_file
• endif
• clean up when finished
• stop
• rm -f /usr/tmp/0_input_
• rm -f /usr/tmp/0_output_

75
pnmtojpeg summary

• Produced a new pbmplus utility
• Used UNIX shell scripting
• Argument handling
• Scratch space /usr/tmp
• Process id handling
• Clean up
• Integrated IDL program and commands
• 21 lines of UNIX commands

76
Summary

• The dot files
• Basics of Shell Scripting
• Special Shell Variables
• Seamless integration of UNIX to other utilities
  (IDL)

77
Other Shell Constructs to keep in mind

• foreach
• while
• case