Chapter 5: Advanced Editors

1
Chapter 5 Advanced Editors

• awk, sed, tr, cut

2
Objectives

• After studying this lesson, you should be able
  to
• awk a pattern scanning and processing language
• sed stream editor
• tr translate one character to another
• cut cut specific columns vertically

3
Awk

• awk is a pattern scanning and processing
  language.
• Named after its developers Aho, Weinberger, and
  Kernighan. (developed in 1977)
• Search files to see if they contain lines that
  match specified patterns and then perform
  associated actions.

4
awk

• Syntax
• awk F(separator) patternaction filenames
• awk checks to see if the input records in the
  specified files satisfy the pattern
• If they do, awk executes the action associated
  with it.
• If no pattern is specified, the action affects
  every input record.
• A common use of awk is to process input files by
  formatting them, and then output the results in
  the chosen form.

5
awk

• A sample data file named countries
• Canada385225North America USA3615237North
  America Brazil3286134South
  America England9456Europe France21155Europ
  e Japan144120Asia Mexico76278North
  America China37051032Asia India1267746Asia
  
• country name, area (km2), population
  density(106/km2), continent

6
awk

• awk -F ' printf "-10s \td \td \t15s
  \n",1,2,3,4 ' countries
• Outputs
• Canada 3852 25 North America
• USA 3615 237 North America
  
• Brazil 3286 134 South
  America
• England 94 56 Europe
• France 211 55 Europe
• Japan 144 120 Asia
• Mexico 762 78 North
  America
• China 3705 1032 Asia
• India 1267 746 Asia

7
Some build-in Variables

• NF - Number of fields in current record
• NF - Last field of current record
• NR - Number of records processed so far
• FILENAME - name of current input file
• FS - Field separator, space or TAB by default
• 0 - Entire line
• 1, 2, , n - Field 1, 2, , n

8
Formatted output

• printf syntax
• printf "control-string" arg1, arg2, ... , argn
• The control-string determines how printf will
  format arg1 - argn.
• The control-string contains conversion
  specifications, one for each argument. A
  conversion specification has the following
  format -x.yconv

9
Formatted output

• -x.yconv
• - causes printf to left justify the argument.
• x is the minimum field width
• .y is the number of places to the right of a
  decimal point in a number.
• conv is a letter from the following list
• d decimal
• e exponential notation
• f floating point number
• g use f or e, whichever is shorter
• o unsigned octal
• s string of characters
• x unsigned hexadecimal

10
printf examples

• printf I have d s\n, how_many, animal_type
• printf -10s has 6.2f in their account\n,
  name, amount
• printf 10s -4.2f -6d\n, name, interest_rate,
  account_number
• printf \td\td\t6.2f\ts\n, id_no, age,
  balance, name

11
awk

• awk opens a file and reads it serially, one line
  at a time.
• By specifying a pattern, we can select only those
  lines that contain a certain string of
  characters.
• For example we could use a pattern to display all
  countries from our data file which are situated
  within Europe.
• awk '/Europe/' countries

12
Match operator

• A sample data file named countries
• Canada385225North America USA3615237North
  America Brazil3286134South
  America England9456Europe France21155Europ
  e Japan144120Asia Mexico76278North
  America China37051032Asia India1267746Asia
  
• awk -F '3 55' countries
• Matching operators are
• equal to ! not equal to
• gt greater than lt less than
• gt greater than or equal to lt less than or
  equal to

13
File Breaking

• Default is on space and tab and multiple
  contiguous white space counts as a single white
  space and leading separators are discarded

14
Logic Operations

• Sample file named cars
• ford mondeo 1990 5800
• ford fiesta 1991 4575
• honda accord 1991 6000
• toyota tercel 1992 6500
• buick centry 1990 5950
• buick centry 1991 6450
• awk '3 gt1991 4 lt 6250' cars
• awk '1 "ford" 1 "buick"' cars

15
Data processing

• Sample file named wages
• Brooks 10 35Everest 8 40Hatcher 12
  20Phillips 8 30Wilcox 12 40
• name, /hour, hours/week
• Calculate /week, tax/week, (25 on tax).
• awk ' print 1,2,3,23,230.25 ' wages

16
Other examples

• who awk ' print 5, 1 ' sort
• prints name and login time sorted by time
• awk -F ' print 1 ' /etc/passwd sort
• print existing user names and sort it
• awk -F ' print "username " 1 "\t\tuid" 3 '
  /etc/passwd
• print user name and user id

17
sed

• sed stands for stream editor.
• sed is a non-interactive editor used to make
  global changes to entire files at once
• An interactive editor like vi would be too
  cumbersome to try to use to replace large amounts
  of information at once
• sed command is primarily used to substitute one
  pattern for another

18
sed

• Typical Usage of sed
• edit files too large for interactive editing
• edit any size files where editing sequence is too
  complicated to type in interactive mode
• perform multiple global editing functions
  efficiently in one pass through the input
• edit multiples files automatically
• good tool for writing conversion programs

19
sed

• Syntax
• sed e command file(s)
• sed e command e command file(s)
• sed f scriptfile file(s)

20
sed

• Whole line oriented functions
• DELETE d
• APPEND a
• CHANGE c
• SUBSTITUTE s
• INSERT i

21
sed examples

• sed 's/Tx/Texas/' foo
• replaces Tx with Texas in the file foo
• sed -e '1,10d' foo
• delete lines 1-10 from the file foo
• sed /Cot/,/0-9/d foo
• deletes from the first line that begins with Cot,
  Coot, Cooot, etc through the first line that ends
  with a digit

22
sed examples

• cat file I have three dogs and two cats sed -e
  's/dog/cat/g' -e 's/cat/elephant/g' file I have
  three elephants and two elephants
• sed e //d foo
• deletes all blank lines
• sed -e 6d foo
• deletes line 6.

23
sed examples

• sed 's/Tx/Texas/' foo
• replaces Tx with Texas in the file foo
• sed -e '1,10d' foo
• delete lines 1-10 from the file foo
• sed '11,d' foo
• A dollar sign () can be used to indicate the
  last line in a file. For example, delete lines 11
  through the end of myfile.

24
sed examples

• sed can also delete lines based on a matching
  string. Use /string/d For example, sed
  '/warning/d' log deletes every line in the file
  log that contains the string warning.
• To delete a string, not the entire line
  containing the string, substitut text with
  nothing. For example, sed 's/draft//g' foo
  removes the string draft everywhere it occurs in
  the file foo.

25
tr

• translates characters from stdin to stdout.
• Syntax
• tr options string1 string2
• Options
• -c complement set with respect to the entire
  ASCII character set
• -s squeeze duplicates to single characters
• -d delete all input characters contained in
  string1

26
tr examples

• Typical usages
• tr chars1 chars2 lt inputfile gt outputfile
• tr chars1 chars2 lt inputfile less

27
tr

• tr s z
• replaces all instances of s with z
• tr so zx
• replaces all instances of s with z and o with x
• tr 'a-z' 'A-Z'
• replaces all lower case characters with upper
  case
• tr 'a-m' 'A-M'
• translates only lower case a through m to upper
  case A though M

28
My first Shell Script

• tr .,?! .
• converts all punctuation to a period
• tr c 0-9a-zA-Z _
• converts all non-characters to _
• tr s a-zA-Z
• squish all consecutive multiple characters

29
tr

• The output of tr can be redirected to a file or
  piped to another filter and its input can be
  redirected from a file or piped from another
  command
• This implies that certain characters must be
  protected from the shell by quotes or \, such as
  spaces ( ) lt gt \ ! NEWLINE TAB
• Example tr o replaces all os with a blank
  (space)

30
tr

• tr -d lets you delete any character matched in
  string1.
• Examples
• tr -d 'a-z' deletes all lower case characters
• tr -d aeiou deletes all vowels
• tr -dc aeiou deletes all character except vowels
  (note this includes spaces, TABS, and NEWLINES
  as well)

31
tr

• tr -cs 'A-Za-z' '\n' ltin_file gt out_file
• It replaces all characters that are not a letter
  (-c) with a newline ( \n ) and then squeezes
  multiple newlines into a single newline (-s). The
  after /n means as many repetitions as needed.

32
cut

• cut - used to cut specific columns vertically
• cut -c2-5 filename 
• cut column numbers from 2 to 5 (all inclusive)
  from the file filename.
• cut -f3-4 filename
• if the filename has field delimiters, then
  individual fields can be cut out using the -f
  option.

33
cut

• A sample file named bar
• madanSSMRC-LMBOhio
• christineSSMRC-LMBNebraska
• This particular examples has 3 fields which are
  'delimited' by a so to get field number three,
  you should run
• cut -f4 -d'' bar

34
Summery

• awk a pattern scanning and processing language
• sed stream editor
• tr translate one character to another
• cut cut specific columns vertically