Geoprocessing with Python for Arcgis 9.3

1
Geoprocessing with Python for Arcgis 9.3

• An Example of radar DSP Product Processing
• Newfel Mazari
• Hongjie Xie

2
What is Geoprocessing ?Why Geoprocessing ?

• Geoprocessing tasks can be time intensive since
  they are often performed on a number of different
  datasets or on large datasets with numerous
  records.
• Using ArcGis Tools to process GIS Data
• ArcMap
• ArcCatalog
• ArcInfo
• Scripting

3
Geoprocessing and Scripting

• Scripting is an efficient method of automating
  geoprocessing tasks.
• Scripting allows the execution of simple
  processes (a single tool) or complex processes
  (piggybacked, multitool tasks with validation).
• Scripts are recyclable, meaning they can be data
  nonspecific and used again.
• Script is a set of computing instructions,
  usually stored in a file and interpreted at run
  time.

4
Tools in Arcgis

• All commands (otherwise known as tools) are
  maintained in toolsets within the ArcGIS
  toolboxes.
• A toolbox can contain tools, toolsets, and
  scripts and is organized according to the
  collection of geoprocessing commands it contains.
• A toolset can contain tools, toolsets, and
  scripts and is organized according to the
  geoprocessing commands it contains.

5
Types of Tools in Arcgis

• System tool
• Model Tool
• Script Tool
• Custom Tool

6
System Tools

• Usually, these tools are installed and registered
  when you install ArcGIS.
• Although third-party developers can also create
  and register system tools.
• System tools are sometimes called function tools
  by developers.

7
Model Tools

• These tools are created by user with Model
  Builder. Some of the tools in the system
  toolboxes are model tools.

8
Script Tools

• Created by user with a scripting language editor
  (typically an enhanced text editor).
• Some of the tools in the system toolboxes are
  script tools.

9
Custom tools

• Built by system developers and have their own
  unique user interface for creating the tool.
• The ArcGIS Data Interoperability extension
  contains custom tools.

10
Scripting for ArcGis

• ArcGIS 9 includes scripting support for many of
  today's most popular scripting environments, such
  as
• Python,
• VBScript,
• JScript,
• Perl.

11
Python

• Open Source
• Object Oriented
• Multi-Platform
• Good Debugging Tools
• Installed with ArcGis
• ESRI samples provided

12
Python

• Python is simple to learn because of its clean
  syntax and simple, clear concepts.
• Python supports object-oriented programming in
  an easy-to-understand manner.
• Documenting Python is easy because it has
  readable code.
• Complicated data structures are straightforward
  to work with in Python.
• Python is simple to integrate with C and
  Fortran.
• Python can be seamlessly integrated with Java.
• Python is free from the Web and has a widespread
  community.

13
Geoprocessing Concepts for Arcgis

• Object almost anything can be an OBJECT
• Attributes
• Methods
• Geoprocessor component of ArcObjects
• manages all the geoprocessing functions available
  within ArcGIS.
• It is an object that provides a single access
  point and environment for the execution of any
  geoprocessing tool in ArcGIS, including
  extensions.
• Using the native arcgis scripting module,
  depending on version of software version you have.

14
Resources for Geoprocessing with Pyhton

• Arcgis Desktop Help
• Arcgis Libraries
• www.python.org

15
How to start Basic Concepts

• Define you geoprocessing problem
• Complex Problem Divide and Conquer
• Write and draw a chart of required steps
• Conceptualize the problems and solutions
• Look at ArcGis Desktop Help
• Find tools in ArcGis

16
Concepts Continued

• The easiest way to start is to create a model,
  and export it to a python script.
• Look carefully at syntaxes

17
Concepts again

• Each Tool has a scripting help associated with it

18
Tools Aliases
19
An Example DSP data processing
An Example of Geoprocessing
20
DSP Example
Radar Cells (grids)
21
DSP Example Radar Cells and 50 Gages
22
Conceptualize the Problem

• Extract radar Values According to Rain Gauge
  Locations
• What are the available tools solutions within
  ArcGis

Radar Cells Polygon Feature Class Gages Point
Feature Class Shapefiles
23
Conceptualize the Problem
Gages have - Ids - Coordinates
24
Radar Attributes
Hundreds of files similar to this one Records
from two radars KEWX KDFX
25
What are the Tools we can USE ?

• Analysis
• Cartography
• Conversion
• Management
• Geostatistical Analyst
• Spatial Statistics

26
Tools We can Use

• To make sure that both radar shape file and gages
  shape file are on the same Coordinate System

27
Tools We can Use

• Define Projection Tool

Intersect Tool
Export Feature Attributes to ASCII
28
First Step Create Your own ToolBox

• Add Necessary tools to your own Toolbox

29
Second Step Create a Model
30
My Model
31
Activate the Model
32
Test Run the Model
33
Check the results
34
Open the ASCII file
35
Strategy for Work

• Divide to 3 Steps
• Define Projection
• Intersect
• Export Attributes to ASCII file

36
Export to Script Define Projection
37
Export to Script Intersect
38
Export to Script Export Attributes to ASCII
Syntax Arguments
39
Lets Look at the one script Define Projection
40
Script Title and General Information

• ------------------------------------------------
  ------------------
• DefineProjection.py
• Created on Thu Apr 02 2009 044801 PM
• (generated by ArcGIS/ModelBuilder)
• ------------------------------------------------
  -------------------

With PythonWin 2.5 IDLE Comments are shown in red
color
IDLE Interactive Development Environment
41
Import Necessary Modules
Comments Line

• Imports
• import arcgisscripting as ARC
• import sys as SYS
• import os as OS
• import locale as LOCALE

ArcObjects
Operating System
File and Folders Path
The locale module opens access to the POSIX
locale database and functionality
42
Create the Geoprocessor Object

• Create the Geoprocessor object
• GP ARC.create(9.3)

Load Required Toolboxes
Load required toolboxes... GP.toolbox
"management"
Load required toolboxes... GP.AddToolbox(
"C/Program Files/ArcGIS/ArcToolbox/Toolboxes/ Dat
a Management Tools.tbx" )
43
Script Arguments and Workspace

• script arguments
• inputWS "C\\testFolder\\test1"
• outputWS "C\\testFolder\\test2"
• set the workspace
• GP.workspace "C\\testFolder\\test1

44
Set the Script to go through a list of files

• Get a list of feature classes in the workspace.
• fcs GP.ListFeatureClasses()
• for fc in fcs
• Process Define Projection...
• Statement for Tool
• .Line 2 of statements
• .Line 3

List all files in the worksapce folder
45
Define the Tool Arguments and parameters

• Process Define Projection...
• coordsys "Coordinate Systems/Geographic
  Coordinate Systems/North America/North American
  Datum 1983.prj
• coordsys "Coordinate Systems/Geographic
  Coordinate Systems/World/WGS 1984.prj
• GP.DefineProjection_management(fcs,"GEOGCS'GCS_W
  GS_184',DATUM'D_WGS_1984',SPHEROID'WGS_1984',637
  8137.0,298.257223563,PRIMEM'Greenwich',0.0,UNI
  T'Degree',0.0174532925199433")
• GP.defineprojection(fc, coordsys)

46
Add Debugging Messages

• Get the resulting messages and print them
• print fcs
• print GP.GetMessages()

It is important to print error messages and
processes for DEBUGGING purposes
47
Same Procedure for the Intersect Script

• Load required toolboxes...
• gp.AddToolbox("C/Program Files/ArcGis/ArcToolbox/
  Toolboxes/Analysis Tools.tbx")
• Script Arguments
• inputWS "C\\testFolder\\test1"
• precip_gages_wgs84 "C\\testFolder\\test2\\preci
  p_gages_wgs84.shp"
• outputWS "C\\testFolder\\test3"
• Set the workspace
• gp.workspace "C\\testFolder\\test1"

48
Intersect Script Continued

• for fc in fcs
• set the output name for each feature class to
  be the same as the input
• output outputWS "\\" "kewx_" fc
• for each file in the list run the Intersect
• gp.Intersect_analysis(fc ""
  precip_gages_wgs84, output, "ALL", "", "INPUT" )
• get the resulting message and print them
• print gp.GetMessages()

49
Dealing with files in Python

• Python has built in function to work with files
• List of files in a folder
• Open and read files
• Read all file at once
• Read line by line
• Write to files
• Read lines, search inside lines
• Read bytes..
• Create files for reading and/or writing

50
Searching for files in folders

• Search for files and get a list.
• import required python modules
• import sys, string, os, shutil
• Define the path of the serch directory or
  folder
• InputFolder "C\\testGP\\test1\\"
• OutputFolder "C\\testGP\\test3\\
• ListFiles open(OutputFolder "\\"
  MyList.txt", "w")
• listfiles os.listdir(InputFolder)
• for fileNames in listfiles
• print fileNames
• MyList.write(fileNames "\n")
• print finished listing files

51
Reading and Writing in files

• Create a new file
• open("C\\testGP\\output.txt", "w")
• Read entire file into a single string S
  input.read()
• Read next line S input.readline()
• Read entire file into list of line strings
• L input.readlines()
• Write string S into file output .write(S)
• Write all line strings L into file
  output.writelines(L)
• Manual close of file output .close()

52
Export Attributes to ASCII Script

• Load required toolboxes...
• gp.AddToolbox( "C/Program Files/ArcGis/ArcToolbox
  /Toolboxes/Analysis Tools.tbx )
• gp.AddToolbox( "C/Program Files/ArcGis/ArcToolbox
  /Toolboxes/Spatial Statistics Tools.tbx )
• gp.AddToolbox("C/Program Files/ArcGis/ArcToolbox/
  Toolboxes/Data Management Tools.tbx" )
• Script Arguments
• outputWS "C\\testFolder\\test4"
• inputWS "C\\testFolder\\test3"
• outputWS "C\\testFolder\\test4"
• outputASC "C\\testFolder\\test4
• Set the workspace
• gp.workspace "C\\testFolder\\test3"

53
Export Attributes Continued

• create a text file for combining and writing
  all outputs
• file1 open( "C\\testFolder\\test4\\kewxradar.tx
  t", "w )
• set file2 as the feature class attributes
  exported to text file
• create text files for all outputs with Zero
  counts and Rain Counts
• file3 open( "C\\testFolder\\test4\\ZeroCountKew
  x.txt", "w )
• file4 open( "C\\testFolder\\test4\\RainCountKew
  x.txt", "w )
• for fc in fcs
• set the output name for each feature class to
  be the same as the input
• output outputWS "\\" fc018 ".txt"
  
• get the count of each intersection file if
  countgt0 then export to ASCII
• count gp.GetCount_management(fc)

54
Attributes Export Setting the files
names,Outputs, Tool parameter

• if count gt1
• set the timestamp for each file
• timestamp fc " "
• print output " The count is " count
• write the timestamp to a text file
• file1.write(timestamp )
• Process Export Feature Attribute to ASCII...
• gp.ExportXYv_stats(fc, "value", "Space", output)
• open the ASCII files, read and write its
  contents to another text file
• file2 open (output,"r" )
• text file2.read()
• file1.write( " " text )
• file2.close()
• else
• print output "count for rainfall radar cells
  value " count "nothing to export )
• print " the script will proceed to the next file
  "
• file3.write( output "\ t" "Count 0" "\n
  " )

55
Export Attributes Continued

• else
• print output "count for rainfall radar cells
  value " count "nothing to export )
• print " the script will proceed to the next file
  "
• file3.write( output " \ t" "Count 0 " "
  \ n " )
• file1.close()
• print gp.GetMessages()

56
Test your Script Test again

• Python is case sensitive for variable names and
  strings
• Indentation is a very sensitive issue in Python
• Note the difference in the use of forward slash
  \ for system paths
• Double Forward Slashes \\ for workspaces
  paths
• Test Again, run and SAVE your Module

57
So Lets give it a try!