Data File Access API : Under the Hood

1
Data File Access API Under the Hood

• Simon Horwith
• CTO
• Etrilogy Ltd.

2
Session Topics

• Data File Access API General Overview
• What is it?
• Why use it?
• Where to get it?
• Architecture Overview
• DFA API Functional Specification
• Technology to the Rescue!
• Code Overview
• DFA API Methods
• The DFAQuery Custom Tag Putting the I in
  API
• Viewing a Sample Application
• Summary

3
DFA API General Overview What is it?

• The Data File Access API
• is a framework for storing, retrieving, and
  maintaining application data in the local file
  system
• is an Application Programming Interface for
  ColdFusion Developers building small to medium
  sized applications on the MX platform
• is an alternative to the traditional approach of
  storing data in a relational database management
  system (RDBMS)
• Allows developers to retrieve data as a
  ColdFusion Query or as XML using either SQL or
  XPATH!

4
DFA API General Overview Why use it?

• The DFA API offers many benefits over traditional
  RDBMS solutions
• platform neutrality
• portability
• performance
• XPath and XSL Support
• XML storage and retrieval allows for
• Data and data-definition re-purposing
• Web Services
• Flash Remoting
• better flexibility and extensibility

5
DFA API General OverviewWhere to Get It?

• The DFA API is available to Macromedia DevNet
  subscribers. It is part of DRK 3.
• Installing the DFA API from the DRK CD will
  install
• ColdFusion Component API
• Custom Tag wrapper
• Sample Application
• Sample DFA API XML skeleton files
• An additional application for data entry and data
  table creation is available at http//www.how2cf.c
  om

6
DFA API Architecture OverviewFunctional
Specification

• DFA API Goals
• Create an Application Programming Interface to
  allow developers to query data stored as XML or
  CSV using familiar syntax (SQL) as well as
  perform other common SQL tasks such as adding,
  manipulating, and deleting data
• Small to medium sized application usage is
  recommended
• Performance must be comparable to that of
  applications that query an RDBMS
• Applications using the API should be able to be
  migrated from one machine to another with ease
• Cross-platform functionality
• Must be flexible and easy to extend

7
DFA API Architecture OverviewTechnology to the
Rescue!

• All API functionality is housed in a single
  ColdFusion Component
• Easy installation (just copy the cfc to the
  server!)
• Easily executed by any application on the server
• Public/private data and functionality
• API can easily be extended via component
  inheritence
• Developers could expose API functionality and
  data via Web Services
• API could be used to provide data to Flash
  Applications via Flash Remoting

8
DFA API Architecture OverviewTechnology to the
Rescue! (contd)

• XML is useful in the real world
• Architecture uses XML to
• Define data tables (table names, columns, etc.)
• Map location of data files to the data table
  definitions
• Define the actual data
• Each of the above (data table definitions,
  locations, and data) is stored in an XML file.
  Data table definitions and appings are loaded and
  stored in memory each as an XML DOM.Data table
  data is loaded into memory (and persisted) the
  first time the data is needed.
• CSV support simply means converting all CSV
  content to XML
• Storing this data as XML enables
• Retrieval of data as XML
• Use of XPATH to retrieve data

9
DFA API Architecture OverviewTechnology to the
Rescue! (contd)

• Query of Queries saves the day!
• The API must support not only XPATH but also SQL
  SELECT Statements for data retrieval. Thanks to
  QoQ, adding support for SQL SELECT was easily
  accomplished by first converting the XML DOM in
  question to a ColdFusion Query variable and then
  executing the SQL against it in a Query of
  Queries
• If the data is to be returned as XML, the result
  set is converted back to an XML DOM before being
  returned.
• Performance is fantastic, performance is
  fantastic, performance is fantastic!!

10
DFA API Architecture OverviewTechnology to the
Rescue! (contd)

• Locking, Memory, and File System Access
• Misinformation dispelled file I/O operations
  are actually very fast
• All required information and data operations are
  stored/performed first in persistent memory in
  order to boost performance
• Named locks are used for all data operations
• Exclusive locks for all write operations to data
  tables in memory and on file
• Read-only locks for all other data operations

11
Code Review Component Methods
Method Description
AddData Adds a row of data to a datatable
addDataTableMapping Adds a datatable mapping to the mapping file
addMasterDataTableDefinition Adds a datatable definition to the master definition file
commitDataTable Saves a datatable currently in memory as an xml file
DeleteData Deletes rows of data from a datatable
DumpIt ltcfdumpgts a value then does a ltcfabortgt. Only used for debugging and should be commented out in production.
parseDataDefinitionMappings Loads the 'datatable'-to-'data-file' mappings for an application
parseDataTableDefinitions Parses the xml file containing all available data table definitions and creates an XML DOM of all data tables
GetData Loads a datatable if doesnt already exist, performs SQL or XPath searches, returns data as XML or a query (or as transformed xml).
getDataTableDefinitionArray Returns an array of structures that define a datatable
GetDef Returns a DOM of all available table definitions. Only used for debugging and should be commented out in production.
12
Code Review Component Methods
Method Description
GetMap Returns the DOM of application specific table definition to directory/url mappings. Only used for debugging and should be commented out in production.
GetNextID Returns next id for a datatable when performing an insert
GetPubData Returns a DOM of all loaded datatables. Only used for debugging and should be commented out in production.
GetTableLocation Retrieves datatable location from application data definition xml mapping DOM
IsCSV Verifies whether or not a string is proper CSV format
isDataTableIDUnique Validates whether or not a data table ID is not currently in use
isDataTableInMemory Is a specific datatable in memory or able to be loaded into memory?
13
Code Review Component Methods
Method Description
LoadCSV Create a datatable from CSV text or from a CSV file
PathTypeOf Determine if a string is a absolute or relative path, or a URL
removeDataTable Removes a datatable from memory
UpdateData Modifies data in an existing datatable
XmlToRS Converts a datatable xml DOM to a CFML query
14
The DFAQuery Custom Tag Putting the I in API

• One goal of the DFA API was to make the API as
  easy as possible for developers to use in their
  applications. This was achieved by creating a
  custom tag wrapper to shield developers from
  the component and to allow them to use the tag
  syntax they are already familiar with.

15
The DFAQuery Custom Tag Putting the I in API
DFAQuery Tag Execution Modes DFAQuery Tag Execution Modes
Mode Tag Operation
Start Tag checks to see if theres an active instance of the API. If so, it does nothing. If not, it checks to see if the API can be initialized and either creates an instance or throws an error.
End Tag performs the requested API operation.
16
Viewing a Sample Application

• The rolodex application is a simple sample
  application that installs with the API. It uses
  the DFAQuery custom tag to interact with the API
  Component in order to allow users to query the
  application for rolodex members, view member
  details, and update member information.
• A second sample application (qbe.cfm) is also
  installed with the API. It is a query by example
  interface that shows how CSV text can be parsed
  into a data table, queried using SQL or XPATH,
  and deleted on the fly.
• Lets briefly examine the underlying code and
  files used by these applications
• http//www.how2cf.com has a third sample
  application that uses nothing but direct
  component method calls in order to create an
  administrative interface for data entry and data
  table creation!!

17
Summary

• The Data File Access API
• Serves as an example of how to properly
  architect a component based application as well
  as how to code an API that makes use of the best
  features of ColdFusion MX
• Offers developers an alternative to the
  traditional approach to storing and manipulating
  data in dynamic web applications. This
  alternative is easily extended to meet
  application specific needs, is not dependent on
  any RDBMS platform, and allows the data tier and
  its business logic to be distributed as part of
  the application all without sacrificing
  performance.