1 of 23

1
ERCAP Web Proposals, Reviews Resource
Allocations at NERSC

• John L. McCarthy
• NERSC User Services, Berkeley National Lab
• InterLab 99
• Stanford Linear Accelerator Center, 4 November,
  1999

2
Outline Summary

• NERSC Environment the Web
• ERCAP Goals, Objectives, Requirements
• Software Toolset
• Web Document Framework (Seeker gt Concur)
• Variant Source Code Control (TCCS)
• Bug Reports Task Management
• Regression Testing (Segue Silktest)
• Oracle Enterprise Manager (Backup, Schema,
  Worksheet, )
• Current Plans Challenges
• On-line demo?

3
The NERSC Environment

• National Energy Research Scientific Computing
• DOE National Facility
• founded 1974
• moved to LBNL 1996
• Cray J90s, T3E, IBM SP, HPSS Mass Storage
• Staff Operations, Systems, Storage, User
  Services
• goal Web interfaces to all documentation
  services
• Energy Research Computing Allocations Process

4
Ercap Business Objects Functions

• PI Proposals (500) Resource Requests
• including LONG (gt2000 character) text items
• Reviewers comments and ratings
• DOE Manager allocation decisions
• Project User Lists
• Staff Verification Editing
• Query access to various information

5
ERCAP 1996 Goals Objectives

• continue database storage, query retrieval
• move from X-Windows to Web interface
• speed up submissions/reviews/awards
• reduce labor costs
• improve quality control

6
Basic Requirements

• Run on Unix/Oracle (legacy system)
• Data Integrity (don't lose or corrupt data)
• Performance Reliability (response lt 10
  seconds?)
• Ease of development (debug, document, maintain)
• Extensibility, Flexibility (new forms, questions,
  menus
• test environment for new modules in context of
  operational system)

7
Functional Requirements

• Login authentication (person, role)
• Access control (read/add/edit/delete different
  types of data)
• Navigation (modeless, menus always visible)
• Query facilities (search for requests, people)
• SQL Select,Insert,Update (on remote Oracle
  database)
• Entity-level locking (prevent update if database
  has changed)

8
Using databases middlewarefor dynamic web
documents
9
Seeker Software Framework ERCAP Architecture
Components
10
Basic Screen Types Base Models
Detail Lister (Addrd)
Query.htm
Update (Addru, Addrux)
Lister (Reqs.htm)
Navigator Frame (menus)
11
ERCAP Frames Interface
Menu Frame
Query Frame
Body Frame
12
ESQL Document Methods Hierarchysimple
inheritance over-rides adds
Preup.htm (base pre-update model) includes
Syscheck.htm Constr.js methods
frame_style parameters start_body
include bdybck.htm connections heading
authorization include authidu
data_request data_display
def1dspu() update_audit_log paging
trailing_links constraints js function
ckconsts()
Upreup.htm (user pre-update model) extends
Preup methods heading include
Authdefu (update auth)
Addru.htm (Address Preupdate) extends
Upreup override_parameters
sub_heading data_request data_display
13
ESQL Document Components
lt! Filename /Lbl/Revu.htm gt ltTITLEgt NERSC
ERCAP Allocation Request Review lt/TITLEgt
lt!extends /Models/Codes/Upreup.htm gt lt!
METHOD PARAMETERS gt include /Lbl/Revb.htm
lt!/METHODgt lt! METHOD DATA_REQUEST gt
lt! d.reviewselect from request_reviews
where request_id'request_id' and
reviewer_personid'cookie.userid' gt lt! n
select name, first_name from names where
personid'personid' gt lt!/METHODgt lt!
METHOD DATA_DISPLAY gt ltform action"/cgi-win/Ac
giEsql.exe/?Lbl/Revux.htm" METHODPOST
TARGETMSGgt ltbgtReviewerlt/bgt n.first_name
n.name ltbrgt ltligt lta href"/Lbl/Help/General.htm
"gtGeneral commentslt/agt ltbrgt lttextarea
wrapphysical cols58 rows10 namegeneralgtrevie
w.generallt/textareagt ltligt lta href"/Lbl/Help/Rati
ng.htm"gtRatinglt/agt Cbutton(general_sum)ltpgt
def1btn() You MUST click this button to
store your review in the database
lt!/METHODgt ltSCRIPT LANGUAGE"JavaScript"gt se
tTimeout("cleanUp()",1000) window.document.forms
0.Pin.value "" lt/SCRIPTgt
extends base doc parameters data request data
display
object.variable
function
widget
ratio of HTML to ESQL ranges from 11 to 151
javascript library
14
Object-Oriented ESQL Components

• Variables
• e.g., d.reviewselect from request_reviews
  where ...
• lttextarea namereadinessgtreview.readylt/textareagt
  
• Functions
• HTML files reusable functionality can have
  parameters
• e.g., review.score.checknumber(1,99)
• Widgets
• HTML files like functions, but used with
  specific columns
• e.g., Listbox() creates popup menu from database
  items

15
Role-based Access Control

• Seeker Dictionary Database Tables
• Documents
• each HTML document has unique ID, description,
  etc.
• Roles
• defines roles (e.g., PI, Reviewer, DOE Manager,
  etc.)
• Person_Roles (system_roles)
• associates person with one or more roles
• Document_Roles
• type of access (read, write, etc.) for each
  document-role pair
• database view can restrict columns and rows
  allowed
• Authorization Methods section in abstract
  class(es)
• for current document and/or
• particular rows and/or columns of a database
  view and/or
• based on relationship of user to data owner
  (e.g. request PI)

16
Bug Reports Task Management
version information
task list
bug report
entered by, resolution, full edit
17
Variant Versions of ERCAP for Development
Testing

• Ercap3 Database Development Version
• Development Netscape/ESQL Engine (special port )
• Individual Developer Document Sets
• John
• Jed
• ..
• Development HTML Document Set
• Test HTML Document Set
• Production HTML Document Set
• Production Netscape/ESQL Engine (standard port )
• Development HTML Document Set .
• Ercap2 Database Production Version
• Netscape/ESQL Engine Development version
  (special port )).

18
Managing Variant Source Code

• tccs/Ercaproot/
• archive/
• Htdocs/
• Lblmenu.htm,v
• ER2rel1_29/
• ER2rel1_30/
• ER2rel1_31/
• .TCCS/
• src/
• Htdocs
• Lblmenu.htm
• .

-r-r--r--
http//www.ora.com/homepages/tccs/ Bolinger
Bronson, Applying RCS and SCCS
19
Regression Testing with PC Silktest

• Run after creation of each new Test version
• Systematic, automated testing of each frame
• teach via manual mouse movements, selections
  typing
• Silktest uses names within frames, not just x,y
  coordinates
• Hierarchical test plan individual frame tests
• but still too fragile ideosyncratic -- e.g.,
  requires restarts
• Various hardware and software platforms
• NT, Windows95, Windows98 Netscape (3,4), Internet
  Explorer(4,5)
• Macintosh Netscape (3,4), Internet Explorer(4)
  via PC Timbuktu
• Solaris Netscape (3, 4) via X-Windows on PC
• http//www.segue.com/html/s_solutions/s_silktest/s
  _silktest_toc.htm

20
Current Plans Challenges

• Self-service user contact information changes
• Migrate to XML XSL (with Oracle 8i)
• Extend to other NERSC accounting data
• Current challenges
• secure server (SSL, HTTPS, with Kerberos or DCE
  authentication)
• version control for DBMS schema evolution
  (tables, views, etc.)
• display document inheritance, DBMS views tables
  hierarchicy

21
Take-home Lessons from ERCAP

• Tools for web/database development
• Automated regression testing
• HTML source code control for variant versions
• Bug reports task management
• Seeker Framework concepts ideas
• Make every module an HTML Document
• Method sections inheritance via extends,
  embedded Sql, control flow
• Augment DBMS dictionary information (domains,
  relationships, ...)
• Access control at document, person, element value
  relationship levels
• Cookies maintain state information across session
  transactions
• Central audit log invoked by elements in base
  tables plus config file
• Optimistic locking for multi-table objects
• Web plus database has unlimited potential!

22
Other Internet Collaboration Tools

• Email ListServe Archive e.g, http//www.list.org
  /
• Web seminars -- slides plus teleconference
• irc for minutes, parallel conversations
• XML and schema fragments for data sharing
• MS NetMeeting

23
Open Forum on Metadata Registries

• When, Where and Why?
• January 17-21, 2000 in Santa Fe, New Mexico
• http//www.sdct.itl.nist.gov/ftp/l8/sc32wg2/2000/
  events/openforum/index.htm
• Sponsors
• International Organization for Standardization /
  
  International Electrotechnical Commission
  (ISO/IEC)
• Joint Technical Committee 1 (JTC 1)
• Subcommittee 32 - Data Management and Interchange
  (SC32)
• Working Group 2 - Metadata (WG2)
• Meeting Host US Environmental Protection Agency.

24
Seeker Development Framework includes several
innovative ideas

• ESQL Engine
• receives/decodes browser/application messages
  NSAPI, ISAPI, CGI
• ESQL Documents Libraries
• HTML, embedded Sql, control flow, Method
  sections Class inheritance via extends
• libraries for javascript (client browser
  functions), icons (gif), widgets (column
  domains), HTML functions
• Seeker Tables, Dictionary
• augments Oracle dictionary information (e.g.,
  domains, relationships, help text, default
  formats)
• Role-based Access Control, Security
• access control at document, person and element
  value levels cookies maintain state information
• Central Audit Log Optimistic Locking
• creation and last modification time, ip address,
  personid, query, etc. for each row in specified
  tables
• Document Management (including menus)
• database, dictionary control, and directory
  structure (with Lbl subtree for ERCAP files)
• Alternate Paths for debugging testing new docs
• test in context of production system without
  disrupting ongoing production documents and
  transactions