Microsoft Commerce Server Platform

1
Microsoft Commerce Server Platform
2
Agenda

• Overview
• User Profiling System
• Targeting
• Data Warehouse Analysis
• Pipelines Transactions
• Administration
• Catalog
• BizDesk
• Solution Sites

3
Overview
Web Site
Web Presence
Retail
Supplier
Corp Purchase
Web Site Tools
Content
Products
Analysis
Order
User Profiles
Marketing
Custom Catalog Order Requisitions
4
Solid Momentum in Market

• Solid Customer wins
• Internet Deployments
• GAP, Eddie Bauer, 800 Flowers, Tower Records,
  Time Warner,Barnes and Noble (Commerce)
  Broadcast.com, Sony Music, Sportsline,
  Ivillage/Ibaby, CBS Marketwatch, PC World, PBS,
  Intellicast, QVC (Commerce), WWF, Quote.com,
  Virgin Records, Capitol Records, Forbes, MCP 20th
  Century Fox
• Reviews Won 16/19 SSCE Reviews
• 7 comparative wins/positives, 9 positive stand
  alone reviews
• Doing well with top tier analysts
  (Gartner/Forrester comparatives)
• Microsoft Web Platform considered most
  scaleable by PC Week Lab (July 1999)

5
Solid Momentum as Platform

• Solution Developers/SIs
• 1,900 solution developers in US trained on
  commerce platform
• 100 Go To MCSPs in US referral program
• 7 of the top firms now support Microsoft commerce
  solutions
• Compaq, EY, Amdahl and DMR Consulting, KPMG,
  NCR, USWeb
• ISP/CSPs
• 17 ISP/CSPs focused on Microsoft commerce
• 8 go-to-market GTE Internetworking, Icon CMT,
  Digex, US Web, Data Return, AIS, Concentric,
  Northwest Nexus
• 9 offer INEX hosting 9 Net Avenue, Anaserve,
  CCI, First Internet Media, IDT, NetEase,
  TotalNet, United Interactive Tech., Web FX 2K
• ISVs
• 70 partners shipping extensions to SSCE

6
Release Theme

• Build a comprehensive integrated platform for
  web-commerce applications
• Profiling
• Targeting
• Product Catalog
• Transactions
• Data Warehousing Analysis
• Improved Developer Productivity
• A set of configurable tools that businesses use
  to target users, manage content, process
  transactions and analyze site usage

7
Sample Customers

• Eddie Bauer (Add web traffic stats for a few)
• 20k shoppers/day (4x peak)
• 20/1 transaction ratio
• 5 total overall revenue
• 13 web server, 2 SQL servers
• 1 800 Flowers
• 25k transactions/day (6x peak)
• 6k concurrent connections
• 80 web server, 2 SQL servers
• 10min average shopping time

8
Closing The Loop
Web Site

• Relationship Mgmt
• Providing Customers what they want, when they
  want it
• Implicit user profiling Targeting

9
Commerce Server Overview
10
User Profiling System
11
Requirements

• Extensible application layer to model business
  entities.
• Profiles, Expressions, Terms (Objects, Rules,
  Definitions)
• Users, Orgs, POs, Requisitions, Products,
  Accounts, etc...
• Scalable user management platform
• 10Million Users, 1 Peak Online, Mixed
  Transactions (login, read, write)
• Data store agnostic
• LDAP, 3rd Party OLEDB/ODBC sources, Commerce SQL
  Prop Store
• Transparent Data Source aggregation
• XML Schema Representation
• Enable integrated business solutions
• Analysis, BizTalk, Transaction Integration
• User Management, Targeting, Business Process

12
BizData Enabled Platform
Targeting Services
Transaction Services
User Management Services
Analysis Services
Ads, Promos
Analysis Objects
PipeLine Objects
Auth Objects
Selection Objects
DTS Tasks
BizTalk
User Profiles
Expressions
Profiles
Profiles
Profiles
User Profile System
13
BizData Definitions

• Profile Definitions
• Profiles define logical business objects
• Users, Organizations, Purchase Orders,
  Requisitions, etc.
• Data model supports business data extensions
• property groups, site terms, profile references,
  attribute tags, etc..
• Enable property mappings to physical data sources
• user.name maps to SQL1.Table2.Column3
• Expression Definitions
• Expressions are simple rules evaluated against
  profile instances.
• E.g. IsFrequentVistor - User.TotalVisits 100
• Site Terms
• Common store of business terms used in profiles
  and expressions.
• User.EyeColor SiteColors Brown, Blue,
  Green

14
BizData Runtime Objects

• Profile Object
• Exposes Profile Instance data with simple object
  model
• ADSI-IADs, ADO-Fields
• Supports simple physical-logical data
  translations
• Data store agnostic (LDAP Directories, Ansi-SQL,
  etc.)
• Supports data store aggregation.
• Extensive Caching (object properties,
  connections)
• User Profile Object is successor to Site Server
  3.0 AUO (Active User Object).
• Expression Evaluator Object
• Enables runtime evaluation of boolean expressions
• Consumes Profile objects, Commerce dictionaries,
  IDispatch objects.
• Extensive data caching (expressions, results,
  profile properties).
• Optimized Pcode.

15
Profile System Architecture
Profile Service
Profile object exposes simple object managing
data entities. ADO Support for Bulk Processing
(Select, Insert, Update).
Profile Object Cache
ADO
Commerce OLEDB Provider
Commerce OLEDB Provider provides standard
interface supporting both the profile service and
ADO. Provider exposes simple SQL semantics
(profile analogous to a logical table
view). Provider enables data translations and
data source aggregation based on profile
definitions.
BizData Defs
Profile Definition Cache
Data Source Aggregation Manager
ADSI/ LDAP
OLEDB/ SQL7
OLEDB/ ODBC
Optimized data source provider plug-ins for
directories (LDAPv3), SQL7, and OLEDB/ODBC
sources supporting ANSI-SQL subset.
LDAP DS
LDAP DS
LDAP DS
3rd Party Databases
LDAP ADS
Commerce SQL Prop Store
16
BizData Profile Design
SAMPLE USER PROFILE User.Name Dave User.Email
Dave_at_Ms.com User.Info.Age
22 User.Info.Preferences Sports, Books
17
User Management Services

• Profile architecture targeted to enable User
  Management.
• User Profiles everywhere (auth, targeting,
  transactions, analysis, etc...)
• Registered User, Anonymous User, User Context,
  etc...
• User Management Requirements
• User Identification and Authentication
  Integration
• Anonymous user tracking leverage Commerce AuthID
  object (encrypted cookies, URLs)
• Register Users
• Integrate Authorization solutions (NT5, Active
  Directory)
• HTTP Forms solutions (Beta)
• Enable single logon across web servers (Beta)
• Highly scalable user management platform
• 10Million Users, 1 Peak Online, Mixed
  Transactions (login, read, write)
• Caching (object, property, connections, ...)
• Optimized database providers (ADSI/IDirectory,
  OleDb/SQL7/StoredProcs, etc...)
• Partitioning, Replication, etc...
• Database agnostic
• Directories (ads), RDBMS (sql), Other (oledb)
• 3rd party (Netscape, Oracle, DB2, etc..)
• Enable data privacy based on working internet
  standards (P3P, etc..).

18
User Profile Enabled Platform
19
Targeting
20
Targeting

• Expressions
• Content Selection Framework
• Predictor
• Direct Mail
• Campaigns
• Advertising
• Discounts
• Related Sell

21
Targeting overview
22
Targeting Requirements

• Fast, 50ms latency, 100requests/sec
• Real time event tracking
• Integrated with
• expression builder/evaluator
• user profile service
• catalog
• Platform for 3rd party extensions
• Comprehensive reporting
• Active extensible content templates

23
Targeting Architecture
Offline
Design Time
Run Time
ASP pages
Basket.asp
Expr Evaluator
Load Prediction Model
Content Selection Framework
Biz Desk Applications
Predictor Client
Expression Builder GUI
Expr Evaluator
Predictor Engine
Load Predictor Data
XML
SQL 7
SQL 7
SQL 7
SQL 7
SQL 7
SQL 7
SQL 7
Site Terms
Profiles Schemas . User . Context
Expressions
Data Warehouse
Catalog
Users
Context
24
Expression Evaluator
Expression Evaluator
Expression (stored in XML)
A and (B or C) or D
Eval Engine
Object Property cache
Expr result cache
P-CODE cache
Expression Load preprocessing
Expression Store Objects
Parse-tree
BizData Defs
P-code array
25
Content Selection Framework
Run Time Content Selection Framework
Design Time Business Desk Modules
Users
Web Page
Content Selector
Profiles
26
Recommendations

• scheduler launches model builder

• app downloads current model

• user goes to checkout page

• ASP calls predictor to add recommendations

• IIS logs feedback basket data to the warehouse

IIS
27
Direct Marketing
List Manager
Direct Mail Editor UI
Web Servers Personalized Mail pages.
SSA Scheduler
Campaign Mgr
Reports
BizDesk
Direct Mail NT Service
Data Warehouse SQL 7
Marketing Database SQL 7
SMTP Server
CDO
Mail Pipe
DTS
DTS
Web Logs
IIS EventProcessor
28
Campaign Architecture
Run Time Content Selection Framework
B2B Integration
Design Time UI Modules
Cache Mgr
Load Content
Users
Write Results
Campaign Mgr. BizDesk
BizTalk Server
MSN

Content Selector
Campaign UI
Web Page
3rd parties
Profiles
Exp Builder
Link Exchange
Filter
Exp Eval
Context
Score
User
Catalog
Real Media
Predictor
CSF Pipelines
Select
Content SQL 7
DW SQL 7
Record
Format
IIS Log
Click Sold, etc
Redir
Record
29
Targeting Applications

• Discounts
• Targets product page and users basket
• Supports Up-Sell, Cross-Sell and Inventory Sell
• Integrated with product catalog
• Advertising
• Real-time performance monitoring of campaigns
• Integrated User targeting
• Direct Mail
• Fast, Scalable, Runs as an Win2K Service
• Has a List Management object to support importing
  and merging of targeted lists
• Tracks mails sent, non-delivered, clicked

30
Data Warehouse Analysis
31
OverviewCommerce Data WarehouseCommerce Analysis
Interactive Analysis
Web Usage
Targeted Mail
Users
Site Data Warehouse
Transactions
Scheduled Reports
Catalog
User Profile Management
Advertisements
Content
Prediction Clustering
Document Tracking
Other
Other
32
Commerce Data Warehouse and AnalysisOverall
Process Flow
Event Data
On-line Services
Administration
ADO
Commerce events
Real Time Events
Commerce OLE DB Provider
Commerce Reports
Commerce Business Desk
IIS
Data Warehouse Service
Commerce data events
WebTV
VB
Other Events
Office 2K
Web logs
DTS Packages
ASP
IP Resolution
SQL Server
Non-Event Data
Other
WhoIs
OLAP Server
Commerce data
Custom Task
Other Data
HTML/XML
ADO / OLE DB
CDO
IE
33
Data Warehouse
34
Commerce Data WarehouseRequirements

• Tight integration with Commerce Server Components
• Data sources
• IIS and Apache log formats
• Transactions/Ads/Promos/Content/User
  Profiles/Products
• Extensible and programmable
• Create new data sources (Import Extend Schema)
• Modify and Manage cubes
• Access through standard OLE-DB interfaces
• Scalability Performance
• Support high volume sites internal customers
• 50 M pages/day, 100 M URLs, 30 M users
• Feedback to on-line systems
• User profile (for targeting via expressions)
• Direct Mail, Real-time Recommendations

35
Commerce Data WarehousePotential internal users

• Web TV (100M URLs, 1B rows)
• Windows Media Server
• Proxy Server
• Microsoft.COM (50M pages/day, 40M users)

36
Commerce Data WarehouseHigh Level Architecture
MS TV Server Events
Scripts
Web Log Import
Campaigns Users Transactions
ADO
Commerce OLE DB 2.0/2.5 Provider
XML Schema Definitions
Bulk Insert/Update
Commerce Storage Service
SQL Server
OLAP Server
37
Commerce Data Warehouse OLE DB Provider
Application
Scripts
XML
ADO
Commerce OLE DB Provider
OLE DB 2.0 2.5 Interface
Command Processor
Metadata
IRow
Bulk Insert Update
Bulk data parser
IRowset
DS/SQL Aggregations
Storage Driver
38
Commerce Data Warehouse Storage Services
Commerce OLE DB Provider
Storage Services
Metadata
Jobs
Sequencers
Command
Command
Data packet
Schema Manager
Calc Engine
Aggregator
Calc Sequencer
Delete Sequencer
Save Sequencer
Infrastructure
Cube Mgr
Collections Job Group Job Manager Async Job
Thread Pool Job Queue Connection Pool Virtual
Buffer Pools
Job Network
Job Network
Job Network
Partition Mgr
Job Mgr
Job Mgr
Job Mgr
39
Commerce Data WarehousePreliminary Performance

• Scales linearly with hardware
• Hardware configuration
• DB machine 4 500 MHz CPU, 1 MB L2 cache, 100
  GB array, 1 GB RAM
• Processing machine 4 500 MHz CPU, 512 MB RAM

40
Commerce Data WarehouseLog Import

• Multi threaded, no contention
• Smart memory management
• Processes commerce extensible log events
• Extensible cookie decryption

Users Map
Recycled String Buffer
Hit Queue
Hit Array
Inference Engine Job
CS OLE DB Provider
External Parser Plugin
Log Parser
User Hash
Storage
Hit Queue
Hit Array
Inference Engine Job
CS OLE DB Provider
Hit Array Pool
Users Map
Recycled String Buffer
Hit Array
41
Analysis
42
Requirements

• Integrated with Commerce Server
• Commerce centric reports out of the box
• Present reports via the BizDesk
• User definable calculations and reports
• Allow ad-hoc reporting
• Easy to use intuitive interface
• Performance Scalability
• Significantly increase reporting speed
• Provide long term trend analysis

43
Current Solution

• Define Calculation
• Calculations basis for all reports, defines
  dimensions, measures, and filters
• Can be named and saved in the calculation
  inventory
• Run/create/edit report in BizDesk
• Create from scratch or from existing report (New
  or Save As)
• Choose calculation from inventory or create new
  calculation, apply formatting to the calculation
• View/print Reports
• select from manager list of reports using
  search/find feature
• Export Calculations
• export to List Manager, Direct Mailer, User
  Profiles

44
Analysis Opportunity

• Leverage MS OLAP server
• Data Warehouse integration with OLAP
• Use third party OLAP Reporting tools
• Work closely with PLATO team
• Test if Shiloh meets our requirements
• Work out release logistics
• Integrate cube mgmt

45
Segmentation (Data Mining)

• Segment users based on implicit behavior
• System based discovery of hidden trends and
  behavior in the data warehouse
• Use discovered segments to drive targeting

46
Segmentation Architecture
47
Segment Viewer Screenshot
48
Pipelines Transactions
49
Requirements

• Backward compatible with 3.0 pipelines
• B2B features
• Support multiple POs per requisition
• Support multiple shipments per PO
• Purchase Order approval process
• Integrate with BizTalk
• OBI standards compliant
• Support 64-bit currency values
• Move component UI to the BizDesk
• Shipping Methods, Tax

50
What Is A Pipeline?

• Framework for modeling a commerce process
• Framework consists of stages
• Stage consists of components
• Each component operates on a commerce data
  object (e.g., OrderForm)

51
Pipeline Editor
Produces PCF files that are loaded into the
pipeline object on an ASP page
52
Example Pipeline, Stages
Basket Pipeline
Total Pipeline
Checkout Pipeline
53
Corporate Purchasing Model OBI - Open Buying on
the Internet
Supplying Organizations
Buying Organization

Employee with Browser
Browses Creates Requisition
Internet
Home Page with Vendor Links
Suppliers
Buying Org

Requisition \Basket
BizTalk Server
Purchasing Server
Catalogs
BizTalk Server
OBI Order Request
OBI Order - EDI / HTTP
Acct System Updated
Order Request Approval
Invoice to AP
LOB System
Workflow
54
Administration
55
Administration

• For system administrators (BizDesk is the
  business manager admin)
• Extensible MMC snap-in
• COM objects
• SQL schema for config data persistence
• Hierarchy of sites, addresses, and Web servers
• Resources are configurable admin entities

56
Admin Design
Admin SQL Database
(Can be remote, so central admin is supported)
Admin COM objects
Admin MMC snap-ins and extensions
57
Objects in hierarchy

• Resource
• Entity with independent config data (e.g.,
  Product Catalog, Direct Mail, Data Warehouse)
• Site
• Collection of Commerce Addresses sharing the same
  resources
• Address
• Single URL, single IIS application. May contain
  multiple IIS Web sites (a Web cluster or farm).
• Web Server
• An IIS Web site with Commerce content

58
Catalog
59
Customer Requirements

• Customers require product catalog management in
  the box
• Integrated with other features
• Marketing (product discounts)
• Data Warehouse (report on product sales)
• BizTalk Server (exchange catalogs with trading
  partners)
• Sophisticated search capabilities
• End User Management tools

60
Product Catalog - Overview

• User definable product types and category
  hierarchy (Data Driven Logical Schema)
• Multiple Search Mechanisms
• Import/Export/Merge via XML
• Catalog Interchange support via BizTalk Server
  (XML-Data Schemas for Catalog)
• High performance (using caching)
• Catalog Management Tools via BizDesk
• Custom Catalogs
• Export product data to Data Warehouse for
  reporting
• API (COM based)

61
Catalog Architecture
Commerce Site
SQL7 Catalog Collection
Catalog BizDesk
Schema Design
Editor
Catalog API Layer
Custom Catalog
CatalogExchange
Catalog DTS
BizTalk Server
BizDesk Framework
Export/Import
Commerce DW
Commerce Catalog XML
62
Catalog Schema
Catalog Collection
Example
Property name
Property Definition
Catalog Schema
Property size
Catalog Data
63
Catalog API

• COM Object model
• Provide high runtime performance through use of
  caching
• Catalog Object
• Manage Schema
• Perform Searches
• Import/Export XML
• Product Object
• Read/write product details
• Category Object
• Read/write category details

64
BizDesk
65
Business Desk

• Web-based framework for hosting business
  management modules
• zero-install, remote access, extensible
• 20 modules shipping for common business
  operations
• Business Users
• Common look and feel in IE 5
• Rich UI components in DHTML behaviors, simple
  navigation model, web based wizards
• Developers
• COM objects for server side navigation, data
  validation security
• SDK for Module Development

66
Business Desk
67
Technologies

• DHTML for client side UI including use of
  behaviors, dynamic properties
• CSS for layout
• XML for configuration of navigation, client
  controls, and transport of data from server to
  client
• XSL for XML transforms
• XML data-binding to simplify draw code
• ADO Recordsets for data initialization
• HTTP/XML communication in some controls (so full
  page doesnt refresh every time)
• HTML Help ActiveX control for help files

68
BizDesk Architecture
Client
Server
Edit Sheet
Commerce business objects
ADO
Edit Field
css
css
Expression Builder
Dynamic Table
css
xml data binding
HTTP/XML
xsl
css
ASP
xsl
Converts recordsets from objects and ADO to xml
for widgets to consume
Profile Builder
List Editor
xml data binding
HTTP/XML
xsl
css
css
Tree View
Widget HTC (behavior) files
xsl
css
dynamic properties
69
Solution Sites
70
Solution Sites Requirements

• Provide applications which demonstrate CS4.0
  Integration
• Provide a Jumpstart for the development process
• Demonstrate best practices code samples for
  CS4.0
• Blank Site
• This site contains a full feature BizDesk site
  and a minimum user site. This is the site a used
  by development when a solution site doesnt meet
  their needs

71
Solution Sites

• Common features
• Business data driven for most site functionality
  (exposed to the developer thru MMC)
• Save Orders
• Table driven shipping and tax component
• Partner Desk Services for User
• Supplier
• Multi-line ship-to support
• Delegated User Admin
• Blanket PO designation

72
Solution Sites

• Corporate Purchasing
• NT/CR authentication supported
• One Requisition can support multiple POs
• Multiple catalog support
• Vendor specific catalog
• Support for Line-item or Order Level Account/Cost
  Center Designation on PO
• POs sent to Vendors via BizTalk
• Retail
• Mktg Campaign Management Integration
• Auctions
• Lightweight Inventory mgmt system

73
Site Mobility - PUP

• Envisioned users
• SP moving a site from their development
  machine(s) to customers machine(s) and/or
  corporate pre-production environment moving site
  into production
• Packer UnPacker (PUP)
• utility that takes the resources of a Commerce
  Site and places them into a compressed package or
  extracts them from a package
• package includes virtual roots, directories and
  files, connection strings, database schema and
  optionally data
• the way a Commerce Site plugs into the MMC
  configuration infrastructure
• by default it moves all resources of a site,
  advanced mode allows packing and unpacking of any
  combination of individual site resources

74
PUP Architecture

• Keywords - extensible, simple, generic
• Every site has any number of resources. Each
  resource supplies an object that implements the
  IPup interface
• The PUP performs the following tasks via methods
  on the IPup interface
• Export() packs a site resource into a package
• Import() unpacks a site resource from a package
  into a site
• Delete() removes a site resource from a site
  (Beta)
• CreateGroupResource() creates a logical group
  resource
• DeleteGroupResource() deletes a logical group
  resource (Beta)

75
PUP Architecture (cont.)

• If a third party vendor wants to add a resource
  to our architecture they supply an IPup object,
  add the resource into the Site Config data-store
  (via SDK code) and the PUP will transport their
  resource along with ours
• The PUP operates in two modes
• Command line
• allows execution of the PUP via script
• used by our MMC snap-in
• GUI
• Simple requires very little user input and
  forces reasonable defaults
• Advanced allows extensive flexibility but
  requires more knowledge of our architecture

76
Solutions Architecture

• Thin layering over Platform objects.
• App helper layer
• AppCfg VBScript friendly COM configuration.
• Helper functions (e.g., Localizable address
  formats)
• Wrapper functions that we plan to feed back into
  platform post-Alpha. (e.g., product catalog
  wrapper functions)
• App Display layer separate the HTML for display
  from ASP code for business logic.
• Robust ASP error handling Customer should never
  see VBScript errors in production mode. Rich
  error display in development mode.

Application Display Layer
Reusable Application Helper Layer
Helper ASP Code Library
Object Wrapper ASP Code Library
AppCfg Object
Commerce Server Platform Objects
77
Wrap-up
78
CS Timeline

• Alpha - 09/99
• Beta Code Complete 10/99
• Beta - 12/99
• RTM - Q1/00

79
Challenges

• Need to backfill some key positions
• Dev Manager, Dev Lead, Test Lead, 3 PMs
• Dependencies
• W2K (IE5, CDO, IIS5) needs to ship on time
• Shiloh resolve integration and schedule
• BizTalk resolve integration and schedule

80
Challenges (cont.)

• Post-Alpha work items
• BizDesk usability review
• Convert pipeline definitions to format for use
  with BizTalk
• Integration with Plato/Shiloh
• Content Manager
• Related Sell (Upsell, Cross Sell)
• Additional Solution Site
• Catalog API, UI for custom catalogs
• Perf/scale work

81
The End!
82
Backup Slides
83
ExpressionsOverview

• Show promotion for new convertible when the
  current user is male AND lives on the west
  coast
• Show ad for golf courses when the current user
  is older than 65 years of age and it is the
  afternoon
• Select all users from user profile data store
  who are between 25 and 30 years of age and live
  in Seattle
• Select all users from user profile data store
  who are between 25 and 30 years of age and live
  in Seattle

• Expressions
• Targeting conditions
• Filter queries
• Business rules
• Other expressions
• Expressions are
• Built on profiles
• Stored in BizDesk modules
• Evaluated at run-time

84
DW Extra Slides
85
Commerce OLE DB ProviderFeatures - Components

• Storage Driver
• Communicates with Storage Services
• Packs client side buffers asynchronously and
  submit for further processing to Storage Services
  (in-proc or remote)
• Packing involves normalizing data laying out
  buffers for optimal server processing
• Contains in-memory copy of the Schema Catalog
• Many intrinsic commands
• Bulk Insert/Update
• Parses XML subset, comma, tab and space delimited
  data files
• Supports XML based or using OLEDB/ADO
• Schema changes/definitions
• Calculations changes/definitions/execution
• Report changes/definitions/generation
• SQL Command (Supports subset of syntax)
• SELECT, INSERT, DELETE, UPDATE

86
Storage Services Features

• Provides rich and high performance logical
  abstractions over SQL
• Supports mapping to legacy DBs and partitions
• Custom web data types URI Date
• High performance relation management
• Supports ROLAP surrogate keys slowly changing
  dimensions
• Run as in-proc to OLE DB Provider or as NT
  Service
• Tight integration with OLAP services
• Robust error recovery from many soft database
  errors
• Extensive runtime information collection for
  further analysis

Reference Slides Storage Service Components
87
Storage ServicesComponents Catalog

• Catalog container for Metadata
• Logical
• Classes (Read Only, Non Persistent, Abstract..)
• Members Vectors/Multi-values, computed members
  de-normalization
• Efficient Relations (1-1, Inheritance, 1-M, M-M)
• Key Def, Partition Key Def, Type Def, Partition
  policy
• Aggregate definitions Calculations data
  versioning (slowly changing dimensions)
• Physical
• Table Def / Column Def Template of Physical
  Table
• Source Def, Partition defs, Indexes
• Optimal storage of relations enables ROLAP
  schemas
• Features
• Dynamic Schema
• Can add modify classes, members, relations on
  the fly
• Automatic physical schema generation from logical
  (manual optional)
• Map logical entities to legacy databases
• Loads and compiles the schema for efficient
  processing
• Schema definition in XML or ADO/OLE DB

Reference Slide Storage Services Logical Schema
88
Storage ServicesComponents - Cube Manager Calc
Engine

• Cube Manager - handles MS OLAP integration
• Manages cube schema
• Synchronizes cube schema and logical schemas
• Member add ? dimension prop/dimension/nothing
• Relation add ? parent becomes dimension
• Manages virtual cubes
• Manages cube data
• Creates/modifies cubes
• Manages cube partitions
• Intelligent incremental update of dimensions/fact
  data in parallel
• Calc Engine query translation and execution
• Translates complex queries specified in XML to a
  series of SQL query jobs or to a single OLAP
  query.
• Uses Job Manager infrastructure to run generated
  queries in parallel when invoking SQL queries.
• Assembles the results and invoke appropriate
  calculation render components.

89
Example Web Request job network processing
90
Storage Service Components
91
Storage ServicesLogical Schema

• Logical Schema manipulated through OLE DB 2.5
  Provider
• Special Schema Rows/Rowsets
• Logical Abstraction through Class, Members,
  and Relations
• Versioning support on the Class level

Class B
b1 b2 b3
x y z d
A
B
A
Mapping members from a class to a single
physical table
Mapping members from a class across multiple
physical tables
Mapping members from multiple classes to a single
physical table
Various class-to-table mapping schemes
92
Storage ServicesLogical Schema

• By writing to a logical schema, we greatly reduce
  the burden on the developer. For example, in log
  import we write to one logical class,
  Request/Hit which stores in potentially 17
  physical tables

93
Storage Services Features - Scaling

• Scaling
• Horizontal and vertical partitions across
  multiple servers/databases/tables
• Partitioning based on simple expressions (date,
  hash) and support for partition policies
• Multi dimension aggregates to reduce storage and
  I/O
• High performance
• Dynamic data layout in memory for cache friendly
  access and high speed in memory processing
• Algorithms to do highly parallel processing as
  well as parallel relation resolving with little
  lock contention
• Hierarchy of work queues for processing and
  database accessing in parallel
• Batch operations parallel BCP operations
• Intelligent use of physical resources
• Parallel jobs against multiple tables/databases/se
  rvers
• Supports row IDs of 1, 2, 4, 6 or 8 bytes wide.

94
Storage ServicesADO Schema Example

• -----------------------------------
• ' Add a new CLASS
• -----------------------------------
• rec.Open "CLASS/NewCls", cn, adModeReadWrite,
  adCreateOverwrite
• rec("IsPersistent") 1
  Persistent class
• rec("CatalogName") "test1 Name of
  the catalog to store
• rec("ClassDefName") "NewCls Name of
  the new class
• rec("SourceDefName") "test_Source Name of
  the source
• rec("GenerateTableDef") 1 Create a
  table
• rec("GeneratePartitionDef") 1 Create a
  partition
• rec("GenerateKeyDef") 1 Create a
  key for the class
• rec("GenerateIdentity") 1 Create
  identity column
• rec("__Commit") 1 Commit
  this class
• Commit the class to the catalog
• cmd.CommandText "CommitSchema
• cmd.Execute

95
Storage ServiceADO Schema Example

• ' ----------------------------------
• ' Add a new MEMBER
• ' ----------------------------------
• rec.Open "MEMBER/NewCls/NewMem", cn, adModeWrite,
  adCreateOverwrite
• rec("MemberDefName") NewMem"
• rec("GenerateColumnDef") 1
• rec("IsPrimaryKey") 1
• rec("DefValAsStr") "DefaultValue"
• rec("TypeName") "WSTR"
• rec("__Commit") 1
• ' Commit Catalog changes
• cmd.CommandText "commitschema"
• cmd.Execute

96
Storage ServicesADO Example

• ' ----------------------------------
• ' Add a new Instance
• ' ----------------------------------
• rec.Open "Instance/NewCls", cn, adModeReadWrite,
  adCreateOverwrite
• rec(NewMem") StringData
• rec(Date") Now
• rec("__Commit") 1

97
Storage ServicesDynamic Resource Management

• Storage Services
• Global
• Partition
• DB
• Server
• Resources
• Batch Level
• Class
• CPU

98
Commerce Data WarehouseStatus

• Bugs
• 50 active for Alpha
• 160 active for Beta
• Beta Features
• Full multi-value support
• Improve robustness, performance scalability
• WhoIS, Title Lookup, Scheduling
• High speed deletes
• Archiving

99
Targeting Extra Slides
100
Expression Requirements

• Based on Profiles
• Represented in XML
• Basis for application targeting
• Fast, 150 evaluations in 15ms
• Common UI
• Common Runtime
• Integrates with CSF
• Support translation into SQL queries
• Integrates with Site-Terms

101
Expression Builder

• Business Manager focused UI for authoring
  expressions
• DHTML launched in application context via the
  Business Desk
• Expression Grammar configurable by application
• Consumes profile schemas site terms
• Data Store
• Used to persist, name and retrieve expressions
• Expressions represented in XML

102
Expression Evaluator

• COM component that evaluates specific expression
• Expressions can be preloaded and precompiled
• Can evaluate multiple expressions per request
  (e.g. CSF)
• Evaluation results cached for duration of request
• Callable with expression Id or expression XML

103
Content Selection Framework

• Open content scoring architecture
• Many reusable components
• Very Fast
• Speed research efforts created
• Fast Dictionaries via hash key lookups
• High performance content object
• Cache Controller
• Active content templates (patent applied for)
• Peer-to-peer cache synchronization (patent
  applied for)
• Open content scoring architecture (patent applied
  for)

104
Prediction Engine

• Uses MS Research technology (patent applied for)
• Distributed model architecture

105
Content Selection Framework

• Platform supporting content decision applications
• Advertising, Discount, Content Delivery, Coupons
• Based on commerce pipelines
• Provides
• integrated targeting support with expression
  evaluator
• content formatting templates
• open platform to build your own content BizDesk
  modules
• Supports multiple types of events

106
Real-time Recommendations

• Expands Intelligent Cross Sell from 3.0
• Value-added functionality specific components
• Product cross-sell
• Personalized product recommendation
• Algorithms can be parameterized
• Confidence levels
• Number of recommendations returned
• ASP wrapper functions make coding easier
• Highly scalable architecture
• Developed jointly with MS Research

107
Direct Marketing

• Direct Marketing is an off-line form of
  targeting.
• It leverages the Data Warehouse Calc engine as
  one way to produce lists of users to send mail
  to.
• Via the List Manager object we can get targeted
  lists of users from Analysis, SQL or flat files.

108
CSF Global.ASA

• Dim CacheMgr
• set CacheMgr Server.CreateObject("Commerce.Cache
  Manager")
• CacheMgr.LoaderProgId(discounts")
  "Commerce.LoadDiscounts"
• CacheMgr.WriterProgId(discounts")
  "Commerce.WriteEvents
• Set ContentTypeMap Server.CreateObject(
  "Commerce.Dictionary" )
• Set Application("MSCSContentTypeMap")
  ContentTypeMap
• Set Pipe server.CreateObject("Commerce.OrderPipe
  line")
• Pipe.loadpipe(Discount.PCF")
• Set ContentTypeMap("advertising") Pipe

109
CSF Web Pages

• Dim CSO, rsDISC
• Set CSO Server.CreateObject("Commerce.ContentSel
  ector")
• Set CSO.Evaluator Application("MSCSExpressionEva
  luator")
• CSO.Size "Banner"
• CSO.TargetFrame "_top"
• CSO.NumRequested 1
• CSO.PageGroup "Home"
• Set CSO.UserProfile UserProfile
• Set CSO.CacheManager Application("MSCSCacheManag
  er")
• Set CSO.ContentTypeMapping Application("MSCSCont
  entTypeMap")
• Set rsDISC CSO.GetContent("Discounts")
• If NOT rsDISC.EOF Then Response.Write(rsDISC("html
  ")) End If

110
CSF Redir.ASP

• set Pipe Application("CampaignsCSFEventPipe")
• sCacheName Request.Querystring("CacheName")
• set dictCache Application("MSCSCacheManager").Ge
  tCache(sCacheName)
• sEvt Request.Querystring("evt")
• ciid clng(Request.Querystring("ciid"))
• set dictOrder("_Performance")
  dictCache("_Performance")
• Pipe.orderExecute 1, dictOrder, dictContext,
  Errlvl

111
Content Delivery (not for Alpha)

• Separation of page logic, format and data
• Allows a business user to easily manage, format
  and target content on a page
• Allows developers
• Simple programming interfaces for the VID
  developer
• High performance and scalability
• Extensible formatting templates

112
Advertising features

• Campaign or Campaign Item impression goals
• Real-time performance monitoring of campaigns
• Exposure limits

113
Discount Campaign Mgt

• Reacts to product page and users basket
• Related Sells Campaign Management
• Supports Up-Sell, Cross-Sell and Inventory Sell

114
Direct Mail Campaign Mgt

• Fast, Scalable, Runs as an NT Service
• Has a List Management object to support importing
  and merging of lists
• Campaign tracking of mails sent, clicked
• Non-delivery processing is in danger of being
  dropped due to insufficient staffing

115
User Profile Extra Slides
116
User Profile System Status

• Alpha
• Currently coding and testing SQL7 and AD support
• Need to test scalability of architecture
• Site Auth object working
• Post Alpha
• Need to implement secure Nt5 forms-auth filter
• Need to enable and aggressively test distributed
  data source support (SQL7, AD)
• Need to implement MMC designers and migrate
  BizData Admin Object to C.
• Need to re-align with latest XML/Schema/Data
  standards.
• Need to implement BizData OLEDB BULK update
  support
• Dependencies
• NT5 Active Directory ADSI
• DAG SQL, ADO, OLEDB
• XML (DOM, Standards, BizTalk, etc.)
• IIS5, ASP, COM
• Issues
• Dev and Test bandwidth remains throttled.
• Scaling and deploying on win2k AD is a challenge
• Profile Svc has no Tx support in Commerce 4.0

117
BizData Runtime Object Features

• Profile Service Object
• Runtime data object exposing simple consistent
  object model for business objects (users,
  context, content, ...).
• Optimized architecture minimizes user profiling
  transaction loads.
• Supports business data extensions (groupings,
  terms, meta tags, etc..)
• Enables Commerce Applications (User Management,
  Targeting, ...)
• Commerce OleDB Provider
• Enables data source interoperability (ActiveDs,
  SQL7, 3rd Party (LDAP, ODBC)
• Enables data source aggregation (objects derived
  from multiple sources)
• Enables physical to logical data translations to
  better reflect business logic.
• Enables optimized data source access (connection
  pools, distributed databases, memory pooling,
  configurable caching, etc...)
• Enables solid integration with Analysis system
  (DTS export flags).
• Expression Evaluator Object
• Enables optimized runtime expressions (business
  rules) based on profiles for targeting and
  personalized applications.
• Optimized caching and pcode enables optimized
  evaluation in runtime.

118
Profile Object Architecture Feature Breakdown
Profile Service Object Features Manages primary
profile object factory (Get, Delete, Create)
Set User GetProfile(Joe, USER) Set Org
GetProfileByType(orgID, User.Info.OrgId,
Organization) Enables cross-page profile object
cache Instantiated as Application Object
under IIS (e.g. Global.asa) Configurable
Object Cache (TTL expiration, Cache
pre-allocations) Configurable Profile
Property Loading On first load of
profile object fetch only properties x, y, z.
Other properties (e.g. Images) are
loaded on demand. Profile Object
(Run-time) Profile object interface supporting
simple property level access (set/get
user.name). Exposes standard object interfaces
for property access ADSI-IAds,
ADO-Fields User.Name joe
User.Address.Zip 98052
User.SetInfo Enables R/W property cache (cache
flush and reload semantics). ADO Support
(Design-time) Simple ANSI-SQL semantics
INSERT INTO user VALUES (namejoe,
emailjoe_at_ms.com) UPDATE user SET (email
joe_at_ms.com) WHERE name joe DELETE FROM
user WHERE name joe SELECT name,
address.zip FROM user WHERE name joe
ANDS only SELECT FROM user WHERE
lastModified June 5, 2000 170000 ORDER BY
guid
119
Profile Object Architecture Breakdown (continued)
Commerce OLEDB Provider Enables ADO support
with simple ANSI-SQL semantics (profile table
object) SELECT, UPDATE, DELETE, INSERT
BULK UPDATE, DELETE (Beta) Manages BizData
Profile Definition Cache Profile Definitions
(users, context, etc..) Enables logical to
physical data source mappings Manages Data
Source Aggregation Plug in Data Source
Providers include (LDAPv3, SQL7, OleDb/ODBC)
Simple COM objects providing data source specific
interface Manages joining or chaining across
across data sources. Profile Data Source
Providers ADSI/LDAPv3 Provider Enables
Active Directory, Membership, ILS, ... Netscape?
Supports NT Secure Login based on security
context of logged in user. Enables
connection pooling (beta) Supports DN
semantics (hiearchical storage) Commerce/SQL7
Provider Enables scalable property store on
SQL7. Leverage stored procedure interfaces
Supports MULTIVALUE properties Support
Database Partitioning (beta) OLEDB/ODBC
Enables ODBC sources supporting basic ANSI-SQL
INSERT, UPDATE, DELETE, SELECT
Supports connection pooling, unicode, ...
SQL7, ORACLE, DB2?, etc...
120
Profile Design Flow

• Design Business Profile Objects using the Profile
  Designer
• Create profiles as a collection of properties
• Users, Context, Products, etc...
• Include extended profile data types
• Site Terms User.Favorite_Color - Site_Colors
  (red, blue, green, etc..)
• Profile References User.Manager,
  User.Organization
• Group related properties User.Home_Address.Zip,
  User.Business_Address.Zip
• Attach application specific property tags
  (attributes)
• User.Age (constraints, UI hints, descriptions,
  etc...)
• Map profile definitions to physical data source
  objects
• Map property to physical data source object
• LDAPv3 Directory (class attribute)
• SQL7 Database (table column)
• ODBC Data source (table column)
• Manage data integrity (Primary Keys, Data Types)
• For profiles aggregating across multiple data
  sources define the Join Keys.
• Non-mapped properties are considered dynamic
  (cached properites) that only last for the object
  session.

121
Simple User Profile DefinitionXML Sample
Definitions"
typeDefName"STRING" keyDef"PRIMARY"
IsExported"1" idref"ADSource.User.email"/
typeDefName"STRING" isSearchable"1"
isExported"1" idref"ADSource.User.cn"/
name"UserID" typeDefName"STRING" keyDef"JOIN"
isExported"1" idref"ADSource.User.GUID/

name"Password" typeDefName"STRING"
"/

creditCard"/

typeDefName"NUMBER" isExported"1"
e"/
typeDefName"DATE" isExported"1"
Visit"/
typeDefName"STRING" isExported"1"
isMultiValued"1"



122
User Profile System Performance Goals
Enable Improved performance for user data access
under IIS Enable cross page caching Enable
optimized memory management Enable connection
pooling Enable optimized data source interfaces
(IDirectory, SQL stored procs, etc.) Enable
support for session state (object cache, ILS
support, IMDB?) Challenge Work with DAG and NT
teams to incorporate optimizations and logic into
data access components. Opportunity Work with
AppServer team to incorporate extensible
architecture and caching solutions into core
architecture? Enable Large User Populations (
10Million Users ADSQL) Enable support for
advanced database configurations
(beta) Partitions SQL7 databases ( 1M users
per SQL x 50 props) Distributed AD Directories
( 1M users per DS) Challenges Work with NT
ActiveDS Team on optimal distributed solutions
that will manage anticipated transaction loads
(logins, searches, modifies) Enable Transaction
Load Balancing Minimize Database access (Smart
Object/Property based Caching) Enable Web server
load balancing (beta) single login, cross
server session state, etc... Challenges NT
security context/Secure Passwords Opportunity
Drive BizData Object architecture to 2nd tier
service Opportunity Align with
AppServer Session Service (investigate plans)

123
User Profile System Performance Numbers

First performance run on Alpha drop TBD ASP
Simple ASP/second ASP/ADO/SQL7 ASP/
ADSI/AD ASP/UPO/SQL7 ASP/UPO/AD
124
User Profile SystemCommerce Bizdata Enabled
Services and Integration

• Targeting Services
• Enables user personalization, targeting
  applications and services.
• Content Selection Framework leverages User
  Profiles and Expression Evaluator object.
• More detail to follow.
• Transaction Services
• Easily integrates with transaction services
  (Pipeline and Biztalk) to enable user context and
  transaction management.
• More detail to follow.
• Site Analysis and Decision Support
• Solid integration with Commerce Dataware house
  with custom DTS task for incremental runtime
  export of Bizdata profile objects.
• User profile integrated with Site Auth and IIS
  logging
• BizDesk application integration enables feedback
  into runtime system.
• Business Data Object Interchange
• Enables integrated solutions with BizTalk
  interchange services .

125
Sample XML Expression YoungActiveUser
User.Age 15 AND User.Age (User.Info.Preferences Contains(extreme
sports) OR (User.Info.Preferences
Contains(mountain biking) OR (User.Info.Prefer
ences Contains(auto racing) ) TYPE'and' RTY ID'User.Info.Age' TYPE'number'
/ 15 AUSE ID'User.Info.Age' TYPE'number' / TYPE'number'25 TYPE'or' ID'User.Info.Preferences' TYPE'string'
MULTIVAL'true' / TYPE'string'extreme sports E ID'User.Info.Preferences' TYPE'string'
MULTIVAL'true' / TYPE'string'mountain biking SE ID'User.Info.Preferences' TYPE'string'
MULTIVAL'true' / TYPE'string'auto racing