Title: How to Get Started with Learning Analytics
1How to Get Started with Learning Analytics
2Who Will Benefit?
- Those organizations that have an imperative to
begin the process of proving the value of
learning - Those professionals that are trying to formulate
a strategy around Measurement and need additional
data to develop that strategy - Vendors that have training products that are
charged with helping their customers connect the
value of the products to the customers business
3Agenda
1. Background Approaches
2. Learning Analytics Maturity
3. Developing a Vision
4. Elements of Project Success
5. Project Evolution
6. Case Studies
4Learning Analytics Approaches
- Opinion-based data
- Relies on collecting data about perceptions of
impact of learning - Challenge - Hawthorne Effect
- Harder to defend when the result is a change in
business process - Operations-based data
- Focuses on the correlation of learning data and
business data - Challenge - Data cleanliness confounds
correlative ability - Need to understand impacts on data
5Business Intelligence
- "The oft-quoted example of what data mining can
achieve is the case of a large US supermarket
chain which discovered a strong association for
many customers between a brand of babies nappies
(diapers) and a brand of beer. -
- The explanation goes that when fathers are sent
out on an errand to buy diapers, they often
purchase a six-pack of their favorite beer as a
reward."
Financial Times of London February 7, 1996
6Measurement
If you cant measure it, you cant manage it.
Nolan Norton Consultants Founders of the
Balanced Scorecard
Driven by demand for rapid content deployment,
plus growing interest in value-added modules like
training analytics and competency management, the
market for e-learning infrastructure systems from
U.S.-based vendors is expected to grow 12 in
2004 to 529.4 million
Simba Information 1/9/2004
7History of Business Intelligence
- Originally conceived as Data Warehousing Data
Mining - Now called Business Intelligence (coined by
Howard Dresner of Gartner Group, 1994) - Dresner defined BI A generation of software
that allows corporations to accelerate the rate
at which managers can physically process
information - Traditionally expensive and hard to do
- Today available to everyone - Techniques, Best
Practices, Tools - Includes data integration, analysis, reporting,
and data visualization
8Current Practices
- Better Techniques
- Technology allows for data integration
- Purchase pre-defined configurations
- Best Practices
- Focus on business metrics
- Group and filter data
- Drill up and down
- Create visually expressive charts
- Better Tools
- Microsoft Business Intelligence Platform
- 9 Tools one of which is Microsoft Office
Professional
9OLAP
- On Line Analytical Processing
- Allows the user to interact with the data
- Multi-dimensional analysis
- Drill up or down through various dimensions
characteristics of the data that you are looking
at - Contrasted with
- Standard SQL Reports
- One time setup
- Choose parameters
- Static results at a moment in time
10Why Do Learning Analytics?
- Turn Data into Information
- Measure Learning Effectiveness
- Learning Activity
- Catalog Effectiveness
- Total Cost of Learning
- Manage Compliance
- Understand Business Impact
- Focus Strategic Alignment Initiatives
- Using Business Metrics as your guide
- Allows alignment of learning with strategy
11Expected Results
- Correlation of business data and learning
intervention data - Use correlations to driver operational changes
- Incremental skills in getting the learning team
to understand operational data - Incremental skills in getting the operations
group to understand learning data
12What NOT to Expect
- Individual prescription based on individual
results - Contrast with Performance Management where
individual performance contributes to the greater
performance metrics - Absolute certainty cause effect
13Correlation of Business Learning Data
Determines
Effectiveness of Learning Experience
Determines
Learning Experiences
14Learning Analytics Maturity
- Level 1 - Influence Individual Action
- Just starting to collect information about
Learning Experiences - Requires tools like LMS or equivalent
- Need to template Business Questions so that the
right data is collected from the outset - Level 2 Understanding the Business
- Acquired tools to collect data but early in the
process - Trying to understand what Business Metrics have
value in the organization - Begin to draft Vision for Learning Analytics
- Level 3 Questioning Effectiveness
- Has collected learning data for 6 months have a
sense of the Business Metrics that have a
reliable correlation to Learning Experiences - Ready to implement Vision for Learning Analytics
15Documenting the Vision
- Business Opportunity
- Provides context for the initiative(s)
- Includes a Vision Statement
- Benefits Analysis
- Solutions Concept
- Roadmap for initiative(s)
- Analysis -gt Risk, Feasibility, Usability,
Performance - Solutions Design
- Proposed Technical Architecture
- Initial Project Scope
- Provides range of features/functions
- Defines out of scope
- Criteria for success
16Vision Template
- Send an email request for document
- Commitment to provide feedback to first cut at
Vision document
17Elements of Project Success
Business Sponsors
Cross Functional
Business Reps
Text
Lots of Data
Learning Analytics Projects
Meta- Data
Skilled Staff
Iterate Projects
Clean Data
WBS
Business Analysis
Text
Computerworld White Paper Shaku Atre, Atre Group,
Inc. 2004
18Project Team Composition
- Business Executives
- Customers
- External business partners
- Learning
- Finance
- Marketing
- Sales
- IT
- Operations
19Involving Business Sponsors/Execs
- Understand the value of the project remove
political barriers - Focus the initiative to a specific set of
business questions manage the scope - Initiate a data-quality campaign within their
organizations - Periodic project reviews
20Iterate Projects
- Develop a Clear Vision
- Go through a Readiness Assessment Exercise
- Operationalize your Learning Analytics
- Integrate Business Analytics
21Readiness Assessment
- Focus is to define initial cut at Business
Questions - Identify Business Owners and Involvement
- Identify Process Outputs and Users
- Identify logical Data Sources and availability
- Identify iterative projects
- Prioritize iterative projects
- Develop SOW for Operationalizing Learning
Analytics Engine
22Operationalizing Learning Analytics
- Identify Learning questions
- Identify sources/uses of data
- Validate data integrity
- Install analytics server
- Validate ETL Schema adjust as necessary
- Set up Template Reports that address initial
questions - Weekly reviews of reports and opportunities
23Integrating Business Analytics
- (Assumes Learning Analytics engine is operational
and OLAP Analysis on learning data is being done) - Identify Business questions
- Identify sources/uses of different data sets
- Validate data integrity
- Determine ETL schema adjust as necessary
- Validate ETL schema
- Set up Template Reports that address initial
questions - Weekly Reviews of reports and opportunities
24Uses of Information
- Adjust Program Design
- Improve Program Delivery
- Influence Application Impact
- Enhance Reinforcement for Learning
- Improve Management Support for Learning
- Improve Satisfaction with Stakeholders
- Recognize Reward Participants
- Justify or Enhance Budget
- Develop Norms or Standards
- Reduce Costs
- Market Learning Programs
Phillips, Phillips, Hodges Make Training
Evaluation Work ASTD, 2004
25Case Study Learning Analytics
- Profile
- Packaging Shipping
- 1100 Retail Centers
- 18,000 Employees
- 250,000 Hours Training for new business
orientation - Intervention
- 3 different certifications assigned by job role
- 12-14 modules for each certification
- Had to be complete in 8 weeks
- Results
- 1100 Concurrent users of the learning content
- 300 of these were simply running reports
- Allowed for self-service reporting analysis
- Challenges
- Had to teach some level of application
proficiency to Retail Store managers - Field support went to regional HR managers (not
IT Help Desk)
26Case Study Business Analytics
- Profile
- Telecomm
- 16 Call Centers
- New product rollouts happening quickly
- Customer defections increasing
- Intervention
- Monitored training activity in 6 call centers
- Obtained business data from the same call centers
- Results
- Data indicated that training impacted sales
6-15 - Four call centers needed to significantly
increase training - Challenges
- Use of correlated data is marginal justification
for significantly altering business operations - Setting up the right environment with the right
data set for analysis is challenging
27Case Study Unintended Results
- Profile
- Retail
- 1254 Store locations
- Product training defined monthly based on
seasonal merchandise - High turnover in personnel
- Intervention
- Monitored training completions at the store level
- Utilized store sales results as the business
operations benchmark
- Results
- Negative correlation between training and store
results - Further inquiry revealed inappropriate
application of training - Challenges
- Broadly promoting results in advance of
understanding the data and what is driving it - Getting complete data sets from large audiences
without investments in management technology
28QA - Discussion
Jim Everidge, President Rapid Learning
Deployment, LLC (770)874-1190 x
222 JEveridge_at_rapidld.com www.rapidld.com