StarTrace: Enhancements and Demo

1
StarTrace Enhancements and Demo

• James Zhang Jesse Trutna

2
Motivation

• Web applications are complex
• Many inter-dependent components on different
  hosts
• Single request turns into many sub-requests
• Previous work demonstrated the value of
  path-based analysis
• Detecting and diagnosing failures
• Identifying problematic interactions and
  bottlenecks
• Obtaining a global view of distributed systems
• Most current production web apps cannot track the
  path of a request

3
StarTrace Review

• End-to-end request tracing framework
• Every request step is associated with x-trace id.
  
• X-trace id has parent/child ids
• Construct tree representing network path
• X-trace ids are written into the logs.
• Apache, Rails, Mysql
• No modification to business logic
• Apache mod, Ruby plug-in, no modifications in DB.
• Low overhead
• No measurable overhead from StarTrace alone.

4
Things We Accomplished

• Implemented StarTrace through ARM
  (Apache-RoR-MySql)
• Incorporated dtrace into StarTrace
• Leverages dtraces ability to top-to-bottom
  tracing.
• Puts x-trace ids into dtrace output.
• Created a StarTrace Demo App
• Set up a simple 3 tier application with StarTrace
  enabled on all the relevant components.
• Reconstructed paths as logs are being written.

5
Marry Dtrace with StarTrace

• Created a tool called Loupe
• Used to automatically start/stop dtrace processes
  for RoR dispatchers and to collect dtrace output.
• Enable the use dtrace to capture more details on
  demand
• Can use the existing Ruby provider for dtrace
• Created a custom StarTrace provider for dtrace
• Can use dtrace to do logging in the absence of
  other logs
• This is still somewhat expensive

6
Tools
7
Ruby Provider for Dtrace

• Existing ruby provider insufficient
• No access to ruby function arguments
• Expensive to activate
• Wrote supplementary ruby provider
• Ruby class with instrumented no-op class methods.
• Essentially allows addition of probe points with
  ruby code, instead of C.

8
Loupe
9
Loupe Implementation Details

• Loupe Architecture
• Loupe server is simple distributed ruby object
  running on app server on port 57473 (STATE)
• Monitoring thread watches process list for
  new/deceased dispatchers.
• Recieves dtrace script via RPC. Loupe executes
  dtrace script on all current dispatchers.
• Aggregate dtrace output into a log file to
  minimize network traffic.
• All Dtrace providers accessible.
• Notably, trace ids in RoR via our Ruby dtrace
  provider.
• But also built in system providers. System
  calls, network behavior, etc, including all
  future extensions.

10
Loupe Overview
11
Lumberjack
12
Lumberjack Details

• Input is a stream of logs
• Apache, Rails, and Mysql Query Log
• All interleaved
• Insert events into a fast embedded database
• Events id, host_ip, timestamp, errors, data
• Different logs have different data
• Ex Apache log has request time, url, http code.
• Ex Rails log has session id.
• Ex Mysql query log has the actual SQL queries
• All stored as YAML in the data field in the
  events db table
• Currently can process 1000 log lines/sec.

13
StarTrace Demo App

• A chance to put everything together
• We have StarTrace/X-Trace enabled on Apache,
  Rails and Mysql.
• X-Trace ids are written into logs, but they need
  to be extracted to be useful.
• Implemented a tool called Lumberjack
• Takes different logs and extract x-trace ids and
  other information and puts it into a database.
• Implemented a web application to visualize paths
  as they happen.

14
Demo Implementation Details

• Used Apache, Webrick and Mysql
• Apache mod_proxy_balancer load balancer
• 6 Webrick instances on 2 VMs webapp server
• Mysql database
• Stream all logs into Lumberjack
• Each event consists of x-trace id, timestamp, and
  some associated data (url, request time, etc.)
• Ajax web application is used to pull data out of
  the database and display the paths as they occur.

15
StarTrace Demo Setup
16
StarTrace Demo
17
Current Limitations

• Events that do not appear in logs are not visible
• Dtrace can help to overcome this.
• Too much data not scalable
• Streaming all logs over network is not ideal.
• Log file formatting is a problem
• Different log formats makes Lumberjack difficult
  to manage.
• Relational DB is not the ideal storage mechanism
  for StarTrace events