Bulk Data API

1
Bulk Data API

• Nick Simha
• Technical Alliance Manager

2
Agenda

• Bulk Data API basics
• Demo
• Best practices
• Resource list

3
What is the Bulk Data API?

• REST based, asynchronous API optimized for
  loading large sets of data.

4
Why is it useful?

• Enable high volume integration with Salesforce
  (volume)
• Enable integration that has to finish in a
  certain window of time (speed)
• Part of suite of features that enable our
  customers to store very large data volumes in
  Salesforce
• Batch Apex
• Skinny tables
• Divisions
• Custom Indexing
• Etc.

5
How does the Bulk Data API works
Loop until all records sent (e.g. 50 times for
500k rows)
Decoupled phases. One doesn't wait for the
other, each can run in parallel
Loop until all files processed
6
How can I call the Bulk Data API?

• Through Data Loader
• From any Web Services Client Java, C etc.
• From the command line!
• Support by our integration partners

7
Using the Bulk API from a client

• Create Job
• Create Batch (es) and add to Job
• Number of batches determined by the amount of
  data and the limits on batch size.
• Close Job
• Retrieve Batch Status
• Retrieve Batch Result

8
Sample Request

• POST /services/async/17.0/job HTTP/1.1
• User-Agent curl/7.19.6 (i386-pc-win32)
  libcurl/7.19.6 OpenSSL/09.8k zlib/1.2.3
• Host na6.salesforce.com
• Accept /
• X-SFDC-Session00D80000000MD0n!AQgAQI1EfPPEyWvwuaD
  _IRpvSlrwm7Kr00e
• Content-Type application/xml charsetUTF-8
• Content-Length 195
• lt?xml version"1.0" encoding"UTF-8"?gt
• ltjobInfo xmlns"http//wwwforce.com/2009/06/asynca
  pi/dataload"gt
• ltoperationgtinsertlt/operationgt
• ltobjectgtContactlt/objectgtltcontentTypegtCSVlt/contentT
  ypegtlt/jobInfogt

See Getting Started Chapter in the API guide.
Use curl trace-ascii ltfile_namegt to capture
messages.
9
Demo

• Load 100K Addresses of medical providers
• Cleanse the data

10
Bulk API - Some Additional Information

• Can Monitor Bulk Loads in Builder
• Monitoring -gt Bulk Data Load Jobs
• Doesnt handle attachments
• Governor limits 500 batches per 24 hour limit.
  10,000 records per batch. So theoretical limit of
  5M records per day.
• Caveat Batch size also needs to be less than
  10MB
• Batch limits can be increased by engineering
• Need to contact your SE

11
Best practices

• Combine Bulk API with Batch Apex to get optimal
  performance
• Faster than complex triggers
• Similar to the demo this is a generic pattern
  that you can use in many scenarios
• Stick with parallel processing unless there is a
  reason not to do so
• See FAQ for scenarios when you would serial
  processing
• Handling Very Large Data Volumes requires a
  comprehensive, holistic approach
• Bulk API is one part of the solution

12
Resource list

• Bulk API doc http//www.salesforce.com/us/develope
  r/docs/api_asynch/index.htm