An Introduction to Apache Avro

1
Apache Avro

• What is it ?
• Language API's
• JSON Example
• Schema Evolution
• Avro Use Example

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
2
Avro What is it ?

• Data serialization system
• Uses JSON based schemas
• Uses RPC calls to send data
• Schema's sent during data exchange
• Integrated with many languages
• Fast binary data format or encode with JSON

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
3
Avro Language API's

• Avro API's exist for the following languages
• Java
• C
• C
• C
• Python
• Ruby

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
4
Avro JSON Example

• This text shows an example JSON schema for a
  customer record
• "namespace" "customer.avro",
• "type" "record",
• "name" "customer",
• "fields"
• "name" "name", "type" "string",
• "name" "age", "type" "int",
• "name" "address", "type" "string",
• "name" "phone", "type" "string",
• "name" "email", "type" "string",

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
5
Avro Schema Evolution

• In a fast changing environment it is possible to
  
• Write data to file with a schema
• Change the schema
• Add extra fields
• Delete fields
• Rename fields
• And still read written file with changed schema

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
6
Avro Use Example

• // define schema for customer
• "namespace" "customer.avro",
• "type" "record",
• "name" "Customer",
• "fields"
• "name" "name", "type" "string",
• "name" "age", "type" "int",
• "name" "address", "type" "string",
• "name" "phone", "type" "string",
• "name" "email", "type" "string",
• // compile the schema
• java -jar /....../avro-tools-1.7.4.jar compile
  schema customer.avsc .
• // create a couple of customers

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
7
Contact Us

• Feel free to contact us at
• www.semtech-solutions.co.nz
• info_at_semtech-solutions.co.nz
• We offer IT project consultancy
• We are happy to hear about your problems
• You can just pay for those hours that you need
• To solve your problems