Using Lego Mindstorms NXT in the Classroom

1
Using Lego Mindstorms NXT in the Classroom

• Gabriel J. Ferrer
• Hendrix College
• ferrer_at_hendrix.edu
• http//ozark.hendrix.edu/ferrer/

2
Outline

• NXT capabilities
• Software development options
• Introductory programming projects
• Advanced programming projects

3
Purchasing NXT Kits

• Two options (same price 250/kit)
• Standard commercial kit
• Lego Education kit
• http//www.lego.com/eng/education/mindstorms/
• Advantages of education kit
• Includes rechargeable battery (50 value)
• Plastic box superior to cardboard
• Extra touch sensor (2 total)
• Standard commercial kit
• Includes NXT-G visual language

4
NXT Brick Features

• 64K RAM, 256K Flash
• 32-bit ARM7 microcontroller
• 100 x 64 pixel LCD graphical display
• Sound channel with 8-bit resolution
• Bluetooth radio
• Stores multiple programs
• Programs selectable using buttons

5
Sensors and Motors

• Four sensor ports
• Sonar
• Sound
• Light
• Touch
• Three motor ports
• Each motor includes rotation counter

6
Touch Sensors

• Education kit includes two sensors
• Much more robust than old RCX touch sensors

7
Light Sensor

• Reports light intensity as percentage
• Two modes
• Active
• Passive
• Practical uses
• Identify intensity on paper
• Identify lit objects in dark room
• Detect shadows

8
Sound Sensor

• Analogous to light sensor
• Reports intensity
• Reputed to identify tones
• I havent experimented with this
• Practical uses
• Clap to signal robot

9
Ultrasonic (Sonar) Sensor

• Reports distances
• Range about 5 cm to 250 cm
• In practice
• Longer distances result in more missed pings
• Mostly reliable
• Occasionally gets stuck
• Moving to a new location helps in receiving a
  sonar ping

10
Motors

• Configured in terms of percentage of available
  power
• Built-in rotation sensors
• 360 counts/rotation

11
Software development options

• Onboard programs
• RobotC
• leJOS
• NXC/NBC
• Remote control
• iCommand
• NXT_Python

12
RobotC

• Commercially supported
• http//www.robotc.net/
• Not entirely free of bugs
• Poor static type checking
• Nice IDE
• Custom firmware
• Costly
• 50 single license
• 250/12 classroom computers

13
Example RobotC Program

• void forward()
• motormotorA 100
• motormotorB 100
• void spin()
• motormotorA 100
• motormotorB -100

14
Example RobotC Program

• task main()
• SensorTypeS4 sensorSONAR
• forward()
• while(true)
• if (SensorValueS4
• else forward()

15
leJOS

• Implementation of JVM for NXT
• Reasonably functional
• Threads
• Some data structures
• Garbage collection added (January 2008)
• Eclipse plug-in just released (March 2008)
• Custom firmware
• Freely available
• http//lejos.sourceforge.net/

16
Example leJOS Program

• sonar new UltrasonicSensor(SensorPort.S4)
• Motor.A.forward()
• Motor.B.forward()
• while (true)
• if (sonar.getDistance()
• Motor.A.forward()
• Motor.B.backward()
• else
• Motor.A.forward()
• Motor.B.forward()

17
Event-driven Control in leJOS

• The Behavior interface
• boolean takeControl()
• void action()
• void suppress()
• Arbitrator class
• Constructor gets an array of Behavior objects
• takeControl() checked for highest index first
• start() method begins event loop

18
Event-driven example

• class Go implements Behavior
• private Ultrasonic sonar
• new Ultrasonic(SensorPort.S4)
• public boolean takeControl()
• return sonar.getDistance() 25

19
Event-driven example

• public void action()
• Motor.A.forward()
• Motor.B.forward()
• public void suppress()
• Motor.A.stop()
• Motor.B.stop()

20
Event-driven example

• class Spin implements Behavior
• private Ultrasonic sonar
• new Ultrasonic(SensorPort.S4)
• public boolean takeControl()
• return sonar.getDistance()

21
Event-driven example

• public void action()
• Motor.A.forward()
• Motor.B.backward()
• public void suppress()
• Motor.A.stop()
• Motor.B.stop()

22
Event-driven example

• public class FindFreespace
• public static void main(String a)
• Behavior b new Behavior
• new Go(), new Stop()
• Arbitrator arb
• new Arbitrator(b)
• arb.start()

23
NXC/NBC

• NBC (NXT Byte Codes)
• Assembly-like language with libraries
• http//bricxcc.sourceforge.net/nbc/
• NXC (Not eXactly C)
• Built upon NBC
• Successor to NQC project for RCX
• Compatible with standard firmware
• http//mindstorms.lego.com/Support/Updates/

24
iCommand

• Java program runs on host computer
• Controls NXT via Bluetooth
• Same API as leJOS
• Originally developed as an interim project while
  leJOS NXT was under development
• http//lejos.sourceforge.net/
• Big problems with latency
• Each Bluetooth transmission 30 ms
• Sonar alone requires three transmissions
• Decent program 1-2 Hz

25
NXT_Python

• Remote control via Python
• http//home.comcast.net/dplau/nxt_python/
• Similar pros/cons with iCommand

26
Developing a Remote Control API

• Bluetooth library for Java
• http//code.google.com/p/bluecove/
• Opening a Bluetooth connection
• Typical address 00165302e575
• Bluetooth URL
• btspp//00165302e5751 authenticatefalseencrypt
  false

27
Opening the Connection

• import javax.microedition.io.
• import java.io.
• StreamConnection con (StreamConnection)
  Connector.open(btspp)
• InputStream is con.openInputStream()
• OutputStream os con.openOutputStream()

28
NXT Protocol

• Key files to read from iCommand
• NXTCommand.java
• NXTProtocol.java

29
An Interesting Possibility

• Programmable cell phones with cameras are
  available
• Camera-equipped cell phone could provide computer
  vision for the NXT

30
Introductory programming projects

• Developed for a zero-prerequisite course
• Most students are not CS majors
• 4 hours per week
• 2 meeting times
• 2 hours each
• Not much work outside of class
• Lab reports
• Essays

31
First Project (1)

• Introduce motors
• Drive with both motors forward for a fixed time
• Drive with one motor to turn
• Drive with opposing motors to spin
• Introduce subroutines
• Low-level motor commands get tiresome
• Simple tasks
• Program a path (using time delays) to drive
  through the doorway

32
First Project (2)

• Introduce the touch sensor
• if statements
• Must touch the sensor at exactly the right time
• while loops
• Sensor is constantly monitored
• Interesting problem
• Students try to put code in the loop body
• e.g. set the motor power on each iteration
• Causes confusion rather than harm

33
First Project (3)

• Combine infinite loops with conditionals
• Enables programming of alternating behaviors
• Front touch sensor hit go backward
• Back touch sensor hit go forward

34
Second Project (1)

• Physics of rotational motion
• Introduction of the rotation sensors
• Built into the motors
• Balance wheel power
• If left counts
• Increase left wheel power
• Race through obstacle course

35
Second Project (2)

• if (/ Write a condition to put here /)
  nxtDisplayTextLine(2, "Drifting left")
• else if (/ Write a condition to put here /)
  nxtDisplayTextLine(2, "Drifting right")
• else
• nxtDisplayTextLine(2, "Not drifting")

36
Third Project

• Pen-drawer
• First project with an effector
• Builds upon lessons from previous projects
• Limitations of rotation sensors
• Slippage problematic
• Most helpful with a limit switch
• Shapes (Square, Circle)
• Word (LEGO)
• Arguably excessive

37
Pen-Drawer Robot
38
Pen-Drawer Robot
39
Fourth Project (1)

• Finding objects
• Light sensor
• Find a line
• Sonar sensor
• Find an object
• Find freespace

40
Fourth Project (2)

• Begin with following a line edge
• Robot follows a circular track
• Always turns right when track lost
• Traversal is one-way
• Alternative strategy
• Robot scans both directions when track lost
• Each pair of scans increases in size

41
Fourth Project (3)

• Once scanning works, replace light sensor reading
  with sonar reading
• Scan when distance is short
• Finds freespace
• Scan when distance is long
• Follow a moving object

42
Light Sensor/Sonar Robot
43
Other Projects

• Theseus
• Store path (from line following) in an array
• Backtrack when array fills
• Robotic forklift
• Finds, retrieves, delivers an object
• Perimeter security robot
• Implemented using RCX
• 2 light sensors, 2 touch sensors
• Wall-following robot
• Build a rotating mount for the sonar

44
Robot Forklift
45
Gearing the motors
46
Advanced programming projects

• From a 300-level AI course
• Fuzzy logic
• Reinforcement learning

47
Fuzzy Logic

• Implement a fuzzy expert system for the robot to
  perform a task
• Students given code for using fuzzy logic to
  balance wheel encoder counts
• Students write fuzzy experts that
• Avoid an obstacle while wandering
• Maintain a fixed distance from an object

48
Fuzzy Rules for Balancing Rotation Counts

• Inference rules
• biasRight leftSlow
• biasLeft rightSlow
• biasNone leftFast
• biasNone rightFast
• Inference is trivial for this case
• Fuzzy membership/defuzzification is more
  interesting

49
Fuzzy Membership Functions

• Disparity leftCount - rightCount
• biasLeft is
• 1.0 up to -100
• Decreases linearly down to 0.0 at 0
• biasRight is the reverse
• biasNone is
• 0.0 up to -50
• 1.0 at 0
• falls to 0.0 at 50

50
Defuzzification

• Use representative values
• Slow 0
• Fast 100
• Left wheel
• (leftSlow repSlow leftFast repFast) /
  (leftSlow leftFast)
• Right wheel is symmetric
• Defuzzified values are motor power levels

51
Q-Learning

• Discrete sets of states and actions
• States form an N-dimensional array
• Unfolded into one dimension in practice
• Individual actions selected on each time step
• Q-values
• 2D array (indexed by state and action)
• Expected rewards for performing actions

52
Q-Learning Main Loop

• Select action
• Change motor speeds
• Inspect sensor values
• Calculate updated state
• Calculate reward
• Update Q values
• Set old state to be the updated state

53
Calculating the State (Motors)

• For each motor
• 100 power
• 93.75 power
• 87.5 power
• Six motor states

54
Calculating the State (Sensors)

• No disparity STRAIGHT
• Left/Right disparity
• 1-5 LEFT_1, RIGHT_1
• 6-12 LEFT_2, RIGHT_2
• 13 LEFT_3, RIGHT_3
• Seven total sensor states
• 63 states overall

55
Action Set for Balancing Rotation Counts

• MAINTAIN
• Both motors unchanged
• UP_LEFT, UP_RIGHT
• Accelerate motor by one motor state
• DOWN_LEFT, DOWN_RIGHT
• Decelerate motor by one motor state
• Five total actions

56
Action Selection

• Determine whether action is random
• Determined with probability epsilon
• If random
• Select uniformly from action set
• If not
• Visit each array entry for the current state
• Select action with maximum Q-value from current
  state

57
Q-Learning Main Loop

• Select action
• Change motor speeds
• Inspect sensor values
• Calculate updated state
• Calculate reward
• Update Q values
• Set old state to be the updated state

58
Calculating Reward

• No disparity highest value
• Reward decreases with increasing disparity

59
Updating Q-values

• QoldStateaction
• QoldStateaction
• learningRate
• (reward discount maxQ(currentState) -
  QoldStateaction)

60
Student Exercises

• Assess performance of wheel-balancer
• Experiment with different constants
• Learning rate
• Discount
• Epsilon
• Alternative reward function
• Based on change in disparity

61
Learning to Avoid Obstacles

• Robot equipped with sonar and touch sensor
• Hitting the touch sensor is penalized
• Most successful formulation
• Reward increases with speed
• Big penalty for touch sensor

62
Other classroom possibilities

• Operating systems
• Inspect, document, and modify firmware
• Programming languages
• Develop interpreters/compilers
• NBC an excellent target language
• Supplementary labs for CS1/CS2

63
Thanks for attending!

• Slides available on-line
• http//ozark.hendrix.edu/ferrer/presentations/
• Currently writing lab textbook
• Introductory and advanced exercises
• ferrer_at_hendrix.edu