Chauffeur - A NXT Based Autonomous Driver with Vision

1 downloads 0 Views 851KB Size Report
Chauffeur - A NXT Based Autonomous Driver with Vision ... used to run the vision algorithm. .... zebra crossing area with semaphores, obstacles, a tunnel,.
Chauffeur - A NXT Based Autonomous Driver with Vision Luis Ferreira, Igor Amado, Marcos Liberal, Alexandre Araujo, and Luis Paulo Reis Faculdade de Engenharia da Universidade do Porto, Departamento de Inform´atica Laborat´orio de Inteligˆencia Artificial e Ciˆencia de Computadores da Universidade do Porto Rua Dr. Roberto Frias, s/n, Porto, Portugal {ei04044,ei04028,ei04050,ext07113,lpreis}@fe.up.pt

Abstract— Nowadays, autonomous driving research is almost ubiquitous. The existing projects range from infrastructure enhancements, assistance systems to fully autonomous vehicles. The most advanced projects in this last area are probably related to the DARPA Urban Challenge contest with the final goal of safe door-to-door transportation in arbitrary environments. This paper describes the chauffeur project, a mobile robot for simple autonomous driving applications, designed having in mind to participate on the Portuguese Robotics Open 2008 autonomous driving contest. Its mechanics and main hardware are based on the Lego Mindstorms NXT kit. Additionally, and because the NXT has limited computing resources, a PDA with built-in camera is used to run the vision algorithm. This architecture brings great flexibility, both at hardware and software level. Lego is known for its outstanding modularity, so it is easy to change the chassis. On the other side, the use of a PDA and of the NXT adds two different independent layers regarding the low level decisions based on the NXT sensors and the high-level decisions based on vision. Experiments performed show that the vision algorithm, used to follow the track, still needs some improvements because it fails every six in ten trials, causing the robot to go outside of the track limits. However, despite the obvious need for improvements, this project can be seen as a successful proof of concept.

I. I NTRODUCTION Autonomous driving projects are enjoying an increasing popularity as a research platform. Nowadays, almost any university research lab has a project somehow involving autonomous driving [1]. On the other side of the coin, there are already some commercially available products like cars that can park itself or with advanced cruise control features (e.g. keep distance from other cars) [2]. Despite all the existing technology, the reality is that it is still impossible to build a car or any other robot fully automatic and completely reliable. The first attempt to build such applications started in 1986, led by Ernst Dickmanns [3][4]. Nowadays it continues with the DARPA1 contests [10]. These contests are held on urban and non-urban environments and the environment is slightly simplified from the reality. During the contest most of the projects fail which means that even the best (the winner, in this case) would probably fail when running in the real world. The real world is such a complex and dynamic environment that most of the current applications are not safe there. The autonomous driving is a big challenge and with very promising future applications. The project described here is about a very basic robot, developed to run on almost static environments as described further in section 4. The remainder of the paper is organized as follows. The next section is about the state of the art. Section 3 describes

the robot and its architecture. Section 4 describes the main application of the robot: the Portuguese Robotics Open autonomous driving contest. The last section includes the paper conclusions and some questions for the future. II. R ELATED W ORK The use and integration of a PDA2 with robot platforms is not new. One of the best known PDA projects is the Palm Pilot Robot Kit [5], developed at CMU3 , which is used in colleges and some universities to introduce students to intelligent robotics. However, it lacks vision or any other advanced sensors. In the last years researchers tried to use existing builtin devices found in off-the-shelf PDAs to control existing applications [1], [7]. All of them where related to soccer. The first was about a two-wheeled robot controlled by the PDA to follow and shoot an orange ball [5]. The other one was about controlling an humanoid using the PDA to communicate with other humanoids and play soccer cooperatively [7]. Both used the camera to track the ball, the field and other elements within the field. The use of Lego Mindstorms kits [8] to drive autonomously using a computer or any mobile device with a camera, like a PDA, is almost new. There are a few projects that talk about using the Lego Mindstorms to drive around a path or road using only light sensors [12]. Although it is simple and cheap, it is also very limiting. The wimo project [16] uses a PDA (with camera) attached to a Lego Mindstorms robot. The PDA software streams the camera images via bluetooth to a remote computer operated manually which sends back the commands to the PDA/robot. There are other project very similar to this one [6] and none of them is completely autonomous. They depend somehow from external control mechanisms. III. AUTONOMOUS D RIVING A PPLICATION The platform and the software agent where developed to participate in mobile robot contests. Despite being oriented to autonomous driving applications, it can be used in other kind of contests (e.g. Micro-Mouse, Robotic Fire Fighting). It is small and provides some flexibility, due to the use of a configurable chassis. So, it can be easily adapted to the specific rules of other contests that demand for a flexible small robot. In this specific area there are a few contests promoted by science associations and universities. Many universities run their local competitions as part of their educational activities. Others use standard platforms like the Lego 2 Personal

1 Defense

Advanced Research Projects Agency

3 Carnegie

Digital Assistant Mellon University

Mindstorms [8]. There’s also those organized by official entities, like the DARPA Urban Challenge [10] which is sponsored by the DARPA agency in the USA4 . In this contest, autonomous cars must complete an urban route with traffic and road signals. The 1st prize is $US 2 million but involves highly complex robots and architectures.

can be already occupied by a white box. The robot must determine which slot is empty and then park itself.

A. Portuguese Robotics Open Autonomous Driving Competition The Portuguese Robotics Open Autonomous Driving Competition is held in Portugal since 2001 [12]. The main challenges are: • autonomous navigation; • trajectory tracking; • obstacle avoidance; • task planning; • object recognition; • sensor integration; • learning and adaptation. 1) Contest Description: The challenge comprises a path with an 8-shaped configuration simulated road (Fig. 2). The track is a simplified version of the real world roads and pathways. However, its configuration and elements try to reflect as much as possible real situations. It has a zebra crossing area with semaphores, obstacles, a tunnel, road maintenance area and a parking area. Zebra Crossing and Traffic Light Panel. The path defines a two-way street, about 1.5 meters wide. It has a zebra crossing with a traffic light panel 2. The robots must obey the indications given by the signals in the traffic light panel1. There are five possible signs: stop; turn left; go straight; end of stop; park.

Fig. 2.

Autonomous Driving area [12]

The competition is developed in 3 phases, each one with increasing complexity. First Phase. The first phase demands simple motion along the path and the accomplishment of 2 complete laps, plus stopping at the zebra crossing. Second Phase. The robots must do the same as in phase 1. However, they must detect obstacles along the track and correct their motion (e.g. deviation). They must also obey to the signs given by the traffic light panel. Third Phase. This is the ultimate challenge. The path has the same elements of the previous phases plus a tunnel and a road maintenance area.

Fig. 1.

Signs available at the traffic light panel [12]

Obstacles. The obstacles are white boxes. They are placed on the road occupying one of the lanes, forcing the robot to deviate. The locations of the obstacles are unknown beforehand. Tunnel. The tunnel cover one part of the track. This poses a new challenge because affects light conditions and road borders. Road Maintenance Area. The maintenance area simulates a real situation where a portion of the road is replaced by cones and stripes. The original path is replaced by a guided corridor and the robot must comply with the newly shaped route. Parking Area. The parking area is a marked rectangle outside of the track. It is divided in two squares, each one consisting of a single park slot. One of the slots 4 United

States of America

Score. The final score is based on the total time achieved by the robot to complete the laps during the 3 phases. There are some penalties to the total time (e.g. collisions). The winner is the one with the minimum score. IV. C HAUFFEUR A RCHITECTURE The chauffeur employs several technologies. It has two different deliberative layers and an additional low-level layer (Fig. 3). The PDA layer is the responsible of the robot vision. It sends information to the layer bellow. This layer is composed by two NXT bricks [8]. The brick is like a ”brain”that controls the motors and sensors. Both bricks interchange information between them and the PDA. In order to accomplish the several tasks the robot employes several behaviors. These behaviors range from a simple avoid obstacle to a more complex detect zebra crossing. The behaviors are grouped into layers of competence. The layers reflect a hierarchy of intelligence. Lower layers encapsulate basic survival functions while higher levels create more goal-directed actions. The control of each layer is based in an subsumption architecture. The behaviors are triggered on a stimulus-response way, mainly given by the

A. Kinematics The robot uses a two-wheel differential drive configuration with two additional free wheels in the rear (Fig.4). The rear wheels have the same motion constraints as free castor wheels, so the same kinematics theory used in differential mobile robots with one or two omnidirectional free wheels can be applied in this robot. B. Hardware Setup The hardware used in the chauffeur can be divided in the following categories: PDA and NXT Brick; Actuators and Motors; Sensors; Communication Devices. 1) PDA: The PDA is an Imate Jasjar (Fig. 5). It has an Intel Bulverde 520MHz CPU, 64MB of SDRAM and 128MB of ROM memory. It has a TFT touchscreen with 640x480 pixels of resolution and weights 285g. In terms of connectivity it is wifi and bluetooth enabled [18]. The built-in CCD5 camera has a maximum resolution of 1.3 Megapixels. Fig. 3.

Overview of the chauffeur architecture

sensors information. They don’t operate concurrently and the lower-level behaviors always override the higher-level ones. Fig. 5.

V. M ECHANICS AND H ARDWARE The robot chassis is entirely built using the Lego Mindstorms pieces, provided in each kit. The kit includes both standard Lego and Lego Technic pieces [8] (Fig. 4). The traction is provided by 4 wheels, also from the Lego kit. The robot is articulated in the middle, that way it can steer by folding to the left or to the right.

Imate jasjar

2) NXT Brick: The NXT Brick has an ARM based 32 bit CPU. It has 64KB of Ram and 256KB of flash memory [8]. It features communication via bluetooth and 7 ports: 3 to communicate with actuators and 4 to communicate with sensors. The chauffeur needs two bricks, as described in the architecture IV. 3) Actuators: The robot uses 6 Lego Mindstorms NXT integrated servo-motors. This motors are used to move and steer the robot. They are also used to control the tilt and rotation angle of the camera. 4) Sensors: Besides the camera, the robot uses two other different kind of sensors. The light sensors are used to provide additional control over the road path following and avoid crossing road limits. This type of sensor is also used to measure ambient light and adjust the vision’s binary algorithm threshold. The distance sensors are used to avoid collisions between the robot and obstacles found along the road. 5) Communication: Lego NXT bricks use an integrated communication protocol, based on the I2C bus architecture, to control motors and sensors. The bluetooth technology is used to exchange vision, sensor and control information between the bricks and the PDA. The delay caused by bluetooth transmission is 80 miliseconds, which makes the robot a viable implementation [8].

Fig. 4.

Three views of the robot 5 Charge-coupled

device

TABLE I C OMPARISON BETWEEN THE L EGO M INDSTORMS NXT PROGRAMMING LANGUAGES [9] Name

Language

Bluetooth

F.P

Mt

Performance

NXT-G NXC Robotc

Graphical C C

Half-Duplex Half-Duplex Full Duplex

Yes Yes Yes

No No Yes

1.00x 5.62x 135x

TABLE II M ESSAGES EXCHANGED VIA BLUETOOTH Message

Syntax

From

To

Start Relative position Zebra crossing Green signal light

”start” ”p angle” ”z” ”go”

NXT #2 PDA PDA PDA

PDA and NXT #1 NXT #1 NXT #1 and #2 NXT #1 and #2

VI. S OFTWARE The projects are always time limited. So, the best approach is using existing technologies and platforms like the lego and PDA. This way, the robot development is faster and easier because there are no electronic concerns. All that is needed is focus in the software development. The software is the most important part of the robot. It is responsible to record the sensor readings, control de motors and compute the best decisions at each moment, given a specific situation. The robot has 2 software agents: one running on the NXT brick and the other processing vision on the PDA. The NXT platform has a few programming alternatives. It is possible to write programs in C, Java, Pseudolanguages and with graphical languages. The table I shows a comparison between three different programming alternatives. The NXT software is implemented using the RobotC [9] compiler/IDE, which is the most suited for mobile robots applications as seen in table I. The PDA’s camera vision processing program is developed using the Windows Mobile 5 SDK with the C++ language. The frames from the camera are captured and processed using Windows directshow [17]. The communication between the PDA and the NXT is performed using bluetooth and a common protocol.

C. Vision Processing The vision is an important feature of the chauffeur agent. It is used to determine the position of the robot within the track, to detect obstacles and graphical signs on a traffic light panel. 1) Track Lanes: The agent bases its actions on its current believed location obtained using odometry (section VI-B) and on the 2D map. The 2D map or the World State of the agent is build using the vision information. In fact, the image received using the built-in camera is a 3D perspective of the world. Doing a 3D to 2D transformation would be CPU intensive which is a limitation on the PDA. In order to use the 3D image it must be converted to black and white, using a threshold value based on the current ambient luminosity. The next step is segmentation in which contiguous pixels with the same color are grouped in cells and regions considered to be too small are fused with their nearest neighbor (Fig. 6 (b) ). The camera’s focal length is not small enough to capture the entire track and the two lanes. Instead, the camera is pointed to one side of the track. That way, it is possible to calculate the relative position of the robot to one of the lanes and use it to steer it. The calculation is based on the heights h1 and h2 and hm (Fig. 6 (c) ). This heights are the number of pixels that exist between the white strip, that represents the track lane, and the image limits (top limits for h2 and hm and botton limits for h1). This three heights are used to calculate a value called relation (equation 1). relation =

(a) step one

A. Interaction Protocols The interaction between the NXT bricks and the PDA is done using a specific protocol. This protocol defines a set of messages that both understand and use to communicate. Table II shows the type of messages exchanged and their basic syntax. B. Odometry The odometry is done using the integrated servo motors and additional sensors (e.g. compass, accelerometer) to provide inertial navigation. The integrated servo motors of the NXT count 360 clicks per revolution (or 1 degree per click), while not tying up sensor ports. Also, the improved tire design allows more accurate turning[8]. In order to calculate the position of the robot it is used dead-reckoning technics based on the information provided by the servo motors encoders.

h1 − hm h2 − hm

(1)

(b) step two

(c) step three Fig. 6.

Determining track lane position with vision processing

The individual power of each of the two motors is calculated using fuzzy logic and the relation value. The Figure 7 shows that the power applied to each wheel is a function of the relation. There are three limit situations in which the robot must turn left, turn right or go in front. The Figure 8 shows this three possible situations. Using the three heights in each situation the relation is calculated (table III) and then, using the fuzzy logic function (Fig. 7), the power for each

(a) step one

Fig. 7. vision

(b) step two

Fuzzy Logic applied to calculate individual motor power using

TABLE III I MAGE PROCESSING VALUES AND CALCULATIONS Figure

h1

hm

h2

relation

left power

right power

8 (a) 8 (b) 8 (c)

61 73 68

56 42 19

45 76 120

-0.45 0.91 0.48

57 54 1

3 60 59

(c) step three Fig. 9.

Detecting a zebra crossing area with vision processing TABLE IV R ESULTS OBTAINED FROM SEVERAL TRIALS

motor is calculated. In situation from the Figure 8 (a) the robot is turned to left towards the white lane. Using the above method the power applied to the left motor is 57 while the power applied to the right motor is 3. This way it will steer fast in order to correct its position. The same principle applies to the other two situations. The goal is always to move in front, correcting any left or right deviations.

Test description

No. Trials

No. Successful Trials

Avoid lateral obstacles Deviate from track lanes Detect zebra crossing Evaluate traffic sign Do close curve Do wide open curve

10 10 5 5 10 10

10 (Using distance sensors) 8 (Using light sensors) 5 0 (Color calibration problems) 7 9

VII. R ESULTS

(a) turned to the left

(b) aligned

(c) turned to the right Fig. 8.

Three different possible positions of the robot within the track

2) Zebra Crossing: The zebra crossing detection algorithm is very simple. This algorithm is executed concurrently with the previous one. So, it rasters the image and analyses if there are two consecutive pixel lines with alternating white and black (Fig. 9). Traffic Light: The interpretation of the traffic light panel needs both color and shape analysis. However, the current version is very simplified and supports only image color analysis. It distinguishes from the colors Red and Green. This is done using the RGB components for Red and Green. The two components are compared to check which one has a bigger value.

The robot implementation was tested under several situations. The main goal was evaluating its performance under distinct conditions and different scenarios. Initially, the first experience was concerned with testing the robot track lane following algorithm using vision information provided by the PDA camera. During this phase the robot was able to perform three complete tracks but only with some human help, because it went out of the track two times. After this, the robot was submitted to other experiments, each one with a specific objective like ”doing a sharp bend/curve”, ”detecting the zebra crossing”or ”avoiding an obstacle”(see table IV). The results obtained showed that the robot is able to avoid lateral obstacles and keep itself inside the track boundaries by correctly detecting the track lanes. It is also capable of successfully detecting the zebra crossing and of following completely the track although with some problems in performing sharp bends. Difficulties arise with detecting the traffic lights signals due to some problems with calibration under variable lightning conditions. VIII. F UTURE W ORK A. Mechanics and Hardware The robot chassis must be improved to gain more stability and better turning angle capabilities. Due to the narrow field of view available on this camera, some adjustments related to the PDA holding mechanism are needed like using a mirror instead of moving the PDA and use a big angular lens/convex mirror to increase the image field of view. The speed of the robot can be improved by using a combination of bigger wheels and gears. However, an

increased speed will lead to more problems to effectively control the robot inside the track boundaries. In order to improve the current odometry system, an INU6 could be used together with the compass and accelerometer sensors available for the NXT kits. B. Software and Algorithms The vision algorithm is not flexible enough. A further analysis on the current image could be made to determine the shape of the lane. Also, a flexible color calibration software is needed to avoid problems with variable lightening conditions, like the failure of the traffic sign recognition algorithm. The robot is, at the moment, almost reactive. A motion planning system could be used to avoid obstacles on the track and to follow the track, based on the curvature angle extracted from the image. IX. C ONCLUSIONS The architecture described in this paper has two main advantages over other robot platforms. The control given by the PDA is completely independent from the underlying infrastructure. This means that it is possible to use a Personal Computer instead of a PDA to run the vision algorithm. Also, it is very modular (kudos to Lego) which gives an infinite number of different chassis ( e.g. sensors, motors, design) each one adapted to a specific application. The implemented algorithms although quite simple are not flexible enough and are still prone to failure. Also, some image compression is needed to increase the frame rate performance (e.g. Run Length Encoding). The use of a PDA and a NXT altogether to build an autonomous robot is a new approach for building small autonomous driving robots. The robot uses new technologies and concepts, some of them never proved together in this setup before. So, it was a challenge to build the complete robot and make everything work on this setup. This is the first of its kind and should be seen as a Proof of Concept, showing that the main architecture proposed works and is flexible enough to be easily improved. R EFERENCES [1] K. Doty and S. Jantz, PDA Based Real-Time Vision for a Small Autonomous Mobile Robot, 2002 Florida Conference on Recent Advances in Robotics, May 2002, Florida International University, USA. [2] B. Seppelt and J. Lee, Making adaptive cruise control (ACC) limits visible, International Journal of Human-Computer Studies, Vol. 65, No. 3, pp. 192-205, March 2007, Duluth, MN, USA. [3] E. Dickmanns., and A. Zapp, A Curvature-Based Scheme for Improving Road Vehicle Guidance by Computer Vision in Procedings of SPIE, Vol. 727, pp. 161-168, October 1986, University of Munich, Germany. [4] E. Dickmanns, Dynamic computer vision for mobile robot control, in 19th International Symposium on Industrial Robots, pp. 314-327, November 1988, Sydney, Australia. [5] K. Mukhar, D. Johnson, and D. Johnson, The Ultimate Palm Robot, 1st edition, McGraw-Hill, USA, 2003. [6] D. Stevenson and J. Schwarzmeier, Building an Autonomous Vehicle by Integrating Lego Mindstorms and a Web Cam, ACM SIGCSE Bulletin, Vol. 39, No. 1, pp. 165-169, March 2007, New York, NY, USA. [7] S. Behnke, J. Muller, and M. Schreiber, Using Handheld Computers to Control Humanoid Robots in Proceedings of 1st International Conference on Dextrous Autonomous Robots and Humanoids (DARH05), May 2005, Yverdon-les-Bains, Switzerland. 6 Inertial

Navigation Unit

[8] F. Klassner and M. McNally, Demonstrating the Capabilities of Mindstorms NXT for the AI Curriculum in AAAI 2007 Spring Symposium Series Report, Stanford, CA. March 26-28, 2007. [9] C. Tu and P. Das, NXT Robots and their Applications in Machine Learning, SPAWAR, March 2007, University of California, San Diego, California, USA [10] S. Thrun, Winning the DARPA Grand Challenge: A Robot Race through the Mojave Desert in Automated Software Engineering, 2006. ASE ’06. 21st IEEE/ACM International Conference, pp. 1111, September 2006, Tokyo, Japan. [11] S. Thrun, Stanley: The Robot that Won the DARPA Grand Challenge in Journal of Robotic Systems, Vol. 23, No. 9, pp. 661-692, September 2006, Chichester, UK. [12] P. Afonso, J. Azevedo, C. Cardeira, B. Cunha, P. Lima, and V. Santos, Challenges and Solutions in an Autonomous Driving Mobile Robot Competition in Procedings of Controlo 2006 - 7th Portuguese Conference in Automatic Control, 2006, Lisbon, Portugal. [13] R. Murphy: Introduction to AI Robotics. 1st edition, MIT Press, Cambridge, MA, USA, 2000. [14] D. Mateus, G. Avi˜na and M. Devy, Robot Visual Navigation in Semistructured Outdoor Environments, in Robotics and Automation, 2005. ICRA 2005. Proceedings of the 2005 IEEE International Conference, pp. 4691- 4696, April 2005, Toulouse, France. [15] F. Martin, The Art of LEGO Design in The Robotics Practitioner: The Journal for Robot Builders, Vol. 1, No. 2, March 1995, Kirkwood, MO, USA. [16] B. Cross, Wimo Bot and the NXT. Available at: http://www.wimobot.com/AssemblyNxt.aspx (2007, January 25) [17] Microsoft Corporation, Directshow. Available at: http://msdn2.microsoft.com/en-us/library/ms783323.aspx (2007, February 7). [18] i-mateTM , i-mate JASJAR. Available at: http://www.clubimate.com/t-details jasjar.aspx (2007, January 22).