ISATP 2001, The 4th International Symposium on Assembly and Task Planning
A Roadmap for Autonomous Robotic Assembly H. Bruyninckx, T. Lefebvre, L. Mihaylova, E. Staffetti, J. De Schutter K.U.Leuven, Belgium
[email protected] Abstract This paper describes the components (“software agents”) that are needed in an autonomous robotic assembly system. Such a system is called “autonomous” if it is able to cope with all uncertainties in the real-world execution (control and sensing) of an assembly task that was planned off-line, and with the (re)planning itself. For each component, the paper answers the following three questions: What is needed in this component? What has already been achieved in (industrial and research) practice? And what are realistic expectations for the short to medium term future? The goal is to facilitate discussions about the future of robotic assembly, by providing a structured overview.
1
Introduction
Assembly is one of the largest applications of robotics world-wide, whose academic research is roughly divided into: 1. Fixture and sequence planning: How to fix the objects to rigid supports, in order to be able to reach all required parts and to achieve the required accuracy? What sequences of actions lead to the desired assembled state? 2. Compliant motion planning: Given a fixture, and, within that context, given a sequence plan, how to make a robot execute this plan reliably? Whenever the uncertainties involved in the task are “large” the execution cannot be performed by a position-controlled device, and force control becomes necessary; such force-controlled execution is often called compliant motion. The former domain of assembly research investigates the (dis)assemblability of complex composite objects, such as On leave from the Bulgarian Academy of Sciences.
J. Xiao University of North Carolina, Charlotte
[email protected]
machine engines, with an emphasis on finding collisionfree paths, not on executing the (dis)assembly by a robot system. This paper, on the other hand, focusses on the compliant motion kind of assembly planning. Current compliant motion controllers invariably operate in very structured environments and without active force control: the relative positions and the geometries of the relevant parts of all objects involved in the assembly (including robots and fixtures) are known and fixed. Industrial robot controllers do not yet offer force control features, for various reasons:
Compliant motion is inherently more difficult than the more popular robotics research topics, such as mobile robots: controlled contact is more difficult to achieve than controlled obstacle avoidance. Compliant motion requires the close integration of different software components that implement control, intelligent sensing and planning skills. Industrial robot controllers are hierarchically structured. This suffices for motion control in known environments, but not for the intelligent control needed in advanced assembly: the robot interacts with its environment and uses the inputs from exteroceptive sensors (e.g., force sensor) to adapt its ongoing action, and even to re-plan its task autonomously. Several decades of experience in sensor-based assembly control have led to the three-level hierarchy of Fig. 1 [1]: 1. The lowest (“reactive”) level deals with the dynamics of the robot and its interaction with the environment [2, 3, 4, 5]. Most assembly systems still rely on passive force “control,” using devices such as the RCC (“Remote Center Compliance” [6]) because of their much higher speed and robustness. 2. The medium (“sequencing”) level comprises (i) estimation and monitoring (control/sensing), and (ii) automatic force-controlled compliant motion scheduling. It needs (physical or learned) models of the
2
Reasoning, error recovering High level Medium level
2.1 Overview Adaptation, monitoring
Low level Task specification
Modelling
Control
Robot, environment
Figure 1: The classical three-level hierarchical control pattern for intelligent robot systems.
robot-environment geometry and interaction dynamics: the task is defined in these models, and hence relies on knowing the model parameters. So, in order to compensate for the uncertainty and variation in these parameters, the controller relies on on-line estimates from the estimation component (Sect. 4); at the same time, that component monitors the estimated parameter values, to detect when to switch to the next subtask [7, 8]. 3. The highest (“deliberative”) level is responsible for decision making, i.e., (re)planning and on-line error recovery. Making intelligent decisions depends heavily on being able to interpret all sensor data (raw data, as well as processed data from the two lower levels) within the context of the task (geometry, dynamics, goals, etc.). Only the lowest level in this hierarchy has been implemented industrially; academic research has realized some “proof-of-concept” implementations in both lower levels, but these lack sufficient robustness for industrial application. The medium level becomes necessary in assembly systems whose uncertainties are too large to be compensated by an RCC. The highest level is still mostly unexplored. The emphasis of this paper is not on novel theoretical or practical results, but on putting structure on the autonomous assembly field: How should an autonomous assembly system be decomposed, and what are the responsibilities of these components? These components are autonomous but cooperating “software agents” (or “holons”), that take care of, respectively, modelling, planning, estimation/monitoring, control, and task coordination. The advantages of components are that they can run on a distributed computing environment, and be developed and maintained independently, by specialists in the field, if their responsibilities and interfaces are well specified. This paper’s goal is to be a first step in this direction in the field of compliant motion for assembly.
What is needed? Assembly in the execution-oriented context of this paper is all about controlled motion of objects in contact. The goal is to achieve a desired contact configuration between the manipulated object held by the robot and the environment. This happens through a controlled sequence of contact formations (C F ) (e.g. vertexface, edge-edge, : : : ). The modelling component provides the following information to the other components: (i) a graph of all possible C F s, (ii) an interface to visualize the graph and any C F in the graph, (iii) a basis of the instantaneous motion degrees of freedom of the manipulated object in a given C F , and (iv) grasping or fixturing configurations. What has already been achieved? There are methods for generating a graph of C F s between polyhedral objects automatically (Sect.3) and for visualizing the CFs. There also exist geometric analyses of the degrees of freedom for C F s consisting of two simple contacts [9]. For a C F consisting of one simple vertex-face, edgeedge, edge-face, or face-face contact, both low-level (model-free) and medium-level (model-based) descriptions of the instantaneous motion degrees of freedom exist [10]. Any instantaneous motion of a rigid body can be described as a twist on a screw, t = (! T ; vT )T , and any set of forces and moments that act on a rigid body can be described as a wrench on a screw, w = (mT ; f T )T , [11]. Any (frictionless) C F is then characterized by the reciprocity condition T
t w = 0;
(1)
any twist allowed by the C F generates no work against any wrench generated in the C F . What is the future? Compatibility with CAD systems (file formats and visualization); more realistic simulation; and interfacing with robot controllers. Vendorspecific proprietary implementations hinder the acceptance of computer-assisted assembly planning and control.
3
Planning
3.1 Overview What is needed? Given the graph of C F s, the planning component finds sequences of “neighbouring” C F s that connect a given “initial” C F to a given “goal” C F . It
should be able to adapt plans on-line, in order to deal with changes and uncertainties based on the information from the on-line components (task coordination, control, and estimation. What has already been achieved? Methods for planning compliant motions off-line for polyhedral objects (see next Section). A lot of research is also done on planning motions for specialized part-mating tasks, but without conection to robot execution. What is the future? The most needed development is the two-way link between planning and on-line control and estimation.
3.2
CF graphs and CF -compliant planning
Generally, an assembly task involves known manipulated object and environment, both assumed to be rigid and modeled as polyhedra. Then, the contact formation can be topologically described by a set of principal contacts (PC), [12]. Given two parts to be assembled, to manually enumerate C F s between them and relations among C F s is tedious even for tasks of simple geometry [13]. While it is relatively straightforward to generate such graphs automatically for convex polyhedra [14] it is far from trivial for non-convex objects. [12, 15] recently introduced a promising approach towards automatic generation of C F graphs for arbitrary polyhedral objects. The approach is characterized by directly exploiting both topological and geometrical knowledge of contacts in the physical space of objects and by dividing the problem into simpler subproblems of generating and merging special subgraphs. In a generated graph, each node denotes a contact state characterized by the topological C F and a representative contact configuration, and each arc connects two adjacent states. Fig. 2 shows an example of generated C F graph between a peg and a corner and displays a number of C F s in the graph. Such a graph simplifies planning to a two-level problem: High-level: graph search for an optimal sequence of contact states (based on desired optimization criteria). Low-level: path planning within each state, i.e., within the set of contact configurations constrained by the same C F , which we call C F -compliant path planning. The output is a concatenation of path segments compliant to the C F s in the sequence respectively. To plan a C F -compliant path also poses special challenges if the configuration manifold (i.e., C-surface patch)
Figure 2: A graph of C F s between a peg and a corner.
of a C F is hard to describe analytically. A general approach is to combine exploitation of contact constraints with randomized planning. This entails two key techniques: one is an efficient strategy [9] to automatically generate random samples of configurations satisfying a given C F , or C F -compliant configurations, and the other is a method to produce C F -compliant interpolations between two C F -compliant configurations. These techniques enable the application of a randomized planner, such as one based on the probabilistic road maps approach [16], to generate C F -compliant paths free of all collisions except the required contact formation [17].
4
Estimation and monitoring
4.1 Overview What is needed? Estimation of the parameters in a given , and matching of the sensor data to a set of possible models and recognize the C F that fits best. CF
What has already been achieved? The most successful and advanced approaches rely on tools from Bayesian probability theory, e.g., the Kalman Filter [7], or Hidden Markov Models [8]. Bayesian probability is particularly suited for these tasks, because it is very straightforward to incorporate a priori knowledge (such as models) into the sensor processing, even for one-of tasks.
Figure 3: Robot placing a cube in a corner.
What is the future? The medium term achievement that would most significantly advance the industrial practice in force-controlled assembly is the creation of a database of primitive but robust assembly tasks, i.e., to list which control/estimation/monitoring actions are available for the C F s between polyhedral objects, and to describe how they are to be implemented. This goal implies the construction of a general “language” in which these elementary assembly actions can be formally described, with integrated control and sensing specifications. Major challenges to academic assembly research are: (i) to cope with the non-linearities that occur everywhere in the estimation procedures, (ii) real-time algorithms; and (iii) robust multi-sensor fusion, i.e., estimate uncertain parameters based on data coming from different sensors (e.g., camera and force sensor).
4.2 Bayesian probability tools The execution of an assembly task is subject to uncertainties (topology and geometry of the C F ) due to measurement noise, inaccuracies in the models, and incomplete observation of the parameters. In addition, the system and measurement models are usually nonlinear. These problems require advanced estimation techniques, of which this section describes (i) the estimation of geometric parameters in the C F model, (ii) the monitoring of a transition between two C F s, and (iii) the recognition of which C F has been reached after the transition.
Estimation of contact formation geometry Twist and wrench measurements give information about the position of the objects in contact. Each C F has a measurement model, linking the measurements to the estimator “state” (i.e., the uncertain model parameters). This measurement model is nothing else but the reciprocity constraint of Eq. (1), applied to the current C F [7]. Kalman filters have been successfully used to estimate the contact formation state in presence of uncertainties, [7]. In order to cope with non-linearities, several adaptations of the original Kalman filter framework exist, such as the (Iterated) Extended Kalman Filter [18], and so-called “Monte Carlo” methods, that approximate the probability density functions (used in Bayesian probability theory to model uncertainty) by sets of discrete samples The Kalman Filter algorithms can be applied when: (i) all functions are smooth, (ii) the derivative matrices can be calculated, and (iii) the uncertainties can be approximated by Gaussian distributions. The filters’ performance and stability depend strongly on the initial conditions. The Monte Carlo algorithms avoid the requirement for Gaussian distribution of the uncertainties and the need of linearization, but they are computationally intensive. In general, the measurements do not observe the full state, and the observation depends on the executed motion. Special active sensing motions should be superimposed on the nominal motion (or the nominal motion trajectory can be interrupted to execute specific sensing actions), in order to have a better observability of the parameters. For example, the robot can estimate the position of the object in its gripper, by making it touch a known “calibration” object. Monitoring of contact transitions Figure 3 depicts a typical example of a force-controlled assembly task. Each C F requires a different control action, different estimation models, etc. Hence, fast detection of transitions between C F s is important. In a Kalman Filter, consistency of the measurements with respect to the model is checked by a Summed Normalized Innovation Squared (SNIS) test [18]. In this way contact transitions can be observed: when gaining or loosing a contact, the wrench or twist measurements will be statistically inconsistent with the model of the current C F . A Hidden Markov Model (HMM) is another approach to detect transitions: to each C F (i.e., a “state” of the HMM) corresponds a range of measurements that are compatible with this C F , and neighbouring C F s can be reached with different probabilities, depending on the executed action. HMMs allow fast assemblies for repetitive tasks, because then the transition probabilities can be trained. Kalman Filters, however, also work for one-off tasks, given sufficiently rich a priori models.
Recognition of contact formations Even when a detailed plan of the assembly is available, the planned sequence of C F s does not always correspond to the real sequence. Hence, the estimation component should be able to order C F s according to their probability. Two Bayesian “matching” methods are available (but have not been tested on real assembly systems yet!): the Interacting Multiple Model (IMM) filter [18], and Markov Localization. The IMM runs several Kalman Filters in parallel (each corresponding to one candidate C F ), combines their individual estimates, and selects the C F that fits best. Markov Localization does not use a “state vector” to collect all information from the past, but, instead, it matches all measurements with predictions from (an adaptively chosen subset of) all possible configurations.
5
6
Task coordination
6.1 Overview What is needed? The components discussed in previous Sections are “servers” that perform services to the user; or rather, to a task coordination “master” that interfaces to the user. The task coordination component takes care of: (i) translation of the high-level user input (“Assemble this product!”) into inputs to the server components; (ii) taking decisions on when to activate which component or when to accept autonomously generated “advice” from them; (iii) the consistency of the models used by all components in the system; and (iv) combining feedback from all servers into a consistent, user-oriented view on the system. The adaptability of the task coordination to the context of the task is very important: only in the correct context, “data” becomes “information.”
Control
5.1 Overview What is needed? The task coordination component has decided which action to execute, and asks the control component to take care of performing this action stably and efficiently. The controller, in general, relies on the accurate knowledge of the parameters in the model corresponding to the current C F . If these parameters are not well known, the control component has to use the services of the estimation component.
What has already been achieved? The lowest, servo level of motion and force control has matured quite well during the last two decades, and has given rise to two basic paradigms: hybrid control and impedance control [4, 5]. (However, none of them belongs to the standard features of industrial controllers yet!) On this lowest level, the controller itself can cope with limited uncertainties, passively (RCC) and actively [2].
What is the future? Really “open” commercial robot controllers are needed. Currently, all robot control manufacturers claim their product is open (in the sense that it has a documented Application Programming Interface (API)), but in reality they do not allow to interface exterior sensors (e.g., a force sensor) to the low-level servo loop, or only with too low sample rates and too long dead-times.
What has already been achieved? Industrial robot control systems have become very reliable and programmable. However, inflexibility is the price paid for this robustness, and hence, only the lowest level of Fig. 1 has been implemented. Academic research on more flexible software components, as needed in the autonomous assembly system sketched in this paper, has not yet delivered a useable, unambiguous paradigm. At least, as far as the coordination and integration of the components are concerned, because object-oriented design and component “brokers” (`a la CORBA or DCOM) form a mature infrastructure for designing, building and linking components. What is the future? The key to using the abovementioned infrastructure in order to build a really autonomous system is the well-known “frame problem”: intelligent behavior boils down to reliably determine the context in which the available data must be interpreted. Some engineering disciplines (e.g., speech recognition, CADCAM design) have shown that this can be achieved in practice, by making the user responsible for restricting the possible contexts to a set of well-understood problems. Such “mini” autonomous assembly systems will probably be developed in academic labs in the near future, where the context is limited to one single robot system and one single task, but with high uncertainty on the geometry and positions of objects. The industrial viability of an autonomous assembly system depends on how easy it is to use it. A key factor in this success is the availability of a database of elementary autonomous tasks, from which more complex tasks can be assembled. Each task in this database (i) is parameterized (such that it can be adapted to the dimensions and
physics of the current objects), (ii) has been fine-tuned and made robust in a given context, and (iii) has well-defined interface and pre- and post-conditions that make it easy for components to interact.
[5] N. Hogan, “Impedance control: An approach to manipulation. Parts I-III,” J. Dynamic Syst., Measurement and Control, vol. 107, pp. 1–24, 1985.
7
[7] J. De Schutter, H. Bruyninckx, S. Dutr´e, J. De Geeter, J. Katupitiya, S. Demey, and T. Lefebvre, “Estimating firstorder geometric parameters and monitoring contact transitions during force-controlled compliant motions,” Int. J. Robotics Research, vol. 18, no. 12, pp. 1161–1184, 1999.
Conclusions
This paper describes the goals, the current state-of-theart, and realistic expectations for the near future in the field of sensor-controlled assembly. A structure to this field is given in terms of building blocks (“components”) with well-defined interfaces. A major lesson from the past is that strict hierarchical software structures are not appropriate for autonomous systems, because they reduce the flexibility and reaction times needed in these systems. But also heterarchical architectures are useless in themselves, because they lack the ability to use context. The modern software engineering solution is then to build “intelligent” components (each possibly having some level of internal hierarchy) that take care of sub-problems, and that deliver services to each other online. Making a robust implementation of such a flexible system is the major challenge, not just in assembly, but in all future robotic systems. The paper has outlined the contents of each of these components, and discussed major past and future developments. The authors are convinced that these components have reached sufficient maturity, such that working “proof of concept” systems can be built in the near future. Acknowledgments. H. Bruyninckx and T. Lefebvre are Fellows of the Fund for Scientific Research–Flanders (F.W.O). Financial support by the Belgian Programme on Inter-University Attraction Poles (IUAP), F.W.O. , K.U.Leuven’s Concerted Research Action, and the U.S.A. National Science Foundation (grant IIS-9700412), is gratefully acknowledged.
References [1] G. N. Saridis, Self-organizing control of stochastic systems. Marcel Dekker, 1979. [2] J. De Schutter and H. Van Brussel, “Compliant robot motion I–II,” Int. J. Robotics Research, vol. 7, no. 4, pp. 3–33, 1988. [3] B. J. McCarragher, G. Hovland, P. Sikka, P. Aigner, and D. Austin, “Hybrid dynamic modeling and control of constrained manipulation systems,” IEEE Robotics and Automation Magazine, vol. 4, no. 2, pp. 27–44, 1997. [4] M. Raibert and J. J. Craig, “Hybrid position/force control of manipulators,” J. Dynamic Syst., Measurement and Control, vol. 102, pp. 126–133, 1981.
[6] D. E. Whitney and J. L. Nevins, “What is the remote centre compliance (RCC) and what can it do?,” in International Symposium on Industrial Robots, pp. 135–152, 1979.
[8] G. Hovland and B. J. McCarragher, “Hidden Markov models as a process monitor in robotic assembly,” Int. J. Robotics Research, vol. 17, no. 2, pp. 153–168, 1998. [9] X. Ji and J. Xiao, “On random sampling in contact configuration space,” in Proceedings of the Workshop on Algorithmic Foundations of Robotics, March 2000. [10] H. Bruyninckx and J. De Schutter, “Specification of forcecontrolled actions in the “Task Frame formalism”: A synthesis.,” IEEE Trans. Robotics and Automation, vol. 12, no. 4, pp. 581–589, 1996. [11] R. S. Ball, A Treatise on the Theory of Screws. Cambridge University Press, 1900. [12] J. Xiao and X. Ji, “A divide-and-merge approach to automatic generation of contact states and planning of contact motions,” in Proceedings of the 2000 IEEE International Conference on Robotics and Automation, 2000. [13] R. H. Sturges and S. Laowattana, “Fine motion planning through constraint network analysis,” in Proceedings of the IEEE International Symposium on Assembly and Task Planning, pp. 160–170, August 1995. [14] H. Hirukawa, Y. Papegay, and T. Matsui, “A motion planning algorithm for convex polyhedra in contact under translation and rotation,” in Proceedings of the 1994 IEEE International Conference on Robotics and Automation, pp. 3020–3027, May 1994. [15] X. Ji and J. Xiao, “Automatic generation of high-level contact state space,” in Proceedings of the 1999 IEEE International Conference on Robotics and Automation, pp. 238– 244, 1999. [16] L. Kavraki, P. Svestka, J. Latombe, and M. Overmars, “Probabilistic roadmaps for path planning in highdimensional configuration spaces,” IEEE Trans. Robotics and Automation, vol. 12, no. 4, pp. 566–580, 1996. [17] X. Ji, On Generation of Contact State Space and Contact Motion Planning. PhD thesis, University of North Carolina - Charlotte, Charlotte NC, 2000. [18] Bar-Shalom and X. Li, Estimation and Tracking: Principles, Techniques and Software. Artech House, 1993.
