Agent-based models for the creation and

Agent-based models for the creation and management of airline schedules Josef Jacobus Langerman

Thesis submitted in fulfilment of the requirements for the degree Philosophiæ Doctor in Computer Science in the Faculty of Science at the University of Johannesburg Promoter: Prof. E.M. Ehlers November 2005

ii

Abstract This thesis reports on research into the applicability of intelligent agents in the airline scheduling environment. The methodology employed was to look at intelligent agent research and then, based on this, to build models that can be used to solve some of the airline scheduling problems. The following was done: •

An agent-based model was developed that can assist airline schedulers in the maintenance of a disrupted schedule. The agent model consists of a hybrid approach combining elements of machine learning and expert systems.

•

A multiagent model was developed that can generate a profitable and flyable schedule. The multiagent model developed extends the traditional control structures of the hierarchical agent organisation to a matrix structure. This new model can be extended to any problem domain that deals with resource allocation and capacity management.

To guide the thinking behind this research, a few questions were posed regarding the problem to be solved: Q1.

Can intelligent agents play a role in the airline industry, with specific focus on the scheduling creation and maintenance process?

Q2.

What will the design of the agent models be if the scheduling needs of an airline have to be addressed?

Q3.

If the models as envisioned in question 2 can be created, what will the practical implications be?

At a conceptual level the research produced three results: R1.

No references were found to multiagent technology in the production or maintenance of airline schedules. This theoretical research into agent systems shows that there is applicability in the scheduling environment, with specific reference to schedule maintenance and generation.

R2.

An agent model was created that combines declarative knowledge with empirical learning to assist human schedulers in the day-to-day maintenance iii

of the schedule. Multiple solutions to a scheduling problem are generated by the agent using embedded scheduling rules.

The agent then uses the Q-

learning algorithm to learn the preferences of the human scheduler. This approach combines the best of expert systems and machine learning. To solve the problem of schedule generation, a multiagent system with a matrix governance model was introduced.

Aircraft and airports were

modelled as buying and selling agents. The business manager agent that assigns individual aircrafts to specific routes was defined.

This was

accomplished by matching individual aircraft capacity to origin-destination demand. The agent model was then expanded to show how the inclusion of a resource manager agent can handle system capacity management. This is a matrix governance model, as an aircraft agent is managed by a business manager agent, as well as by a resource manager agent. The initial results from the prototype show that this model can generate profitable and flyable schedules. The multiagent model developed extends the traditional hierarchical agent organisation to that of a matrix structure. The contract net protocol used for typical multiagent coordination was adapted to work in this new control structure. This new model can be extended to any problem domain that deals with resource allocation and capacity management. R3.

A few airlines use expert systems to handle schedule disruptions.

By

introducing machine learning, a flexibility is achieved that is currently not available. The approach proposed for schedule generation is not guaranteed to provide optimal results like traditional operations research techniques, but it is useful for high-level analysis, long-term planning, new hub or alliance planning and research. It also has potential as a catalyst for integrated planning.

Keywords: Multiagent systems, airline scheduling

iv

ACKNOWLEDGEMENTS I would like to take this opportunity to express my gratitude to my supervisor, Prof. Elize Ehlers, for her kindness, guidance and patience on this long journey. I wish to express my heartfelt thanks to my wife, Irene, who had to bear with me during my studies. My deepest thanks to my mother and father for their ongoing support and encouragement. It is also my pleasure to acknowledge the ample financial support from the University of Johannesburg, Electronic Data Systems and Standard Bank of South Africa. None of this would have been possible without the support of South African Airways, who provided me with an ideal environment in which to learn about the complexities of airline scheduling.

v

Contents 1

2

Introduction..........................................................................................................1 1.1

Background and context ................................................................................1

1.2

Multiagent software architectures..................................................................1

1.3

Problem domain .............................................................................................2

1.4

Thesis structure ..............................................................................................3

1.5

Conclusion .....................................................................................................4

Intelligent software agents ..................................................................................5 2.1

Definition of intelligent software agents........................................................5

2.2

Multiagent systems ........................................................................................7

2.2.1

Cooperation and collaboration .............................................................10

2.2.2

Multiagent coordination.......................................................................13

2.3

2.3.1

Optimisation problems.........................................................................18

2.3.2

The testing of simulations....................................................................18

2.4 3

Agent-based simulation and modelling .......................................................15

Conclusion ...................................................................................................20

Multiagent systems as ecologies of computation .............................................21 3.1

Market-based ecologies................................................................................23

3.2

Decision making in a market-based world...................................................24

3.2.1

Price as an incentive mechanism .........................................................24

3.2.2

Utility as the means to distinguish between different options .............25

3.2.3

Profit maximisation..............................................................................26

3.3

Economy-based pricing models ...................................................................26

3.4

Allocation and pricing mechanisms.............................................................27

3.4.1 3.4.1.1

English auction.................................................................................30

3.4.1.2

Dutch auction ...................................................................................31

3.4.1.3

First-price, sealed-bid (discriminatory) auctions .............................31

3.4.1.4

The Vickrey auction (uniform second-price)...................................32

3.4.2 3.5

Auctions ...............................................................................................28

Tatonnement ........................................................................................33

Conclusion ...................................................................................................34

vi

4

5

6

7

Machine learning ...............................................................................................35 4.1

Introduction to learning................................................................................35

4.2

Learning in psychology and philosophy ......................................................36

4.3

Symbol-based learning.................................................................................37

4.3.1

Concept learning ..................................................................................37

4.3.2

Decision tree learning ..........................................................................37

4.3.3

Bayesian learning.................................................................................38

4.3.4

Reinforcement learning........................................................................39

4.4

Connectionist learning .................................................................................39

4.5

Social and emergent models of learning ......................................................40

4.6

Conclusion ...................................................................................................41

Airline scheduling ..............................................................................................42 5.1

The airline scheduling problem ...................................................................42

5.2

Scheduling environment ..............................................................................44

5.3

Impact of the airline schedule ......................................................................46

5.3.1

Cost and revenue impacts ....................................................................46

5.3.2

Customer satisfaction...........................................................................48

5.3.3

On-time performance ...........................................................................49

5.4

Implications for airlines ...............................................................................50

5.5

Conclusion ...................................................................................................50

Detailed airline scheduling ................................................................................51 6.1

High-level overview.....................................................................................51

6.2

Components of the schedule ........................................................................52

6.3

Approaches to solving the problem .............................................................55

6.4

Tools available .............................................................................................55

6.5

Conclusion ...................................................................................................57

Airline schedule disruptions..............................................................................58 7.1

Disruption management ...............................................................................58

7.2

Operations control........................................................................................59

7.3

Dealing with schedule disruptions ...............................................................59

7.4

Conclusion ...................................................................................................60

vii

8

A solution to airline schedule disruptions........................................................61 8.1

Introduction..................................................................................................61

8.2

Reinforcement learning................................................................................63

8.2.1

Q-learning ............................................................................................64

8.2.2

Application of theory to solve the operational scheduling problem ....67

8.3

Comparison of algorithms............................................................................68

8.4

Conclusion ...................................................................................................70

9

A matrix, market-driven scheduling model.....................................................71 9.1

The model for schedule creation..................................................................71

9.2

A practical example .....................................................................................75

9.3

Evaluation ....................................................................................................77

9.4

Comparison with the standard FIPA CNP ...................................................78

9.5

Conclusion ...................................................................................................79

10

Allocation of aircraft through the use of a business manager agent.............80

10.1

Overview of the environment ......................................................................80

10.2

Aircraft allocation model .............................................................................81

10.3

Experimental set-up .....................................................................................82

10.4

Experimental results using three aircraft .....................................................82

10.5

Experimental results using six aircraft.........................................................86

10.6

Conclusion ...................................................................................................89

11

Capacity management through the use of a resource manager agent ..........90

11.1

Overview of the environment ......................................................................90

11.2

Capacity management model .......................................................................92

11.2.1

Functionality to be maximised or minimised.......................................92

11.2.2

Determining an objective function.......................................................92

11.3

Solution algorithm .......................................................................................93

11.4

Calculating the new capacity .......................................................................95

11.5

Examples of performance ............................................................................95

11.5.1

Performance starting with high capacity..............................................95

11.5.2

Low capacity........................................................................................98

11.6

Sensitivity analysis.....................................................................................101

11.7

Conclusion .................................................................................................102 viii

12

System implementation ...................................................................................103

12.1

System requirements ..................................................................................103

12.2

Architectural framework ............................................................................105

12.3

Static model and supporting framework ....................................................108

12.4

Dynamic view of the simulation ................................................................111

12.5

User interface .............................................................................................114

12.6

Conclusion .................................................................................................115

13

Critical evaluation of the research .................................................................116

13.1

Research objectives....................................................................................116

13.2

Airline scheduling ......................................................................................117

13.2.1

Context...............................................................................................117

13.2.2

Scope..................................................................................................117

13.2.3

Related work ......................................................................................117

13.2.3.1

Airline scheduling ......................................................................117

13.2.3.2

Disruption scheduling ................................................................119

13.3

Contribution of this research to airline scheduling ....................................119

13.4

Multiagent systems ....................................................................................120

13.4.1

Context...............................................................................................120

13.4.2

Scope..................................................................................................120

13.4.3

Related work ......................................................................................121

13.4.3.1

Automated resource allocation approaches ...............................121

13.4.3.2

Multiagent organisational frameworks ......................................121

13.4.3.3

Contract Net Coordination Protocol ..........................................123

13.4.3.4

Hybrid learning techniques ........................................................124

13.5

Contribution of this research to computer science.....................................125

13.6

Satisfying the research objectives ..............................................................125

13.7

Conclusion .................................................................................................127

14

Future trends....................................................................................................128

15

Conclusion ........................................................................................................130

15.1

Disruption scheduling ................................................................................130

15.2

Schedule creation .......................................................................................131

15.2.1

Airline scheduling problems ..............................................................131 ix

15.2.2

15.3 16

Components of the model ..................................................................131

15.2.2.1

Simulation environment.............................................................132

15.2.2.2

Aircraft agents............................................................................132

15.2.2.3

Airport agents.............................................................................133

15.2.2.4

Business manager agent.............................................................133

15.2.2.5

Resource manager agent ............................................................133

Closing words ............................................................................................133

List of references ..............................................................................................135

x

List of figures Figure 1: The FIPA contract net ..................................................................................12 Figure 2: The coordination process..............................................................................15 Figure 3: Verification and validation in a simulation model .......................................18 Figure 4: Simplified view of computational economic modelling ..............................24 Figure 5: An example of a decision tree for credit risk assessment.............................38 Figure 6: Airline planning decisions............................................................................44 Figure 7: Integration architecture between scheduling systems ..................................45 Figure 8: Scheduling impact on air carrier investment model [95] .............................48 Figure 9: Factors driving customer satisfaction...........................................................49 Figure 10: Customer complaints ..................................................................................49 Figure 11: Hybrid system.............................................................................................62 Figure 12: System and environment interaction ..........................................................64 Figure 13: The Q-learning algorithm (Watkins & Dayan) ..........................................65 Figure 14: The Q-learning algorithm (Mitchell).........................................................66 Figure 15: Convergence rates when users make no mistakes......................................69 Figure 16: Convergence rates when users make mistakes 25% of the time ................69 Figure 17: Static agent organisation framework ..........................................................73 Figure 18: Dynamic agent organisation framework ....................................................73 Figure 19: Matrix CNP ................................................................................................78 Figure 20: Schematic overview of the simulation environment ..................................80 Figure 21: Schematic overview of the simulation environment incorporating the resource manager agent.............................................................................90 Figure 22: Determining new capacity..........................................................................93 Figure 23: Forecasting performance ............................................................................94 Figure 24: Average profit with initial high capacity....................................................96 Figure 25: Average daily capacity starting with initial high capacity .........................97 Figure 26: Average spilling with initial high capacity.................................................97 Figure 27: Average passengers carried with initial high capacity ...............................98 Figure 28: Average load factor with initial high capacity ...........................................98 Figure 29: Average profit with initial low capacity.....................................................99 Figure 30: Average daily capacity with initial low capacity .....................................100

xi

Figure 31: Average spilling with initial low capacity................................................100 Figure 32: Average passengers carried with initial low capacity ..............................101 Figure 33: Average load factor with initial low capacity ..........................................101 Figure 34: Sensitivity analysis for capacity increase factor.......................................102 Figure 35: Inheritance relationship between main classes in the simulation.............111 Figure 36: Negotiation sequence diagram .................................................................112 Figure 37: Resource allocation sequence diagram.....................................................113 Figure 38: Main application user interface ................................................................114 Figure 39: User interface showing scheduling information.......................................115 Figure 40: Extended FIPA contract net .....................................................................124

xii

List of tables Table 1: Mandatory and optional agent properties ........................................................7 Table 2: Comparison between biological and market ecosystems ..............................23 Table 3: Comparison of different types of auctions.....................................................33 Table 4: Impact of the schedule on an airline ..............................................................47 Table 5: Components of the scheduling process..........................................................52 Table 6: Available tools ...............................................................................................56 Table 7: Supporting products.......................................................................................57 Table 8: Differences between a production system and the proposed hybrid system .63 Table 9: Demand matrix ..............................................................................................74 Table 10: Demand from one airport to another at given time......................................75 Table 11: Sample aircraft configuration file................................................................82 Table 12: Demand information for January 2004 from 00:00 to 23:00 (extract) ........83 Table 13: Schedule generated for three aircraft (extract) ............................................83 Table 14: Results of comparison.................................................................................84 Table 15: Results of random allocation ......................................................................84 Table 16: T-test - two-sample assuming unequal variances for few aircraft (Pax) .....85 Table 17: T-test - two-sample assuming unequal variances for few aircraft (load factor) ........................................................................................................85 Table 18: Aircraft configuration file for six aircraft....................................................86 Table 19: Schedule generated for six aircraft (extract)................................................87 Table 20: Results of allocation algorithm....................................................................87 Table 21: Results of random allocation .......................................................................88 Table 22: T-test - two-sample assuming unequal variances for more aircraft (Pax) ...88 Table 23: T-test - two-sample assuming unequal variances for more aircraft (load factor) ........................................................................................................89 Table 24: High capacity initial aircraft configuration..................................................96 Table 25: Low capacity initial aircraft configuration ..................................................99 Table 26: High-level overview of classes in the simulation ......................................109 Table 27: Evaluation of the research objectives ........................................................126

xiii

List of equations Equation 1: The goal of distributed artificial intelligence .............................................8 Equation 2: First formulation of the objective function...............................................92 Equation 3: Second formulation of the objective function ..........................................92 Equation 4: Objective function of the simulation ........................................................92 Equation 5: Objective function used in forecasts ........................................................94 Equation 6: Method of increasing the capacity............................................................95

xiv

Abbreviations ACSEM

Air Carrier Service Evolution Model

AGIFORS

Airline Group of the International Federation of Operational Research Societies

ARP

Aircraft rotation problem

AseP

Airport selection problem

ASM

Available seat mile

ASP

Arrival scheduling problem

CAP

Crew assignment problem

CNP

Contract net protocol

CPP

Crew pairing problem

CSV

Comma separated value

DAI

Distributed artificial intelligence

DSP

Departure scheduling problem

EDS

Electronic Data Systems

ESS

Evolutionary stable strategy

FAP

Fleet assignment problem

FIPA

Federation for Intelligent Physical Agents

IATA

International Air Transport Association

LCAP

Leg capacity assignment problem

LFAP

Leg frequency assignment problem

MEP

Market evaluation problem

MSEP

Market selection problem

NAS

National Airspace System

OCC

Operations control centre

RPM

Revenue passenger mile

xv

1

Introduction

Chapter 1 is devoted to a discussion of the aims of the research undertaken with regard to multiagent systems and the airline scheduling problem.

1.1 Background and context An important issue in scheduling is how to handle problems that are so large and complex that they cannot be solved without some decomposition into subproblems. Of particular interest are those cases for which there are significant dependencies and dynamic interactions between the subproblems. The airline scheduling problem is exactly such a problem. The commercial and operational benefits of finding a way to do this properly are huge and therefore warrant research. Traditionally, the airline scheduling problem has been approached by using techniques from the field of operational research [1, 2]. The aim of this research is to look at how intelligent software agents and techniques from the field of artificial intelligence can be used to solve this problem.

1.2 Multiagent software architectures Intelligent agent software research is part of the broad field of distributed artificial intelligence (DAI). Multiagent systems can be considered as organisations composed of autonomous and proactive agents that interact and cooperate with one another in order to achieve a common goal [3]. The fundamental concepts of multiagent systems are intentional and social rather than implementation-orientated [4]. Existing software development techniques like object-orientated analysis and design are unsuitable for this task [5]. There is a fundamental mismatch between the concepts used by object-orientated developers and the agent-orientated view. In particular, existing approaches fail to adequately capture an agent’s flexible, autonomous problem-solving behaviour, the richness of an agent’s interactions and the complexity of an agent system’s organisational structures.

Chapter 1

Introduction

Researchers believe that the application of multiagent systems based on the concept of DAI is one of the most promising control architectures for next-generation manufacturing [6].

This optimism can easily be extended to airline scheduling.

Distributed heterogeneous agents make use of flexible control mechanisms for creating and coordinating the resulting society of agents. This society provides the foundation for the creation of an architecture that possesses the capability to benefit airline scheduling departments by enhancing a system’s reliability, maintainability, flexibility and stability, as well as providing a means for real-time planning and scheduling. Of interest for this research are architectural styles specifically suited for multiagent systems. Agents are autonomous, persistent computing entities that have the ability to choose which tasks to operate on and when to perform them [7]. Organisational structure will be a good starting point as organisational theory describes the internal structure and design of organisations and the independent organisational actors that pursue agreed goals. An organisation can be defined as a consciously coordinated social entity with a relatively identifiable boundary that functions on a relatively continuous basis to achieve a common goal or a set of goals [4]. In this research the organisational structures found in literature were studied and a matrix organisation structure that can be used by multiagent systems is presented. To provide a control or coordination mechanism for the matrix model, the contract net protocol (CNP) was extended to be able to operate in a matrix structure. This is novel both to multiagent systems and the airline scheduling problem domain. In some problem domains experts must make choices when presented with multiple solutions. Using agent technology and techniques from machine learning and expert systems, a generic model is provided that can be used in cases like this.

1.3 Problem domain In this research the focus is on airline scheduling problems and how to apply agent theory to them. The airline scheduling problem domain is ideally suited for the application of agent technology as it can be broken down into separate problem domains.

Each problem domain can be treated as a self-contained optimisation

problem, contributing to a global problem solution [8]. For example, the scheduling

2

Chapter 1

Introduction

department defines a schedule, the technical department assigns aircraft to the schedule and the crewing departments assign crew to the schedule. Each process is dependent on the other and there is constant interaction between these departments. The goal is to obtain a system that maximises some global quality measure, but that might be suboptimal from the point of view of the system components. Agents are very well suited to this problem domain as they can be organised to solve the constituting problems and coordinate and negotiate on a higher organisational level to achieve a global solution acceptable to all parties.

1.4 Thesis structure Chapters 2 and 3 provide an overview of intelligent software agents. These chapters provide the theory and context for multiagent systems, the concepts of market-based ecologies and agent-based modelling, which form the bedrock of the solutions to the airline scheduling problem that is presented later in the thesis. Chapter 4 serves as an overview of the field of machine learning. The theory of machine learning is used to develop a solution to the problem of how airlines can handle disruptions to the schedule in an automated way. Chapter 5 provides an overview of the airline scheduling problem and its centrality with regard to airlines and their commercial and operational activities. Chapter 6 describes some of the approaches to solving the airline scheduling problem and also a way of decomposing it into manageable pieces. Chapters 7 and 8 outline how intelligent agents can be organised to solve the problem of operational disruptions to the schedule and subsequent recovery from these disruptions. Chapter 7 gives an overview of the operational/disruption scheduling environment, while Chapter 8 describes the agent model employed to address these problems. Chapter 9 outlines the proposed matrixed, market-driven approach to solving the airline scheduling problem. Market price systems constitute a class of mechanisms that provide effective decentralisation of decision making. Most existing methods for scheduling in airlines are based on centralised or hierarchical decision making. The market-driven approach is, in contrast, based on a distributed and locally autonomous decision structure. Chapter 9 details how intelligent agents can be organised to solve 3

Chapter 1

Introduction

the airline scheduling problem, specifically the aspects of market evaluation; market, airport and leg selection; frequency and capacity assignment; departure and arrival scheduling; aircraft rotation and fleet assignment. The notable exclusion from the scope of this research is crew scheduling. Chapter 10 gives a detailed account of the design of the business manager agent. The business manager agent is the mechanism employed to assign aircraft to flights. Experimental results are presented to show that the schedules created are costeffective, reduce passengers spilled (passengers not picked up) and maximise passengers carried. Chapter 11 introduces the resource manager agent, which is used to manage capacity in the model. Too few aircraft will reduce the number of passengers carried, while too many will be unprofitable as many seats would be empty. The resource manager agent tries to find the balance between too many and too few aircraft. Experimental results are documented to show the effectiveness of the resource manager agent. Chapter 12 then outlines the system implementation details. The design principles, static and dynamic models of the implementation are detailed. Chapter 13 evaluates the scientific or technological contribution of this research to multiagent theory as well as airline scheduling. Chapter 14 provides a window onto the future developments in airline scheduling. Chapter 15 presents detailed conclusions that can be drawn from the research.

1.5 Conclusion At the conclusion of this thesis, the researcher will have expounded on and explained how the matrix multiagent organisation and the hybrid learning approach can be used in the airline scheduling environment, viz. schedule generation and disruption management.

The decentralised information structure, the distributed decision-

making authority and the cooperative relationship amongst the different decisionmaking entities would make agent-based airline scheduling a new paradigm with great potential for meeting today’s airline scheduling challenges.

The models to be

presented can be used in any domain where resource allocation and capacity management happen simultaneously or where people must make choices based on solutions presented to them by an automated system. 4

2

Intelligent software agents

The research reflected in the thesis deals primarily with how intelligent software agents can be organised to solve complex scheduling problems. In this chapter the enabling technology to get this done, namely intelligent software agents, is considered. The concept of an “agent” is defined, and multiagent systems and the modelling of agent systems are discussed. The chapter will conclude by looking at parallels between biological and economic systems and agent-based systems.

2.1 Definition of intelligent software agents There are numerous definitions of agents. Russel and Norvig [9] define an agent as “anything that can be viewed as perceiving its environment through sensors and acting upon that environment through effectors”. This is, in essence, the same as Maes’s definition [10], but Maes adds that an agent tries to fulfil a set of goals in a complex, dynamic environment. Nikolic [11] is basically of the same view. He also adds to the definition by stating that agents must be autonomous, i.e. they have control over their internal state and their behaviour.

Having control over their behaviour is one characteristic that

distinguishes agents from objects. Objects have states and behaviours, but do not contain behaviour activation or action choice. According to Woolridge [12], agents differ from objects in the sense that: •

Agents are autonomous: Agents embody stronger notions of autonomy than objects, and they decide for themselves whether or not to perform an action on request from another agent.

This correlates with Nikolic’s agent/object

differentiator. •

Agents are smart: Agents are capable of flexible (reactive, proactive, social) behaviour, whereas the standard object model has nothing to say about such types of behaviour.

•

Agents are active: A multiagent system is inherently multithreaded, in that each agent is assumed to have at least one thread of active control.

Chapter 2


Ferber defines an agent as a physical or virtual entity [13]: •

Which is capable of acting in an environment.

•

Which can communicate directly with other agents.

•

Which is driven by an individual object or by a satisfaction/survival function which it tries to optimise.

•

Which possesses resources of its own.

•

Which is capable of perceiving its environment.

•

Which has only a partial representation of its environment.

•

Which possesses skills and can offer services.

•

Which may be able to reproduce itself.

•

Whose behaviour tends toward satisfying its objectives, taking account of the resources and skills available to it and, depending on its perception, its representation and the communication it receives.

Ferber’s definition clearly captures physical entities as well, with the exception of mobility. It is interesting to note that in nature, agent intelligence is always associated with mobility, e.g. animals. Plants, which have for all practical purposes no mobility, also have no intelligence. Another angle of looking at agents is, according to Jennings and Woolridge [14], to say that an intelligent agent is an agent that is capable of flexible autonomous behaviour. Flexible to them means: •

Responsive: Able to perceive the environment and respond in a timely fashion.

•

Proactive: Exhibit goal-directed behaviour and take appropriate initiative.

•

Social: Able to interact/communicate when appropriate with humans and other agents.

The organisational structures for the control of intelligent agents is investigated in this research. Whenever the word “agent” is used, it refers to an intelligent agent or software agent. By examining the mandatory and optional properties of agents, agents can be categorised and the different definitions can be positioned. Table 1, borrowed from Lange and Oshima, shows this categorisation [15].

6

Chapter 2

Intelligent software agents Table 1: Mandatory and optional agent properties

[15] Mandatory Properties Reactive: Senses changes in the

Optional Properties Communicative: Able to communicate

environment and acts according to

with other agents.

those changes1. Autonomous: Has control over its own

Mobile: Can travel from one host to

actions2.

another.

Goal-driven: Proactive.

Learning: Adapts according to previous experience.

Temporally continuous: Continuously

Believable: Appears believable to the

executing.

end-user.

An important aspect of agents is that they facilitate cognitive modelling or behavioural modelling as opposed to role-based modelling [16].

Behavioural

modelling is based on behaviour fulfilling a purpose. Role-based modelling is based purely on purpose. A major thrust of this research is to model these behavioural aspects to gain a better mapping of real-world problem solving into computer systems. The autonomy and proactivity of agents facilitate a system in which entities take action without the need for centralised control.

2.2 Multiagent systems Various definitions have been proposed for the term “multiagent system”. Giorgini, Kolp and Mylopoulos describe a multiagent system as an organisation of coordinated autonomous agents that interact in order to achieve particular, possibly common goals [4].

Durfee, Lesser and Corkill define multiagent systems as a loosely coupled

network of problem solvers that work together to solve problems that are beyond the individual capabilities or knowledge of each problem solver [17]. Ferber describes a multiagent system as composed of a population of autonomous agents, which interact with each other to reach common objectives, while each agent simultaneously pursues 1 In robotic terminology, this is done through sensors and effectors. Some researchers regard software agents as software robots, often calling them “softbots”. 2 This attribute can also be seen as independence.

7

Chapter 2 individual objectives [13].

Intelligent software agents Oliveira, Fischer and Stepankova define multiagent

systems as a collection of possibly heterogeneous, computational entities, each having their own problem-solving capabilities, which are able to interact in order to reach an overall goal [18]. Thus a multiagent system consists of groups of loosely coupled agents that work together on tasks. Huhns and Singh [19] describe the goal of DAI as “creating a system that interconnects separately developed collaborative agents, thus enabling the ensemble to function beyond the capabilities of any of its members”. Equation 1 describes this notion formally.

Equation 1: The goal of distributed artificial intelligence

V( agenti) > max(V(agenti))

where V represents “value addedness”. This could have been an arbitrary definition involving attributes such as speed, worst-case performance, reliability, adaptability, accuracy, or some combination of these [20]. Important characteristics of multiagent environments are [21]: •

Local and non-local views: Each agent has a view of itself and this view of self is called a local view. Agents may also have views on non-local information. This is information about other agents with which the local agent interacts.

•

Imperfect information: Dynamism, limited bandwidths and bounded rationality have the result that agents do not have perfect views of other agents. Only approximated information about the characteristics of their own actions may be available, i.e. estimations of what will happen if they do a certain task.

How does a multiagent system differ from, say, a distributed database or even distributed objects? In a distributed database system the database is distributed across many machines but is accessible through a single user interface. In this case the different databases also work together to solve the task of retrieving information. The multiagent system consists of individual agents, each of whom have a choice of with

8

Chapter 2


whom they want to cooperate, how they will plan for an activity, how much they will charge for a service, etc. A partition of a distributed database, or for that matter an object, does not have the richness of action choice. In a multiagent system the agent may even be under the control of different corporate or legal entities. An agent that searches for the best price for an airline ticket may negotiate a deal with an airline agent to secure the best possible deal. In this case each of these agents belongs to a totally different legal entity. Montana, Herrero, Vidaver and Bidwell identify two characteristics that distinguish multiagent systems from ordinary distributed systems [22]: •

Uniformity: Multiagent systems require a greater degree of uniformity among their different components (i.e. agents) in terms of both behaviour and communication.

•

Goal-driven: Agents, unlike ordinary modules in distributed systems, continuously attempt to achieve certain goals without the need for outside invocation.

Now that an understanding of multiagent systems has been established, the need for multiagent or collaborative systems can be determined. The motivation for having collaborative systems may include one or several of the following [20, 34]: •

To solve problems that are too large for a centralised single agent.

•

To allow for interconnecting and interoperation of multiple existing legacy systems, e.g. expert systems, decision support systems.

•

To provide solutions to inherently distributed problems, e.g. air-traffic control.

•

To provide solutions which draw from distributed information sources.

•

To provide solutions where expertise is distributed.

•

To have coordination, which is necessary because with decentralisation in agentbased systems anarchy can easily set in.

•

To meet global constraints. Global constraints exist when the solution being developed by a group of agents must satisfy certain overall conditions if it is to be deemed successful.

9

Chapter 2 •


To establish cooperation and coordination, which can significantly increase efficiency. Information discovered by one agent can be of use to another so that both agents can solve the problem twice as fast as one agent on its own.

•

To research other issues, for example understanding interaction among human societies.

Research into multiagent systems covers a host of topics, including economic theories (agoric computing) [23, 24], communication or messaging protocols [25, 26], formal notions of intent [27] and architectures for building agent systems [28, 29]. Economic theories are based on Adam Smith’s invisible hand argument where equilibrium is reached through the interaction of many self-interested agents. Price setting through negotiation or auctions is used to achieve the abovementioned state of equilibrium. Communication and messaging protocols are important because if there is no communication between agents, there cannot be any collaboration. Research in this area is also directed towards ontologies for communication [30, 31]. The concepts mentioned in the above paragraph are important aspects of building multiagent systems and many of them provide broad and theoretical underpinnings. However, for agents to behave appropriately and to coordinate their activities and interact cooperatively in a meaningful fashion, they must be able to reason about their actions and temporarily sequence them. When actions involve multiple agents, the scope of the action sequencing problem is expanded. The local agent must now consider how its problem solving relates to that of the other agents. Generally the process of exchanging information, deciding how the processes relate, evaluating whether it is worth working together and forming commitments or agreements to do the work is called multiagent coordination.

2.2.1 Cooperation and collaboration Durfee, Lesser and Corkill [32] describe the problem of cooperation as how a loosely coupled network of problem solvers can work together to solve problems that are beyond their individual capabilities. Each problem-solving node in the network is capable of sophisticated problem solving and can work independently, but the problems faced by the nodes cannot be completed without cooperation [34]. How tasks, information and resources are distributed among agents so that they are able to 10

Chapter 2


work cooperatively to achieve overall system goals is called collaboration, according to Ferber [13]. Cooperation and collaboration are therefore closely related. The main technique used for cooperation and collaboration is the contract net protocol, which is described in the next section. The CNP is a widely used protocol in DAI, as it has proved to be a flexible and low communication interaction protocol for task assignment. The situation it is best suited for is that of a single task to be assigned among a number of agents [33]. The basic metaphor used in the CNP is, as the name of the protocol suggests, contracting. The CNP was originally proposed by Smith, who took his inspiration from the way that companies organise the process of putting contracts out to tender in public markets [34]. The original CNP has been applied to resource and task allocation problems. Examples of this include meeting scheduling [35, 36] and flexible manufacturing [37, 38, 39].

The current interpretation of the CNP used by a whole range of agent platform interpretations is the Federation for Intelligent Physical Agents (FIPA) interpretation of the contract net. The FIPA interpretation is a minor modification to the original protocol in that it adds rejection and confirmation speech acts [40]. Figure 1 shows the interaction diagram for this protocol. Using the FIPA terminology, the agent with the task is the initiator and agents that compete for acquiring are the task participants. The protocol requires the initiator to send a “call for proposals”, including a task description to all participants. They now either specify what they require for the task to be done or refuse to do the task at all. The initiator then accepts one of these proposals and rejects the others. The agent who was successful in having the task allocated to it is then required to inform the initiator about the results or the failure to deliver the results.

The advantages of the CNP include: •

Dynamic task allocation: Self-bidding leads to better agreements.

•

Natural load balancing: Busy agents need not bid or can put the costs high enough to discourage allocation.

11

Chapter 2 •


Dynamic removal and introduction of agents: Agents can easily be added or removed from the system with little effect on the functionality of the overall system.

•

Failure recovery: As control is distributed, single points of failure can be reduced.

Initiator

Participant

Call for proposals Refuse Deadline Propose Reject-proposal Accept-proposal Failure Inform: done Inform: result

Figure 1: The FIPA contract net

[40] The CNP has some shortcomings as well [34]: •

Indefinite waiting: If the initiator waits for all the bids before evaluating them, it can wait indefinitely. To overcome this problem, Ferber proposes assigning to each task announcement a deadline for the receipt of bids, and all those arriving after this date are directly rejected and not taken into account in the selection step [13].

•

Binding commitments: Once an agent agrees to a contract it has to follow through with it no matter what the future holds. Although a contract may be profitable to an agent when it is established, it may not be profitable after some future events have occurred. In the majority of realistic scenarios, agents are

12

Chapter 2


situated in dynamic environments. They may become aware of new information, or other agents may attempt to interact with it, etc. To solve this problem the CNP has been extended to the so-called “levelled commitment contracts”. Instead of conditioning the contract on future events, a mechanism is built into the contract that allows de-committing.

This is achieved by specifying the level of

commitment through de-commitment conventions [41, 42, 43]. •

Problem of earlier commitments: Between the time a bidder submits its proposal to the manager and the time the bidder is awarded the contract or rejected, it is committed to carrying out the task. While submitting any future proposals, it must take into account the earlier commitments not yet awarded.

To solve this

problem, Lee, Chang and Lee extended the CNP to a time-bound negotiation framework or a finite time-guarantee protocol by attaching the commitment duration to the task announcement and bid messages [44].

2.2.2 Multiagent coordination Coordination is a set of supplementary activities which need to be carried out in a multiagent environment and which a single agent pursuing the same goals would not accomplish [45]. For Nwana [20], coordination is a process in which agents engage in order to ensure that their community acts in a coherent manner. Coherent in this sense means that the agent’s actions are better coordinated and do not conflict with one another. The main issues involved in multiagent coordination are the optimisation of problemsolving activities, the avoidance of conflicting situations and how to distribute tasks to agents [12]. Coordination generally requires several steps during which agents compare their different tasks and decide what to do. These steps are outlined in Figure 2 [21]. The different types of operations found in multiagent coordination are: •

Detection: Agents exchange information about the activities they are planning to perform and information about activities that may be performed some time in the future.

•

Analysis and choice: Not all tasks are equally valuable to an agent. Agents have choices about which operations to perform and how to perform them. They also

13

Chapter 2


have limited resources and therefore it is generally not possible to coordinate activities with all other agents even if the local agent so wished. •

Commitment: Formation and negotiation. Once the decision has been made locally to collaborate, the local agent then enters a phase of commitment proposal and exchange with the other agents involved. The sequence of communication and messaging is dependent on the collaboration protocol.

During the

commitment formation process the local agent must analyse the proposed commitments to understand the ramifications on its current set of commitments and current set of scheduled actions, and decide whether to refine the commitment (proposing a change to the commitment) or to accept/reject the commitment. •

De-commitment: After forming a commitment, it is possible that the agent will decide to break the commitment because new opportunities have arisen or the problem-solving context has changed.

If the agent decides to break a

commitment, it may enter into a de-commitment process with the agents involved. •

Intermediate and final results sharing: Often interactions and commitments originate from the need to share results between agents.

When results are

produced, the producer agent must convey the required information to the consumer agent. Again, the mechanics of how this occurs are dependent on the protocol. In some cases, agents will provide status updates and intermediate results and in some situations the consumer agent may query the producer to request updates and intermediate results.

Coordination in multiagent systems is normally assumed to happen at run time. This means that agents themselves must be capable of recognising these relationships and, where necessary, managing them as part of their activities. This contrasts with the more conventional approach in computer science, where the designer explicitly attempts to anticipate possible interactions in advance and designs the system to avoid negative interaction and exploit potential positive interactions [12]. Approaches like partial global planning, joint intentions, mutual modelling and norms and social laws can be used for dynamic coordination [12]. The steps required for effective coordination and the decision of what to do are outlined in Figure 2.

14

Chapter 2


Exchange local information Interactions exist

No interactions exist

Detect interactions

End

Decide whether or not to coordinate Coordinate

Don’t coordinate

Commitment formation dialogue

End

Analyse implications of proposed commitment and refine Formed commitment

Did not form commitment

Perform other duties until task performance time arrives or until de-commitment occurs Still committed

“Changed mind”

Perform committed tasks Share results

End Analyse implications of proposed commitment and refine

De-commit End

End

Figure 2: The coordination process

[21]

2.3 Agent-based simulation and modelling Advancing the state of simulation requires appreciating the unique value of simulation as a third way approach to science in contrast to induction and deduction. Simulation,

15

Chapter 2


similar to deduction, starts with a set of explicit assumptions, but unlike deduction it does not prove theorems. Instead, a simulation generates data that can be analysed inductively. Basically, simulation is a way of doing thought experiments.

The

assumptions may be simple; the consequences may not be at all obvious. Simulation can be an effective tool for discovering surprising consequences of simple assumptions [47]. Bradley, Fox and Schrage define simulation as “driving a model of a system with suitable inputs and observing the corresponding outputs” [46]. Simulation and modelling can be used for diverse purposes. These purposes include prediction, performance, training, entertainment, education, proof and discovery [47]. •

Prediction: Through simulation complicated inputs can be processed by taking into account some hypotheses and outputs can be generated as predictions.

•

Performance: Simulation can be used to perform tasks by mimicking human behaviour, e.g. through artificial intelligence techniques.

•

Training: Simulation can be used to train people by providing a reasonably accurate and dynamic interactive representation of a given environment. Flight simulators are a good example of this.

•

Entertainment: Simulations of completely imaginary worlds or even realistic worlds can provide entertainment. Flight simulators and adventure games are good examples of this.

•

Education: The main use of simulation in education is to allow users to learn relationships and principles for themselves.

•

Proof: Simulation can be used to prove an existence proof. Conway’s Game of Life [48] shows that extremely complex behaviour can result from very simple rules.

•

Discovery: Prediction, proof and discovery form the basis of the scientific method. Using simulations for prediction can help validate or improve the model on which the simulation is based. Simulation can also work for the discovery of new relationships and principles.

16

Chapter 2


Simulation might also not be appropriate in certain circumstances. These include where common sense can be used to solve the problem, where the problem can be solved analytically or where it is easier to perform direct experiments. An important type of simulation in the social sciences is agent-based modelling. It is characterised by the existence of many intelligent software agents that interact with each other with little or no central direction or control. The emergent properties of such a model are the results of “bottom-up” processes, rather than “top-down” direction. The goal of such agent-based models is to enrich our understanding of fundamental processes that may appear in a variety of applications [47]. A central argument for agent-based simulation and modelling can be expressed as: “Why don’t we model it as the real world?” [49] The above quote from Parker shows why agent-based modelling and simulation can be useful, as intelligent agents offer a way of modelling the entities and their interactions in the real world.

Agent-based modelling allows the elimination of

generalisations and an exploration of the world through discrete entities and their interactions. There are two approaches to an agent-based model, i.e. Model the Model and Mirror the World [49]. In the Model the Model approach a conceptual model of a system is created and modelling tools are used to realise that conceptualisation. In agent-based modelling it might be discovered that a set of simple rules or behaviour iterated across a collection of agents produces a particular dynamic and it may be observed that a similar phenomenon occurs in real life. It may therefore be inferred that this dynamic is caused by many simple interactions. This type of explanation is very different from building a model and observing that it seems to have correlations with the real world [49]. In the Mirror the World approach an effort is made to model reality by building a mirror world that can be examined. Instead of trying to understand the emergent behaviour of interacting objects and deducing natural laws, known interactions between natural entities are modelled. This allows the prediction of the behaviour of a natural system without the level of information and understanding that is necessary for theoretical models [49, 11].

17

Chapter 2


2.3.1 Optimisation problems The discussion so far has been based primarily on modelling natural systems, for example the flocking of birds or modelling social behaviour between autonomous natural agents. This paradigm can, however, be extended to optimisation problems. By modelling the problem using intelligent software agents the system for the subsequent emergent behaviour can be observed. By tweaking and changing the rules by which the intelligent software agents operate, the emergent behaviour can be governed to some extent. The emergent behaviour can be the solution to a problem. Surprisingly, emergent behaviour is not always directly observable when designing the rules by which the software agents operate. To illustrate, consider the optimisation problem of producing a flyable, cost-effective airline solution. By modelling and simulating the system using intelligent software agents, the quality of the solution provided can be observed. Simulation in the transportation industry is not something entirely new.

Researchers have used

simulation in the fields of national airspace, aircraft offloading operations, baggage handling airline operations, passenger flow, airline interaction effects, etc. [50].

2.3.2 The testing of simulations

Real system Calibration & validation

Conceptual validation

Conceptual model Assumptions on system components. Structural assumptions that define interactions between system components. Input parameters and data assumptions. Model verification Operational model (Computerised representation) Figure 3: Verification and validation in a simulation model [50]

18

Chapter 2


Figure 3 gives a high-level overview of the process to verify the validity of a simulation model. To determine if a simulation should be used for a given situation, its credibility should be established by evaluating the fitness of the model or simulation for the intended use. In simplest terms verification and validation are two distinct but interrelated processes that gather and evaluate evidence to determine, based on the simulation’s intended use, the simulation’s capabilities, limitations and performance relative to the real-world objects it simulates. In addition to verification and validation, some researchers and organisations add the criterion of accreditation. Verification, validation and accreditation can formally be defined as follows: Verification: The Software Engineering Institute defines verification as confirming “that work products properly reflect the requirements specified for them” [51]. Validation: The Software Engineering Institute defines validation as confirming “that the product, as provided, will fulfil its intended use” [51]. Accreditation: The official certification that a model, simulation or federation of models and simulations and its associated data are acceptable for use for a specific purpose [52].

Each one of the abovementioned processes can also be informally captured by the simple questions: •

Verification: Did I build the thing correctly?

•

Validation: Did I build the right thing?

•

Accreditation: Should it be used?

Verification is concerned with building the model correctly.

It is utilised by

comparing the conceptual model to the computer representation that implements that conception. It asks the questions: Is the model implemented correctly in the computer? Are the input parameters and logical structure of the model correctly represented? Validation is concerned with building the right model. It is utilised to determine if a model is an accurate representation of the real system. Validation is usually achieved 19

Chapter 2


through the calibration of the model. This is an iterative process comparing the model to actual system behaviour and then using the discrepancies between the two and the insights gained to improve the model. This process is repeated until model accuracy is judged to be acceptable. Validation should be performed throughout the software development cycle. As an aid in the validation process Naylor and Finger [53] formulated a three-step approach which has been widely followed: 1. Build a model that appears reasonable to knowledgeable persons. 2. Validate the model assumptions. 3. Compare the model input-output transformations to corresponding input-output transformations for the real system.

2.4 Conclusion In this chapter the main concepts of intelligent software agents and multiagent systems were briefly presented. This chapter dealt specifically with what an agent is and investigated the characteristics of agents and how agents can form part of larger groups in multiagent systems.

The resultant problems of cooperation and

collaboration were discussed. Understanding multiagent systems leads to an understanding of how simulation and modelling could be used to model real-world scenarios and to provide a mechanism to predict behaviour. To determine whether the model is appropriate, its credibility needs to be established by evaluating its fitness for the intended use. This is done through a process of verification, validation and accreditation. In the next chapter a particular way of viewing multiagent systems, namely ecologies of computation, will be discussed. The concepts introduced through a specific type of computational ecology, the market ecology, will be used as the building blocks in the airline scheduling model, and presented later in the thesis.

20

3

Multiagent systems as ecologies of computation

Multiagent systems can also be viewed as a type of ecology of computation. The term “ecology” is defined in the Concise Oxford Dictionary as “dealing with the relations of organisms to one another and their physical surroundings” [54]. This definition has been understood in much the same form since at least the time of the early evolutionist’s writings in the 1870s [55]. A great deal of research is beginning to consider computational ecologies or models for computations that are based on relationships and self-organisation [55, 56, 57]. Distributed computational systems are acquiring the same characteristics as social and biological organisations. Using techniques from biology, Dellaert and Beer show how self-organisation can be modelled on a molecular, cellular and organism level using intelligent agents [58]. These systems consist of self-regulating entities which make decisions based on local imperfect knowledge about the world and on information which is at times inconsistent and delayed. These entities become a community of concurrent processes which, in their interactions, strategies and competition for resources, behave like whole ecologies [59]. Through spontaneous ordering processes these artificial ecologies can develop increasingly effective solutions to computational problems. By modelling the airline scheduling environment as an ecology, it will be shown that effective schedules can be generated. When studying ecologies an important aspect to consider is evolutionary stable strategies (ESSs). An ESS is a strategy which, given that it dominates an ecology, cannot be invaded by any other strategy. ESS drives the behaviour of an ecosystem. The task faced by a designer of such systems is to relate ESS to the emergent properties of the system - including properties sought by the designer. When modelling ecologies, or more specifically, entities competing for resources, there are two choices – adapt a cooperation model either from nature or from human society. Social models have different ESSs to biological models. To create an airline schedule in this research, a social model-based approach is used which, in this case, is the market ecosystem.

Chapter 3


In biological ecosystems the conservation of mass and energy is the prime physical constraint. Among animals, the critical resources like biomass and free energy are downwardly conserved. This means that they can be transferred and reduced by the transactions animals are capable of, but not increased. In biological ecosystems animals must eat other animals or plants to survive. This is not a synergistic or symbiotic process – the incentives are not towards cooperation between predator and prey. These incentives lead to a race in which the predators develop better weapons and the prey develop better armour [60]. The “attack-defence” strategy is the ESS of most biological ecosystems. In the idealised market system physical constraints include rights of property, unforgeable currency and trademarks. These constraints provide the foundations for idealised markets. Players in the idealised market must obtain valuable resources or goods in order to sustain activity. An entity which simply gives away goods would steadily lose resources compared to one that does not. Such a strategy is not an ESS. In order to receive something, an entity must offer something in exchange. For both sides to want to trade, each must value the goods received more than the goods given. Each trade in such a market can be seen as moving the system towards a condition in which no transaction that will make both parties better off remains to be done. In economic terms this is known as Pareto optimality. Miller and Drexler [60] argue that biological ecosystems involve more predation, while idealised market ecosystems involve more symbiosis.

In this thesis the

researcher sets out to prove that the symbiosis between entities can result in good schedules. It is thus apt to use market ecosystems as a template for the modelling. Table 2 depicts the comparison between biological and market ecosystems.

By

studying the idealised market ecosystems column it can be seen that the market ecosystem has a firm footing in the theory of economics. The English economist, Lionel Robbins, recently defined economics as “the science which studies human behaviour as a relationship between (given) ends and scarce means which have alternative uses” [61].

22

Chapter 3

Multiagent systems as ecologies of computation Table 2: Comparison between biological and market ecosystems

Biological ecosystems Conservation of mass and energy.

Idealised market ecosystems Rights of property, unforgeable currency and trademarks.

Incentives are not toward cooperation

Incentives are towards cooperation

between predator and prey.

between producer and consumer.

ESS is "attack and defence". Entities

ESS is "productive and wary". Entities

must attack other entities, as well as

must be productive and honest as a

defend themselves against attack.

producer and wary as a consumer.

Prey have no voluntary choice about

Producers will voluntarily seek out

their consumption.

those who want to consume what they have. Consumers will voluntarily seek out producers they want to support.

Long-term negative sum game.

Synergistic, symbiotic order.

3.1 Market-based ecologies Tesfatsion describes decentralised market-based economies as “complex adaptive systems, consisting of large numbers of buyers and sellers involved in massively parallel local interactions. These interactions give rise to macroeconomic regularities such as shared market protocols and behavioural norms that in turn feed back into the determination of local interactions. The result is a complicated dynamic system of recurrent causal chains connecting individual behaviours, interacting network and social welfare outcomes” [62]. This intricate two-way feedback system between the micro- and macrostructure has been recognised in economics from at least the time of the great economist Adam Smith [63]. Decentralised market economies have the following features and complexities [64]: •

Large number of economic agents involved in distributed local interactions.

•

Two-way feedback between microstructure and macro regularities mediated by agent interactions.

•

Potential for strategic behaviour. 23

Chapter 3


•

Pervasive behavioural uncertainty.

•

Possible existence of multiple equilibriums.

•

Critical role of institutional arrangements.

Experimental treatment factors (initial conditions)

Economy evolves over time

Macro regularities

Figure 4: Simplified view of computational economic modelling

Economies can also be modelled using computational techniques. Figure 4 shows a high-level schematic of the basic modelling process.

3.2 Decision making in a market-based world 3.2.1 Price as an incentive mechanism We live and work in communities which could not exist without the cooperation and coordinated efforts of their members. Our world is an interdependent world because it is a world of specialisation. If we are specialising, we must be able to exchange our goods for other goods which we want, but do not produce ourselves. Without some easy and convenient means of exchange, this elaborate network of exchange would collapse overnight. The unique function of money is to act as a medium of exchange [65]. In this discussion price means a metric measurement referring to the value or worth of something.

Rising prices induce sellers to augment their activity and buyers to

diminish theirs; falling prices produce the opposite effect [66]. A supplier’s goal is to optimise its individual satisfaction (profit) derived from its choice of resource

24

Chapter 3


allocations to consumers. Each consumer is endowed with money that it uses to purchase required resources. Each producer owns a set of resources and charges customers for the use of its resources. The price a producer charges for a resource is determined by its supply and the demand of the agents for the resource. The price system ensures that a realisable allocation of resources is achieved. Each agent is also subject to a budget constraint. Each agent in an economic system is endowed with a certain amount of wealth. Economics, as well as artificial intelligence, assumes rational agents. In the context of economic markets this means that an agent cannot spend more than its wealth. Resource pricing and budget constraints are the chief mechanisms that will enable agents to be coerced to act responsibly.

3.2.2

Utility as the means to distinguish between different options

Central to decision making and planning is the concept of utility. Utility is a property common to all things desired. Economists of the last century used the term “utility” to refer to a person’s preferences, or to the satisfaction that a person receives from consuming units of a particular resource.

Simplistically, utility is a measure of

happiness. Any rational agent can be described as possessing a utility function. Agents that possess an explicit utility function can make rational decisions [67]. They do this by comparing the utilities achieved by different actions. A utility function is therefore a function that maps a state of the world onto a real number, which describes the associated degree of happiness. Utility functions are constrained by six axioms, viz. orderability,

transitivity,

decomposability.

continuity,

substitutability,

monotonicity

and

For the purposes of this thesis a detailed description of these

properties will not be given.

Each rational agent should choose an action that

maximises its expected utility. This principle is called the principle of maximum expected utility. In a sense this principle could be seen as defining all of artificial intelligence [9]. The question now arises as to how to define utility in terms of the airline scheduling process. An obvious example is money as a utility measure. The almost universal exchangeability of money for all kinds of goods and services suggests that money

25

Chapter 3


plays an important role in human utility functions. It can be assumed that an agent prefers more money to less, all other things being equal. There are a few possibilities for using money as a utility measure in the airline scheduling process. An agent can earn money by carrying more passengers or it can conserve money by picking suitable aircraft for certain routes. This metric can be used to define plans for agents. The shape of the utility function also determines if an agent will be risk-averse, riskneutral or risk-seeking [9]. Part of the research in the thesis is aimed at trying to find suitable utility functions for airline scheduling agents.

3.2.3 Profit maximisation In the previous paragraph it could be seen that agents can use utility functions to decide on an appropriate action. In a market-based system buyers would use utility functions to decide which product or service to buy. Buyers are not the only entities making decisions. Sellers must decide how much of their goods to sell, to whom, when and at what price.

A buyer’s motivation for establishing a strategy is to

maximise utility. In contrast, sellers attempt to maximise revenue.

3.3 Economy-based pricing models When looking at market-based ecologies, essentially economic models are being considered.

Using economic models, there are two primary ways of allocating

resources to entities – exchange-based economies and price-based systems.

In

exchange-based economies entities are initially endowed with some resources. These entities (also called agents) then exchange resources until the marginal rate of substitution of the resources is the same for all the agents. The agent trades resources in the direction of increasing utility. The Pareto optimal allocation is achieved when there is no transaction left to be done which will make any party better off [68]. There are several ways of allocating resources in an economic price-based system. The primary ones are supply and demand theory, bidding and auction theory and command mechanisms. Command economies resemble the economic theories as used in communism and Marxism, and will not be taken into consideration in this thesis. The aim is to investigate distributed problem solving and not centralised problem solving, of which the command economy is a part.

26

Chapter 3


In supply and demand theory, resources are priced based on economic variables like supply, demand and the overall wealth in the economic system. endowed with some wealth.

Each agent is

Each agent computes the demand from the utility

function and the available budget. The aggregate demand from all the agents is sent to the suppliers who then compute the new resource prices. If the demand for a resource exceeds the supply, the supplier raises the price of the resource. If the supply exceeds the demand, the price must be decreased. The agents then again compute their demands given the current prices and present them to the suppliers. This process continues iteratively until the equilibrium price is achieved where demand equals supply [69]. Auctions and bidding processes are also used for resource allocation based on prices. An optimal auction is a bidding mechanism designed to maximise a seller' s expected profit [70]. The basic philosophy behind auctions and bidding is that the highest bidder always gets the resources and the current price is determined by the bid prices. The different auction mechanisms include Vickrey auctions, first-price sealed-bid auctions, English auctions and Dutch auctions.

3.4 Allocation and pricing mechanisms In economic models there are two ways to allocate resources among competing agents. One of them is the exchange-based economy (non-pricing-based economy) and the other is the price-based economy [10]. In the exchange-based economy, each agent is initially endowed with an amount of resources. They exchange resources until the marginal rate of substitution of the resources is the same for all the agents. In a price-based system, the resources are priced on the demand, supply and wealth in the system. One method for allocating goods in an exchange-based economy is the Walrasian auction. The idea behind the Walrasian exchange auction is to have an auctioneer adjust the prices of the various types of resources in the auction until supply equals demand for every type of resource. The auction takes place in a pure exchange economy, that is, bidders do not own money. They pay for the resources they actually want with resources they already possess. As such, prices are in fact mere exchange

27

Chapter 3


ratios between resources. The available budget is determined by the worth of the resources relative to those they desire under the going prices [71]. In contrast to the exchange economy, pricing-based allocations can be used. Prices using this set-up can be set by either the seller or through the use of auctions. In many markets the sellers of a product set the price of the product. They estimate the demand by assuming that market conditions are near equilibrium and perturbing the price to arrive at a new one. Gupta, Stahl and Whinston assume that trading near equilibrium is adequate and more efficient than computing the actual equilibrium [72].

3.4.1 Auctions Bidding and auctioning resources is another form of resource allocation. Auctions have been analysed extensively in literature [73, 74, 75] and there are many websites for buying and selling items using auction mechanisms. According to the Oxford dictionary, the term “auction” comes from the Latin root, "auctio", which means increase [54]. This root is often misleading as not all auctions have ascending price schemes. There are many different auction formats including the familiar ascending bid, but also including the descending, sealed-bid, simultaneous, handshake and whispered forms of bidding, some of which will be discussed. Auctions are useful when selling a commodity of undetermined quality [76]. They can be used for single items (indivisible items) such as a painting, and for multiple units of a homogeneous item (divisible items) such as jewellery. Auctions are useful in circumstances in which the goods do not have a fixed or determined market value, i.e. when a seller is unsure of the price he/she can get. Auctioning has advantages for both buyers and sellers. From the seller’s perspective, choosing to sell an item by auctioning it off is more flexible than setting a fixed price and less time-consuming and expensive than negotiating a price.

In a price

negotiation, each bid and counter-bid is considered separately, but in an auction the competing bids are offered almost simultaneously. List prices are useful when there is perfect competition and there are a sufficient number of buyers and sellers such that no economic agent has any degree of market power. Auctions are thus especially useful when markets are thin and valuation is uncertain.

28

Chapter 3


Almost anything can be auctioned, including land, livestock, wine and flowers. The common denominator is that the value of each item varies enough to preclude direct and absolute pricing [77]. An auction is thus a method of allocating scarce goods, a method that is based upon competition. In essence, a seller wishes to obtain as much money as possible, and a buyer wants to pay as little as necessary.

An auction offers the advantage of

simplicity in determining market-based prices. It is efficient in the sense that it usually ensures that resources accrue to those who value them most highly and also that sellers receive the collective assessment of the value [78]. What is unique about an auction is that the price is set not by the seller, but by the bidders. On the other hand, it is the seller who sets the rules by choosing the type of auction to be used. An auction is unusual also in that, unlike other methods of selling, generally the auctioneer does not own the goods, but acts rather as an agent for someone who does. Frequently, the buyers know more than the seller about the value of the item. Sellers, not wanting to suggest a price first for fear that their ignorance will prove costly, hold an auction to extract information they might not otherwise realise. Generally, experts agree that there are four major one-sided auction formats: •

English,

•

Dutch,

•

First-price sealed-bid, and

•

Vickrey (uniform second-price).

Both buyers’ and sellers’ perspectives must be taken into consideration.

When

choosing a type of auction, various factors must be considered. Cheating, speed, perishable and non-perishable goods and article value are all taken into account. In some auctions the buyer must be present, yet this is not always possible if the bidders are in a different country, which is often accommodated for. "Winner’s curse" is widely recognised as being that phenomenon when a winner pays more for an item than it is worth.

Auction winners are faced with the sudden

realisation that their valuation of an object is higher than that of anyone else. In auctions in which no bidder is sure of the worth of the goods or items being auctioned, the winner is the bidder who made the highest guess. If bidders have 29

Chapter 3


reasonable information about the worth of the item, then the average of all the guesses is likely to be correct. The winner, however, offers the bid furthest from the actual value [79]. All bidders have private valuations and tend to keep that information private. 3.4.1.1 English auction The English auction is known also as the open-outcry auction or the ascending-price auction. It is used commonly to sell art, wine and numerous other goods. The auctioneer begins with the lowest acceptable price (reserve price) and proceeds to solicit successively higher bids from the customers until no one will increase the bid. The item is “knocked down” (sold) to the highest bidder [80]. According to Reynolds [81], auctioneers will sometimes maintain secrecy about the reserve price, and they must start the bidding without revealing the lowest acceptable price. One possible explanation for the secrecy is to prevent rings. Bids are often not made aloud, but signalled (tugging the ear, raising a bidding paddle, etc.). Many traders prefer the semi-anonymity, as known experts in a certain field may not want others to know they are bidding because it would probably increase the bidding interest. When a decision to accept signals is made, a system of price intervals must be introduced so that seller and buyer understand the signals. In certain situations, an auctioneer has wide discretion. One variation on the open-outcry auction is the open-exit auction in which the prices rise continuously, but players must publicly announce that they are dropping out when the price is too high. Once a bidder has dropped out, he/she may not re-enter. This variation provides more information about the valuations (common or public) of others than when players can drop out secretly. In another variation, an auctioneer calls out each asking price and bidders lift a paddle to indicate a willingness to pay that amount. The auctioneer then calls out another price. In the ascending-bid format, auctioneers can exert great influence. They can manipulate bidders with their voice, tone and personality, and can increase the pace or even refuse to notice certain bidders if, for example, they believe someone is a member of a ring.

30

Chapter 3


From a seller' s point of view, the key to any successful auction is the effect of competition on the potential buyers. In an English auction, the underbidder usually forces the bid up by one small step at a time. Often successful bidders acquire an object for considerably less than their maximum valuation simply because they need only increase each bid by a small increment. The seller thus does not necessarily receive maximum value, and other types of auctions may be more profitable for the seller than the English auction [78]. Another disadvantage of the English system is that a buyer must be present, which may be difficult and/or expensive. This type of auction is also highly susceptible to rings. 3.4.1.2 Dutch auction The descending-price auction, commonly known as the Dutch auction, uses an open format rather than a sealed-bid method and has a descending-bid structure. In a Dutch auction, bidding starts at an extremely high price and is progressively lowered until a buyer claims an item by calling "mine", or by pressing a button that stops an automatic clock. When multiple units are auctioned, normally more takers press the button as price declines, which means that the first winner pays more for his/her prize than later winners. When the goods are exhausted, the bidding is over. The Dutch auction is believed to be superior to the English auction from a seller perspective in that if the bidder with the highest interest really wants an item, he/she cannot afford to wait too long to enter a bid. This means he/she might bid at or near his/her highest valuation [81]. 3.4.1.3 First-price, sealed-bid (discriminatory) auctions This type of auction has a primary characteristic of being sealed, unlike the English or Dutch varieties, and thus hidden from other bidders. A winning bidder pays exactly the amount he/she bid and usually each participant is allowed one bid, which means that bid preparation is especially important. Generally a sealed-bid format has two distinct parts, a bidding period in which participants submit their bids, and a resolution phase in which the bids are opened and the winner determined. An important distinction must be made as to quantity (how many goods are being auctioned). The name "first-price" comes from the fact that the award is made at the highest offer when a single unit is sold. When multiple units are being auctioned, it is

31

Chapter 3


called "discriminatory" because not all winning bidders pay the same amount. Each bidder submits one bid in ignorance of all other bids and the highest bidder wins and pays the amount he/she bid. In a discriminatory auction, sealed bids are sorted from high to low, and items awarded at highest bid price until the supply is exhausted. Winning bidders can thus pay different prices. From a bidder' s point of view, a high bid raises the probability of winning but lowers the profit if the bidder is victorious. A good strategy is to shade a bid downward closer to market consensus, a strategy that also helps to avoid winner' s curse. This type of auction is advantageous to a bidder to gather information about a competitor' s valuation before the auction [81]. 3.4.1.4 The Vickrey auction (uniform second-price) The uniform second-price auction is commonly called the Vickrey3 auction. Like the first-price auction, the bids are sealed and each bidder is ignorant of other bids. The item is awarded to the highest bidder at a price equal to the second-highest bid (highest unsuccessful bid).

This method benefits both buyer and seller by

guaranteeing that bids reflect the fair value of the item. When auctioning multiple units, all winning bidders pay for the items at the same price (the highest losing price). Although this method seems unusual, bidders fully understand the rules and modify their bids as circumstances dictate. In the case of a Vickrey auction, bidders adjust upward. No one is deterred for fear that they will pay too high a price. Aggressive bidders receive sure and certain awards but pay a price closer to market consensus. The price that the winning bidder pays is determined by competitors'bids alone and does not depend upon any action the bidder undertakes. Less bid shading occurs because people do not fear winner' s curse. Bidders are less inclined to compare notes before an auction. A potential drawback is that this system requires total honesty from the auctioneer(s). If the auctioneer is not trustworthy, he/she could open the bids, find the winner and insert a new bid just barely under that to ensure higher revenues [73, 77].

3

William Vickrey died on 11 October 1996, three days after the announcement that the 1996 Bank of Sweden prize in economic sciences in memory of Alfred Nobel was being awarded to him.

32

Chapter 3


Table 3 compares the different types of auctions in terms of the types of goods that can be auctioned, the advantages, disadvantages and rules governing the auction.

Table 3: Comparison of different types of auctions Action name English (ascending price) Open

Dutch (descending price) Open First-price (sealed-bid) Discriminatory auction when multiple items are being auctioned Vickrey auction (secondprice sealedbid)

Types of goods Perishable and non-perishable

Advantages

Disadvantages

Rules

Inexperienced bidders may end up paying way more than the goods are worth (winner’s curse).

1. The buyer must be present, which may be difficult and/or expensive. 2. The seller does not necessarily receive maximum value.

Seller announces reserve price or some low opening bid. Bidding increases progressively until demand falls. Winning bidder pays highest valuation. Bidder may reassess evaluation during auction.

The seller often gets maximum value or close to it.

Buyers are forced to place bids early in the process for fear of losing the auctioned item(s).

Seller announces very high opening bid. Bid is lowered progressively until demand rises to match supply.

Refinancing credit and foreign exchange

If the bidders are riskaverse, then the firstprice (and also the Dutch) yields greater revenue.

Winning bidders can (and usually do) pay different prices. This system is susceptible to collusion.

Bids submitted in written form with no knowledge of bids of others. Winners pay the exact amount they bid.

Refinancing credit and foreign exchange

Bidders adjust upward. No one is deterred for fear that they will pay too high a price.

Uncertainty as to what is actually paid by winner as the winners do not pay their own bidding price.

Bids submitted in written form with no knowledge of the bids of others. Winner pays the second-highest amount bid.

Divisible and non-divisible

Perishable and non-perishable Divisible and non-divisible

Uniform price auction when multiple items are being auctioned

3.4.2 Tatonnement To find market equilibrium an iterative process called tatonnement can be used. Each market participant has a demand function, which inputs a vector of prices and outputs a vector of quantities that a participant would buy. If participants have convex utility

33

Chapter 3


functions an auctioneer can find equilibrium given a supply of commodities by repeatedly asking each customer their desired quantity of goods. This approach presents a few problems. The first is that demand functions can reveal sensitive information that a participant might not wish to reveal explicitly, and the second is that the search for equilibrium can be computationally expensive.

3.5 Conclusion Although evolutionary reasoning is most often applied to biological ecosystems, this chapter has shown how it can be expanded to market systems. A market-based world is one of the foundations of the economy, which has the primary concern of allocating scarce resources. In this chapter market-based ecologies, how prices and utility are used as decisionmaking criteria and how prices for goods are determined were considered. In the matrix model resource allocation the business manager agent will allocate resources through a pricing mechanism.

The resource manager agent in turn will use

profitability to manage aircraft capacity.

34

4

Machine learning

Chapter 4 gives a theoretical overview of machine learning. The machine learning theory presented here will be used and expanded in a later chapter to present a solution to the airline disruption scheduling problem.

4.1 Introduction to learning The computer, which only a few decades ago spent its fledgling years as a mere computational tool, has developed into today' s super complex micro-electronic device with far-reaching changes in processing, storage and communication of information. The main objectives of system theory in the early stages of development concerned the identification and control of well defined deterministic and stochastic systems. Over the decades, interest gradually shifted to systems which contained a substantial amount of uncertainty since most systems belong to this class. Since the ability of living organisms to cope with uncertainty is well known, it is only natural that efforts were also made to incorporate similar features in industrial systems. Learning, which is a natural activity of living organisms, deals with the ability of systems to improve their responses based on past experience [82]. Simon defines learning as “any change in a system that allows it to perform better the second time on repetition of the same task or another task drawn from the same population” [83]. Machine learning has proven to be of great value in a variety of problem domains. Machine learning techniques are especially useful in [84]: •

Data mining: Databases with large amounts of information can be analysed as they may contain valuable implicit regularities that can be discovered automatically.

•

Poorly understood domains: Machine learning can help where humans do not have the knowledge to develop effective algorithms, e.g. face recognition from images.

Chapter 4 •

Machine learning

Domains where the program must dynamically adapt to changing conditions: Algorithms can adapt to changes in the environment by employing machine learning techniques.

Machine learning is multidisciplinary and draws on ideas from a set of disciplines, including artificial intelligence, statistics, computational complexity, information theory, psychology and philosophy.

4.2 Learning in psychology and philosophy In psychology, the process of learning deals with the formation of concepts, thoughts and images as well as the relationship between experience and the organisation of the mind. The major conceptual division within psychology as well as philosophy is between rationalism and empiricism. The principal tenets of empiricism are that experience is the only source of knowledge, all complex ideas are made up of simpler ideas and the said complex ideas are connected together through the association of experiences. Philosophers such as Aristotle, Locke, Berkeley and Hume have elaborated on empiricism as explanations of mental phenomena. Learning theories such as those of Pavlov, Thorndike and Skinner are based on the associationistic framework.

Classical

empiricism was an attempt to show that all significant knowledge about the world is based on a posteriori truths. A belief, proposition or argument is said to be a posteriori if its truth or falsity can be established only through observation [85]. Rationalism, on the other hand, states that raw experience together with associative learning principles is not generally adequate to interpret events. Classical rationalism was an attempt to show that all significant knowledge about the world is based on a priori truths, which most of the rationalists associated with innate ideas. A belief, proposition or argument is said to be a priori if its truth or falsity can be established independently of observation. Definitions, mathematics and the principles of logic are usually held to be a priori [85]. Associationistic theories can also be divided into stimulus response theories (behaviourism) and cognitive theories. The importance of motivation, reward and punishment in learning and performance received great emphasis in the behaviouristic revolution led by Watson in the United States. In the 1930s and 1940s Skinner

36

Chapter 4

Machine learning

developed his principles of radical behaviourism and argued that psychology should deal directly with observable behaviour [85]. Around the 1950s, psychology abandoned the radically behaviourist point of view in what came to be termed the cognitive revolution. This new school of thought dealt with processes such as perception, attention, memory, thought and problem solving. There are a number of epistemological and philosophical issues that underlie learning [86]. These include: •

Generalisation problem: How are invariant patterns in data identified in a manner sufficient to use these invariants to fit new previously unseen data?

•

Inductive bias problem: This bias refers to how program designers use their own intuitions and heuristics in the design of representations, models and algorithms that support the learning process.

•

Empiricist dilemma: If there are no predetermined biases in the learning in the system, how can anything useful be learned, or how do we know when something is learned?

Following Luger’s taxonomy of learning systems [86], symbol-based learning, connectionist learning and social and emergent learning will be discussed below.

4.3 Symbol-based learning Symbol-based learning is primarily concerned with inductive learning. In essence this means learning from a generalisation from a set of examples. This can also be seen as learning a function from examples of its inputs and outputs.

4.3.1 Concept learning Concept learning is an example of induction-based learning. It can be formulated as a problem of searching through a predefined space of potential hypotheses for the hypothesis that best fits the training examples. The searches can in most cases be efficiently organised by taking advantage of the naturally occurring structure of the hypothesis space [84].

4.3.2 Decision tree learning Decision tree learning is one of the most widely used and practical methods for inductive inference. It is a method for approximating discrete valued functions that is 37

Chapter 4

Machine learning

robust to noisy data and capable of learning disjunctive expressions [84]. A decision tree takes as input an object or situation described by a set of properties and outputs a yes/no decision. Decision trees therefore represent Boolean functions. Functions with a larger range of outputs can also be represented [9]. Given a set of training examples a number of different decision trees can be produced that fit the data. Using a contemporary version of Ockham’s razor, i.e. accept the simplest answer that correctly fits the data, a decision tree can be chosen, in this case the smallest decision tree that correctly classifies the data. Figure 5 shows an example of a simple decision tree.

Income? R0 to R5k

R5k to R15k Credit history?

High risk Unknown

Bad High risk

Low risk

Good Moderate risk

Figure 5: An example of a decision tree for credit risk assessment

4.3.3 Bayesian learning Bayesian learning provides a probabilistic approach to inference. It is based on the assumption that quantities of interest are governed by probability distributions and that optimal decisions can be made by reasoning about these probabilities together with observed data. The Bayesian view of learning is extremely powerful, providing general solutions to the problems of noise, overfitting and optimal prediction [9, 84]. Mitchell describes the features of Bayesian learning methods as [84]: •

Each observed training example can incrementally increase or decrease the estimated probability that a hypothesis is correct.

38

Chapter 4 •

Machine learning

Prior knowledge can be combined with observed data to determine the final probability of a hypothesis.

•

Bayesian methods can accommodate hypotheses that make probabilistic predictions.

•

New instances can be classified by combining the predictions of multiple hypotheses, weighted by their probabilities.

•

Even in cases where Bayesian methods prove computationally intractable, they can provide a standard of optimal decision making.

4.3.4 Reinforcement learning Reinforcement learning addresses the question of how an autonomous agent that senses and acts in its environment can learn to choose optimal actions to achieve its goals. Each time the agent performs an action in its environment, a trainer might provide a penalty or reward to indicate the desirability of the resulting state. Interaction with the world demonstrates cause and effect, the consequences of actions and even how to achieve complex goals [84, 86]. The task of reinforcement learning is to use rewards to learn successful agent functioning, which is difficult because the agent is never told what the right actions are, nor which rewards are due to which actions. Within the agent framework the agent must be hardwired to recognise a percept as a reward rather than just another sensory input [9]. Reinforcement learning, and especially the Q-learning algorithm, will be used as one of the mechanism to solve the problems of disruptions to an airline schedule. A much more in-depth treatment can therefore be found in Chapter 8.

4.4 Connectionist learning In connectionist systems, the explicit use of symbols in problem solving is deemphasised. Connectionist learning systems are inspired by neurally or biologically inspired approaches to learning. In this approach it is believed that intelligence arises in systems of simple, interacting components (biological or artificial neurons) through a process of learning and adaptation by which the connections between components are adjusted. Problem solving is parallel in the sense that all the neurons within the collection process their inputs simultaneously and independently [86]. Other names 39

Chapter 4

Machine learning

that have been used for this field include parallel distributed processing, neural networks, adaptive networks and collective computation. Mitchell states that connectionist learning approaches are appropriate for problems with the following characteristics [84]: •

Instances are represented by many attribute-value pairs.

•

The target function output may be discrete-valued, real-valued or a vector of several real- or discrete-valued attributes.

•

The training examples may contain errors.

•

Long training times are acceptable.

•

Fast evaluation of the learned target function may be required.

•

The ability of humans to understand the learned target functions is not important.

4.5 Social and emergent models of learning Whereas connectionist systems are inspired from ideas from neurology, social and emergent models of learning are based on ideas from evolution. The primary idea that is borrowed from evolution is the shaping of a population of individuals through the survival of its most fit members. The power of selection across a population of varying individuals has been demonstrated in the emergence of species in natural evolution, as well as through the processes underlying cultural change. These ideas have been formalised through research in cellular automata, genetic algorithms, genetic programming and artificial life [86]. Using genetic algorithms a hypothesis is often described as bit strings whose interpretation depends on the application. The search for an appropriate hypothesis starts with a population of initial hypotheses. Members of the current population give rise to the next generation population by means of operations such as random mutation and cross-over. After each step, the hypotheses in the current population are evaluated relative to a given measure of fitness [84]. In this sense genetic algorithms resembles a computer simulation of natural selection. Genetic algorithms are typically used to solve combinatorial optimisation problems.

40

Chapter 4

Machine learning

4.6 Conclusion Chapter 4 provided a cursory overview of the field of machine learning covering symbol-based, connectionist and evolutionary learning. The concept of reinforcement learning will be used and expanded in Chapter 8 to solve the problem of how to handle disruptions to a schedule.

41

5

Airline scheduling

In this chapter a high-level description is given of the airline scheduling problem. It will become obvious why it is a crucial and also complex process in the airline industry. Its complexity is examined.

The impact of the schedule on financial

performance, customer satisfaction and on-time performance is also examined.

5.1 The airline scheduling problem Scheduling is one of the most complex tasks that faces an airline. The product that an airline sells is a seat to a destination at a certain time. This seat is the product of a complex and involved process called airline scheduling. The driving forces of the scheduling process are economic factors, marketing considerations, safety regulations and the availability of personnel, facilities and equipment. Wells [87] describes the schedule planning process as “[t]hat what is necessary to take the company’s marketing goal for a particular period and turn them into a saleable schedule that creates volume of new traffic; beats the competition; makes the most efficient use of personnel, facilities, and aircraft; serves the cities on the system; and earns ever-increasing profits.

Scheduling may be the most difficult job in any

airline.” The aim of airline scheduling is the same as that of any business, namely the maximisation of profit in the long term. The difference between income and cost is referred to in this research as profit. The capacity of an airline schedule to satisfy market demand in terms of quality as well as quantity largely determines the income potential of an airline [88]. The airline schedule aims to minimise costs through the optimal allocation of resources. It plays a pivotal role in maximising income and minimising cost. This means larger profits for the airline as well as a greater return on equity for the shareholders. The product that an airline sells is a seat that moves from one point to another. The elements that make the airline’s seat attractive to the buyer are the origin/destination and the time of flight, i.e. the schedule. This simple statement hides a wealth of complexity, not found in many industries. To unpack this statement the question must

Chapter 5

Airline scheduling

first be asked: What is an airline schedule? Once it has been defined, the importance of the schedule to an airline can then be considered. As mentioned in the previous paragraph, the element that most significantly enhances the ability of an airline to sell seats is its schedule. If people want to fly to a destination and no flights are available, they cannot fly, regardless of the service, price, type of aircraft and other factors that might be considered as important. The schedule forms the heart of an airline’s operations. To create and operate a schedule, an airline’s marketing goals must be turned into a saleable schedule that can create and sustain passenger and cargo traffic, earn profits, beat the competition, make efficient use of personnel, facilities and aircraft, collaborate with suppliers, airport authorities and air-traffic control, and ensure that all of this can be executed flawlessly. Producing an efficient schedule that satisfies most of the various airline departmental stakeholders and the marketplace in a profitable manner and operating that schedule might be the most difficult task in any airline. The schedule generating process can be seen as “the art of defining system-wide flight patterns which deliver an optimum service to the public in terms of quantity and quality, subject to operational, commercial, financial and political and regulatory constraints” [88]. Broadly, the constraints can be categorised as: •

Financial constraints: A schedule must support and contribute to the financial targets of the airline, e.g. the bottom line.

•

Operational constraints: A schedule must be practical and operable. Operational constraints include the number of aircraft and crew available, types of aircraft available, airport capabilities, ground equipment available, skill level of staff, etc.

•

Commercial constraints: It is no use building a schedule that nobody wants to fly. The schedule must therefore take into account the sales and marketing strategies of the airline.

•

Political and regulatory constraints: These constraints include flying rights to certain sectors, employment conditions like labour agreements, aircraft service cycles, etc.

This is not an easy problem. Typical US domestic carriers: •

Operate 5 000 flights per day, 43

Chapter 5

Airline scheduling

•

Serve over 10 000 markets, and

•

Offer over 4 000 000 fares.

Over and above this they: •

Change schedules twice each week,

•

Change 100 000 fares on a typical day, and

•

Offer their products for sale more than one year in advance.

The total number of products requiring definition and control is approximately 500 000 000 and this number is increasing due to the proliferation of distribution channels and customer-specific controls [89].

Schedule planning Route development Frequency planning Timetable development Fleet assignment Aircraft rotations

Pricing

Crew scheduling

Revenue management

Airport resource management

Sales and distribution

Operations control

Types of decisions

• • • • • •

STRATEGIC

Fleet planning

Types of Decision

TACTICAL

SHORT TERM

Time horizon

LONG TERM

5.2 Scheduling environment

Figure 6: Airline planning decisions [90]

Figure 6 shows the typical scheduling decisions of an airline, together with the time horizon over which these decisions are made [90]. 44

Chapter 5

Airline scheduling

Airline scheduling is a continual task starting long before a flight takes off. The process flow of airline scheduling will be discussed in more detail in the next chapter. What is important at this stage is that airline marketing and sales, i.e. pricing, revenue management and sales and distribution, are driven by the schedule. Whereas airline scheduling asks the question “can I fly a passenger from one destination to another, at a specific time?”, revenue management asks the question “can I sell the right seat to the right passenger at the right price?”. Flight Scheduling

Crew

Cargo

Yield Management

Flight Operations

Maintenance & Engineering

Pricing

Airports’ Gates/Staff Plan

Cabin Services

Distribution

Flight Operating System System

Revenue Accounting

Reservation System

Aircraft Comms

Historic Flight Data

Historic Traffic/ Revenue Data

Figure 7: Integration architecture between scheduling systems [91]

Whereas Figure 6 shows an abstraction of the process flows within an airline, Figure 7 shows the integration aspects between different airline systems. This is a typical realworld example as presented by an airline practitioner at the Airline Group of the International Federation of Operational Research Societies (AGIFORS) 2000 Conference. This example shows the complexities and interdependencies between all the different airline systems directly and indirectly involved in the planning process [91].

45

Chapter 5

Airline scheduling

5.3 Impact of the airline schedule The true determination of the value of the schedule is how it balances the various considerations, with profitability being the most important.

It is also normally

assumed that every flight takes off and lands as scheduled. In this section these assumptions are analysed and the customer’s perspective of the airline is correlated with operations, which in turn depend on the schedule.

5.3.1 Cost and revenue impacts The optimisation of schedules can have a huge cost-saving impact on an airline. Moving to the other side of the income statement, route selection, which forms part of scheduling, can have a significant effect on the potential revenues of an airline. One of the areas where savings are possible is through the better utilisation of equipment. In March 2002 it was reported that United Airlines could save as much as US$25 million by improving the assignments of planes to flights using a new algorithm developed in collaboration between the Massachusetts Institute of Technology, the University of Florida and United Airlines [92]. This model not only looked at traditional flight assignment, but also at the effect that “through assignment” might have. Consider the effect in using an example of three city pairs (London, Paris, Zurich). Instead of allocating an aircraft to the London to Paris flight and another one from Paris to Zurich, an airline might decide to create a single flight from London to Zurich with a stop-over in Paris. If too small an aircraft is used, it results in loss of revenue: if too big, it might be costly and inefficient. This problem might seem trivial until it is considered that United Airlines have roughly 2 300 flights a day, service 150 cities and use 500 aircraft that are spread across 10 different types of aircraft. Improvements in the allocation of assets using fleet assignment and through planning can therefore have a huge impact on the bottom line [92]. Improvements are not limited only to aircraft assignment. Ellis Johnson, one of the world’s leading airline schedule experts, reported that excess crew costs for American Airlines had been driven down to 0% - 3% in 2002. Ten to fifteen years before, it was about 8% – 10%. Each 1% is worth about US$10 million per year. In 1997 American Airlines had 9 000 pilots costing about US$1.2 billion. The same report cites that improvements in fleet assignments have increased profits by tens of millions of US dollars [93]. 46

Chapter 5

Airline scheduling

Air New Zealand has effected cost improvements of more than NZ$15.6 million (about US$7 million) a year by innovations in the area of crew scheduling. The project brings improved service to passengers and increased ability to meet air crews’ assignment preferences. For a relatively small carrier this is a substantial amount of money. For this achievement they have been honoured by the Institute for Operations Research and Management Sciences (INFORMS) [94]. Flight disruptions have an impact not only on the customer but also financially on the airline.

The Handbook of Airline Operations [95] reports that the annual cost

estimates of disruptions amount to about 3% of the airline revenue. Airlines would do well to look at minimising disruptions and also finding innovative ways of recovering from these disruptions speedily. The impact of the schedule on the airline is summarised in Table 4: Table 4: Impact of the schedule on an airline

Primary impact Revenue

Cost

Determines market share

Drives direct operating costs Determines human resources needs

Determines volume of supply

Secondary impact Impacts on time performance Affects staff morale Influences competitor behaviour Influences price and load factors Impacts customer satisfaction Figure 8 depicts a typical airline investment model [96]. The shaded areas are those processes that are directly influenced by the scheduling process.

ASM is the

abbreviation for available seat mile, which is calculated by multiplying the total seats available by non-stop mileage. RPM is the abbreviation for revenue passenger mile, which is calculated by multiplying the number of on-board passengers by total non-

47

Chapter 5

Airline scheduling

stop mileage travelled. Both these concepts represent the revenue and cost cycles of the airline.

Estimated demand coefficients Per capita income Population Unemployment Fare yield

Fare yield

Fare equation

RPM time series

Total operating revenues Employment

Demand assumptions Per capita income Population Unemployment

Technology assumptions Productivity Unit costs Year of introduction Initial penetration Terminal penetration

ASM time series

Adjusted operating profit margin

Aircraft fleet Fuel cost

Total operating costs

Flight labour cost

Maintenance cost Flight equipment cost Ground equipment cost Indirect cost

Figure 8: Scheduling impact on air carrier investment model [95]

5.3.2 Customer satisfaction A good indication of the quality of an airline’s operations is the customers’ perception of the service. Figure 9 shows the factors driving overall customer satisfaction for US airlines. This data is compiled from frequent flyers [97]. Just by studying this, it can be seen that at least 34% of factors that drive customer satisfaction are directly schedule-related, i.e. on-time performance and flight availability. The schedule also has an effect on staff morale. If a flight attendant did not get the flights he/she applied for or has been overworked, the effect will be carried through to the customer. The schedule or crew roster will then also affect part of the flight attendants’ contribution to customer satisfaction. By combining the schedule-related factors with the flight attendants factors, it can be stated that the schedule contributes between 34% and 43% to customer satisfaction. 48

Chapter 5

Airline scheduling

25% 20%

On-time performance Airport check-in

15%

Flight attendants Seating area

10%

Gate factors Aircraft interior

5%

Flight availability/scheduling Food service

0% Factors Driving Customer Satisfaction

Frequent flyer programmes

Figure 9: Factors driving customer satisfaction [96]

Another good metric is to look at what customers actually complain about. Figure 10 shows that flight problems, which include delays, are a major contributing factor to customer complaints.

40% 35% 30%

Flight Problems

25%

Customer care

20% 15%

Baggage

10%

Res, Ticketing and Boarding

Other

5% 0%

Refunds Customer Complaints

Figure 10: Customer complaints [96]

5.3.3 On-time performance By studying the information the US Department of Transport produces, it appears that on-time performance is not up to standard. For the 12-month period ending August 2001, 75.4% of fights were on time (not delayed by more than 15 minutes) and 2.4% of flights were cancelled. The statistics for the period September 2001 onwards are skewed by the terrorist attacks of 11 September 2001. (40% of flights were not on time and 25% of all flights were cancelled for that month.) A high number of flight delays are not only an American occurrence. In 2000, 35% of French domestic 49

Chapter 5

Airline scheduling

carriers’ flights suffered from 15-minute plus delays, down from a record of 39% in 1999. The average delay was as high as 41 minutes. Travellers found this particularly irksome as the flight time between the country’s busiest city pairs does not exceed 90 minutes, according to AviationNow [98].

5.4 Implications for airlines From the above it is very clear that scheduling improvements can have an extremely beneficial effect on the bottom line of an airline. Acquiring better tools for scheduling flights and assigning resources to these flights, etc. will be money well spent. It is also clear that customers are not happy with airlines’ operational performance. This could be due to extremely tight schedules built for optimal financial benefit. To prove this would be extremely difficult. What is clear though is that in view of the poor operational performance, better recovery solutions to schedule disruptions must be implemented.

5.5 Conclusion In this chapter airline scheduling was defined and where it slots into the overall operational processes of the airline was determined. The impact that the schedule has on the financial performance of the airline, customer satisfaction and on-time performance was discussed. In the next chapter airline scheduling will be discussed in more detail, with specific regard to the subprocesses and the flow of these processes that make up airline scheduling.

50

6

Detailed airline scheduling

In the previous chapter a high-level definition of airline scheduling was given and the impact that the schedule has on an airline was determined. In this chapter the airline scheduling problem is investigated more thoroughly and some ways and means that have been used to solve this problem are studied.

6.1 High-level overview The scheduling department of an airline has to answer four questions: •

Which airports should be connected by a direct flight, that is which legs should be flown?

•

When and how often should a leg be operated, that is what are the departure and arrival times?

•

Which aircraft should be selected to fly a leg at a certain time?

•

Which crew should operate the aircraft flying a certain leg at a certain time?

The problem can be solved by taking one of two approaches: either cycling through schedule construction and schedule evaluation until a satisfactory schedule is constructed or the available planning time is up, or dividing the scheduling process into strategically, tactical and operational planning and progress to day of operation through these different stages. Beyond the scope of this chapter is disruption management or operational scheduling. Airlines in the past were concerned with building optimal schedules, with very little slack built in or thought of how to handle disruptions to the schedule. This chapter will focus on techniques for building optimal schedules.

The discussion of

operational scheduling or the handling of airline schedule disruptions will be deferred to the next chapter. During the planning process crew and aircraft are seen as separate entities which can be scheduled and optimised independently.

Crew scheduling consists of the

subproblems of crew pairing and crew rostering. In the crew pairing phase, pairings (trips) starting and ending at a home base are constructed. In total the pairings must

Chapter 6


cover all positions to be covered in the flights defined by the timetable. The purpose of crew rostering is to assign all pairings and other activities to named individuals. Crew scheduling must be completed several weeks before the day of operation [99]. Tail assignments, i.e. the assignment of actual aircraft to flights, are typically done a few days before day of operation.

6.2 Components of the schedule One of the strengths of the building industry is that standard materials can be used to build almost anything. A brick is standard and it can be used to construct a building of practically any size and shape.

This analogy can be extended to the airline

scheduling process – by defining the constitutive parts it should be possible to combine these parts or components to tailor-make a solution for an airline. Table 5 lists what are considered to be the standard components of the scheduling. process. Table 5: Components of the scheduling process

[8] Abbreviation

Description

MEP


MSeP


AseP


LseP

Leg selection problem

LFAP


LCAP


CPP


CAP


DSP


ASP


ARP


FAP


52

Chapter 6


The standard components as classified by Antes [8] and outlined in Table 5 will be used in the next few paragraphs to explain the scheduling process. The first step in airline scheduling is to evaluate potential markets. This component is called the market evaluation problem (MEP).

This step is important from a

profitability and growth perspective of an airline. [8] After looking at all the possible markets identified during the MEP process, the airline needs to select its target markets, the service levels and the pricing. This is called the market selection problem (MSeP). [8] The result of the market selection exercise may indicate a disconnected network. The airline therefore needs to decide on some of the connecting legs it might want to operate. This is called the leg selection problem (LSEP). This is done usually by taking into consideration the airport selection problem (AseP). Taking into account partnerships and available ground assets the airline will determine the airports in a specific market to which it wants to fly. [8] The next step is to determine the frequencies and capacities used in operating each leg. A frequency for a certain leg gives the number of times in a planning period the leg should be flown. For example, the published OAG schedule shows the frequency per week of flights between an origin and destination airport. This component is called the leg frequency assignment problem (LFAP). A capacity for a leg specifies its total transport capacity for the planning period.

By determining each leg’s

capacity, the leg capacity assignment problem (LCAP) is solved. [8] In order to solve the problem of which crew is to operate which leg, the crew pairing problem (CPP) and crew assignment problem (CAP) have to be solved. A pairing is a sequence of connected legs originating and ending at the same crew base giving duties to a single set of crew. A pairing is constrained normally by different flight and duty rules, for example for every ten hours of flight a crew member must rest eight hours. The CPP means that a minimum cost set of pairing must be found such that each leg is offered by at least one pairing. The crew assignment process assigns individual crew to each pairing, ensuring that each leg is covered. The crew member gets a plan of action for a period of time. This problem is also called the rostering problem. [8] During the scheduling process departure and arrival times for each leg must be selected. This constitutes the departure scheduling problem (DSP) and the arrival 53

Chapter 6


scheduling problem (ASP). These times have to give room for sufficient flying time and enough time to switch planes between legs.

The decision for arrival and

departure times is also influenced by customer preferences and airport restrictions such as curfews. [8] Aircraft might be grouped according to their technical characteristics to form fleets of similar aircrafts. All aircrafts of a fleet are said to be of the same type of aircraft. Examples of aircraft fleets are all planes of the type Boeing 737 or all planes of the type Airbus 320. Airlines normally try to standardise on as few fleets as possible because of aircraft maintenance and training considerations. Given the legs, along with their capabilities, the fleet assignment problem (FAP) determines a fleet for each leg capable of operating the leg and providing sufficient capacity. [8] Fleet assignment is an extremely important building block of the airline scheduling process. It is widely used within the airline industry. American Airlines and Delta have reported 1% profit margins through fleet assignment. Given a flight schedule and an available fleet of aircraft, fleet assignment maximises operating profit subject to the following physical and operational constraints: •

Cover: Each flight in the schedule must be assigned exactly one type of aircraft.

•

Plane count: The total number of aircraft assigned cannot exceed the number available in the fleet.

•

Balance: Aircraft cannot appear or disappear from the network.

Since the schedule of most airlines is to be repeated every week, connected sequences of legs should be found for each aircraft such that the aircraft cycle through the schedule starting at the beginning of each week. Such a sequence chosen for a certain aircraft is called a rotation. The aircraft rotation problem (ARP) is to determine a set of rotations for a part of the decomposed schedule in such a way that each leg is served by a rotation and the maintenance policy, if followed, for each aircraft. [8] The different components of the scheduling process can be combined to form logical groupings. The network design problem is a combination of the airport and leg selection problems along with the leg frequency and capacity assignment problems. The crew pairing and crew assignment problem together form the crew scheduling problem. The flight scheduling problem consists of the leg frequency assignment and

54

Chapter 6


the departure and arrival scheduling problems. Finally, the aircraft routing problem is defined by the problem group consisting of leg capacity assignment, fleet assignment and aircraft rotation. [8]

6.3 Approaches to solving the problem Now that the basic components of the airline scheduling process have been established, the next issue is how they can be used to solve the problem.

The

combination of components A and B is denoted as A⊕B. Sequential treatment of block A followed by B is denoted by A→B. The operator ⊕ has a higher precedence than the operator →. Using this notation, different ways of arranging the basic scheduling components can be found [8]. A small airline can schedule using: LSeP ⊕ LFAP → DSP → ARP A stepwise approach: LSeP ⊕ LFAP → DSP → ARP → CPP ⊕ CAP USAir Approach MSeP ⊕ LSeP ⊕ LFAP → DSP → FAP ⊕ ARP (⊕ ASP) CPP → CAP Suhl’s strategy MEP ⊕ MSeP ⊕ LSeP ⊕ LFAP ⊕ LCAP → DSP ⊕ ASP → FAP ⊕ ARP → CPP ⊕ CAP In this section all the different building blocks that make up the process of generating an airline schedule were explained. There are computerised tools to solve each of these problems. In the next section some of the more common tools available will be investigated.

6.4 Tools available Table 6 lists some of the Electronic Data Systems (EDS) applications that match the different scheduling components.

55

Chapter 6


Table 6: Available tools

Abbreviation

Description

Product

MEP


MseP


AseP


Sabre SlotManager

LseP

Leg selection problem

Sabre Through Assignment Model

LFAP


Sabre PC Airflite

LCAP


Sabre PC Airflite

CPP


Sabre Crew Management Sabre Crew Pairing Management System

CAP


Sabre Crew Management Sabre Crew Manpower Planning Sabre Bid-line Optimization

DSP


Sabre SlotManager

ASP


Sabre SlotManager

ARP


Sabre Aircraft Routing Model

FAP


Sabre O&D FAM

For any airline there should also be supporting processes to facilitate the scheduling process. Table 7 lists the systems that support the scheduling process. These range from publication, forecasting and profitability tools to simulation applications that mimic the schedule on the reservation terminals.

56

Chapter 6

Detailed airline scheduling Table 7: Supporting products

Description

System

Schedule publication

Airsync Simwise

Airline reservation simulation. This product simulates, from the CRSSim schedule, where a flight will end up on the computer reservation systems. Airline profitability model

APM

Integrated forecasting system

IFS

6.5 Conclusion This chapter gave an overview of the main components of the airline scheduling process and the approaches to solving the scheduling problem.

By treating the

scheduling components as building blocks, solutions can be found by arranging and combining them in different configurations. An overview was also given of some of the commercially available scheduling tools.

In this chapter creating a perfect

schedule was studied; in the next chapter how to deal with disruptions to this schedule will be discussed.

57

7

Airline schedule disruptions

In chapter 7 the role that a software agent based on rule-based and learning algorithms can play to assist an airline with schedule disruption problems will be defined. During the last decade airlines focused on developing efficient flight schedules with very little slack built into the system [100]. This makes it difficult for airlines to accommodate any changes to the schedule. Planned schedules often get disrupted by factors like unavailability of crew, extreme weather and technical problems. Without slack in the schedule, the schedule has to change to incorporate the day-to-day realities. Airline disruption management, often known as operational scheduling, focuses on how to handle this problem. With the renewed focus on airline costs, interest in developing cost-effective mechanisms to deal with this problem has increased.

7.1 Disruption management In this chapter the focus is on how to maintain the schedule which is barraged by a multitude of daily operational realities. The operational scheduling department at any airline is responsible for handling all disruptions to the schedule. Typical disruptions include mechanical problems, severe weather, crew getting sick, industrial action, congestion at airports, etc. Operational schedulers are supposed to solve problems caused by the disruption in a matter of minutes. This means making changes to the schedule which took an array of specialists months to construct. Solving one problem normally has an effect on the other flights in the systems (domino effect). The operational scheduler must try to minimise this impact. Whenever a disruption occurs, the operational scheduler must try to achieve the following conflicting objectives [100]: •

Operational problems: Total flight delay (time) as well as the number of flights delayed must be minimised.

•

Commercial problems: The total number of passengers affected by the disruption must be minimised.

Chapter 7 •


Financial problems: The financial impact of the disruption must be minimised. The financial impact includes cost incurred by the airline for booking passengers on other flights, booking passengers into hotels when no flights are available etc.

•

Get back to plan as soon as possible: Getting back to plan is conventional airline wisdom. In some cases it is necessary because it may not be possible to change crew schedules without advance notice. It may, for example, be essential to ensure that a crew member can fly the rest of his/her duties even if the current assignment is disrupted.

The original plan should have been optimal in the

situation before the disruption occurred.

7.2 Operations control Most large airlines have an operations control centre (OCC) which is used to monitor the day-to-day coordination of the schedule execution. Representatives from the various key airline functions work together in this department to ensure the airline’s operations are handled smoothly. Typically the OCC is responsible for [100]: •

Flight dispatch: The flight dispatcher follows the preparation and progress of a number of flights and raises alerts with other areas when problems occur.

•

Crew tracking: Crew check-ins must be monitored and crew pairing must be changed in case of delays or cancellations. The crew tracking is normally divided into cockpit and cabin crew.

•

Technical operations: The technical operations representatives are normally responsible for the unplanned service and maintenance of the aircrafts.

•

Passenger service: The OCC has the responsibility to ensure that passenger inconvenience is taken into account during the normal decision-making process.

When developing solutions to airline schedule disruptions, the main user will be the OCC.

7.3 Dealing with schedule disruptions There are basically two ways to deal with schedule disruptions: •

Build robustness into the schedules

59

Chapter 7 •


Re-optimise the schedule after disruptions occur

When considering building robustness into the schedule it is a fact that delays propagate along aircraft routes. Only limited slack can be added. If the slack is located appropriately, delay propagation can be prevented.

By routing aircraft

intelligently, slack can be allocated better. This means that slack is added where advantageous and reduced where less needed. Rosenberger, Johnson and Nemhauer [101] use the concept of robustness during the schedule design and maintenance scheduling process. Yen and Birge use a similar approach for crew scheduling [102]. Re-optimising after schedule disruption is the other approach to dealing with schedule disruptions. Several automated approaches can be used to help with this. The most common approaches are based on operational research techniques [103] and expert system technology [104]. The main advantage of using a computational approach is that there is less room for mistakes. In complex cases it is much faster than humans. The computational approach can also capture human expertise. In most airlines, where operational schedulers are a scarce commodity, the capturing of human expertise is certainly an advantage.

7.4 Conclusion Chapter 7 gave a very brief introduction to the problem of disruption scheduling. To alleviate the problem of disruptions the airline planning departments can build robustness into the schedules. The department responsible for disruption management on the day of operation is the OCC. They have the responsibility for re-optimising the schedule after a disruption occurs. In a later chapter the concept of reinforcement learning and rule-based systems will be used to solve the problem of how to reoptimise the schedule.

60

8

A solution to airline schedule disruptions

In this chapter the role that learning algorithms can play in the operational scheduling environment is defined. The rationale underlying the learning approach is based on the desire for adaptive decision making in highly stochastic environments. This chapter examines some of the theory behind learning algorithms, in particular reinforcement learning.

It will also be shown how a particular instance of

reinforcement learning, viz. Q-learning, can be applied to the operational scheduling problem.

8.1 Introduction The operational scheduling department at any airline is responsible for handling all disruptions to the operational schedule. Solving one problem caused by the disruption normally has an effect on the other flights in the systems (domino effect). The operational scheduler must try to minimise this impact. Several automated approaches can be used to solve the operational scheduling problem. Examples of such approaches are [105]: •

Production systems, e.g. intelligent rescheduling systems.

•

Operational research techniques.

•

Decision trees built by learning algorithms.

•

Hybrid systems.

In Chapter 7 the components of the operational scheduling problem and the computational approaches to solve this problem were presented. In this chapter a hybrid approach is proposed to solve the operational scheduling problem. Static computational algorithms are not sufficient to solve the operational scheduling problem effectively, but combined with a learning approach a dynamic solution to the problem can be created. With the dynamic approach the changing preferences of operational schedulers can be captured.

In the normal operational scheduling environment production systems

(rule-based systems) generate a set of solutions to the scheduler. They formalise

Chapter 8


knowledge into rules containing an IF part and a THEN part (also called a condition and action) [106]. These solutions are ranked with regard to some fixed objective function [104]. The learning algorithm will capture which choices the user made and under which circumstances those choices were made. This process is dynamic in the sense that as user preferences change over time, the learning algorithm will capture this change in behaviour. The combination of a production system with a learning system captures both the knowledge of human schedulers (heuristics) as well as their behaviour (which solution they will choose under which circumstances).

Heuristics

Production system

User picks a solution

Agent learns user response Agent makes suggestion as to best solution

Learning system

Set of solutions

Problem

Figure 11: Hybrid system

Figure 11 gives a schematic high-level overview of the hybrid system solution, which combines the features of a production or an expert system with those of a learning system. For the problem domain of this research, the “Problem” block represents the disrupted schedule, the “Heuristics” block the rules of the expert or production system, and the “Set of solutions” the set of solutions that can be followed to fix the disrupted schedule. The difference between a pure production system and a hybrid system is summarised in Table 8.

62

Chapter 8


Table 8: Differences between a production system and the proposed hybrid system

Production system Strengths

Hybrid system Strengths System is dynamic.

Rules are easy to understand.

All the strengths of a production system, except inferences and explanations are more difficult to derive.

Inferences and explanations are easily derived. Modifications and maintenance are relatively easy. Weaknesses

Could require fewer rules than a production system. Weaknesses

Complex systems require many rules.

Inferences and explanations are more difficult to derive.

System is static. Difficult to combine uncertainty with rules.

8.2 Reinforcement learning In Chapter 4 machine learning in general was considered. In this section the material already presented will be expanded on and it will be shown how to adapt it for the operational scheduling problem. A simple model of agent-environment interaction will be discussed, followed by a review of the principles of Markov decision processes and Q-learning. Techniques based on reinforcement learning have been used to build systems that learn to perform non-trivial sequential decision tasks. Most of this work has focused on learning tasks that can be described as Markov decision processes. A Markov decision process intuitively corresponds to a control task in which at each point in time (a) the system directly observes the state of the environment and (b) the effects of actions depend only upon the action and the current state [84]. Figure 12 shows a model of system-environment interaction which is widely used in reinforcement learning. In this model the agent and the environment are represented

63

Chapter 8


by two synchronised finite state automata interacting in a discrete time cyclical process [107]. At each time point, the following sequence of events occurs: 1. The system senses the state of the environment. 2. Based on this current state, the system chooses an action to perform. 3. Based on the current state and the action selected by the system, the environment makes a transition to a new state and generates a payoff. 4. The payoff is passed back to the system.

Environment Rejectproposal

Payoffs

Actions

System

Figure 12: System and environment interaction

[107] At each time step the system senses the current state, selects an action, observes the new state and rewards that result. Rewards are used as feedback for learning. One way to specify the system behaviour is in terms of a control policy which describes for each state which action to perform.

In reinforcement learning, the agent' s

objective is to learn a control policy that maximises some measure of the total reward accumulated over time.

8.2.1 Q-learning Q-learning is an incremental reinforcement learning method. It is a good representative for reinforcement learning because it is simple, mathematically well founded and widely used [107]. A simple Q-learning algorithm is illustrated in Figure 13. The first step of the algorithm is to initialise the system' s action-value function Q.

64

Q is the agent' s

Chapter 8


estimate of the optimal action-value function. If prior knowledge about the task is available, the information may be encoded in the initial values, otherwise the initial values can be arbitrary.

After initialisation the system enters the main

controlling/learning loop. The first step is to sense the current state, x. Next, the agent selects an action a to execute. Most of the time, this action will be the action specified by its policy f(x), but for the purposes of exploration the system will occasionally choose an action that appears suboptimal. For example, policy f might be followed with probability p or a random action may be chosen. Exploration is necessary to guarantee that the system will eventually learn an optimal policy. The system then executes the selected action and notes the reward r and states y that results. Finally, the action-value estimate for the state-action pair (x, a) is updated. Q

a set of initial values for the action-value function (e.g. uniformly zero).

Repeat forever: x

the current state 1. Select an action a to execute that is usually consistent with f(x) but occasionally an alternate. 2. Execute action a, and let y be the next state and r be the reward received. 3. Update Q(x, a): Q(x, a)

(1- )Q(x, a) +

[r + U(y)],

where U(y) = Q(y),f(y)). Here for each x ∈ S:f(x)

a such that Q(x, a) = maxb a Q(x, b).

Figure 13: The Q-learning algorithm (Watkins & Dayan)

[108] Q(x, a)

(1- )Q(x, a) + [r + U(y)] is a weighted sum combining the one-step

estimate [(r + U(y))] with the old estimate for Q(x, a). In this formula a is the learning rate, r is the reward and y is a discounted reward factor. Discounted reward means that rewards received s steps hence are worth less than the reward received now, by a factor of

S

(0 < < 1). Q-learning is guaranteed to converge to an optimal

policy for any finite Markov decision process if, in the limit, every state action pair is 65

Chapter 8


tried infinitely often and if the learning rate decreases according to a proper schedule [108]. In the application of the Q-learning algorithm in this research, the schedule by which the learning rate decreases is

αn =

1 where n>1. Watkins and Dayan proved that given learning rates 0 ≤ αn < 1 n

and ∞

α n ( x , a ) = ∞,

∞

i

i =1

i =1

[α ni ( x ,a )]2 < ∞, ∀x, a then Qn ( x, a ) → Q * ( x, a ) as n → ∞,∀x, a

with a probability of 1 [108].

A simplified version of the algorithm by Watkins and Dayan referred to by Mitchell is shown in Figure 14. For each s, a initialise the table entry Qˆ (s, a) to zero. Observe the current state s Do forever: •

Select an action a and execute it

•

Receive immediate reward r

•

Observe the new state s

•

Update the table entry for Qˆ (s, a) as follows:

Qˆ (s, a)

r+

Qˆ

max

Qˆ (s',a'),

a'

•

s

s'

Figure 14: The Q-learning algorithm (Mitchell)

[84]

66

Chapter 8


In the algorithm by Mitchell the symbol Qˆ refers to the learner’s estimate of the actual Q function. Later in this chapter the robustness of these two algorithms will be compared.

8.2.2 Application of theory to solve the operational scheduling problem In the previous section a brief look was taken at the theory behind reinforcement learning and, in particular, Q-learning. In this section the application of this theory to the operational scheduling problem is discussed. The main tenet of this section is to describe how a schedule can be represented as a state, as required by the Q-learning algorithm. The basic purpose of the learning algorithm is to learn which schedule a user will select under which circumstances. In real life the user is presented with a set of schedules, all of which are valid solutions to a problem. The user will select only one of these schedules as a final solution. The aim is to learn which solution will be chosen. The environment must be modelled as a Markov decision process.

A Markov

decision process is described by the tuple (S,A,T,R), where S is a set of possible states, A is the set of possible actions, T is the state transition function and R is the reward function. At each time, the environment occupies exactly one state from S, and accepts one action from A. S and A are usually assumed to be discrete and finite. For the operational scheduling process these variables can be defined as: S:

A set of numerical values representing a schedule.

A:

Choose a solution which has the shortest time to get back to normal. Choose a solution where the least number of flights is delayed (magnitude). Choose the solution which has the least number of commercial penalties (cost).

T:

This function gets the next set of solutions (schedules).

R:

The reward is 2 if the system' s choice of schedule corresponds to that of the user, else -1. R ∈ {-1, 1}.

67

Chapter 8


The set of solution schedules must be coupled to a state. This state is a numeric value which forms part of set S. One way of achieving this coupling is to see each schedule as a set of attributes. The attributes of a schedule are: R – Time to recovery M – Number of flights delayed C – Cost

With a suitable heuristic, the real values of R, M and C can be projected to any value between 0 and 9. Bi =Ci + 10Mi + 100Ri where i is the index of the schedule. The state values of each of the solutions can be added to form the state value S. Whereas Bi is representative of the state of a single schedule, S is the representative state value of the set of schedules presented to the user. If n is the number of solutions, then S =

n

i −1

10 Bi

i =1

A further enhancement to the algorithm is to make the values of S unique in the sense that a state value S of 1234 is the same as 4321, or any other combination of the same digits.

8.3 Comparison of algorithms In the previous section two algorithms were outlined that implement the basic Qlearning algorithm, i.e. the one presented by Mitchell and the one by Watkins and Dayan. In this section the performance of these algorithms in two different situations is compared. Assume that the user consistently makes the same choice when presented with the same set of schedules. Both learning algorithms should converge. Figure 15 shows the convergence rates in this situation.

68

Chapter 8


User Choice with No Mistake 2.5

2

1.5

1 Q(x,a) 0.5

Watkins & Dayan Mitchell

0 1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

-0.5

-1

-1.5 Epoch

Figure 15: Convergence rates when users make no mistakes

Inconsistent User Choice (25%) 2.5

2

1.5

1 Q(x,a) Watkins & Dayan Mitchell

0.5

0 1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

-0.5

-1

-1.5 Epoch

Figure 16: Convergence rates when users make mistakes 25% of the time

Assume that users do not consistently select the same schedule but 25% of the time they make another choice. This is closer to a real-life situation, as people inherently do not make consistent choices in difficult situations.

69

Figure 16 shows the

Chapter 8


performance of the algorithm proposed by Mitchell and the one by Watkins and Dayan. Both algorithms still try and converge, but the recovery rate for the Watkins and Dayan algorithm is smoother.

8.4 Conclusion In this chapter it was shown how learning algorithms, specifically Q-learning, can be used to create a dynamic system to help solve the operational scheduling problem. The solution proposed is a hybrid system which combines production system technology with a learning system. Production systems are typically systems in which the knowledge representation method is formalised into rules containing an IF part and a THEN part (also called a condition and an action).

The mathematical

foundation of the Q-learning algorithm was shown to be well founded. A description of the encoding schemes for the schedule was given and it was shown that the model can be used to learn the choices of the user effectively.

70

9

A matrix, market-driven scheduling model

In this chapter an organisational structure is presented that is able to create a flight schedule.

9.1 The model for schedule creation To solve the airline scheduling problem, the use of a market-driven approach is proposed.

Market price systems constitute a class of mechanisms that provide

effective decentralisation of decision making. Most existing methods for scheduling in airlines are based on centralised or hierarchical decision making. This is called monolithic scheduling. In the market-based approach the scheduling model is based on a distributed and locally autonomous decision structure. The basic idea is to localise and distribute the functionality of scheduling, leaving the complexity of operational decisions to local decision makers, while maintaining a simple and generic central coordination mechanism [109]. The capabilities of the agents and their organisation in an overall decision-making structure determine the behaviour of the system as a whole. Because it concerns the collective behaviour of self-interested decision makers, the design of this decentralised structure is fundamentally an exercise in economics or incentive engineering. In a market-orientated approach to distributed problem solving, the activities and resource allocations for a set of computational agents are derived by computing the competitive equilibrium of an artificial economy [110]. In the model proposed here scheduling decisions are localised and distributed. The complexity is left to local decision makers.

A simple and generic coordination

mechanism must be put in place to ensure that there is coordination between all the decision makers. Local decision makers base their decisions on a local utility (i.e. preferences, objectives and constraints).

Each local decision maker has a local

optimisation problem which is to maximise expected total reward subject to the local constraints. Each local decision maker makes a decision which he/she thinks will optimise the utility function. This decision is sent as a bid to the central coordination mechanism, which in this case is a type of auctioneer. The auctioneer uses these bids

Chapter 9


to perform resource allocation based on an auction process. The aim of the auctioneer is to optimise a global utility function which maximises the efficiency of the overall system. A totally decentralised agent model can handle the schedule creation problem [105]. A shortcoming of this approach is that there is no global optimisation of the schedule. This model consists basically of airport and aircraft agents. The airport agents are simply required to move a number of passengers from one airport to another. The airport agent therefore represents a matrix showing how many people want to fly from that specific airport to another (see Table 9 for an example of such a demand matrix). In the decentralised agent model, aircraft agents bid directly against each other to satisfy the requirement to move a number of passengers from one point to another. For example, if Airport A has a demand for 100 people at 7:00 a.m. to fly to Airport B, several aircraft can bid to satisfy this demand. If aircraft X bids with the best-cost structure, the bid would then be awarded to aircraft X. The schedule will then show “7:00AM flight from Airport A to Airport B using aircraft X”. The problem with this approach is that each agent tries to optimise a subset of the problem, without looking at the effect on the global solution. To overcome this problem, an organisational structure has to be introduced to regulate agent behaviour. The new model consists of airport agents, a business manager, a resource manager, a delivery manager and aircraft agents. The organisation created will have one aim: to maximise profit (defined here as revenue less cost). This organisation will be called NewCo. NewCo has as officers a business manager, a resource manager and a delivery manager. Employees of NewCo consist of aircraft agents which join the company from time to time, depending on the business need. Customers of NewCo are the airport agents. The organisation structure is based on a matrix. Each aircraft agent is under the control of a resource manager. Only resource managers can hire or fire aircraft agents. Figure 17 gives a very simple high-level overview of the resource manager, business manager, aircraft and airport agents.

72


Airport 1

Business manager

Resource manager

Aircraft 1

Aircraft 2

Aircraft n

Demand

Chapter 9

Airport 2

Airport n

Figure 17: Static agent organisation framework

Whereas Figure 17 shows the static interrelationship between the different agents, Figure 18 shows a high-level overview of the interaction between the different entities. Airport

Aircraft

Business manager Register

Res. manager

Register

Publish demand Recommend Negotiate Schedule flight

Manage capacity

Figure 18: Dynamic agent organisation framework

73

Get info

Schedule

Chapter 9


In later chapters these interactions will be dealt with in much more detail. The next few paragraphs describe the main entities.

Airport agent: The airport agent needs to fly passengers from one specific airport to another at a certain time. The airport agent builds the demand up through marketing analysis, etc. The demand vector for an airport is reflected in Table 9. An airport agent registers its demand with NewCo’s business manager.

Table 9: Demand matrix

Departure time

To Airport 2

To Airport 3

(Cape Town)

(Johannesburg)

07:00

100

180

08:00

70

100

Business manager agent: The business manager agent ultimately assigns aircrafts to fulfil the need of the different airports. The business manager makes the different demand matrices available to the aircraft agents. The aircraft agents bid for these different routes. The business manager evaluates the different bids and assigns an aircraft to an airport. For every passenger flown, the business manager receives revenue from the airport agents and pays a fee to the aircraft agent. The business manager will therefore try and optimise the profit of NewCo.

Resource manager agent: An aircraft agent contracts directly with the resource manager. If demand exceeds supply, the resource manager will add aircraft agents to the system. If there are too many aircraft agents in the system, the resource manager agent will reduce the number of aircraft agents, or if there are too few, the manager might increase the number.

Aircraft agent: The aircraft agent is an agent which represents a real aircraft. Each aircraft has the capacity to carry different numbers of people. An aircraft agent has a cost structure which consists of a fixed overhead plus an incremental cost per passenger.

74

Chapter 9


The next section will show how the model described in the previous paragraphs can operate in reality.

9.2 A practical example In this example there are only three airport agents, viz. Cape Town, Durban and Johannesburg.

These agents represent real airports in South Africa.

The

Johannesburg airport will serve as a base. This means that all aircraft will initially depart from Johannesburg. Table 10 shows the demand from one airport to another at a given time. The left-most vertical column shows the departure airport and the other vertical columns show the destinations. The intersection between the departure and arrival airports shows the passenger demand. Table 10: Demand from one airport to another at given time

07:00 Johannesburg

Cape Town

Durban

100 0 80

50 75 0

Johannesburg Cape Town Durban

08:00 Johannesburg 0 170 100

Cape Town 120 0 100

Durban 70 105 0


09:00 Johannesburg 0 150 90

Cape Town 110 0 90

Durban 60 95 0


0 150 80

From 24:00 that morning the following aircraft are available: 1 x 200 seater 2 x 150 seater 2 x 100 seater 1 x 50 seater

75

Chapter 9


For the purpose of this study it is assumed that all aircraft fly at the same speed and that the flying time is the same between all the airports, i.e. one hour. To start with, the airport agents register their requests with the business manager agent. The business manager agent then puts the flight up for auction. The most successful bid will be the one with the lowest overhead and can fly the most passengers. Another way of looking at this is to maximise the load factor. Using these rules, the following flights are scheduled at 07:00: Johannesburg - Cape Town (100 seater) Johannesburg - Durban (50 seater) This solves the demand at 07:00 at Johannesburg, but what about the other airports? No other airport has an aircraft available to schedule a flight. The business manager therefore puts out a request to bid to the aircraft agents. The time for which they want to bid is 06:00 because a flight between any of the airports takes an hour. Agents are therefore asked to bid for a 06:00 flight from Johannesburg airport to one of the other airports (as all the aircraft are located at Johannesburg). The aircraft that might bid are: 150 seater: because there is a 07:00 flight from Cape Town to Johannesburg. 100 seater: because there is a 07:00 flight from Durban to Johannesburg. 100 seater: because there is a 07:00 flight from Durban to Cape Town. 50 seater: because there is a 07:00 flight from Cape Town to Durban. The business manager agent awards the 150 seater to the Cape Town – Johannesburg flight, the 100 seater aircraft to the Durban - Johannesburg flight and the 50 seater to the Durban – Cape Town flight. The flights that are scheduled at 06:00 are then: Johannesburg – Cape Town (150 seater) Johannesburg – Durban (100 seater) Johannesburg – Durban (100 seater) Johannesburg – Cape Town (50 seater) After the bidding process the following flights can be scheduled at 07:00: Cape Town – Johannesburg (150 seater)

76

Chapter 9


Durban – Johannesburg (100 seater) Durban – Cape Town (100 seater) Cape Town – Durban (50 seater) This whole process is repeated at 08:00. As can be seen, agents do not have to bid for aircraft all the time. The demand can be satisfied using resources that are available at the airport at the time the flight must depart.

9.3 Evaluation Instead of a purely market-based organisational structure where agents only use market mechanisms to compete with each other, a shift has been made to a hierarchy where agents still contract using a market-based mechanism but in a more controlled way. Coordination in this model is based on supervision and a price mechanism. The model presented is a unique solution for scheduling flights and comprises a business manager agent, aircraft agents and airport agents and a resource manager agent. The business manager agent assigns aircraft software agents to flights from airports represented by airport software agents using an assignment methodology that attempts to minimise the cost of a particular flight and minimise cost across all of the flights. Each aircraft agent represents a physical aircraft and it includes the capacity of the aircraft, costs per passenger and the overhead costs of flying the aircraft regardless of the number of passengers on a flight. It submits bids to the business manager to service the flights that the business manager agent is scheduling. Each airport software agent represents an airport in a geographic location and it includes basic passenger demand information, at a minimum an indication of a destination airport, a departure time and the number of passengers for the flight. It communicates the flight information to the business manager agent for flights that are not assigned an aircraft software agent and need to be scheduled. In a purely market-based environment it is difficult to ensure that the integrity of the agents is preserved, i.e. agents might exploit the system. By introducing the resource manager concept, agents can still join or leave the organisation, but the difference is that they are under supervision. The resource manager will add and remove aircraft

77

Chapter 9


agents based on the profitability of these agents. Secondly, the problem of look-ahead in the schedule, which is similar to the greedy algorithm approach to solving the shortest path problem, is better resolved by introducing the business manager agent concept.

9.4 Comparison with the standard FIPA CNP Bus. man.

Res. man.

Aircraft

Schedule

Call for proposals Refuse Deadline Propose Reject-proposal Accept-proposal Failure Inform: done Inform: result Update Terminate X Continue

Ask Response

Create Aircraft

Figure 19: Matrix CNP

If the ideas of a matrix control structure are applied to the standard CNP, the diagram as presented in Figure 19 is produced. This is an extension of the FIPA CNP as presented in 2.2.1. 78

Chapter 9


The differences between what was presented and this extension is that in Figure 18 the business manager recommends an appropriate airport agent and aircraft agent. The negotiation then happens between the airport agent and the aircraft agent. This takes the processing load off the business manager, as the negotiation is not centralised. In Figure 19 the airport agent’s demands are represented by the business manager and the business manager handles the negotiation with the aircraft agents. The model presented in Figure 19 can be used as a general model for problems that combine elements of resource allocation and capacity management. The business manager agent can be replaced by an entity that matches supply and demand between producer and consumer agents. Producer and consumer agents represent any entities that produce and consume goods. Examples of this could be computer processors and the applications running on them, manufacturing equipment and the jobs to be executed by manufacturing equipment, etc. The aircraft agent then represents these producer and consumer agents. The resource manager can be replaced by an agent that manages the supply capacity and the schedule agent by an agent that can act as a central repository of information.

9.5 Conclusion Up to now the airline planning problem was decomposed into more manageable problems and then sequentially solved. With the approach outlined in this chapter this somewhat artificial approach is avoided by creating a flyable schedule in an integrated fashion. An agent-based model that is based on a matrix model of control to schedule aircraft was extended.

The FIPA CNP was extended and it was shown how this is a

generalisation of the scheduling model used in this research.

79

10 Allocation of aircraft through the use of a business manager agent This chapter discusses how the business manager agent assigns aircraft to match the demand of the different airports. The results of aircraft allocation with few and many aircraft will be shown to see if the simulation models provide results that are better than just the arbitrary assignment of aircraft.

10.1 Overview of the environment Figure 20 illustrates the different components that will be used in this chapter to explain how the business manager allocates resources.

Aircraft configuration file

Aircraft agent n

Simulation environment

Business manager Agent Airport agent n

Demand file

Figure 20: Schematic overview of the simulation environment

A more detailed Unified Modelling Language-based architectural overview and a clearer description of all the agents and components will be given in Chapter 12. For the purposes of this chapter a brief overview is given of how all of the items fit together to create the scheduling environment.

Chapter 10

Allocation of aircraft through the use of a business manager agent

A rough overview of the process flow is as follows: 1. The simulation environment, in this case the main Java application, reads an aircraft configuration file. This file specifies how many aircraft agents should be created initially. 2. The simulation environment creates the business manager agent. 3. The simulation environment creates three airport agents, namely JNB, DBN and CPT.

These are the International Air Transport Association (IATA)

abbreviations for the Johannesburg, Durban and Cape Town airports in South Africa. This example will be limited to these three agents. 4. Each airport agent registers its demand with the business manager agent. 5. Each aircraft agent negotiates with the business manager agent to service a demand from one airport to another. 6. The business manager agent allocates an aircraft to a demand.

10.2 Aircraft allocation model Each aircraft has a capacity. As a first pass, an effort should be made to match the appropriate capacity to the demand. The airline would typically try to maximise the number of passengers carried, minimise the number of passengers spilled and maximise the load factor. ‘Passengers spilled’ refers to the number of passengers that could not be transported from one airport to another. Take, for example, an aircraft with a capacity of 50 passengers allocated to the Durban to Cape Town route, for which there is a demand of 75 passengers. For this specific flight the number of passengers carried would be 50 (as the aircraft could not carry more), the number of passengers spilled would be 25 and the load factor for this specific flight would be 100%. In this model the aircraft with the closest capacity to the demand is allocated. In the negotiation model the aircraft bid with a price for an origin-destination pair. This is simply a function of the capacity of the aircraft, as the price the aircraft will bid is

81

Chapter 10


proportional to the capacity.

In the following section the simplified results of

allocating according to capacity are shown.

10.3 Experimental set-up In the following section the different experiment runs testing the allocation method of aircraft to passenger demands are presented. An experiment comparing the allocation model with a random model for three and for six aircraft is run. The reader might ask why so many iterations of the experiment need to be run, as algorithms are normally deterministic in nature. In the model agents are used which have separate states of execution from one another. These agents submit bids to the business manager agent, which then matches the bids (aircraft) to the demand (airports). Because the agents are implemented as separate threads, it means that the timing of the requests is no longer deterministic but random. The business manager agent makes the allocation at the moment it receives the requests. Due to this randomness, the output of the schedules has to be compared many times to see if there is really a trend or just a random occurrence.

10.4 Experimental results using three aircraft In the first experiment only three aircraft with the capacities shown in Table 11 are used. The break-even factor has not been used. Table 11: Sample aircraft configuration file

Aircraft name

Capacity

Base airport

Time from Breakwhich aircraft is even available factor

ZAQA20

20

JNB

01.01.2004.00.00 0.7

ZAQB50

50

CPT

01.01.2004.00.00 0.7

ZAQC150 75

DBN

01.01.2004.00.00 0.7

The demand for 1 January 2004 from 00:00 to 23:00 is shown in Table 12. In this section an extract is shown. The full demand information can be viewed in Appendix A.

82

Chapter 10


Table 12: Demand information for January 2004 from 00:00 to 23:00 (extract)

Date

Origin

Destination Passengers

01.01.2004.00.00 CPT

DBN

5

01.01.2004.00.00 CPT

JNB

20

01.01.2004.00.00 DBN

CPT

5

01.01.2004.00.00 DBN

JNB

10

01.01.2004.00.00 JNB

CPT

20

01.01.2004.00.00 JNB

DBN

10

01.01.2004.01.00 CPT

DBN

5

01.01.2004.01.00 CPT

JNB

20

01.01.2004.01.00 DBN

CPT

5

01.01.2004.01.00 DBN

JNB

10

The business manager agent, using the resource allocation mechanism outlined above, then generated the following schedule: Table 13: Schedule generated for three aircraft (extract) Origin

Destination

Departure time

Arrival time

Aircraft

Pax

Capacity

Spill

JNB

CPT

Thu Jan 01 00:00:00 GMT 2004

Thu Jan 01 01:00:00 GMT 2004

ZAQA20

20

20

0

CPT

JNB

Thu Jan 01 00:00:00 GMT 2004

Thu Jan 01 01:00:00 GMT 2004

ZAQB50

20

50

0

DBN

JNB

Thu Jan 01 00:00:00 GMT 2004

Thu Jan 01 01:00:00 GMT 2004

ZAQC150

10

75

0

JNB

CPT

Thu Jan 01 01:00:00 GMT 2004

Thu Jan 01 02:00:00 GMT 2004

ZAQB50

20

50

0

JNB

DBN

Thu Jan 01 01:00:00 GMT 2004

Thu Jan 01 02:00:00 GMT 2004

ZAQC150

10

75

0

CPT

JNB

Thu Jan 01 01:00:00 GMT 2004

Thu Jan 01 02:00:00 GMT 2004

ZAQA20

20

20

0

CPT

JNB

Thu Jan 01 02:00:00 GMT 2004

Thu Jan 01 03:00:00 GMT 2004

ZAQB50

20

50

0

DBN

JNB

Thu Jan 01 02:00:00 GMT 2004

Thu Jan 01 03:00:00 GMT 2004

ZAQC150

10

75

0

JNB

CPT

Thu Jan 01 02:00:00 GMT 2004

Thu Jan 01 03:00:00 GMT 2004

ZAQA20

20

20

0

83

Chapter 10


By generating ten iterations of the algorithm outlined above and comparing them with a random schedule generated, the following results were achieved: Table 14: Results of comparison

Iteration Pax

Spill

Load factor

1

2 245

685

64

2

2 405

825

63

3

2 245

685

64

4

2 205

825

63

5

2 205

825

63

6

2 205

825

63

7

2 370

510

68

8

2 205

825

63

9

2 245

685

64

10

2 205

825

63

Average

2 253.5

751.5

63.8

The results of the random allocation were as follows: Table 15: Results of random allocation

Iteration

Pax

Spill

1 2 3 4 5 6 7 8 9 10

2 190 2 135 1 990 2 100 2 055 1 900 2 090 1 955 2 190 2 080

1 580 2 030 1 590 1 660 1 915 1 830 1 605 1 580 1 665 1 730

Average

2 068.5 1 718.5

Load factor 62 61 57 60 59 54 60 56 62 59 59

By comparing the two samples using a simple T-test with a 95% confidence factor, the results in Table 16 were obtained.

84

Chapter 10


In this case the null hypothesis is that the allocation mechanism is no improvement over the random assignment with regard to the number of passengers carried. As this is a one-tail calculation, the result for the “T-Stat” parameter should be greater than the “t Critical one-tail” value. In this case it is significantly higher. Another way of putting this is that the chance that the allocation model is not a fluke or a lucky chance is 99.99% (1-0.000077).

Table 16: T-test - two-sample assuming unequal variances for few aircraft (Pax) Best Total number of passengers carried (mean) Total number of passengers carried

Random

2 253.5

2 068.5

5 389.167

9 239.167

10

10

(variance) Observations Hypothesised mean difference

0

Df

17

t Stat

4.83698

P(T