Oct 2, 1994 - Este documento describe brevemente los componentes del c odigo fuente del Cache ... Los principales m odulos y funciones de cada.
facultad de informatica
universidad politecnica de madrid
Implementation of a Parallel File System: CCFS a Case of Study J. Carretero F. Perez P. de Miguel F. Garcia L. Alonso
TR Number FIM/84.1/DATSI/94
h ( X h (" a ! hX X (! b hhh ( X a c # ( H
% ? @ ( e J BAE H ( b aaX " ! ( Xhhhhh H ( c # ! ( b " (
% aaXXXX hh H ? @ ( e ! J A B E ( ( b c " # ! H ( XX hhhhhh !"#? % E BAJe @cbHaa ((( ( ! XXX hhhh ( H aa "#?% ((( XXX hhhh " !! E B AJe@cbbHH a ! ( ((( XX ! " #?% E B A Je@c b H aa ! X b " XXX H # ! E B A J e@cc b HHaaa XX !! "" # ?% b X a !
% H ? @ e J A B E b aa c " # ! H ! b "
% a H c ? @ # ! e J A B E aa b " ! # ? % b HHH E B A J e @ c aa !! "" b c # ! aa HH ? % ! E B A J e @ b " c # ! a b " ! % HH ? @ e J A B E c # b " % b HH c ? @ " # e J A B E b " c # H ? % A J e @ B E b " H c # b "
% ? @ e J A B E b c " #
b % ? @ " e J A B E c # b " % c ? % # e JJ AA BB EE b
c ? @ # " e @
c Facultad de Informatica de Madrid 1994 This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for non-pro t educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of the Facultad de Informatica de Madrid, UPM, , in Campus de Montegancedo, Madrid, Spain; an acknowledgement of the authors and individual contributors to the work; and all applicable portions of the copyright notice. Copying, reproducing, or republishing for any other purpose shall require a license with payment of fee to the UPM - Universidad Politecnica de Madrid. All rights reserved.
Implementation of a Parallel File System: CCFS a Case of Study Technical Report Number: FIM/84.1/DATSI/94
10-02-1994
Department
Dep. de Arquitectura y Tecnolog'ia de Computadores Keywords Parallel le systems, multicomputers, multiprocessors, distributed processing, disk systems, RAIDs Publication Notes Project partially supported by the GPMIMD Esprit Project Address Boadilla del Monte, 28660 Madrid email: jesus@ .upm.es
Abstract
This document brie y describes the components of the Cache Coherent File System (CCFS) source code. CCFS has three main components: Client File Server (CLFS), Local File Server (LFS), Concurrent Disk System (CDS). The main modules and functions of each component are described here. Special emphasys has been put on interfaces and data structures.
Resumen
Este documento describe brevemente los componentes del codigo fuente del Cache Coherent File System (CCFS). CCFS tiene tres componentes principales: Client File Server (CLFS), Local File Server (LFS), Concurrent Disk System (CDS). Los principales modulos y funciones de cada componente son descritos en este documento. Especial enfasis se ha dedicado a los interfases y a las estructuras de datos.
Contents 1 Introduction 1.1 1.2 1.3 1.4
CCFS Software : : : : Language Bindings : : Implementation State Document Layout : : :
1 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
2 CCFS Prerequisites
2.1 Hardware Prerequisites : 2.2 Software Prerequisites :
3 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
3 CCFS Code Structuration 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8
General Overview : : : Binaries : : : : : : : : System Con guration Utilities : : : : : : : : Test Facilities : : : : : Devices : : : : : : : : Libraries : : : : : : : : Source Code : : : : : :
3 3
5
: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
4 CCFS Source Code Structuration 4.1 General Overview : : : 4.2 Miscellaneous Modules 4.2.1 cfs alloc.c : : : 4.2.2 cfs cache.c : : : 4.2.3 cfs dir.c : : : : 4.2.4 cfs dnlc.c : : : 4.2.5 Cfs init.c : : : 4.2.6 cfs inode.c : : : 4.2.7 cfs misc.c : : : 4.2.8 cfs param.c : : 4.2.9 cfs perm.c : : : 4.2.10 cfs rw.c : : : : 4.2.11 cfs super.c : : : 4.3 LFS Modules : : : : : 4.3.1 lfs calls.c : : : 4.3.2 lfs ops.c : : : :
1 1 1 2
5 5 6 6 6 6 6 7
9
: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
i
9 9 10 10 10 11 11 11 12 13 13 13 13 14 14 15
CONTENTS
ii
4.3.3 lfs serv.c : : : : : : : : 4.4 CLFS Modules : : : : : : : : 4.4.1 clfs serv.c : : : : : : : 4.4.2 clfs calls.c : : : : : : : 4.4.3 clfs lib.c : : : : : : : : 4.4.4 clfs rw.c : : : : : : : : 4.4.5 clfs misc.c : : : : : : : 4.4.6 clfs lp.c : : : : : : : 4.4.7 clfs open.c : : : : : : : 4.4.8 clfs stadir.c : : : : : : 4.4.9 clfs proc.c : : : : : : : 4.4.10 clfs time.c : : : : : : : 4.4.11 clfs link.c : : : : : : : 4.4.12 clfs protect.c : : : : : 4.4.13 clfs mount.c : : : : : : 4.4.14 clfs extended.c : : : : 4.5 CDS Emulation Modules : : : 4.5.1 cds manager.c : : : : : 4.5.2 cds local.c : : : : : : : 4.5.3 cds conf.c : : : : : : : 4.5.4 cds aux.c : : : : : : : 4.6 Kernel Services Modules : : : 4.6.1 channel.c : : : : : : : 4.6.2 mqueue.c : : : : : : : 4.6.3 portnameserv.c : : : : 4.6.4 pthread.c : : : : : : : 4.7 Include Modules : : : : : : : 4.7.1 Miscellaneous Includes 4.7.2 LFS Includes : : : : : 4.7.3 CLFS Includes : : : : 4.7.4 CDS Includes : : : : : 4.7.5 Kernel Includes : : : : 4.8 Utility modules : : : : : : : : 4.9 Test Modules: Examples : : :
5 Bibliography
10-02-1994
J. Carretero et al.
: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
16 17 17 17 19 20 20 21 21 21 21 22 22 22 22 22 23 23 23 24 24 25 25 25 25 26 26 27 27 27 27 28 28 28
31
UPM { DATSI
Chapter 1
Introduction 1.1 CCFS Software This document shows a brief description of the CCFS software , developed in the GPMIMD project (ESPRIT project P5404). This software has been developed following the detail instructions given in [2],[4] [3] [1], and the next three detailed design documents:
Client File Server (CLFS) detailed design (FIM/82.1/DATSI/94) Local File Server (LFS) detailed design (FIM/81.1/DATSI/94) Concurrent Disk System (CDS) detailed design [4] [3] [1] (FIM/83.1/DATSI/94)
So, following the design description, the CCFS software has a set of miscellaneous modules providing general functionality, and three main specialized components with a di erent approach to data:
The CLFS, which manages general data objects (distributed or not) The LFS, which manages only le system objects (distributed or not) The CDS which manages only at le system objects A detailed description of each level modules is provide in chapter 3
1.2 Language Bindings Language bindings for system functions are de ned in C language.
1.3 Implementation State The delivered implementation of CCFS satis es all the requirements stated in the GPMIMD technical annex. It provides a distributed parallel le server with cache coherence. A POSIX interface, with some extensions, has been provided to the user applications. The whole functionality of the CCFS is implemented. The CDS is emulated on UNIX le systems. It is actually running on a network of workstations executing Solaris. To port it to the GPMIMD machine communication and kernel services should only be ported. 1
2
Chapter 1. Introduction
J. Carretero et al.
1.4 Document Layout This document describes the structure and components of the CCFS software. CCFS hardware and software dependencies are shown in chapter 2, software structuration and module contents are described in chapters 3 and 4, system installation and execution is explained in chapter ??, and basic restrictions and main design modi cations are shown in chapter ??.
10-02-1994
UPM { DATSI
Chapter 2
CCFS Prerequisites The delivered demonstration software requires both hardware and software mechanisms to properly execute. These restrictions, while few, are very important to CCFS installation and execution.
2.1 Hardware Prerequisites The following hardware con guration is needed to install and execute CCFS:
Several Sun Sparc Stations with at least 16 MBytes of memory Server Sparc Stations should have disks A network connecting the Stations
Be aware that a T9000 machine is a necessary prerequisite for the nal CCFS version that will be provided in the GPMIMD project extension.
2.2 Software Prerequisites Main software prerequisites were already stated in [4] [3] [1]. Summarizing this documents, the delivered CCFS software will require:
A UNIX system per Sparc Station with Solaris 2.3 OS A C compiler into each node Threads Sockets IPC mechanisms
3
4
10-02-1994
Chapter 2. CCFS Prerequisites
J. Carretero et al.
UPM { DATSI
Chapter 3
CCFS Code Structuration 3.1 General Overview All CCFS code is located under the directory /ccfs. CCFS modules are grouped in several categories, depending of the functionality included into each one. Each group is located under a new directory. The directories that may be found in a CCFS installation:
bin, which include all functionality shared by the LFS and the CLFS. sysconf, which include all functionality used by the LFS. utils, which include all functionality used by the CLFS. tests, which include all functionality used by the CDS. dev, which includes all functionality depending on the system hardware or software (communications, threads, ...)
lib, which include all de nitions used in the CCFS source code. src, with some utils related with system installation and customization. All these groups are brie y summarized in this chapter.
3.2 Binaries Executable les to start up a CCFS system are included in this directory. This les do not include commands or utilities. So, the only les included in this directory are:
LFS y CLFS serv Name server, which will be located only on the ROOT LFS node CCFS initialization process, which will be located only on the ROOT LFS node (CCFS init). 5
Chapter 3. CCFS Code Structuration
6
J. Carretero et al.
3.3 System Con guration Several les de ning system con guration information of the components of CCFS are included into this directory. CDS emulation con guration is described in the following les:
disk.conf, which contains emulated disk devices information. group.conf, which contains information about the groups of disks existing in the system. partn.conf, which contains information about the partitions existing in the system. subpartn.conf, which contains information about the subpartitions existing in the system.
LFS con guration information is shown in the le lfs.conf. Name and location of LFSs is de ned into this le. CLFS con guration information is shown in the le clfs.conf. Name and location of CLFSs is de ned into this le. The name server, CDS, and Kernel addresses should be known by the root LFS, and the should be included in the boot.conf. In a rst system, xed addresses will be used.
3.4 Utilities This directory contains executable les which provide several facilities to manage CCFS. Installation (CCFS Con g, CCFS init,...) procedures and CCFS commands (mkfs, mount, ...) are provided in this group of les.
3.5 Test Facilities Executable les of some test procedures used during CCFS development and functional evaluation are provided into this directory. Some of them are read, write, lfs ls, cmount, ... Source code of these procedures in included in the /ccfs/src/tests directory (see chapter 4).
3.6 Devices Emulated disk devices are included into this directories. They are not real devices but UNIX les. They will be implemented on actual devices as far as a real CDS exists.
3.7 Libraries CCFS libraries used to build a new executable of LFS or CCFS. Some libraries included into this directory are:
libchann.a, including objects modules for channel management. libmsg.a, including objects modules for message management. 10-02-1994
UPM { DATSI
3.8. Source Code
7
J. Carretero et al.
libcds.a, including objects modules for cds emulation management. libcfs.a, including objects modules for LFS and CLFS reconstruction; this library includes objects described as miscellaneous in chapter 4.
3.8 Source Code Source les of CCFS. See chapter 4.
Report No. FIM/84.1/DATSI/94
10-02-1994
8
10-02-1994
Chapter 3. CCFS Code Structuration
J. Carretero et al.
UPM { DATSI
Chapter 4
CCFS Source Code Structuration 4.1 General Overview All source code of CCFS is located under the directory /ccfs/src. CCFS modules are grouped in several categories, depending of the functionality included into each one. Each group is located under a new directory. The source code directories that may be found in a CCFS installation with code available are the following:
misc, which include all functionality shared by the LFS and the CLFS. lfs, which include all functionality used by the LFS. clfs, which include all functionality used by the CLFS. cds, which include all functionality used by the CDS. kernel, which includes all functionality depending on the system hardware or software (communications, threads, ...)
includes, which include all de nitions used in the CCFS source code. utils, with some utils related with system installation and customization. tests, with the source code of some programs used to tests the system functionality. All this groups are brie y summarized in this chapter, including a brief description of each module and the functions exported by this module. A most complete description of this modules and functions is provided in the appendixes.
4.2 Miscellaneous Modules This group included the following modules: cfs alloc.c, cfs cache.c, cfs dir.c, cfs dnlc.c, cfs fsck.c, cfs init.c, cfs inode.c, cfs misc.c, cfs mkfs.c, cfs mount.c, cfs param.c, cfs perm.c, cfs rw.c and cfs super.c The total number of lines of this modules is almost 16K lines, being cfs dir.c with almost 3k lines, cfs inode.c with 3.5k lines, and cfs super.c with 2.5k lines main exponents of this group. 9
Chapter 4. CCFS Source Code Structuration
10
J. Carretero et al.
4.2.1 cfs alloc.c
This module manages the block bitmap of a partition. Main topics related with block allocation and deallocation are included into this module. Functions exported by this module are included below. Do alloc blocks (); Do alloc inodes (); Do free inodes (); Do free blocks (); Cfs map alloc ();
4.2.2 cfs cache.c
This module manages the block cache of a node . Main topics related with cache initialization, block management, ush and block invalidation are included into this module. Functions exported by this module are included below. Ccfs cache init (); Ccfs cache get block (); Ccfs cache put block (); Ccfs cache ush (); Ccfs cache invalidate (); Ccfs cache u inval (); Do start cache consistency (); Do end cache consistency ();
Do invalidate cache Do activate cache
4.2.3 cfs dir.c
This module includes all the functions managing directories. Functions exported by this module are included below. Init directory (); Cfs dirlook (); Cfs direnter (); Cfs dirremove (); Cfs dirnextentry (); Cfs eat path (); Cfs dirmangled (); 10-02-1994
UPM { DATSI
4.2. Miscellaneous Modules
11
J. Carretero et al.
4.2.4 cfs dnlc.c
This module include all the functions managing name caches. Cache name lookup. Given a directory inode and a name, speeds up inode search. Functions exported by this module are included below. Init dnlc (); Cfs dnlc enter (); Cfs dnlc lookup (); Cfs dnlc remove (); Cfs dnlc purge (); Cfs dnlc purge vp ();
4.2.5 Cfs init.c
This module includes all functions related with system initialization. Functions exported by this module are included below. Read conf init (); Clfs init (); Init all lfs (); Init all clfs (); Do get con guration ();
4.2.6 cfs inode.c
This module includes all functions related with inode management. Functions exported by this module are included below. Inode hash init (); Cfs iusers (); Do inode get (); Do inode put (); Do inode users (); Do inode lock (); Do inode consist (); Do inode unlock (); Do inode update (); Search inode hash (); Do is part busy (); Inode here (); Inode put (); Report No. FIM/84.1/DATSI/94
10-02-1994
Chapter 4. CCFS Source Code Structuration
12
J. Carretero et al.
Inode lock (); Inode unlock (); Inode truncate (); Cfs iget (); Cfs ihold (); Cfs iput (); Cfs ilock (); Cfs iunlock (); Cfs iupdat (); Cfs itrunc (); Cfs ialloc (); Cfs itimes (); Cfs ifree (); Indirtrunc ();
4.2.7 cfs misc.c This module includes several miscellaneous functions related with le attributes. Functions exported by this module are included below. O to block (); Cfs bmap (); Cfs blko (); Cfs blkno (); o minor (); o inc (); o dec (); File pos cmp (); Cfs roundup (); fbrelse (); fbread (); fbwrite (); fbawrite (); cfs bcmp (); di o (); 10-02-1994
UPM { DATSI
4.2. Miscellaneous Modules
13
J. Carretero et al.
4.2.8 cfs param.c
This module includes the functions related with the management of message parameters. Functions exported by this module are included below. time stamp (); Get answer mqueue name (); Get parameters (); Get fs call (); Get return parameters ();
4.2.9 cfs perm.c
This module includes all functions related with le access permissions. Cfs iaccess (); Cfs suser (); Cfs groupmember ();
4.2.10 cfs rw.c
This module includes all functions related with data block management. Reading and writing blocks are the two main functions of this module. Do write blocks (); Do read blocks (); Cfs readwrite ();
4.2.11 cfs super.c
This module include all functions related with super block management. These functions are used by the LFS. Init table super (); Load remote super block (); Unload super block (); Read super block (); Write super block (); Get super block (); Load inode bit maps (); Load block bit maps (); Unload inode bit maps (); Unload block bit maps (); Report No. FIM/84.1/DATSI/94
10-02-1994
Chapter 4. CCFS Source Code Structuration
14
J. Carretero et al.
Free inode (); Free block (); Alloc inodes (); Alloc blocks (); Do show partn (); Bit block belong to this node (); Block belong to this node ();
4.3 LFS Modules This group included the following modules: lfs calls.c, lfs ops.c, and lfs serv.c. They are closely related with LFS, being mostly interface library, multithreaded server, and LFS system calls code. The total number of lines of this modules is almost 7K lines, having lfs calls.c almost 3.5k lines, lfs ops.c almost 3k lines, and lfs serv.c almost 0.5 klines. LFS operations directly a ects the LFS external objects, which are the main data structures. LFS has ve external objects managers as shown in the detailed design document:
File System manager I-node manager Directory manager Block manager Control manager
4.3.1 lfs calls.c
This module manages the LFS system calls which have been described in the detailed design documents. Functions exported by this module are included below. Do call load super(); Do call unload super(); Do call propagate super(); Do call cancel super(); Do call alloc blocks(); Do call free blocks(); Do call get inode(); Do call put inode(); Do call alloc inode(); Do call free inode(); 10-02-1994
UPM { DATSI
4.3. LFS Modules
15
J. Carretero et al.
Do call lock inode(); Do call unlock inode(); Do call update inode(); Do call is part busy(); Do call propagate inode(); Do call dir addentry(); Do call dir search(); Do call dir delentry(); Do call lookup name(); Do call read blocks(); Do call write blocks(); Do call start cache consistency(); Do call end cache consistency(); Do call invalidate cache(); Do call activate cache(); Do call lfs enosys(); Do call init(); Do call exit();
4.3.2 lfs ops.c LFS library operations are described in the following modules. LFS library functions are stub functions of the real LFS functions. Their main goal is to provide communication with the appropriate LFS. Library functions use only parameters to interchange signi cant data. The function value is used to return a 0 to indicate a correct operation, or an positive number indicating an error code. All LFS functions have a typical RPC structure of a RPC scheme, describing the following pattern:
Marshaling the input parameters on a message bu er Sending the message to the appropriate LFS Receiving the answer for the LFS Unmarshaling the output parameters from the message bu er Returning the output parameters or an error code (if needed)
LFS library provides support for CLFSs and other LFSs. There is no di erent between both former elements to a particular LFS, because both of them may be clients of a LFS. All LFS library functions are included in the lfs ops.c source le. A detailed description of this functions, in manual page format, is provided in the detailed design documents. Report No. FIM/84.1/DATSI/94
10-02-1994
Chapter 4. CCFS Source Code Structuration
16
J. Carretero et al.
Lfs load super block(); Lfs unload super block(); Lfs propagate super block(); Lfs cancel super block (); Lfs read blocks (); Lfs write blocks (); Lfs alloc blocks (); Lfs free blocks (); Lfs is part busy (); Lfs get inode (); Lfs put inode (); Lfs lock inode (); Lfs unlock inode (); Lfs update inode (); Lfs propagate inode (); Lfs inode users (); Lfs alloc inodes(); Lfs free inodes (); Lfs dir addentry (); Lfs dir search (); Lfs dir delentry(); Lfs lookup name (); Lfs exit (); Lfs init ();
4.3.3 lfs serv.c
The Local File Server is designed as a server executing a main loop that:
Waits for a request message to arrive Extracts the system call number and uses it as an index into the functions table Calls the appropriate procedure, waiting for the answer (thread creation) Replies the request
The main server of LFS is developed into this module. System call messages are received and processed here. Threads are used to execute an speci c operation. Each LFS creates a thread to execute each incoming request. So, threads do not receive directly incoming requests from outside the LFS. Incoming request to the LFS have two possible origins: a CLFS or another LFS. See the detailed design document for a skeleton of the LFS main server. 10-02-1994
UPM { DATSI
4.4. CLFS Modules
17
J. Carretero et al.
4.4 CLFS Modules This group included the following modules: clfs calls.c, clfs lib.c, clfs rw.c, clfs misc.c, clfs serv.c, clfs lp.c, clfs lib.c, clfs open.c, clfs stadir.c, clfs proc.c, clfs time.c, clfs link.c, clfs protect.c, clfs mount.c, and clfs extended.c. The total number of lines of this modules is almost 20k lines, with two major modules clfs calls.c and clfs lib.c including almost 4k and 3k lines respectively. The other modules are devoted to the execution of speci c operations. The clfs extended.c module includes extra operations needed by the other modules. It may probably be enlarged in the demonstration system.
4.4.1 clfs serv.c
The CLFS main server is implemented into this module. After a declaration and a initialization stage, an in nite loop is executed to receive requests, processing the CLFS call, and sending the reply to the client. Threads, ports, and semaphores are used. The semantic for these services is the one used in POSIX, and it has been speci ed in [4] and [3]. The CLFS Basic Server is responsible for the interface with external requests from users, CLFSs and LFSs. It is possible to treat several requests at the same time and satisfy some requests without waiting for the others blocked. Each manager loops on the following sequence:
Waits for a request message to arrive Extracts the system call number and uses it as an index into the functions table Calls the appropriate procedure, waiting for the answer (thread creation) Sends the reply to the client
The communication protocol between CLFS and users is based upon a very simple RPC mechanism.
4.4.2 clfs calls.c
This module manages the CLFS system calls which have been described in the detailed design documents. Functions exported by this module are included below. Do call clfs init(); Do call clfs exit(); Do call clfs create process(); Do call clfs exit process(); Do call clfs exec process(); Do call clfs acquire super block(); Do call clfs release super block(); Do call clfs propagate inode(); Report No. FIM/84.1/DATSI/94
10-02-1994
Chapter 4. CCFS Source Code Structuration
18
J. Carretero et al.
Do call clfs invalidate cache(); Do call clfs activate cache(); Do call clfs showpartn(); Do call clfs access(); Do call clfs chmod(); Do call clfs chown(); Do call clfs fcntl(); Do call clfs fstat(); Do call clfs lseek(); Do call clfs stat(); Do call clfs utime(); Do call clfs umask(); Do call clfs close(); Do call clfs creat(); Do call clfs dup(); Do call clfs dup2(); Do call clfs link(); Do call clfs mk fo(); Do call clfs open(); Do call clfs pipe(); Do call clfs read(); Do call clfs rename(); Do call clfs unlink(); Do call clfs write(); Do call clfs chdir(); Do call clfs mkdir(); Do call clfs rmdir(); Do call clfs chroot(); Do call clfs mknod(); Do call clfs mount(); Do call clfs umount(); Do call clfs truncate(); Do call clfs eseek(); Do call clfs aspace(); Do call clfs mcreat(); Do call clfs mopen(); 10-02-1994
UPM { DATSI
4.4. CLFS Modules
19
J. Carretero et al.
Do call clfs open sub le(); Do call clfs aread(); Do call clfs awrite(); Do call clfs iowait(); Do call clfs no cache();
4.4.3 clfs lib.c
This module provides the system call interface to CCFS. It is an extended POSIX interface which has been described in the detailed design documents. Functions exported by this module are included below. Ccfs access (); Ccfs chdir (); Ccfs chmod (); Ccfs chown (); Ccfs close (); Ccfs creat (); Ccfs dup (); Ccfs dup2 (); Ccfs fcntl (); Ccfs fstat (); Ccfs link (); Ccfs lseek (); Ccfs mkdir (); Ccfs mk fo (); Ccfs open (); Ccfs pipe (); Ccfs read (); Ccfs rename (); Ccfs rmdir (); Ccfs stat (); Ccfs utime (); Ccfs umask (); Ccfs unlink (); Ccfs write (); Ccfs fpathconf(); Ccfs pathconf(); Report No. FIM/84.1/DATSI/94
10-02-1994
Chapter 4. CCFS Source Code Structuration
20
J. Carretero et al.
Ccfs opendir(); Ccfs readdir(); Ccfs rewinddir(); Ccfs closedir(); Ccfs getcwd(); Ccfs mount (); Ccfs unmount (); Ccfs mknod (); Ccfs chroot (); Ccfs tell (); Ccfs truncate(); Ccfs ftruncate(); Ccfs eseek(); Ccfs aspace(); Ccfs mcreat(); Ccfs mopen(); Ccfs open sub le(); Ccfs aread(); Ccfs awrite(); Ccfs iowait(); Ccfs no cache();
4.4.4 clfs rw.c
This module includes the functions used to perform read and write operations. Functions exported by this module are included below. Do clfs read(); Do clfs write(); Read write(); New block ();
4.4.5 clfs misc.c
This module includes several miscellaneous functions related with le names and descriptors management. Functions exported by this module are included below. Get last component (); Is mounted(); Clfs eat path (); 10-02-1994
UPM { DATSI
4.4. CLFS Modules
21
J. Carretero et al.
Do clfs fcntl(); Do clfs dup (); Do clfs dup2 ();
4.4.6 clfs lp.c
This module contains the functions that manipulate le descriptors. Functions exported by this module are included below. Clfs init lp(); Clfs get fd(); Clfs get lp(); Clfs nd lp();
4.4.7 clfs open.c
This module includes the functions used to create, open, close, and seek on les. Functions exported by this module are included below. Do clfs creat(); Do clfs mknod(); Do clfs open(); Do clfs close(); Do clfs lseek(); Do clfs mkdir();
4.4.8 clfs stadir.c
This module includes the functions that perform the le status operations and change the current and root directories. Functions exported by this module are included below. Do clfs chdir(); Do clfs chroot(); Do clfs stat(); Do clfs fstat();
4.4.9 clfs proc.c
This module contains the procedures that manipulate the process table. Functions exported by this module are included below. Clfs inic proc(); Clfs get process(); Clfs create process(); Clfs exit process(); Report No. FIM/84.1/DATSI/94
10-02-1994
22
Chapter 4. CCFS Source Code Structuration
J. Carretero et al.
4.4.10 clfs time.c
This module includes functions used by le system calls dealing with time. Functions exported by this module are included below. Do clfs utime();
4.4.11 clfs link.c
Link, unlink and rename system calls are managed in this module. Do clfs link(); Do clfs unlink(); Do clfs rename();
4.4.12 clfs protect.c
This module deals with protection in the le system. Functions exported by this module are included below. Do clfs chmod(); Do clfs chown(); Do clfs umask(); Do clfs access(); Read only();
4.4.13 clfs mount.c
This module includes the functions that perform mounting and unmounting operations. Functions exported by this module are included below. Do clfs mount(); Do clfs unmount();
4.4.14 clfs extended.c
This module includes the functions that performs all the POSIX interface extensions. Functions exported by this module are included below. Do clfs eseek(); Do clfs aspace(); Do clfs mcreat(); Do clfs mopen(); Do clfs open sub le(); Do clfs aread(); Do clfs awrite(); Do clfs iowait(); Do clfs no cache(); 10-02-1994
UPM { DATSI
4.5. CDS Emulation Modules
23
J. Carretero et al.
4.5 CDS Emulation Modules This group included the following modules: cds aux.c, cds conf.c, cds local.c, cds manager.c. The total number of lines of this modules is almost 3k lines. Functions included in the CDS emulation are not the ones speci ed in the detailed CDS design. They are convenience functions thought to provide a higher level functionality to the LFSs. Those functions are implemented using basic CDS functions.
4.5.1 cds manager.c
This module contains the CDS interface functions excluding those con guration functions included in cds conf.c module. Functions exported by this module are included below. CreateGrp(); OpenGrp(); CloseGroup (); SetGrpId(); AddDisk(); RmDisk(); DiskCount(); ListDisks(); DiskInfo(); CreatePartn(); DeletePartn(); PartnCount (); OpenPartn(); GetPartn (); ClosePartn (); GetPartnInfo (); LocateBlk (); ReadBlocks (); WriteBlocks();
4.5.2 cds local.c
This module includes some local functions of the CDS. Functions exported by this module are included below. do device (); read block(); write block(); Report No. FIM/84.1/DATSI/94
10-02-1994
Chapter 4. CCFS Source Code Structuration
24
J. Carretero et al.
4.5.3 cds conf.c
This module contains the functions managing con guration parameters of the CDS. Functions exported by this module are included below. InitDiskTable(); InitSubPartnTable(); InitPartnTable(); ReadGroupConf(); ReadSubPartnConf(); ReadPartnConf(); WriteDiskTable(); WriteGroupTable(); WriteSubPartnTable(); WritePartnTable(); PrintGroupTable(); PrintSubPartnTable(); PrintPartnTable(); UpdateFileCon guration(); SetCon guration(); OpenGroups(); OpenPartns(); Cds init();
4.5.4 cds aux.c
This module includes some miscellaneous functions of the CDS. Functions exported by this module are included below. Find slot disk table(); Find slot group table(); Exists grp(); Find free slot group table(); Check disk grp(); Do grp(); Exists partn(); Find free slot partn table(); Find slot subpartn table(); Check sub partn(); 10-02-1994
UPM { DATSI
4.6. Kernel Services Modules
25
J. Carretero et al.
Do subpartitions(); Find slot partn table(); Count subpartitions(); Find order sub();
4.6 Kernel Services Modules This group includes all functionality depending on the system hardware or software (communications, threads, ...) This group includes four modules and some more auxiliary ones. Only four main modules will be studied in this document, since the other are auxiliary functions which are only referenced from the main modules. The four named modules are: channel.c, mqueue.c, portnameserv.c, and pthread.c. The total number of lines of the whole modules is almost 3k lines.
4.6.1 channel.c This module includes all the operations related with channels. cqopen(); cqclose(); cqsend(); cqreceive();
4.6.2 mqueue.c This module includes all the operations related with message queues. mqprintmess(); mqopen(); mqclose(); mqsend(); mqreceive(); msgalloc(); msgfree(); mqgetpid();
4.6.3 portnameserv.c Independent process providing unique names for the communication queues in the system. Is a typical client-server scheme, with module providing the server process. Clients are the LFSs and the CLFSs. Report No. FIM/84.1/DATSI/94
10-02-1994
Chapter 4. CCFS Source Code Structuration
26
J. Carretero et al.
printtables (); getport (); freeport (); destroy (); createport (); init dir(); init tables();
4.6.4 pthread.c POSIX like thread facilities. They are programmed using the kernel ones. pthread once(); pthread attr create(); pthread create(); pthread detach(); pthread join(); pthread yield( ); pthread exit(); pthread mutexattr create(); pthread mutex init(); pthread mutex lock(); pthread mutex trylock(); pthread mutex unlock(); pthread mutex destroy(); pthread condattr create(); pthread cond init(); pthread cond signal(); pthread cond broadcast(); pthread cond wait(); pthread cond destroy();
4.7 Include Modules This group includes all de nitions used in the CCFS source code. 10-02-1994
UPM { DATSI
4.7. Include Modules
27
J. Carretero et al.
4.7.1 Miscellaneous Includes
cfs buf.h, disk bu er cache de nition. cfs const.h, global CCFS constants and types. cfs cred.h, security structure de nition; it is called credential. cfs dir.h, de nition of the directory structure, the directory slot structure, and some macros
used to manage directories. cfs dnlc.h, name cache de nition. cfs errno.h, error codes returned by the kernel simulation (comm, threads, and kernel interaction). cfs global.h, CCFS global variables for each LFS and CLFS. cfs init.h, information needed initially by the CCFS; it is formed up from several include les. cfs inode.h, le inode de nition, including hash table of inodes and several constants and
ags. cfs super.h, super block de nition, including hash table of superblocks and several constants and ags. cfs types.h, CCFS de ned types. cfs vattr.h, attributes used in directory management to give creation and deletion attributes.
4.7.2 LFS Includes
lfs init.h, information needed initially by a LFS; it is formed up from several include les. lfs struct.h, LFS message structures to be used in LFS system calls.
4.7.3 CLFS Includes
clfs const.h, number of lp entries, number of le descriptor per process, and number of
process per clfs. clfs lp.h, table clfs lp de nition; le to inode translation. clfs init.h, information needed initially by a CLFS; it is formed up from several include les. clfs proc.h, CLFS per-process information; A slot is reserved for each potential process. clfs struct.h, CLFS message structures to be used in CLFS system calls.
4.7.4 CDS Includes
cds var.h, constants, variables and types for the CDS emulation modules.
Report No. FIM/84.1/DATSI/94
10-02-1994
Chapter 4. CCFS Source Code Structuration
28
J. Carretero et al.
4.7.5 Kernel Includes
communications.h, constants, variables and types for the messages and channels modules. portnameserv.h, constants, variables and types for the name server process modules. pthread.h, constants, variables and types for the thread implementation module.
4.8 Utility modules This group includes some utils related with system installation and customization.
makedev mkfs fsck install
4.9 Test Modules: Examples This group includes the source code of some programs used to tests the system functionality.
test lib.c clmount.c build.c read.c write.c lfs ls.c cainode.c c node.c cfblock.c addentry.c cablock.c cmount.c cumount.c cwrite.c cread.c 10-02-1994
UPM { DATSI
4.9. Test Modules: Examples
29
J. Carretero et al.
delentry.c getinode.c inodetest.c lookup.c searchentry.c showpartn.c destroy.c recvchann.c sendchann.c sendmsg.c recvmsg.c
Report No. FIM/84.1/DATSI/94
10-02-1994
30
10-02-1994
Chapter 4. CCFS Source Code Structuration
J. Carretero et al.
UPM { DATSI
Chapter 5
Bibliography [1] J. Carretero, F. Perez, P. De Miguel, F. Garcia, and L. Alonso. A Multiprocessor Parallel Disk System Evaluation. In R. Puigjaner, editor, Proceedings of the ICDDS'93 Conference. Elsevier, September 1993. [2] J. Carretero, F. Perez, P. De Miguel, F. Garcia, and L. Alonso. CCFS Design Rationale. Technical Report ESPRIT Project P5404, CEE, 1993. [3] P. De Miguel, J. Carretero, F. Costoya, F Garcia, and L. Alonso. CCFS Kernel Services. Technical Report ESPRIT Project Number P5404, Report WP13, CEE, February 1993. [4] P. De Miguel, J. Carretero, F. Costoya, F Garcia, and L. Alonso. Concurrent Disk System Simulation. Technical Report ESPRIT Project Number P5404, Report WP11, CEE, February 1993.
31
