Oregon State University Department of Computer Science

5 downloads 327594 Views 135KB Size Report
The survey was initially applied to two test populations. .... "Novice" subjects were first and second-year undergraduate CS students enrolled in ... users. Among these are the time available to invest in program development, the degree of new.
Oregon State University Department of Computer Science Technical Report CSTR 94-80-3 What Users Need in Parallel Tool Support: Survey Results and Analysis June 1994 Cherri M. Pancake Curtis Cook

Oregon State University Department of Computer Science Corvallis, OR 97331

What Users Need in Parallel Tool Support: Survey Results and Analysis0

Cherri M. Pancake and Curtis Cook Department of Computer Science Oregon State University Corvallis, OR 97331 pancake,[email protected] March 7, 1994

Abstract Considerable effort and expense have been devoted to developing tools that support parallel programming. Recent evidence suggests that users do not find the current generation of tools useful for their program development needs, but offers no guidance on how that problem might be addressed. This paper reports the results of a survey applied to a significant (N=448) cross-section of the parallel user community at Supercomputing '93. The responses establish a number of general patterns in program development and tool use among scientists, engineers, and computer scientists. Our analysis suggests several likely directions for future tool development efforts.

As Thomas Moran noted more than ten years ago, software developers often adopt the attitude that "... the best way to deal with the user is simply to take more care in considering the user -- all the system designer needs is to be given the time to do so. The designer is, after all, human and has the intuitions to predict what will be easy for the user. It is mostly common sense, anyway, isn't it?" [6:2] This attitude is prevalent in the parallel computing industry. Existing parallel tools and runtime support environments were designed by computer scientists and systems engineers who concentrated their efforts on meeting the serious technological challenges presented by parallel systems. In response to user complaints, they are now taking the tools and libraries they found useful in their own development work and "re-packaging" them for the user audience by adding graphical or quasi-graphical interfaces. The result? Users simply do not find the current generation of tools useful for their program development needs [9, 11]. The final session of a recent NSF/ARPA workshop1 which brought together tool developers and users clarified the problem: A lot of smart people are developing parallel tools that smart users just won't use. Significant levels of effort and expense are being devoted to the development of tools to support parallel programming. Why, then, are the results so disappointing? We believe that there is a significant mis-match between user needs and tool support. This paper addresses the issue of parallel tool requirements. It describes a survey applied to a significant cross-section of the parallel user community. The results provide a basic framework indicating how users approach parallel programming, where they direct most of their efforts, and what tools or other 0 The survey described here is part of a broader research project dealing with the parallel user community, sponsored by Intel Corporation, IBM Corporation, CONVEX Computer Corporation, and the Oregon Advanced Computing Institute. This report will be included in the Proceedings of the Scalable High Performance Computing Conference (May, 1994). 1 1993 Workshop on Parallel Computing Systems: A Dialog between Users and Developers (Keystone, Colorado, April 1993).

1

resources they use. The survey was initially applied to two test populations. In November of 1993, it was applied on a much larger scale at Supercomputing '93. This paper offers a preliminary analysis of the questions relating to division of programming effort and the use of tools to support those efforts. Background and Motivation Most current tools for debugging and tuning parallel programs were designed by systems programmers, who were motivated by their own needs in developing and tuning compilers, operating systems, and runtime libraries. In response to user demands, the tools and libraries that systems programmers found useful have been adapted for the general user community. The approach has not been successful -- where "success" refers to the frequency and significance of tool use. Across the board, virtually all parallel users prefer hand-coded instrumentation over current tool offerings [12]. They complain that tools are hard to learn, are tedious to use, and fail to provide the information users really need and want. Parallel tool developers, on the other hand, claim that parallel users have unreasonable expectations about the ease of using tools, do not give tools a "real chance," or even mis-apply them to inappropriate tasks. Clearly, users and tool developers are not seeing eye to eye. In some cases, the tool's user interface is at fault. Tool information often is presented in a fashion that reflects the tool's organization, rather than the logical patterns employed by users. The information sought by the user may be available only indirectly, via multiple operations or through assimilation from multiple sources. At best, the tool is considered clumsy and hard to learn; at worst, the user assumes it cannot provide the desired information. The root of the problem lies much deeper than the interface, however. The audience for parallel tools is made up of scientists, engineers, and other technical programmers. They do not approach programming in the same way as their computer science counterparts, nor are they tolerant of tools that are complicated or non-intuitive. The problem is compounded because tool developers do not ask users the "right" questions. If a tool designed for a particular programming task isn't applied to that activity, the developer asks the user why it's not being used. What needs to be asked instead is how the user does go about the task, why it is performed in that way, and what he/she does to simplify or streamline the effort. As Norman pointed out: Too often, technology is constructed for its own merits, independent of its uses ... [or] the user. This works fine when the technology is used to optimize some complex computation, or to discover algorithms.... It is not appropriate when the whole point of the technology is to act as a direct aid to people's work activities. When this is the case, then the technology will only succeed if the people and the activities are very well understood [8:89].

For example, our recent studies of parallel applications suggest that even the basic organization of code segments into procedures differs significantly between the computer science and scientific programming communities [10]. Scientific programmers are likely to partition statements into procedures according to how the code was acquired, its size, or its assumed level of correctness. Code fragments acquired from other people or modified from other applications commonly appear as independent procedures. When loops or routines become too long to be easily managed or edited, sub-portions may be extracted as functions or subroutines. Similarly, fragments that have been debugged thoroughly (or conversely, statements likely to contain errors or performance bottlenecks) are isolated into procedures. The internal logic of this approach -- physically isolating elements that are assumed to be "not currently of interest" from those that are "under active work" --

2

eludes most computing professionals, who develop code hierarchically via functional decomposition or some other structured approach. Why is this a problem? Current tools reflect the developers' perspective of program organization. Code navigation aids, program editing facilities, and even procedure selection lists assume that the structure and naming of procedural units mirror in some way the logical structure of the application. User applications, however, do not look or act like the modularly designed programs written by computer scientists. Therefore, representations derived automatically via analysis of procedural organization are unlikely to provide a good fit with the user's concept of program logic. The realization that parallel users approach programming differently is essential to good tool design, but offers little in the way of practical guidance. Clearly, tools should be designed according to how the user actually programs, not how the developer thinks the user should program. But applicable information on programming habits is remarkably absent from the computing literature. There is a substantial body of work derived from empirical studies of programmers, but the subjects, materials, and tasks differ dramatically from the parallel user community in key ways: (1) Subjects were from a more uniform population (cf. [5, 1, 13]). A large majority of the individuals observed in the empirical studies were computer science students or students from other departments who were enrolled in computing courses. Occasionally they were professional programmers employed in commercial data processing. Such groups clearly are not representative of parallel users -- who for the purposes of this discussion are those individuals using parallel machines, not as part of the educational experience or for exploring the characteristics of concurrency per se, but to exploit the performance potential of multiple processors. At the present time, parallel users are typically individuals engaged in scientific or engineering research, or computational specialists (often mathematicians) involved in the development of algorithms and libraries to support technical programming activities. They span a wide variety of disciplines and levels of sophistication with respect to serial/parallel computing. (2) Subjects could be partitioned into peer groups on the basis of programming experience (cf. [4, 5, 17, 15]). "Novice" subjects were first and second-year undergraduate CS students enrolled in programming courses. "Expert" subjects were often seniors or graduate students in CS departments; occasionally they were professional programmers. Neither classification is comparable with parallel users. They may have substantial serial programming experience in professional or research settings, so they can hardly be compared with freshman students. At the same time, they are occasional or infrequent programmers who do not match the characteristics of full-time computing specialists (neither student nor professional). (3) The coding examples used were scaled down dramatically to facilitate study (cf. [14, 18, 19]). In most cases, the programs presented to subjects for analysis, debugging, modification, or development comprised just 15-200 lines of code, with only limited complexity. Small programs provide for better experimental control, but it is far from clear that the results can be generalized to real-world problems of any size. They are particularly inappropriate in comparison with parallel applications, which typically involve tens or hundreds of thousands of lines and hundreds of files and are unlikely to exemplify structured, well-modularized programming styles. (4) Limited to serial programming techniques (cf. [7, 14]). A high proportion of the studies dealt with program comprehension, but always in terms of serial code -- and in many cases, just code fragments. The complexity of the program development process alters drastically when parallelism is

3

introduced, and the problems are heightened by the relative instability of current parallel runtime environments. (5) The implicit goal of previous studies was to improve program understandability or maintainability (cf. [14, 16]). This makes sense for the student populations and data processing situations where empirical studies were carried out, but other factors are of more concern for parallel users. Among these are the time available to invest in program development, the degree of new learning required, the desire to attain a solution that is newer/better than previous ones, and -- most importantly -- the ultimate performance of the program, either in terms of execution time or of the problem size that can be accommodated. Consequently, existing empirical studies of programmers are not a satisfactory basis for establishing user requirements for parallel tools. We must study the parallel community itself if we are to identify solutions that will really work. Administration of the Survey The survey reported here is the first instance of an instrument designed specifically for application to a significant cross-section of the parallel user community. Our situation is similar to that of a group who recently studied the use of network support for scientific researchers; they found that "Despite ... confident predictions for the future, there has been no systematic empirical research on how computer network support for science is related to important scientific outcomes, including productivity. We therefore decided to do such a study" [3: 90]. One of the authors is a former ethnographer with considerable experience in survey instrument design and application. To obtain a high response rate, the instrument was kept short -- one page in length and requiring 3-5 minutes to complete. Initial items established a simple characterization of the respondent, including the resources employed in learning parallel programming. Most items dealt with the level of effort spent in various parallel development activities and what resources are used to facilitate those efforts. A final item determined what kind of tools (e.g., purchased, developed in house, etc.), if any, are employed for each activity. To acquire a maximum amount of information in such a brief instrument, most questions involved multiple answers with relative ranking indicators. Two pre-tests were applied at user group meetings sponsored by computer manufacturers in order to verify the consistency and comprehensibility of item wording. The concept of tools was left intentionally broad in order to determine if users were developing their own forms of automated analysis or assistance. Further, the instrument was intended to establish baseline information on how parallel users actually develop programs and the resources they use to do so, rather than details on how/why they use specific tools. Instead, a series of follow-up surveys, tailored to specific subcommunities of users, will pursue the findings in more detail. Supercomputing '93 (SC'93) was selected as the venue for our effort because it represents the largest single gathering of users, developers, and manufacturers of parallel computing systems. As an inducement, respondents received a small premium when they returned the completed form to a booth at the conference.2 Since not all SC'93 attendees would be experienced parallel programmers, the instrument was printed doublesided. Parallel users were asked to fill out one side, while serial or vector programmers completed the other. 2 The premiums were provided by six manufacturers of parallel computers: CONVEX Computer Corporation, Cray Research Inc., Intel Corporation, IBM Corporation, Meiko Scientific, and Thinking Machines Corporation. We gratefully acknowledge their assistance, and that of Rebecca Walpole and Lisa Loeffler of Oregon State University, who spent three days collecting the forms. We also thank Matt Killinger and Lisa Loeffler for their work in keying the responses.

4

We expected the survey to delineate the differences between how computer scientists and non-computer scientists differ in their approaches to parallel programming and their use of tools. As shall be seen, this was indeed the case. A total of 782 persons completed the survey, approximately half of whom were scientists or engineers. This paper addresses just those respondents who considered themselves parallel users. The raw results for that sub-population, which included 448 individuals, are tabulated in [2]. Each respondent was asked to characterize his/her primary area of work as science, engineering, computer science, mathematics, or "other." Of the 448 parallel users, almost half (199) classified themselves as computer scientists. Science (99) and engineering (87) together accounted for almost another half. Mathematics was represented by a relatively small number (24) of respondents. Of the remaining (39) individuals, 20 marked more than one category and 19 failed to respond; these are considered as multidisciplinary/other in our analysis. Survey Results The results described below are derived from a detailed analysis of the final two items of the instrument, dealing with division of parallel programming effort and the use of tools. It is also of interest, however, to note the responses concerning methods of program development (see Table 1). Approximately two-thirds of the respondents from each discipline -- science, engineering, computer science, math, other -indicated that they most commonly develop their parallel applications by modifying an existing serial or parallel program. Fewer than one-third develop such applications from scratch, while the remaining 10% make use of component libraries to build their applications. With the exception of the mathematicians, the results are quite uniform across disciplines. Table 1. Most Common Method of Parallel Code Development3

Sci

Eng

CS

Math

(N=99)

(87)

(199)

(24)

(39)

(448)

45%

53%

49%

50%

49%

49%

modify parallel program

19

18

17

13

18

18

build from libraries

9

9

11

13

10

10

start from scratch

32

30

32

29

23

31

modify serial program

Other

All

The allocation of effort in developing parallel applications is less consistent. Respondents were asked to estimate the proportion (as a percentage) of their time spent in six activities: fixing up the serial version of the program, writing the initial parallel version, improving the underlying model, debugging, tuning program performance, and setting up execution runs. The results appear in Table 2. Two groups skew the results in opposing directions. The mathematicians appear to spend significantly more time than their counterparts from other disciplines in writing the parallel program, and significantly less time preparing it for execution. The scientists' responses counteract this effect, since they spend less time on the parallel program and more in execution setup. 3 Based on number of "1" (highest ranked) responses in each category. Respondents were asked to indicate all methods used, ranking them in order of frequency. Because some respondents marked "1" for multiple categories, the column sums may exceed 100%.

5

Table 2. Allocation of Parallel Programming Effort4

Sci

Eng

CS

Math

Other

All

(N=99)

(87)

(199)

(24)

(39)

(448)

13%

16%

16%

14%

10%

15%

write parallel program

22

27

24

31

29

24

improve basic model

14

12

11

18

14

12

debug

19

20

23

20

22

21

tune performance

18

18

18

12

16

17

set up program runs

14

8

9

5

9

10

fix serial program

After establishing the level of effort involved in each of the six parallel programming activities, the survey instrument addressed the extent to which tools are used to support those activities. Respondents were asked to indicate what (if any) types of tools they use for each activity: purchased tools, tools available in the public domain, tools developed within their organization, or tools they have developed personally. The results are summarized in Tables 3-5. As can be appreciated in Table 3, a significant number of persons reported using no tools at all. The fact that half the respondents carry out improvements to the underlying logical model without the assistance of tools is not particularly surprising, since few current tools address such tasks. On the other hand, debugging and performance tuning tools are widely available, yet almost one-third of the survey population profess to using no tool support for these activities. The proportions are even higher for basic programming activities (fixing the serial version or writing parallel code) and for setting up parallel program runs. It is also interesting to note that in several activities, computer scientists are somewhat less likely to use tools than are scientists or engineers. Table 3. Number of Respondents Who Use No Tools5

Sci

Eng

CS

Math

Other

All

(N=99)

(87)

(199)

(24)

(39)

(448)

42%

38%

40%

38%

33%

40%

write parallel program

39

39

44

29

44

41

improve basic model

41

44

54

58

51

49

debug

32

30

31

29

28

31

tune performance

31

22

30

33

31

29

set up program runs

43

38

48

33

38

44

fix serial program

Tables 4 and 5 contrast the origins of the tools used for each program development activity. The percentages in Table 4 reflect the proportion of respondents who use tools supplied by outside sources (either commercial or in the public domain), while those in Table 5 indicate the use of tools developed personally or by someone else within the respondent's organization. These categories are not mutually exclusive; that is, some respondents report using tools from both types of sources. 4 Responses that did not add up to 100% were averaged and then normalized to do so. 5 Respondents who either checked the "use no tools" column or left all other entries blank for a given activity.

6

The actual values calculated for "external" tool use vary from one discipline to another. Nevertheless, there is a clear general trend: purchased and public-domain tools are most common for debugging, followed by performance tuning, serial modifications, parallelization, model improvement, and execution setup (in that order). "Internal" tool use, on the other hand, varies more significantly and yields no consistent pattern across disciplines. Consider, for example, the activities where internal tool use dominates. For scientists, those activities are execution setup and model improvement; for engineers and computer scientists, performance tuning; for mathematicians, execution setup; for multidisplinarians/other, debugging. Table 4. Number of Respondents Who Use "Externally Developed" Tools6

Sci

Eng

CS

Math

Other

All

(N=99)

(87)

(199)

(24)

(39)

(448)

42%

38%

40%

46%

36%

40%

write parallel program

40

33

29

38

31

33

improve basic model

27

30

21

17

15

23

debug

42

46

48

46

44

46

tune performance

43

40

43

50

44

43

set up program runs

21

26

20

29

33

23

fix serial program

Table 5. Number of Respondents Who Use "Internally Developed" Tools7

Sci

Eng

CS

Math

Other

All

(N=99)

(87)

(199)

(24)

(39)

(448)

21%

29%

28%

33%

36%

28%

write parallel program

24

32

30

50

31

30

improve basic model

36

32

28

29

38

31

debug

31

32

30

38

44

32

tune performance

29

46

36

25

31

35

set up program runs

36

38

34

54

38

37

fix serial program

Analysis of Results The survey responses establish a number of general patterns in program development and tool use among parallel programmers. Roughly half of all respondents develop their parallel applications by modifying an existing serial program, while fewer than a third start from scratch. In general, users expend some two-thirds of their program development effort in writing the parallel portions and debugging/tuning the resulting program. They are most likely to employ tools for debugging and tuning, and least likely to use 6 Respondents were asked to indicate all tool types, ranked by frequency, used for each activity. Since most multiple responses turned out to be two or more "1"s (rather than "1", 2", etc.), we treated all occurrences as "1"s. Thus, any respondent marking any number in the "purchased tools" or "public-domain tools" column for activity Ai was counted as a user of externally developed tools. 7 Any respondent marking any number in the "in-house tools" or "personal tools" column for activity A . i

7

them in efforts to improve the model on which a program is based. At a more detailed level, it is equally clear that users tend to apply just one class of tool (purchased, public domain, in-house, or personal) to each programming activity, since fewer than 6% reported using multiple classes. The respondents who use multiple classes of tools are most likely to do so for debugging. Interestingly, a computer scientist is almost twice as likely to use multiple tools -- for any activity -- than is a scientist or engineer. From this point on, the results exhibit some variation according to the respondents' disciplines. Since the mathematicians were underrepresented (under 6%) and we did not have sufficient information to categorize the "multidisciplinary/other" group satisfactorily, the remainder of the discussion will be limited to the scientists, engineers, and computer scientists. Where possible, scientists and engineers are grouped together, forming a subpopulation that is roughly the size of the computer science group (i.e., 186 versus 199). In cases where there are significant differences between scientist and engineer responses, the two are distinguished. As can be appreciated in Table 4, "external" tools are used most frequently for debugging and tuning, followed closely by modifications to the serial program. An in-depth analysis revealed some interesting patterns in this use. Overall, purchased and public-domain tools are used in similar proportions, except for parallel modifications, where public-domain tools dominate 2:1. There are significant differences among the three disciplines, however. Public domain tools are quite popular among computer scientists, being used 2:1 in preference to purchased tools (except in debugging, where usage is roughly equal). In contrast, among engineers purchased tools are the clear winner, dominating by anywhere from 1.5:1 to 2:1 across activities. The scientists lie somewhere in between, with the effects not as marked. For serial modifications and debugging, purchased tools dominate by (1.3-1.5):1; for parallel modifications and execution setup, public domain tools win, at a slightly higher proportion; the two classes are roughly even for tuning and model improvement activities. It is intriguing to note that very few respondents use both purchased and public-domain tools for any activity (only .5%-3.5% of scientists/engineers and about three times that many computer scientists). The use of "internal" tools was also analyzed in depth. Table 5 shows that these are used most frequently for tuning and execution setup activities, and to a lesser extent for model improvement. Overall, tools developed in-house dominate over those developed personally by the user, but again there is some variation by discipline. Among engineers, in-house tools uniformly wins by a large amount, typically on the order of 3:1 or 4:1. Computer scientists also markedly prefer in-house tools for serial or for parallel modifications (roughly 2:1), and to a lesser extent for tuning and model improvement (1.5:1), but there is no particular preference for debugging or execution setup. Scientist report a slightly higher use of personal than in-house tools for execution setup, although in-house tools win 2:1 for debugging, tuning, model improvement, and serial modifications; in-house tools dominate even more strongly (4:1) for parallel modifications. Again, there are very few respondents who employ both subclasses (some 2%-4% in each group). A caveat is in order as we begin discussing our interpretation of the survey results. The SC'93 audience cannot be construed as truly representative of the HPC user community. The fact that respondents were attending a multidisciplinary conference that showcases leading-edge computing technology implies that they are probably more flexible and more computer science oriented (or at least relatively comfortable with computer scientists) than would be the case at, say, an American Chemical Society meeting. Therefore, the numbers should be taken as "optimistic" upper bounds in terms of what they indicate about current tool use. Past surveys conducted among a smaller, more conservative audience of Fortran users, for example, showed that about 95% regularly use hand-coded instrumentation for debugging, performance tuning, or both; almost 90% indicated that they rely exclusively on PRINT statements (rather than tools) [12]. In comparison, only 22% to 33% of the SC'93 respondents said they do not use any tools for debugging and tuning efforts (see Table

8

3). Another contrast with previous surveys is that a larger proportion (31% on average) of respondents choose to develop parallel applications from scratch, rather than modifying an existing program or using library components. In spite of this perceived bias, several important trends emerge from the SC'93 survey. In particular, we were disturbed by the frequency with which "internal" tools were used in preference to "external" ones. This comparison is of concern because it reveals how likely a user is to find an existing purchased or publicdomain tool that meets his/her needs. Figure 1 depicts the relative incidence of internal vs. external tool usage among scientists, engineers, and computer scientists. The scale is arranged to reflect the fact that someone in the organization (the user or his/her colleagues) has to invest significant time and effort to develop an internal tool, in comparison with purchasing or acquiring an external one. A value of 0.00 means that a user is equally likely to use a purchased/public-domain tool or one developed internally. Negative values indicate a tendency toward external tools; positive values, toward internal tools. That is, -.5 means that a user is 50% less likely to use an internal tool (or 1.5 times as likely to choose an external one), while +1.0 means the user is 100% more likely (twice as likely) to use an internal tool. If currently available tools are really useful to a significant portion of the user community, the purchased/public-domain tools should suffice for, say, 2 out of 3 users (i.e., have a rating in the range of -0.66 or lower). Clearly, parallel tools ultimately should aim at satisfying a much larger community (perhaps 8 out of 10), but the present state of tool availability -- tools tend to be limited to a narrow range of languages and/or machine platforms -- suggests that a more optimistic figure is not really practicable. +1.00

prefer internal

CS Sc i Eng

+0.50

prefer external

0.00

-0.50

Reasonable (?) expectation level for par allel tools

-1.00

Fix Serial

Write Parallel

Improve Model

Debug

Tune

Set Up Runs

Figure 1. Likely preference for internal vs. external tools Unfortunately, the results show that current ratings are not even close to satisfying a reasonable proportion of users -- even given the "tool-ready" nature of the SC'93 respondents. No subpopulation expressed more than a 42% preference for external tools for any programming activity. In fact, with the exception of serial modifications, all activities are ranked at about -.33 or higher; that is, at best one in three

9

users is likely to choose an external rather than an internal tool. Worse yet, the ratings are greater than 0.00 for more than half the group/activity combinations. What makes the implications more disturbing is the fact that these users, unlike the more conservative HPC users surveyed previously, show a strong inclination toward tools. Although Table 3 indicates that tool usage is low for some programming activities, this is not because some users simply refuse to use tools. Figure 2 shows that of the 448 respondents, only about one in eight (12%) reports using no tools for any activity. An even smaller proportion employs tools for just one activity. Over half of the respondents currently use tools for more than one task, and among computer scientists, the number swells to almost twothirds. A significant proportion (varying from 21% to 31%) applies tools to all six of the programming activities.

10 0% used for all 6 tasks

50% used for 2-5 tasks

used for one task not used for any tas k

Sci

Eng

CS

Math Oth

All

Figure 2. Use of tools for multiple programming activities. It is not certain from this instrument how the use of any particular tool is distributed across activities. A followup survey will be required to determine if the same tool tends to be applied to multiple tasks, or if the user who finds one tool helpful is then predisposed to try other tools as well. But a comparison of the information in Figure 2 and Table 3 reveals that the problem of poor "tool fit" lies not in users' failure to try tools, but rather the irregular distribution of tool use across programming activities. (Of course, considering the difficulty of developing parallel applications, even a figure of 12% for non-tool users is probably too high.) Finally, it is important to compare where programmers are spending their development effort with where they are employing tools to facilitate those efforts. The responses establish that, for all disciplines, efforts are apportioned to parallel modifications, debugging, tuning, serial modifications, model improvement, and execution setup, in that order. Further, the first three activities account for two-thirds of programmer efforts. If we look at how tools are employed across those activities (Table 3), however, it becomes obvious that some 40% don't use any tools for the most time-consuming activity, parallel modifications, and almost one-third of the respondents don't use any tools for debugging or tuning. The implication is clear: tools are not as effective as they could be in improving programmer productivity, since they are under-utilized in the most effort-intensive activities.

10

Conclusions The SC'93 instrument posed the kind of basic questions -- concerning the allocation of effort in developing parallel applications and preferences in tool origin -- that should be of importance to any tool developer. The answers offer a good indication of where developers should be concentrating their own efforts if they are to attract parallel users. Several areas are promising. First, tools could provide a much better match for program development methods. For example, since a significant proportion of parallel programmers start by modifying a serial program, tools should help them compare the behavior, performance, and results of serial vs. parallel versions. Second, tools should do a much better job of meeting user requirements. The fact that users turn to internal, often ad hoc tools instead of commercial or publicdomain products is evidence that they find the latter too hard to use or poorly suited to their program development activities. It cannot be simply a matter of generalized vs. domain-specific tools, since a number of current public-domain offerings are domain specific; in this context, it is interesting to note that computer scientists, not scientists and engineers, are the ones who are likely to give public-domain tools a try. Third, tool development can potentially yield higher pay-offs when directed at those program development activities where users spend the most time. Although our survey cannot provide a blueprint for tool success, its results help clarify the nature of current tool (under-)utilization. In the absence of statistically relevant information, developers have had to rely on intuition and introspection to determine user requirements for parallel tools. The SC'93 survey is the first in a series of instruments which will establish the kind of baseline evidence on user patterns that is sorely lacking in today's parallel computing community.

References 1.

Adelson, B.: When Novices Surpass Experts: The Difficulty of a Task May Increase with Expertise. Journal of Experimental Psychology: Learning Memory, and Cognition. 10 (3): 483-495 (1984). 2. Cook, C., C. Pancake, R. Walpole: Supercomputing '93 Parallel User Survey: Response Summary. Department of Computer Science, Oregon State University, Technical Report 94-80-2 (1994). 3. Hesse B., L. Sproull, S. Kiesler and J. Walsh: Returns to Science: Computer Networks in Oceanography. Communications of the ACM, 36 (8): 90-101 (1993). 4. Mayer, R. E.: The Psychology of How Novices Learn Computer Programming. ACM Computing Surveys. 13 (1), 121-141 (1981). 5. McKeithen, K., J. S. Reiman, H. Reuter and S. C. Hirtle: Knowledge Organization and Skill Differences in Computer Programmers. Cognitive Psychology. 8: 336-356 (1981). 6. Moran, T. P.: An Applied Psychology of the User. ACM Computing Surveys. 13 (1), 1-12 (1981). 7. Myers, G. J.: A Controlled Experiment in Program Testing and Code Walkthroughs/Inspections. Communications of the ACM. 21 (9): 760-768 (1978). 8. Norman, D.: Collaborative Computing: Collaboration First, Computing Second. Communications of the ACM, 34 (12): 88-90 (1991). 9. Pancake, C. M.: Graphical Support for Parallel Debugging. Software Support for Parallel Computation, ed. J. Kowalik and L. Grandinetti. Springer-Verlag, pp. 216-228 (1993). 10. Pancake, C. M.: Customizable Portrayals of Program Structure. Proceedings of the 3rd ACM/ONR Workshop on Parallel and Distributed Debugging, pp. 64-74 (May 1993).

11

11. Pancake, C. M.: Why Is There Such a Mis-Match between User Needs and Parallel Tool Products? Keynote address, 1993 Workshop on Parallel Computing Systems: A Dialog between Users and Developers (April 1993). 12. Pancake, C. M. et al.: results of user surveys conducted on behalf of Intel Supercomputer Systems Division, two divisions of IBM Corporation, and CONVEX Computer Corporation (1989-1993). 13. Pennington, N: Comprehension Strategies in Programming. In Proceedings of the Second Workshop on Empirical Studies of Programmers, pp. 100-113 (1987). 14. Pennington, N: Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs. Cognitive Psychology. 19: 295-341 (1987). 15. Sheil, B. A.: The Psychological Study of Programming. ACM Computing Surveys. 13 (1), 101-120 (1981). 16. Sheppard, S. B., B. Curtis, P. Milliman and T. Love: Modern Coding Practices and Programmer Performance. IEEE Computer. 12 (12): 41-49 (1979). 17. Soloway, E. and K. Erlich: Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering. SE-10 (5): 595-609 (1984). 18. Vessey, I. and R. Weber: Research on Structured Programming: An Empiricist's Evaluation. IEEE Transactions on Software Engineering. SE-10 (4): 397-407 (1984). 19. Youngs, E. A.: Human Errors in Programming. International Journal of Man-Machine Studies. 6: 361-376 (1974).

12