SoSResultsSummary

1. Q: Should any of the existing topic sections be removed from HPCandBDSurvey?

6 evaluators recommended keeping all existing topic sections.

1 evaluator recommended removing removing two sections:

• Software Environment, provided reason not complete: "The survey should be f" (focused?)
• Bandwidth and ScienceDMZ: "too technical"

SurveyCzar response: All topics will be retained, per recommendation of 6 out of 7 evaluators (one of whom commented that "all are important")

2. Q: Any changes to existing topics?

2 evaluators suggested changes to the HPC storage section; both suggestions concerned cost of disk space. Comments:

• Many researchers have expressed their desire to buy disk space for a reasonable price and without having to pay annually for the space. It might be of interest to ask a question in the survey to identify this need.
• The cost of additional space is high.

SurveyCzar response: A series of questions has been added to section on HPC storage as well as the section on BD computational an d storage. These questions are directed to users involved in purchasing decisions. Background information is provided about Tier 1 and Tier 2 storage options. Users are queried about suitability of costs for their rese arch needs, maximum costs they are willing to pay, and preferred payment schedule.

3. Q: Please provide any other topics that you think should be added to the survey. Say why the topic(s) should be included, and briefly describe the content.

1 evaluator suggested a topic:

• Should ask if researcher has experience with non-NJIT HPC facility and with choices of AWS Google, Azure, Penguin, and Other [w/input box]. Should ask if researcher is interested in evolving (or alternate) HPC technologies such as Google tensor processing unit (TPU), Intel Phi. Also ask if they have an opinion on other filesystems technologies, e.g., IBM GPFS or Google Drive for HPC storage.

SurveyCzar response: A series of questions has been added to the section on HPC computational resources. Users who indicate that they use non-NJIT HPC computational resources are asked to 1) specify the resources, 2) specify the inadequacies of the NJIT resources that drive them to use outside resources, 3) provide additional comments, and indicated their interest in newly available processors (Intel Core i7 or i9, Google TPU, Intel NNP, Intel Xeon Phi processor, AMD Epyc. The questions provide multiple opportunities for unrestricted comments.

4. Q: Word clarity and suggestions

5 evaluators rated clarity of wording as excellent; 2 rated it as good.

1 evaluator strongly recommended changing the wording on multiply-appearing questions about on/off premises shared/dedicated resources.

Existing wording:

• On-premise shared The resource is located at NJIT, is provided by NJIT, and is shared amongst its users
• On-premise dedicated The resource is located at NJIT, and is dedicated to the purchaser of the resource
• Off-premise shared The resource is not located at NJIT, is provided by NJIT, and is shared amongst its users
• Off-premise dedicated The resource is not located at NJIT, and is dedicated to the researcher the purchaser of the resource

Note: Off-premise providers include Amazon Web Services, Azure, Google Cloud Platform, IBM Bluemix, Oracle Cloud, Penguin Computing<

Suggested wording:
• Increase HPC resources provided by NJIT and which are shared among users
• Increase HPC resources at NJIT that are dedicated to a particular user
• Increase the use of commercial cloud computing through access purchased by NJIT and shared among users
• Increase the use of commercial cloud computing through access purchased by user
• Use publicly available HPC resources (e.g., at a national supercomputing center)

SurveyCzar response: The wording of the question has been changed, and the answer choices streamlined; see below.

You have indicated that at least one HPC parallel computational resource on [Kong/Stheno etc.} is inadequate. Potential remediation options involve increases in resources that differ in whether they are:

• Paid for by NJIT versus paid for by individual researchers
• On-premise versus off-premise
• Public resources requiring a successful researcher proposal

Note: Off-premise providers include Amazon Web Services, Azure, Google Cloud Platform, IBM Bluemix, Oracle Cloud, Penguin Computing

Please rank the following options.

• Increase NJIT-provided HPC resources shared among users
• Increase user-purchased HPC resources at NJIT
• Increase NJIT-provided commercial off-premise resources shared among users
• Increase user-purchased commercial off-premise resourcesIncrease use of publicly available HPC resources (e.g., at a national supercomputing center); successful proposal by researcher is required
5. Q: Visual clarity and suggestions

5 evaluators rated visual clarity as excellent; 2 rated it as good.

One evaluator suggested including a progress bar.

SurveyCzar response: LimeSurvey has a progress bar option, but it is a misleading indicator in a highly customizable survey; the bar shows the user's position in the entire sequence of possible survey questions, rather than in that user's particular question subset. Thus, a given user who has answered 90% of his/her particular questions might see the progress bar showing only 30%, followed shortly by an apparently erratic jump to 80%, etc. In short, the progress bar is not useful for this survey.

6. Q: Device and ease of navigation

All evaluators took the survey on a large-screen device, and reported navigation as easy, well-suited, no pro blems, etc. The concern was that users taking the survey on a mobile device would have a different navigation experience, e.g., find it difficult to refer to background information while answering a question, as all information could not be seen simultaneously without repeated scrolling. However, as no one used a mobile device, there is no feedback addressing this concern.

7. Q: How many minutes did the survey take?

Three evaluators took less than 15 minutes; two took 15 - 20 minutes; one took 26 - 30 minutes; one took more than 30 minutes. Hence, five out of seven evaluators completed the survey within 20 minutes.

Additional Comments

1. Remove these questions if not needed: 1) What is the general classification of computations for which you use IST-managed HPC? 2) What is the specific description(s) of the computations for which you use IST-managed HPC?

SurveyCzar response: These questions allow fine-grained analysis of the survey results. For instance, we can see if particular research interests correspond with type of resource usage, adequacy assessments, etc.

2. What department and what college don't make sense for staff, or provide a "N/A or Other" choice.

SurveyCzar response: Academic staff do belong to particular departments

3. Pages are very spread out, has giant 3" margins on my display. The "listing of software...below" was very oddly formatted, three narrow columns of data split by two 4" gray columns, was something missing? Also the "software name" is unnecessarily word-wrapped.

SurveyCzar response: The display issues have not been observed and/or reported by other testers. As to the large table of available software: The display, particularly the gray columns, is the best solution to date of a Lime bug in the table editor; we found no better way to separate groups of information-filled columns. If we can find a better work-around, we will implement it. However, like the "giant 3" margins", these problems have not been observed elsewhere, where the gray columns are less than an inch wide and there are no word-wrapping oddities.

If large margins are present on very wide displays they can be shrunk by horizontally contracting a table.

4. In several of the ranks a choice is "Increase external-successful proposal required" - it is not clear how this differs from the two "Increase off-premise" choices. Later, in storage, there is no "Increase external"?.etc.

SurveyCzar response: See wording clarity suggestions, 4. above

5. Surprised "adequacy of (?) Wiki" doesn't ask for user input on how/why which I picked "Much better needed". Suggest references to Wiki have a pop-out link to the documentation.

SurveyCzar response: a user input how/why question will be added. A pop-out link would be desirable, but unfortunately that behavior is browser-setting dependent and can result in the link opening in a new tab instead of a new window, so the user loses reference to the question for which the link provides information. There were several places in the survey where external links were originally used, but theses links had to be abandoned.

6. The "Science DMZ" was a surprise to me, some explanation might be good.

SurveyCzar response: An explanation of Science DMZ is already provided.

7. Under position status, choices of tenured/tenure-track/non-tenured---(leave out) lecturer, of which some do research, some only teach?

SurveyCzar response: a) Actually, the choices are tenured/tenure-track/non-tenure-track. Czar thinks that non-tenure-track entails lecturer, whether researcher or instructor. b) The Would you like to proceed question indeed only appears if you choose Only members of my research group use IST-managed HPC and BD resources; I myself do not use them directly. However, the follow-up question sa ys Would you like to proceed anyway. Czar believes anyway demystifies the question. However, as a sop to the evaluator, wording has been changed to Would you like to proceed on behalf of your research group...?

8. I am surprised the three "top suggestions" are mandatory.

SurveyCzar response: The question is no longer mandatory, and users may provide up to five suggestions.

9. The ranking interfaces are quite different with and without Javascript; if block such as NoScript is used then interface does a terrible job; later discovered red bar stating Caution: Javascript is disabled in your browser. You may not be able to answer all questions in this survey. Please, verify your browser parameters." but most of the bar is above the top margin and thus unreadable.

SurveyCzar response: The email invitation will emphasize that JavaScript must not be blocked.

10. HPCandBDSurvey is outstanding. I appreciate the effort of ARCS team. Thanks.

SurveyCzar response: It is hoped that the survey results will be an important influence on the NJIT's direction in HPC and BD.