From: LEPTON::PALAZZI "29-Aug-1990 1704" 29-AUG-1990 17:38:42.83 To: @wmap.dis CC: PALAZZI Subj: draft of ECP/PT work map, as promised CERN, European Laboratory for particle Physics ECP Division PT Group WORK MAP OF THE PROGRAMMING TECHNIQUES GROUP AND EXTRAPOLATION TO THE LHC P. Palazzi, 23 August 1990 Draft, not for distribution --------------------------------------------------------------------------- Contents: 1. Current activities 2. Project affinities 3. Technologies involved 4. Towards the LHC Appendix A: Detailed project descriptions: ADAMO, ALEPH/ADBS, ALEPH/DALI, FIND, PDG DATABASE, GISMO, GSS/SW, L-3 SLOW CONTROL, STING. Appendix B: Software Engineering Methods and Tools Appendix C: Hypertext Appendix D: Object Oriented Programming Appendix E: Distributed Processing 1. CURRENT ACTIVITIES --------------------------------------------------------------------------- Most of the members of the new PT Group in the ECP Division are currently engaged in established projects. These are, alphabetically: Project People In collaboration with --------- ------------------- --------------------- ADAMO P. Palazzi RAL, ZEUS ALEPH/ADBS F. Jacot ECP/SA, PPE ALEPH/DALI H. Drevermann PPE, Copenhagen FIND K. Gieselmann CN GISMO K. M. Storr et al. PPE, SLAC, Washington GSS/SW F. Chevrier et al. ECP/LI LAA/MSL B. Van Eijk et al. L-3 SLOW CNTRL J.M. Le Goff et al. PPE PDG DATABASE K. Gieselmann PPE, Berkeley STING D.M. Sendall et al. AT All these projects are described in detail in appendix A, apart from LAA/MSL. Moreover, possible new activity areas and projects are being proposed by present or future group members and fellows. They are explained in appendices B to E. 2. PROJECT AFFINITIES --------------------------------------------------------------------------- Comparing the different projects of ECP/PT, two important fetaures clearly appear: 1) most of them are part of an experiment and are currently in a running stage. 2) all of them could be grouped into five main fields of interest: - Information and Documentation Management on various platforms - Data Bases and Data Structures Management and User Interfaces - Data taking, Data control and Data distribution of slowly changing parameters - Graphic User Interfaces for event display and data monitoring - Evaluation & Use of New Programming Techniques for future projects Inside the same field, projects will benefit one from the other since they have different backgrounds or experience, and have used various approaches to reach the same goals. In a first approximation the possible correlations are listed below: INFORMATION MANAGEMENT : STING, FIND, HYPERTEXT DATA BASES, DATA STRUCTURES : ADAMO, ALEPH/ADBS, PDG DB DATA CONTROL : L-3 SLC, GSS GRAPHIC USER INTERFACES : ALEPH/DALI, L-3 SLC GRAPHIC, GSS GRAPHICS SW DEV TECHNIQUES : SE METHODS AND TOOLS, GISMO, DISTRIBUTED PROCESSING These affinities will be specially relevant when new projects will be started in the LHC context. 3. TECHNOLOGIES INVOLVED ------------------------------------------------------------------------ Most of the modern software techniques available for HEP have been used by people now in the ECP/PT group, with the exception of Neural Networks and Massive Parallel Processing which where not quite available on the market at the time these projects went into the design phase. All the techniques can be grouped into the following families: - Data Bases of various kinds - Expert Systems, from High Level ones to Low Level and quasi-real time ones - Object Oriented Languages on different platforms: MAC,NEXT, etc.. - Real Time Techniques under VMS and OS9 - Traditional and Modern Graphic User Interfaces on various Workstations - Communication Packages: DECNET, TCP/IP and RPC on different computers Many of these tools are currently being inproved drastically, and it is likely that we will find in the near future suitable platforms to use most of them effectively for existing or new experiments. 4. TOWARDS THE LHC ------------------------------------------------------------------------ It appears that the inpact of software technologies on the LHC can be important at five different stages: - Design and Simulation - Data Acquisition of the events and the related triggering strategies - Data Acquisition of the control parameters - Data transfer and Storage of events - Event processing and analysis, and graphic display on a desktop computer The stages are pipelined, with important overlaps. Consequently the same techniques can be used in very different contexts, making the process of software production inside the experiment more efficient. The good performance of an experiment will depend more and more on the choice of the proper techniques. Clearly ECP/PT can bring important contributions to the five stages, namely - A major effort of organization and information will have to be undertaken if one wants to survive the complexity of an LHC experiment. In this context an efficient Documentation Distribution and Management Scheme will be highly appreciated. - It is easy to anticipate that DATABASES of all kinds will be among the preferred tools for LHC data storage, apart from event data. Again the strong background of PT in this field could be an important asset. - No experiment can run without a well structured, efficent and reliable Data Control System. The different types of slowly changing parameters will very likely remain the same and probably only there numbers will change by an order of magnitude. Almost all the work already done in this field can then very easily be translated into the LHC world thanks to the expected improvements of the tools presently used in the LEP experiments. This is especially true for Expert Systems and Object Oriented Languages. - Graphic User Interfaces are growing in importance, potentially allowing everyone from any desktop available at CERN to perform all the tasks related to event viewing, data histogramming and data control regardless of the complexity of the network of computers involved behind the scene (thanks to distributed processing) - Finally software engineering methods and tools are needed in all the phases of the lifecycle of any non-trivial project, in order to produce better systems at a lower cost, within a predictable time and manpower frame. The PT group will have the know-how to try out and recommend suitable techniques for program design, construction, testing, project management and documentation. APPENDIX A: DETAILED PROJECT DESCRIPTION ---------------------------------------------------------------------------- The following projects are described, as contributed directly by the people involved: ADAMO ALEPH/ ADBS ALEPH/ DALI FIND, PDG DATA BASE GISMO GSS/ SW L-3 SLOW CONTROL STING For most of the projects, the following items are covered: 1. description 2. current status 3. personnel allocation including non staff 4. programming techniques and software products used 5. problems 6. short-term evolution 7. implications for the LHC ALEPH/ ADBS, FIND and PDG DATA BASE are covered by a short description only. ADAMO P. Palazzi --------------------------------------------------------------------------- 1. description -------------- ADAMO is a system to design and manipulate data, targeted at particle physics online and offline software. It is based on the ideas of the Entity-Relationship (ER) model, and has been originally developed in the ALEPH collaboration. It is now used also by ZEUS at DESY, FENICE at Frascati, Obelix and LAA at CERN. It consists of: - data design and dictionary management tools: - language binders for FORTRAN, C, VAXFORTRAN, PROLOG, SQL; - a package to manipulate data from FORTRAN programs; - I/O drivers for ZEBRA, BOS and ORACLE; - an interactive data manipulaton system coupled to PAW. - installation kits for various operating systems. 2. current status ----------------- The version 3.1 is operational on VMS, VM, MVS/NEWLIB and various UNIX platforms. 3. personnel allocation including non staff ------------------------------------------- ADAMO is a joint project with RAL, other Institutes taking part in the ZEUS experiment, and the World Lab. P Palazzi CERN/ECP/PT coordination and documentation; S M Fisher RAL engineering and maintainance; Ch Roberts RAL interactive manipulation module, tests; Several member of ZEUS have ported the system to various OS platforms and helped with the corresponding installation kits. Two students from COPPE, Rio de Janeiro, also supported by the World Lab, are working on their master thesis related to ADAMO: C Maidanchik: merging specifications, code and documentation, possibly also automatic test generation; G Xexeo : combining ADAMO with the IDAL interactive language by T. Burnett 4. programming techniques and software products used ---------------------------------------------------- The system was designed using ER, SA and a crude form of OOD. It uses ZEBRA internally for storage management, was developed on VAX VMS, using CMS, PCA, LSE, DTM. The interactive interfaces are written with KUIP. 5. problems ----------- A programmer based at CERN is required, to take over engineering and maintainance. A Users Group with an electronic Newsletter is needed. 6. short-term evolution ----------------------- The documentation of version 3.1 is being converted from SCRIPT + native help files to LATEX + KUIP. The design of version 4.0 is in progress, featuring extensions to the model for OOP, and an internal rewriting of the kernel to make it more robust and suitable for concurrent data access and distributed processing. The port to CRAY is almost complete, other UNIX versions (NEXT, DECstation) are on the way. 7. implications for the LHC --------------------------- The use of the ER model is now pervasive in the Software Industry, and adequate support tools for it are appearing on the market. The experience with ADAMO indicates that a global data strategy based on the ER model is of great help in producing and maintaining software for an experiment. The data structures are language independent, and transfer of data across different programming environments is easy. It appears that such an approach could be very beneficial for an LHC collaboration. ALEPH/ ADBS F. Jacot ------------------------------------------------------------------------- ADBS is a database system to describe the ALEPH detector. The information is used by several programs to simulate events, reconstruct them and display them graphically The experiment consists of several subdetectors . This requires a large number of alignment and calibration constants varying with time, maintained by ADBS. On the same data-base is also kept the description of all data structures created in any ALEPH program. All ALEPH data structures are described according to the Entity-Relationship model, using the ADAMO Data Definition Language (DDL). Using ADAMO, tables of data objects and their relationships are mapped onto banks of a memory-manager and also a documentation system based on a common data-dictionary is provided. The data-base is updated each time the dictionary changes or the detectors'constants are modified. I am also involved in the Bookkeeping system for Aleph data production, a facility to access at any time the information on all tapes, cartridges, or disk files containing relevant information for the offline analysis of Aleph. The information is provided automatically by the production programs. The Bookkeeping data-base is updated daily, on IBM, by an automatic program, controlled by an IBM exec file and copies are done on VAX. ALEPH / DALI H. Drevermann --------------------------------------------------------------------------- 1 and 2. description and current status --------------------------------------- DALI is the major event (and detector) viewing system of ALEPH with about 150 users at CERN. It is used extensively for scanning of events. It serves to generate colour transpaencies for talks and black and white pictures for publications. Coloured pictures of DALI are often used for public relation purposes. DALI may be used interactively and via batch. It is conceived to run fast ( some seconds per picture ) and efficiently even on cheap workstations in order to allow many physicists to use DALI on "private" workstations. Conventional 3D techniques, which need smooth rotations, are not used, as rather expensive workstations are needed. Instead nonlinear transformations are employed to produce clear images. The 3 dimensional behaviour of tracks is studied by use of several 2D pictures, which are simultaneously displayed similar to technical drawings. In this case colour is used to correlate hits or tracks etc. on different views. The concept of orthogonal views is introduced, which means that two projections, the Y/X and the rho'/Z projection, are sufficient to understand an event in detail. Due to the cylindrical detector shape of ALEPH and due to the radial form of the events the concept of angular graphics is employed, which leads to special projections and special interactive means. The 3 dimensional character of hits and tracks in the TPC is best depicted via the "V-plot", which is a true (non perspective) 3D picture in a mathematical sense. This may be seen from the fact that the cartesian coordinates (X,Y,Z) of all hits and tracks may be retrieved from the V-plot. The V-plot technique has recently been applied successfully to simulated data of the fixed target experiment NA35. Histograms of the calorimetric information may be drawn onto the pictures. The interaction in DALI is done either by use of menues, which give at the same time explicit help information, or by use of a command language, which was developped, to be as fast and short as possible. Several commands may be typed one after the other, but they are checked instantaneously one by one. A scheme of userlevels allows to learn how to use DALI by increasing the user level and thus going from more basic to more complex commands. The menue is conceived in a way to ease the learning of the command language, when using the menue. Commands may be grouped together in macros interactively or by editing a macro file. Macros may run in a loop for demonstrations etc.. These features and the error tolerance of DALI bring about a user friendly system, which suffers only from the high sophistication of DALI. A log-file stores screen output and operator commands even if they are given via the menue. Physical objects (hits, tracks etc.) may be selected by picking. The inverse action: moving the cursor to a selected object, is available,as well, which may be used to correlate objects on different projections (pick and move). Physical quantities (e.g. the invariant mass) may be calculated for interactively selected tracks etc.. Lists of banks, of tracks etc. are available and listed objects may be related to the object images via colours or via picking. Size, symbol type, line width etc. may be set interactively. Text editing and picture editing is further available as well as means to define and alter colours in any detail. Pictures may be stored as metafiles in colour or in black and white and be read back for further modifications or printing. Events are read from BOS in all available formats and may be stored for further treatment. References: H.Drevermann,ALEPH 89-53, Softwr 89-5, Cern 1989 C.Grab , Aleph 89-127, Note 89-7, Cern 1989 C.Grab , Aleph-note 89-12, Cern 1989 H.Drevermann & C.Grab IJMP C,Vol.1,No.1 (1990) 147-163 3. personnel allocation including non staff ------------------------------------------- H.Drevermann CERN 100% program development, user help ... C.Grab (fellow) CERN until 1.1.91 60% program devel., installation ... B.Nilsson Copenhagen 25% user interface, Dec windows ... R.Vogl Innsbruck sum.student 10% data base, detector display ... 4. programming techniques and software products used ---------------------------------------------------- DALI is written in FORTRAN. It has actually 69000 lines of code, which are increased to 83000 lines if commons are expanded from include files. The different projections, the text and picture editor etc. are separated in different processors, which are connected via a bus structure. New procesors can easily be added. A scheme to add and call user supplied processors exists necessitating only to relink DALI. Input and output are grouped together in seperate packages. The replacement of BOS by another system will pose no problem. However many projections are specially developped for LEP-like experiments. A recent modification of DALI to be used by the fixed target experiment NA35 was easy to accomplish to produce useful pictures. A more elaborate adaptation of DALI to NA35 needs however substantially more effort. The program is based on UISDC and UIS (for metafiles) and on GKS in the case of batch mode, where it runs on the IBM. The change to another graphical base is extremely fast and simple as long as graphical output alone will be used, as DALI is built on only a few basic graphical output functions. The use of colour metafiles requires the use of segments, which are normally not used. The conversion of a stripped down version of DALI, which does not rely on graphical input, was done successfully several times. It produces the same picture as DALI itself, however by use of more demading operator interaction. The conversion of the graphical input to other systems may be more difficult. The interactive package necessitates the use of VMS-QIO's. The batch program needs on the IBM some VM specific system routines. CERNLIB routines are used as well Input data are stored via BOS and read using the ALEPH-JULIA package. 5. problems ----------- Replacement of C. Grab. 6. short-term evolution ----------------------- Visualisation of objects which still misses implementation: a: track extrapolation from the ECAL to the Muon detector. b: TPC wires c: TPC pads: individual pad information d: Endcap view e: Track-calorimeter correlation f: neutrals g: jets h: ..... Refinement of the help procedures. Conversion from UISCD/UIS to DEC-windows. The accomplishment of these tasks will take more than one year. Their priority is not yet defined. 7. implications for the LHC --------------------------- DALI utilizes special methods to viualise jets of many tracks. It has been shown recently that 385 non-radial straight tracks traversing a volume of 2m*1m*1m in different directions can easily be visualised by use of a modified V-plot. This might be usable for LHC if tracks in tracking devices, which give 3D coordinates, are to be visualised. The concepts of orthogonal projections and angular graphics would be very useful, if event viewing is foreseen. Due to its special projections it might be a diagnostic tool for subdetector groups. The mode of operator interaction, which was specially developped for DALI, should be considered, when conceiving new graphic packages for LHC. FIND K. Gieselmann ------------------------------------------------------------------------- FIND, running on VM/CMS, is an information and help facility available on the CERN Central computers. I am responsible for monitoring the usage of this tool, and this involves evaluating users' needs, creating and writing helpfiles as well as improving existing ones, and access to a large number of SERVICE machines. The project leader is B.Pollermann from CN Division. PDG (Particle Data Group) DATA BASE K. Gieselmann ------------------------------------------------------------------------- I am technical associate and contact person at CERN, responsible for the maintenance of the MESON section of the Particle Data Group database in Berkeley. This work requires knowledge of ORACLE /SQL language and of the RPP Database Encoding Schema, as well as a basic understanding of Particle Properties. The project leader for CERN is L. Montanet from PPE Division. GISMO K.M.Storr --------------------------------------------------------------------------- 1. description -------------- A project to investigate the concept of Object Oriented Programming (OOP) and its applicability to the problems of High Energy Physics by developing a detecto design, simulation, and reconstruction package. The aim of the project is to design and develop a single program package in which the following, normally separate, functions are combined, and are accessible from a single graphical user interface. - Graphical detector design - Event generation - Detector simulation - Event reconstruction - Detector response and event display The idea is to take advantage of the inheritance, encapsulation, and re-useabili concepts of OOP to allow stepwise refinement, independent module development, and code sharing. It is expected that this will speed up the detect design and code development processes, will limit re-invention, ie. the parallel development of 'similar' code in different programs, and hence ensure a high degree of code and data consistency and integrity throughout the analysis chain. It is not intended to rewrite the complete physics analysis chain, but rather to interface to existing tried and tested algorithms, kernel library functions, and physics generators. The initial aim of the project is to use this exercise as a learning tool to acq skills, learn new techniques, and to assess how best they can be employed. In this sense the idea is to investigate concepts rather than to aim at a product. 2. current status ----------------- The project is less than 2 months old and to date much of the time has been spent on a preliminary design of GISMO, and on learning about, and investigating the capabilities of, the NeXT application kit software and objective C. A schematic storyboard has been designed and work has started on the implementation of a complete single path through it. 3. personnel allocation including non staff ------------------------------------------- Bill Atwood (CERN/Aleph, on leave of absence from SLAC): sub-detector design, definition, and methods. Toby Burnett (CERN/Aleph, on leave of absence from Univ. Washington): classification of objects, design, documentation. Robert Cailliau (CERN/ECP): graphics, software engineering techniques. Mick Storr (CERN/ECP): project administration/coordination, NeXT system support, GISMO user interface. 4. programming techniques and software products used ---------------------------------------------------- The aim of the project is to investigate the applicability of OOP techniques to problems of HEP. The hardware and software platform chosen for the project is the NeXT computer, NexTStep, and obective C, with FORTRAN being used where necessary, eg. when existing algorithms are used. 6. short-term evolution ----------------------- An initial phase of 6 months, started 1/7/90, was foreseen in which to develop t design of GISMO and to demonstrate the implementation of one complete path through the storyboard. The project should then be reviewed to assess whether it should continue, in which direction, eg. towards the production of a product, or whether it should be shelved. 7. implications for the LHC --------------------------- Because the aim of this project is to investigate whether the concepts of OOP ca be applied to the software development of future, eg. LHC, HEP experiments, GISMO should already be viewed as part of the LHC research and development effort. GSS/SW F. Chevrier -------------------------------------------------------------------------- 1. description -------------- The name of the project is GSS (General Supervisory Systsem). The aim of the GSS project is to watch over the environnement in the 4 LEP experimental sites in order to prevent accidents happening . The GSS/SW project had to develope the GSS software programs. 2. current status ----------------- The system is operational on the 4 experimental sites. By the end of 1990, the GSS program sources will handed over to Edo Sbrissa's GSS maintenance team. 3. personnel allocation including non staff ------------------------------------------- Francois CHEVRIER: project leader Jean Pierre PUGET: graphic facilities Belinda CHAN and Iain PLANK: communication programs. Laurence VINOT: expert system Chantal LESTIENNE: creation of the graphic images, designs the man machine interfaces, documentation. Jeannine FRAISSE: system management (7 Vaxes). 4. programming techniques and software products used ---------------------------------------------------- We are using expert system programming techniques. The products are namely : GENESIA-I-TR AND GENESIA-II, developped by EDF and marketed by STERIA (Paris) Laurence VINOT is a doctoral student. She is working on the real time aspects of an expert system and collaborates with EDF and STERIA about real time expert system methods and new improvements of GENESIA products. 6. short-term evolution ----------------------- For the 6 next mouths, the people involved will continue to work full time on the GSS poject. 7. implications for the LHC --------------------------- There might be many implications of the sotware we produced and the methods we used in view of the LHC experiments. - Real time management of the LHC experiment data acquisition systems. - Trigger tuning or particle identification. - Automatic code generation - Collaboration between a neural network and an expert system. - Any project which needs intelligent reasoning, learning mecanism... L-3 SLOW CONTROL J.M. Le Goff --------------------------------------------------------------------------- 1. description -------------- The L-3 SLOW CONTROL monitors and controls all the Slowly changing para- meters related to the operation of the experiment. It includes the control of the L-3 MAGNET and the L-3 GAS SYSTEMS. The informations collected come from various subdetectors machines (VAX/VMS & VME/OS-9 computers). It also receives informations from GSS and the LEP Machine. A microVAX 3200 is dedicated to cen- tralize and analyse all the informations received according to the physical relations of various parameters with others. Depending on the current situation in the experiment, the SLOW CONTROL will take various actions like computer broadcasts to inform users and eventually rack powering off and/or detector parts in case of major troubles. A portable Graphic Display allows the user to inquire at any moment the current situation of the L-3 experiment. 2. current status ----------------- Production .The complete system has been now running for one year with satisfactory results. 3. personnel allocation including non staff ------------------------------------------- - Staff: H. Cabel : ORACLE SLOW CONTROL DATA BASES Upgrades and Maintenance. R. Stampfli : VME OS-9 Operations and developments for GAS & MAGNET. - Fellow: F. Diez-Hedo : TCP/IP Communications between MAC and VAX/VMS and others. - Cooperant: R. Barillere: Object Oriented C on MAC for the Graphic Package. - Technical students: G. Berthet : High level Expert System for Daily analysis of the alarms. C. Bussod : same as above I. Rodriguez : same as above - Visitor from MIT/LNS: H. Milcent : VMS and all real time aspects of the SLOW CONTROL 4. programming techniques and software products used ---------------------------------------------------- on VAX VMS : Low Level (real time) Compiled Expert System (GENESIA 1 EDF FRANCE) High Level Expert Systems order 1 logic (GENESIA 2 EDF FRANCE) ORACLE, PRO*C for automatic triplets generation. VMS real time programming including Shared Memory and Process synchronisation. DECNET & WALLONGONG TCP/IP interprocess communication on VME OS-9: Drivers Descriptors in Assembly, shared Memory and interprocess synchronisation in C language, DECNET interprocess communication. MACVEE parallele I/O access to VME BUS (will be replaced by TCP/IP) on MAC IIx: THINK C Object Oriented Language, MAC TCP, ResEdit 2.0 CANVAS 2.2 QuickDraw and MAC / VAX interprocess communication via TCP/IP. 5. problems ----------- Except for Staff people all the others will be gone for the next LEP running period (in 1991). TCP/IP communications through the L-3 Bridge. Solution: Use DECNET to get out of the L-3 area and then use VAX TCP/IP. 6. short-term evolution ----------------------- on VAX VMS : Extended Software actions through the LEP services: WATER VENTILATION and ELECTRICITY in order to take actions before the situation deteriorate too much. (this mostly depends on the LEP SERVICES communication Status with the LEP Experiment). on VME OS-9: Extension of the Hardware safety system (BBL3) and of the software related to it (readout and analysis). on MAC IIx : Extension of the MAC IIx / computer to other machines using TCP/IP i.e: with IBMVM ,VME OS-9 and UNIX Machines. Version 3 of the Graphic display including on line changing gauges and thermometers. 7. implications for the LHC --------------------------- Most of this software is directly portable to the LHC experiments i.e.: the MAC Graphic System and the ORACLE strategy. All the VAX VMS Software call also be portable provided VMS will be choosen at the top of the data taking stream, but I would recommend to start working with more powerful and more realtime expert systems which are starting to become available on the market. On VME OS-9 except for the driver parts which is CPU related most of the software could be reused since we mostly use DAC, ADC and PIA (In/Out register boards). This system being very versatile and fairly modular I believe that most of it could be reused or rearranged according to the environment provided in the LHC. I any case we are already using bits and pieces of this software for some other little project like the MONITORING, CONTROL and DATA ACQUISITION of the EF-L-3 Spectrometer which we have decided to move to VME OS-9 in view of the tests of new R & D crystals for the LHC detectors (see P. LECOQ DRDC / 90-25/P2) STING (Software Technology INterest Group) D.M.Sendall --------------------------------------------------------------------- 1. description -------------- The aim of STING is to allow people who are using new software technologies to exchange information and experience on methods and products. This should help them to pool resources in an informal way, spread knowledge of new techniques and ideas, and guide the policy of support groups. Practical questions of organisation will be discussed with the participants. In addition to meetings, presentations and newsletter articles, we hope to set up a computerised infrastructure to help exchange information. We propose to be fairly open in our initial range of subjects, and to be guided by the response we get. 2. current status ----------------- There has been a very encouraging response to the initial announcements, with over 50 replies covering almost all CERN Divisions and many outside institutes. The range of topics is correspondingly wide. Our first news bulletin will go out in a few days, and we plan to hold a first meeting early in September. At this meeting we hope to agree on a first definition of aims and activities, as well as a calender of meetings and presentations. We shall also have a first discussion of training needs, and of methods for communication and access to documentation. 3. personnel allocation including non staff ------------------------------------------- For the present Mike Sendall is working about 30% on this, with the other two originators (A.Daneels/AT and P.Palazzi/ECP) working for a similar fraction of their time. As the project proceeds we assume that other contributors will devote a substantial amount of their time to it. We shall probably need help to set up the infrastructure once it has been defined, and perhaps also some administrative help, if only at a fairly informal level. 4. programming techniques and software products used ---------------------------------------------------- For the moment the infrastructure of the project is based on electronic mail on VM, VMS and UNIX, plus NAMES files on VM. This will not be adequate beyond the first launch, and mechanisms will be needed for keeping a database of users, topics, documents, products etc, and for distributing and accessing news and documentation. Decisions on these will depend to some extent on users' preferences, but it might be interesting to use the project as a test bench for new techniques such as hypertext. 5. problems ----------- A modest project of this sort needs an infrastructure, including such things as the management of electronic mail distribution lists, document management and distribution, possibly news and conferencing schemes, and the setting up of a small database linked to the official databases already in existence. The confused state of policy and support in these areas means that every such project has to cobble together its own solution, which is likely to be ephemeral and unsatisfactory. 6. short-term evolution ----------------------- Goals for the next 6 months: - Agreed statement of aims - Good programme of presentations - Working infrastructure for news and documents - Active working groups in several key areas - Recommendations to technical training committee - Some coordinated evaluation of products launched 7. implications for the LHC --------------------------- We shall not be producing software as such. However, if we assume that modern techniques and software produced using those techniques will be essential for the success of LHC experiments, we shall have a role to play in such areas as: - Spreading knowledge and exchanging experience - Evaluating tools and methods - Related support activities APPENDIX B: SOFTWARE ENGINEERING METHODS AND TOOLS J. Bunn A PROPOSAL FOR A NEW ACTIVITY -------------------------------------------------------------------------- The Programming Techniques (PT) Group will assist experiments in the selection and use of software methods and tools. Good progress has already been made in this field by the HEP community in recent years. The consolidation and improved coordination of this work will be undertaken by the Group. This will require full consultation with the current users in order to identify new needs, required improvements, obsolete items and so on. The Group will thus act as a focal point for discussion about, and choice of, software methods and tools. Other users will be encouraged to start employing software engineering (SE) techniques. However, the emphasis will always be on *accessibility*, *ease of use* and *simplicity* of any recommended techniques. The Group intends to make full use of suitable commercial and public domain tools, and will negotiate licenses where appropriate. Goals ----- o To improve HEP software design o To help ensure portability of code o To aid in making full use of existing software modules o To promote wider use of software engineering methods. The Group hopes to provide experiments intending embarking on new software project with advice on which methods and tools will be most useful for them. Up until now, there has been no clearly identified source for such information. The intention is to make a study of software tools available commercially, in the public domain, and from within the HEP community, with a view to making available a coherent set to interested parties. Existing HEP tools and methods will be consolidated and maintained. (These presently include the ADAMO data management suite and FLOPPY/FLOW programs.) It may well happen that, for some parts of the software design/validation/testing phases, the need for a tool not available commercially or in the public domain is identified. In these cases, PT Group would, after evaluating the needs, produce the required tool. Strategy -------- The Software Engineering field is rapidly developing. PT Group will attempt to keep programmers in experiments informed of significant developments. It will encourage them to make use of suitable commercial tools for which licenses will be procured. Training and documentation will be provided for these tools, and for existing proven methods, such as the ADAMO data management system. PT Group will advise on all phases of the "software life cycle", from problem analysis, through software design, through implementation, with debugging tools, through automated testing, to installation and maintenance of code. APPENDIX C: HYPERTEXT / HYPERMEDIA R. Cailliau A PROPOSAL FOR A NEW PROJECT -------------------------------------------------------------------------- Hypertext is a form of information storage that is a web of interconnected nodes. The nodes contain information of some kind (document, program, picture, sound, ...), the web lines are ways of getting from one node to another. Searching in hypertext is aided by various mechanisms, each of different speed and efficiency (there can be indexes and keyword lists). However, the user can also navigate by following paths or just hopping along existing links in a random manner. Thus there are no restrictions on how a node of information is reached (in more classical systems one usually has to know part of the location of the information). In addition, users can construct their own additions to the set of links. The information nodes reside on different machines, the links across machines are handled by hypertext servers resident in the machines. In the context of LHC practically all information will have been generated by machine. However, there is no unified way of accessing this information by machione: different platforms are used and different formats have been used to create the documents. Hypertext seems at the right stage of maturity to introduce it as the unification of these otherwise incompatible sources of data. The pilot project would aim at giving a minimum level of access to LHC related information from any workstation. No attempt would be made to implement a large range of features, thus one would not provide uniform access to bitmapped pictures on all platforms, or to fully formatted text, since the formats for pixels, character fonts etc. are too divergent on different systems. However, one would attempt to provide for a small number of the most popular formats, and just use plain alphanumeric text in other cases. This is the only road to success. Phases are: - look at commercial products & get educated, - select the level of services that can reasonably be implemented and the platforms on which they would be provided, - buy all components that can be bought, - integrate, - populate. Population should be done by incorporating existing documents only. The first phase takes about 6 months. The introduction of this technique cannot be done in isolation for one project: it has to be done in conjunction with other services and groups such as CN, the administration of CERN, and the accelerator divisions. Technical details of the project are to be found in a proposal by T. Berners-Lee of CN, with whom I would work closely. APPENDIX D: OBJECT ORIENTED PROGRAMMING D. R. Myers AND ITS APPLICABILITY TO ECP SOFTWARE PROJECTS -------------------------------------------------------------------------- Although Object Oriented Programming has been around for a long time, it continues to gain adherents. For its use at CERN, one needs to evaluate the applicability of the paradigm to some of our particular problems, as well as the various tools and languages available. Personally, I see C++ becoming the most widely used OO language because it is so widely available and is supported by AT&T, even though the most suitable language may well be Eiffel. Unfortunately, the implementation of Eiffel available at present leaves much to be desired, and it will be necessary to follow developments to see if things improve. In addition to C++ and Eiffel, one cannot forget Objective C, which is used on the NEXT, and, for purists, SMALLTALK 80, which also continues to become available on more machines. Although Objective C currently is only supported by a single small company, it could become more important if NextStep takes off on the IBM 6000 platform. Although languages like Modula 2 and Ada contain elements of OOP, I do not think that they are relevant to the present discussion. On the side of applications one could target three areas initially: (1) Event Reconstruction and Simulation (2) OO design (3) Data Aquisition Point (1) is already being addressed by the GISMO project, which also should provide some experience with Objective C and NextStep. I have no specific proposal for point (2). However, it does seem that if system design were to be done in an OO fashion, and the programming carried out in an OO language, then one might be able to avoid the step of converting bubbles to Fortran. Maybe this is just a pious hope, but it is clearly something that the Eiffel people had in mind. Perhaps someone can be found who would like to investigate this area? As to point (3), the OC group (or its reincarnation), were already interested in such an investigation, and I should be interested to collaborate if a project can be defined. In addition to these three areas, the REASON project at SLAC is already looking at the question of data analysis, and some people in the accelerator divisions could well argue that they have been using OO techniques for years, although not via any of the 'standard' OO languages. Certainly it should be possible to maintain contact with SLAC, and maybe something could be done also in the area of accelerator controls. I would suggest that within the Programming Techniques group an attempt is made to organize projects in the three areas listed above, sometimes in collaboration with other groups. The important point is that work on OOP is coordinated from one place, that a forum is available for those people working in the area to share experiences (*), and that [informal] reports are written from time to time on particular topics, or on good/bad experiences people have had with different products or using different techniques. It is undoubtedly the case that programming in an OO language is very different from the procedural style to which we are all accustomed. It will take time and experience to discover the best (or any reasonable) way to define Class hierarchies and Messages. A point to be born in mind is that whilst it is extremely fruitful for both sides to have visitors work on projects, if OO techniques are to take hold within CERN it will be necessary to have a 'critical mass' of (semi) permanent CERN staff sufficiently well versed in the subject. -------------------------------- (*) Q: How many programmers does it take to write an OO program? A: Five; One to write the code, and four to share the experience. (With apologies to Californian electricians.) APPENDIX E: DISTRIBUTED PROCESSING I. Zacharov (Fellow) AND RELATED MATTERS -------------------------------------------------------------------------- The most likely computational model in the LHC era will consist of loosly coupled workstations (tape/file servers) running the UNIX operating system. The investigation headed by Les Robertson lead to a prototype proposal, based on the NQS batch sceduling system (J-P.Baud, J.Bunn, D.Foster, F.Hemmer, E.Jagel, J.Joosten, O.Martin, L.Robertson, B.Segal, R.Tobbicke, "Shift", CN-Division,CERN). 1) My project in this framework is to automate the implications of the data parallelism. I'm investigating how the data can be organized automatically to run the same job in parallel on several workstations to speed up the same computational task (e.g. event reconstruction or analysis). For this perpose I have developed a software device driver based on the UNIX SYS5 Streams. This device driver intercepts the I/O operations of a (fortran) program and instructs an "Event Server" to lock an appropriate event for that job. This "Event Server" is developed using the NCS (Network Computing System) and it can run anywhere on the network. This work has started in jan/feb. 1990. It is pausing, waiting for a sutable application to be tailored to. It is not clear how the speed of the operation will be for a realistic application on the network. A problem is also the organization of the output and most notably the hbook type of output. 2) Together with Gail Hanson et al. (OPAL) I'm trying to setup a computer farm for OPAL's computational needs. The idea is to distribute all events equally on a large number of workstations. The analysis jobs are started where the data are. Jamie Shiers is helping to make an event directory based on FATMAN. I'm writing a remote hbook server based on NCS to collect the hbook output of the related groups of programs. This project is several weeks old and there might be other people involved. The farm should stand in one year from now. The hardware configuration is not established yet except that it is based on UNIX. The programming techniques to be used to support computation on such a farm must be developed. My remote hbook server is just a try in this direction. But it should contribute to the understanding of the distributed computing environments. 3) The programming language of the LHC experiments can be other than F77. Moreover, in the framework of the computational model stated above it should be better something else than F77. In the ECFA subcommitee of Rene Brun I have compared the implications of Fortran-90 standard and the C++ for the HEP programming. I'm following up the question if the programming based on the Inheritance properties can be exploited by HEP. Inheritance property is not available in the Fortran-90 language standard. But the technique of Inheritance and Objective Programming fits very well in the distributed computing environment.