"Bandwidth Gluttony - Distributed Grid-Enabled Particle Physics Event Analysis"
A demonstration at the iGrid2002 Conference
Primary Contacts: Julian Bunn and Bill Allcock
Caltech: Julian Bunn (firstname.lastname@example.org) Iosif Legrand (Iosif.Legrand@cern.ch) Harvey Newman (email@example.com) Steven Low (firstname.lastname@example.org) Sylvain Ravot (Sylvain.Ravot@cern.ch) Conrad Steenberg (email@example.com) Suresh Singh (firstname.lastname@example.org)
DataTAG: Jean-Philippe Flatin (J.P.Flatin@cern.ch)
Using distributed databases at Argonne, Caltech, CERN, possibly UCSD and other HEP institutes, we will show a particle physics analysis application that issues requests for remote virtual data collections, causing these to be moved across the WAN using both striped and standard GridFTP servers. As the collections are instantiated on the client machine in Amsterdam, the analysis results are rendered in real time. This scheme is a preview of a general "Grid Enabled Analysis Environment" that is being developed for CERN's LHC experiments.
The European Nuclear Research Center (CERN) will be bringing the Large Hadron Collider (LHC), the most powerful accelerator in the world, on line in 2006. This system is projected to produce 20 Petabytes of data per year. To handle such large volumes of data, the High Energy Physics (HEP) community is proposing a multi-level or tiered distributed computing system. Tier 0 is CERN and will ultimately hold all raw and reconstructed data for the project. Tier 1 sites are national level, Tier 2 are large universities, Tier 3 are smaller universities, and Tier 4 is the individual researchers. Any data produced at a lower tier, such as analysis results and Monte Carlo simulations, will frequently need to be pushed up to higher levels.
The Globus Project has made extensions to the existing FTP protocol to add security, improved performance, and reliability. This development is called GridFTP. This has been proposed as the lingua franca for transport on the Grid. The HEP community is using GridFTP as part of the LHC project.
Here is a screenshot taken during iGrid2002, showing the ROOT screen while the data files are being downloaded from the remote sites:
The client hardware on the show floor will consist of an 8 node portable cluster. Each node in the cluster is a 2U Compaq DL380G2 server with 1.13 GHz PIII, 512MB RAM, and 6x36GB SCSI hard drives with onboard HW RAID, using RAID 0, dual redundant power supplies, and redundant cooling fans. The node is mounted in a shock mounted case with casters, along with KVM, 1U LCD screen and keyboard for local maintenance, and BayTech network accessible power controllers for remote power cycling, should that be necessary.
Six of the eight nodes in the cluster has a SysKonnect 9821 copper Gigabit NIC. The other two (one of which is the "mayor" or head node) have SysKonnect 9822 dual port copper Gigabit NIC's. These are connected to an Extreme 5i switch, which is a 16 port Layer 3 capable switch rated to be a non-blocking line speed switch. The switch has 12 copper ports, and 4 GBIC ports. During the show, each switch will be connected to our booth router, a Black Diamond on loan from Extreme via (2) Gigabit Fiber uplinks. Our booth router will be connected to the iGrid showfloor network via (4) 1 Gigabit fiber drops. We traveled multiple networks from there to the various sites, but most traffic came across Abilene, ESNet, and Startap.
The client application running on the cluster will initiate GridFTP transfers of ROOT data files from servers at Argonne, Caltech, CERN and possibly UCLA and Rio de Janeiro. The client application is ROOT with Clarens modifications. The ROOT data files are simulated JETMET data from the CMS Experiment at CERN.
The following diagram shows the international network setup currently envisaged:
Striped GridFTP Serve at Argonne
The Argonne server is a load balancing GridFTP
proxy server. It allows a system to provide a single point of contact that
harnesses the network resources of any number of GridFTP servers. The
architecture consists of a single frontend that listens on a TCP port, and
manages client connections. As clients make requests for file transfers the
frontend selects the appropriate backend server for the job, based on resource
load and availability, and forwards the request to the selected server. All
though the backend servers may exist anywhere that the frontend server can make
a TCP connection to, it is most reasonable to have all nodes in the system on a
local area network.
Tier2 Center at Caltech
The Caltech Tier2 server is built from a variety of 2U and 1U rack-mounted slave nodes based on PIII and Athlon processors, connected over a private LAN via Gbit uplinks to two servers with substantial (over 4 TBytes) disk arrays, which are in turn connected via Gbit fibre links to CalREN2 and from there to Abilene. More details on the Caltech Tier2 are available.
Networks and Coordination