• Data analysis tasks are characterised by large CPU time per event
• Implies a small data flow rate (most of the time is spent calculating, rather than doing I/O)
• Thus we pre-stage and filter the required data first
  • The staging and filtering is a part of re-clustering the events either for individual user, or analysis group needs, and is done in a separate process so that:
    • free up any central tape or disk drive asap   
    • minimize contention with other users, associated with central access of data   
    • start moving "small" rather than"large" datasets around,  asap.
    • The data to be reclustered and moved can be determined from a single job for the user or analysis group
• When processing, we want the access to the data to belocal in time as well as space: This means reducing the probability of having to open a container and fill a cache, only to have the cache flushed before many objects in the container are de-referenced.
  • Thus, to reference raw data or lower level hits (perhaps due to an unusual condition or analysis requirement), then the analysis causes a flag (or store a tag) to be set, which defers the access to the required container until later (when sufficient other analysis tasks require access to that container).
  • We avoid handling the unusual condition in quasi-real time, during the fast analysis on smaller, or more easily reachable, container(s).
• We move the events into 300 MBytes of local memory on the processor on which they will be analysed
  • Moving 300 Mbytes to a local memory takes only 3000 seconds at 1 Mbps.
  • A "service" for 100 users, with some 100 Mbps should be sufficient.
  • Moving the data to the user in this way, takes advantage of many desktops for processing.
  • We can use similar considerations for data movement over a LAN, at say 10 Mbps/user (5 minutes to get 30k events). Need 1 Gbps to run such a service.
  • This avoids contention with other users for the same data
  • Users could get many "chunks"of data per day.
  • We can fit 30000 events of size 10 kBytes into the memory
• We then start the analysis task on the processor, assumed to be 2000 MIPS.
  • We assume the time to analyse the event is proportional to the time taken to reconstruct it: 20000 MIPS-seconds per raw 1 MByte event.
  • So we need 200 MIPS-seconds for our 10 kBytes events
  • The analysis runs at the rate of 10 events per second
  • The analysis is complete in 3000 seconds

.