Marcio and I want to report on how the TeraShake
results were received at the SCEC annual meeting in Palm Springs.
First of all we should point out that the meeting
was attended by nearly 400 people, including practically all the
best earthquake seismologists in the country (33 states) and several
from abroad (9 countries). This is a VERY tough audience for this
kind of work.....They positively loved it! The team was formally
thanked by SCEC director Tom Jordan, who expressed the enthusiasm
of the entire SCEC community.
Yesterday there was a press conference attended
by all Palm Springs TV stations, and the majority of the LA TV stations,
a Mexican station, as well as quite a number of newspaper reporters.
We showed one of the movies that Amit had generated overnight, and
this was quite a sensation. A number of meeting participants who
had not attended the press conference told me that they had seen
the movie on the 5 pm news, sometimes more than once. (I did not
see it myself, being busy correcting an error made in building the
base map!). Evidently several papers carried the story. The Palm
Springs "Desert Sun" had a good story, that included accurate
quotes. I will get copies of these stories and some of the video
through SCEC when they have a minute to catch their breath!
Amit Chourasia ---who apparently does not need any
sleep--- spent an enormous amount of effort in fixing the movies
and generating high resolution movies. Marcio and I spent most of
the day downloading them, getting the last one only 20 minutes before
my afternoon presentation. That clinched it. Many people who had
been skeptical of such large scale simulations came to us using
words like "fantastic", "amazing", etc.....and
of course we spent 3 hours after dinner giving private showings
at the evening poster session.
This success is due mostly to the phenomenal team
work that has taken place over the past few months. Since these
activities are not always visible to outsiders, I thought I would
share the summary that Marcio and I put together. This is best done
through examples:
Kim made his AWM code freely available, and Steve
and Geoff helped with understanding its workings and how best to
validate the output.
The SDSC, scientific applications group led by Amit
Majumdar, with support from Giri, Yifeng, Leesa helped with porting
the code; parallelizing the calculation and the IO; and generalizing
the code for scaling up to a large run. Robert Harkness offered
invaluable insights regarding IO management.
Yifeng put in a huge amount of work --including
nights and weekends over the summer-- to implement various substantial
improvements to the code, that will benefit the SCEC community for
years to come.
Geoff and Boris provided the checkpoint/restart
capability, executed cross-validation runs using earlier versions,
and together with David, helped define the metadata. Yuanfang integrated
MD5 data fingerprinting and, together with Steve and Amit, the on-the-fly
visualization library in the TeraShake code.
The Scientific Applications Group mentioned above,
and the high-end systems group, with Patricia, Larry and Chris,
executed DataStar benchmarks to ascertain the best resource configuration
given the computing and IO characteristics of the desired run. Don
Frederick took care of reserving these resources, both for the run
itself and for the subsequent visualization.
Yifeng dimensioned the run; Larry mad sure that
enough cache space on GPFS would be available; Don took care of
the SAM-QFS cache; Tom provided support for the Sanergy client to
access the SAMFS from the DataStar; George benchmarked and designed
the archival process, ie the process of transferring data from GPFS
to the SAM-QFS archive and conducting on-the-fly validations.
Special mention should be made of Bryan's fix to
the dd client, that sped up the archival process enormously.
Nancy took on the task of identifying and allocating
resources, and keeping the momentum through periodic meetings and
telecons. ....and someone (Richard?) obviously did a great job of
coordinating the flurry of activity between the various groups.
The run itself prompts me to acknowledge explicitly
uncommon effort by several team members:
-- The run master has been Yifeng. He launched the
job on Tuesday September 14, at 18:00, and, except for one shift
covered by Giri, took care of the babysitting until Sunday, September
19, at 02:00.
-- George took care of draining the GPFS cache regularly,
moving 43 Tbytes of data safely to archive storage, and registering
the data with the SRB. That task was completed on Monday, September
20, a mere 36 hours after the end of the calculation. The support
of the SRB group was critical in this achievement.
-- Amit started visualizing the data almost immediately,
generating several intermediate animations that convinced us of
the correctness of the simulation. This culminated in a huge effort
after completion of the run, to produce a complete suite of animations
from scratch using a corrected basemap, in time for the press conference
and the SCEC session on ITR.
-- Yuanfang computed all the data products used
in the specialized graphics to assess seismic hazards.
In the end Yifeng and Amit are to be singled out
for extraordinary contribution to the ultimate success of this project.
I speak for the entire SCEC ITR collaboration in thanking them especially.
...and all through the process-- from March 4 to
September 22!-- ... Marcio and Phil have played a truly critical
role in prodding us to keep moving forward, insuring that the lines
of communication remained open, and that the right people got involved
across the ITR collaboration. Without their calm and patient guidance
through the process , we would not have reached our goal.
As those of you who remember me from the time I
served on the NPACI EC (Reagan is one!) I have desired to see a
large earthquake simulation for over a decade. This dream has been
accomplished. I have to convey to you the enormous pleasure I have
taken in participating in this effort, due to the obvious enthusiasm
for the project that was so obvious among the entire team.
Already, many people have asked me what we will
do next. It is very clear that the earthquake research community
is now absolutely "taken" with the capability that has
been demonstrated here. Many will want to participate. Everyone
wants the movies on the web, and many want to know how to get to
the archived output. Others want to team up on future simulations.
I suggest that we schedule an all-team post-mortem
meeting where we review the lessons learned; identify ways to do
it better, faster, and cheaper next time; identify follow-on work
that we can facilitate, such as data mining by the community; and
start planning the next few simulations, which are already being
called for by leaders in earthquake research.
We have a lot of work in front of us. I look forward
to it!
Warmest thanks and regards to all of you,
Bernard
Jean-Bernard Minster
Professor of Geophysics
WWW: http://igpp-sw.ucsd.edu/
