Dome shutter failure recovery progress
This page provides updates on the progress of the recovery efforts following
a serious dome shutter failure which happened in the early morning of Saturday
April 14, 2012.
Wednesday 20 June 2012
Clear skies! The Observatory was released for normal operations by the Director
of Engineering today at 14:10. After 67 days of difficult work and heroic
efforts, the teams reported that the Shutter is now in an operational state.
Observing will start tonight, if weather cooperates. A very big thank you
to every member of the staff who was involved in this enormous and successful
Tuesday 19 June 2012
Work is ongoing on the final adjustments of the shutter motors. The plan is to
begin operations on Wednesday night, if all goes well.
Monday 18 June 2012
High winds at the summit prevented any work on the shutter. Activities should
Sunday 17 June 20123
The crew worked all day yesterday to put us back on the sky and was able to
perform a first complete shutter mouvement since the failure. Unfortunately,
some of the components had tangancial forces applied to them so some adjustment
is needed. Observing is dealyed to at least Tuesday June 19 2012.
Friday 15 June 2012
The shutter crew is working hard on getting us ready to observe Saturday night.
The plan is to make sure that no tangancial forces are applied on the gearbox
Tuesday 12 June 2012
Refurbishment of #8 hub continues today. On #8 hub, 5 of 6
pins needed attention. Looking at the currents from last week, #7 seems to have
failed 2 times, it had a 1.5 second delay on opening.
Friday 8 June 2012
The daycrew got the new motor on #5 but the one taken off
was a different one from the other units. Tangancial loads maybe the cause
for the bending shafts. The teams are looking into this.
Wednesday June 6 2012
Motor #5 was removed and will be disasembled tomorrow. as plan is being prepared
to test the shutter motions.
Tuesday 29 May 2012
The shutter was opened for the first time today since the failure. The crew
did several tests and learned a lot about the behavior of the system. In
Waimea, the controls team went over the proposed changes to the PLC after
the tests done Thursday last week.
Wednesday 23 May 2012
Work is ongoing on a list of items to be completed to attempt opening
the shutter next week.
Monday 21 May 2012
Controls group investigating operating and did the shutter hit
the hard stops. There is no evidence that the shutter has hit the
hard stops since OAP has been used for operation. Drive motor unit #1 is in,
now bolted and 3 phase power was hooked up. It is ready to be tested/adjusted.
Thursday 17 May 2012
The daycrew has the rebuild unit #1 located up on the shutter and secured
in place. The inspection of the rack tooth profiles continues.
Tuesday 15 may 2012
Work continues on the repairs of the dome shutter. The control group now
understands in detail all unexplained shutter actions during initial
investigations. Also, forensics analysis from a metallurgical lab on
the sheared output shaft concludes that the cause is a severe torsional
overload. On the mechanical side, detailed inspection of the shutter
rack has started. The rack will be checked for damage as well as each
teeth profile using rack tooth profile gauges.
Friday 11 May 2012
The control group has a now a good understanding of the events that occured
on the day of the failure, thanks to the test controller they assembled. The
controller also allowed them to understand
some shutter related events that occured before the failure. The mechanical team
is also making progres. The rebuild of unit #1 is now complete with the successful
installation of the pinion gear of the shaft. The summit crew inspected and put
back unit #2 and #6. The plan for next week is to start a close inspection of
the shutter rack. Work will resume on Monday.
Wednesday 09 May 2012
Great progress was made today by the two tiger teams. The controls team finished
building their test unit. They started performing tests and already explained events
that occured on the day of the failure that were, up to now, difficult to explain.
They will continue testing for a couple of days. On the mechanical side, the
summit crew has unts #6 and #2 off the shutter and have put
motor #5 back in. In Waimea, some adjustements were needed to fit motor #1 to
the new gearbox assembly. The work continues on the examination of the pinion
rollers and of the shutter rack.
Tuesday 08 May 2012
Work has resumed on Monday and is progressing. One of the new gearboxes was
successfully mounted on the #1 drive frame. More work is needed to complete
the assembly and to have a usable gearbox. Work has started on the examination
of the pinion rollers using dye tests.
The control group moving forward with the test controller. They are also
completing the video investigation of the shutter rack.
Friday 04 May 2012
The inspection of motors #2, 5 and 6 at the summit has started and is advancing well.
Non-destructive tests on pinion hubs 1, 3 and 4 are in the works. We will look
into dye penetrant and ultrasonic tests. The hope is that these will give
insight into the causes of the failures. The control team has started building
the test controllor for the shutter motor. Also implementations were made in
the OAP software to increase the amount of information captured for future dome
Work will resume on Monday
Wednesday 02 May 2012
The work continues for both the mechanical and the control teams. It was found
that motor #1 pinion gear was misaligned with the gearbox frame. The gearbox
was transferred to a mechanical shop in Hilo for diagnosis. We are also
awaiting diagnosis on the broken shafts. The plan to move forward and the schedule
are still in the works.
Monday 30 April 2012
The #1 Motor and gear assembly are now down in Waimea. The teams spend the day planning the
upcoming work and assessing our knowledge of the problem. Work is underway to
assess the state of the gear assembly. In the meantime, the third gear reducer
is now on it's way to Waimea.
Saturday 28 April 2012
Some members of the control group went to the summit to take video of both sides
of the dome shutter rack. This will help assess the state of the racks and
evaluate if they contributed to the failures. The videos will be analyzed
Friday 27 April 2012
The daycrew was able to finish the removal of motor #1. They worked hard until
late in the afternoon to get this done. The motor was brought back down
to Waimea for inspection. Many thanks to everyone for their extra efforts on
this Friday night. The control group has continued their work on explaining the
odd shutter behavior seen before and during the failure.
Thursday 26 April 2012
The control team continues to find important clues regarding the events of
April 14. On the mechanical side, work continues with the removal of
Wednesday 25 April 2012
The control team has found important clues to explain some of the weird behavoir
of the shutter that was witnessed at the time of the failure. They continue
their work toward gaining a better understanding of the shutter controls. The
mechanical team is ready to draw motor #1 out. This will happen tomorrow. The
plan is to reassess the strategy after inpection of that shutter motor.
Tuesday 24 April 2012
The mechanical team has started the removal of Motor #1 for inspection while
the control team is in the process of updating the documentation of the shutter
control system wiring. At this point, there does not appear to be any anomalous
control system function that could explain or are likely implicated in the
failures of the drive system gear reducer / pinion mechanics.
The Dome DIMM was removed from it's current position at the bottom of the shutter
and relocated away from the work area.
Also, a camera was installed at the top of the cute to record the repair activities.
Monday 23 April 2012
The control group made good progress in understanding the shutter controls.
They developed a plan to build a standalone test unit with the overload breaker,
fuses and drive unit to help understand the operation of the motor
controller. On the mechanical side, the plan is now to dismantle and
inspect the four lowest moters on the shutter. Work is underway to do this
Saturday 21 April 2012
The teams are taking a well deserved break. By the end of the week, the
mechanical team was making progress on the plan to acccess the lowest 4 drives
so that they can either be serviced or replaced. An inspection of the motors
revealed that motor #1 pinion gear may be out of alignment. They are also
looking into consulting with other companies. In the mean time, the group is
also looking into the availability of gearboxes. For the moment, there are no
plans to order more gearboxes. The Control group has made progress into
understanding the shutter control systems. Work will resume on Monday.
Thursday 19 April 2012
The Tiger teams are working hard. The Control team has made progress in trying
to understanding the shutter system and how it is controlled and what its limits
are. The Mechanical team is moving forward with the planning to access the
shutter motors. The new gear reducers from Suitomo have arrived. They will
need to be mechanically modified before being suitable for installation on the
shutter. Shutdown is still thought to last on the order of a few weeks.
Monday 16 April 2012
Planning of the repairs were completed and Tiger teams were formed to study the
access to the drive assemblies for replacement and repairs of the drive and
gearbox. The teams will also investigate the possible causes for the failure.
At this moment, work is expected to last for at least a few weeks. Gearbox
replacements were identified and ordered from Suitomo. They should arrive at
CFHT within a few days.
Sunday 15 April 2012
It was determined that the failure was a rupture of the pinion drive shaft of
motor #4. Work on the plans to repair the shutter and understand the causes of
the failure have begun. A first meeting was held at headquarters to layout
plans for the upcoming work.