|
WIRCam Hardware and Software Changes History
|
Table of contents:
The WIRCam instrument has evolved since its official first light using 2
engineering-grade chips on March 27th 2005. First science operation with 4
science-grade chips was in November 2005. The most important hardware changes
occured before the September 2006 observing run (before semester 06B started)
when the gain of the instrument was increased and before the July 2007
observing when cables were changed to eliminate negative crosstalk.
The following table lists all hardware or software changes on the instrument or
the telescope that may have affected the science images. We will slowly fill in
the blanks of what happened before June 2007.
WIRCam Hardware and Software Changes
Date of change | System | What changed? |
September 2017 | Filter Wheel Failure | On Sunday,
03 September 2017, the filter wheel stopped working. It was possible to get
the filter to J by increasing the motor torque limits, so observing proceeded
for the night. The next day, the system was investigated and it was found
that the lower wheel was moving roughly by hand indicating that the problem
was internal. The instrument was removed from the telescope and allowed to
warm up the next day; 5 September 2017.
After fully warmin the instrument, it was disassembled and it was found that
the sapphire ball bearings in the lower filter wheel were all pretty much destroyed. It is likely that one ball failed and the rest were damaged during
troubleshooting attempts. The race was also damaged by the failure. The inner
bearings (stainless steel) were unaffected. In addition, the broken sapphire
was completely confined to the lower wheel.
Everything was thoroughly cleaned and the lower, outer race and sapphire
bearings were replaced. The upper sapphire bearings were replaced proactively.
The lower, Teflon ball cages were not replaced as there were not sufficient
spares available. They were thoroughly cleaned and put back into service.
Initial tests of the lower wheel were that it did not move smoothly.
Measurements of the races showed that the gap on the inner circle was smaller
than for the upper (original) races. This meant that the lower wheel was
riding on the Teflon ball cages and not on the bearings. The cages were
machined down from 1.8 mm thick to ~1.4 mm thick.
These changes worked. The system was reassembled on 12-13 September and
tested warm on 15 September. The system was tested cold on 19 September.
|
September 2011 | Chip #60 bias offset |
Beginning with the WIRCam camera run 11Bw02 (Sept 6, 2011), Chip #60 (top, left
chip in array) is showing a low level (< 1% in ADU counts) step offset running
along the y-direction (dec axis). The location of the
step changes from one image to the next, and the ADU value is marginally
correlated with the exposure time; the offset is seen in all filters, so the
origin is likely related to the bias voltage.
Various tests were carried out to trace the origin of this offset but the exact
cause is still unclear at the time of writing (Feb 2012). The offset level is
low and the effect on photometry in stacked images is well below the percent
level. In addition, a correction using the median value effectively removes the
offset in processed images. Given the low level of the offset, the effective
correction method and the marginal effect of the photometry, the engineers have
decided to monitor the chip's performance and not to attempt any correction at
present.
All science data observed from 11Bw02 onward are corrected for this offset
before being distributed to the PIs.
|
September 6, 2011 | Swapped Master/Slave Detector Cables |
On the first night of the WIRCam camera run 11Bw02 (Sept 6, 2011), it was
noted that the Master and Slave cables had been swapped by mistake during the
detector work carried out in August. This lead to significant noise features
seen especially as horizontal bands in Chip #54 (bottom, left). The cables were
reconnected properly the next morning and data taken on the night of 7 Sept no
longer showed the banding.
Twilight flats are not available for the science images taken during the
night of Sept 6/7. However, the full set of domeflats in all filters, taken at
the time of the instrument exchange, are available. Therefore all the science
data taken that night have been processed with the processed master domeflats
only. On inspection, the processed images show only low level residual banding;
the PIs of the affected programs are being informed of this at the time of data
distribution.
|
August 2011 | Hardware and Controller |
To make the system ITAR compliant, the DSP code was burned onto the EEPROM
inside the SDSU controller. Booting of the code is now done from these
PROMs. The master and slave code were accidentally reversed in this process
but this was corrected on 1 September 2011. All spare cards were tested at
the time of the change to EEPROM booting, so all cards in the SDSU were
accessed.
|
July 2009 | Software and Headers |
The Staring Mode was implemented. New header keywords such as DEFOCUS to
indicate the defocus amount and WCSTARE to indicate if the Staring Mode is in
operation or not, WCSTAREX,Y,E to indicate the target pixel position and
extension number.
| June 4 2008 | Software and Headers |
The four on-chip guiding windows that are saved used to have problems: many
frames were either skipped and lost or duplicated. Doug Teeple tracked the
problems and made sure that neither case occur anymore. A new header keyword
has been created to monitor if frames are skipped: WCSKIP. Please note however
that guide cubes are generated to files, and no program writing to the
filesystem can "absolutely" guarantee that slowdowns might not occur and cause
skipped frames.
| June 3 2008 | Software |
A new command, gset, has been added to enable deliberately choosing specific
guide stars so that guide cubes can be used for science projects. Thanks to Tom
V.
| Mid March to April 7 2008 | Hardware |
The edge crosstalk has been removed in hardware and the noise level has been
slightly reduced. More very soon on this.
| February 29 2008 | Image Headers |
An error in the values of WCGDDEC1,2,3,4 was fixed. Previously, the DEC values
were divided by 15 before being converted to sexadecimal values.
| October 26 2007 | Image Headers |
The SLDATE?? keywords have been added. They represent the exact time when each
image slice was obtained, starting with 01 and up to 28. Per Doug Teeple's
email: "The DATE and HSTTIME keywords are now accurate to within a few
milliseconds of the start of exposure of the first corner of a cube. New
keywords SLDATExx (where xx is the slice number starting from 01) have been
added to identify the start of each slice exposure time to within a few
milliseconds. When the cube is built from the slices, the SLDATExx keywords
appear in the primary header and all extension headers consecutively after the
DATE keyword." They correspond to the time at the start of the reset.
| October 2 2007 | Image Headers |
Per Doug Teeple's email: "The DATE and HSTTIME headers now reflect the time at
the beginning of exposure (start of the reset) of the first slice (fixing a bug
where the time was the end of exposure)"
| July 3 2007 | Hardware |
One video board (the lower one y=1-512, on chip #54 - ext 3) has been swapped
back to the one that was before. Noise level was higher and it experienced
transient disfunctionning. This video board is special, it has a load resistor
of 15k instead of 10k for all others.
|
June 29 2007 | Hardware |
Cabling to the video board has been changed, i.e. the reference voltage bias
(VREF) is no longer shared with the VSOURCE bias on the first video board of
each array. This should totally remove the negative crosstalk that has plaggued
WIRCam since day 1. The DSP code was adjusted (voltage changes) to minimize
first horizontal line level offset of each amplifier. The system noise has been
reduced by connecting all video boards ground wires. This should improve the
noise level by a couple of ADUs. Five video boards have been replaced or
swapped in the process.
|
June 19 2007 | Hardware | Video
board #3 of extension 4 (third from the bottom - pixels y=1025-1536) was
swapped for one having a load resistor of 10k-ohm like all other boards instead of
previous 5k-ohm. It was shown that this board suffered from more positive
crosstalk than all other boards. That should therefore reduce the crosstalk on
that board. Also, many loose capacitors were resoldered on all boards but this
should have no impact on images if not for lower noise levels. All the video
boards were removed and reinserted in the process as well as one timing board.
|
June 6 2007 | Image headers | The
default World Coordinate System (WCS) solution has been changed and now uses
the values determined by Terapix. The 6 keywords to have changed are: CRPIX1,
CRPIX2, CD1_1, CD1_2, CD2_1, CD2_2 for all 4 extensions. |
Big gap to be filled here | | |
January 19 2007 | DSP/QSO | All
science exposures in the Low OH1/2 filters are now taken by default with
Fowler=4. This is to obtain low readout noise for long exposures. But even
short exposures are with fowler=4 for simplicity. |
Big gap to be filled here | | |
July 14 2006 | DSP code/Detcom | Marc
writes that minor DSP code changes have been made to support the new version of
dtecom and the new Lotus PCI driver. DSP version 2.3. Detcom now introduces
bumping to position the guide stars in the center of the field within range of
the ISU travel. |
June 28 2006 | DSP code | Marc
made a last minute change to "provide a patch to deal with a recurrent,
intermittent problem with the guide window not working properly on only one
chip after power-on." |
May 3 2006 | DSP code | Marc
sends a long email to wircam@cfht about a new DSP code. Results are 1adu lower
readout noise )by slowing down the pixel rate), shorter minimum etime of 2.25se
rather than 2.84s. He mentions a vertical band at the position of the guide
window on extension #4 that is introduced in his code that was not there in the
April run. He discusses a problem on booting the arrays with a 1 pixel shift
that is not reproducible. This affects the positionning of the guide frame
(star is then off center by one pixel) but does not affect the science image.
|
March 30 2006 | DSP code | Marc
Baril sends an email to wircam@cfht stating that he finally has a code he likes
using the enhanced mode. He describes how the high noise in some amplifiers is
coming from digital signal of the other amps being sent out while reading the
affected amps. This is between two observing runs. "Currently, we are reading
the array twice on every read and discrading the first read to ensure
absolutely pristine images, the minimum exposure time is therefore 2.8s. The
exposure time requested is the exposure time obtained within 1ms." |
March 15 2006 | DSP code | A
"fast readout code" for which the first set of darks was taken March 21 and
shows that the background gradient in extension 4 is now gone (as seen from
images with the guider turned on). |
January 18 2006 | DSP code | Marc
writes about the exposure time not being calculated properly. The requested and
actual exposure times seem to have been different up until now. Fortunately,
both times are stored in the headers so we know what the actual etime was.
|
December 14 2005 | DSP code | The IMULT
flash is noticed on the guide frames. Incsequential for science images except
in the quality of guiding perhaps. |
December 12 2005 | DSP code | Marc
mentions synchronization problems. He found a bug in his DSP code and fixed it.
It would have impacted long etimes. The loss of sync amounts to less than
100usec per exposure. |
December 9 2005 | DSP code | Marc
mentions that he and Jeff have found and fixed numerous bugs in the DSP clocks.
The guide window should be a lo cleaner. He mentions problems with the guider
the previous night where there were occasional flashes in the guide frames.
|
December 6 2005 | DSP code | Marc
runs a new DSP code to fix a problem with the guider not being readable during
science readout. Now it is readable. This introduces issues with exposure times
not being exaclty what we request. |
November 28 2005 | DSP code
| Marc mentions that the two controllers may not have been synchronized
in the past which means that exposure times on the two sets of detecttors may
have been slightly different (msec, not sec). |
| | |
| | |
| | |
| | |
| | |
Cooling Cycles and Filter Changes
Date | Action | Comment |
February 24, 2025 | Cool Down | Cool down after cold head swap. |
February 18, 2025 | Warm Up | Warm up to replace cold head which had lost significant cooling power. |
November 4, 2024 | Cool Down | Cool down after UPS work. |
October 10, 2024 | Warm Up | Preventative warming during building UPS work in case of power issues. |
August 16, 2023 | Cool Down | Filter KCont put in place of CH4On, Cold head swapped out preventatively. |
August 1, 2023 | Warm Up | Filter exchange for semester 2023B requested. |
April 12, 2023 | Cool Down | Cooled after pumping warm for a week. |
March 29, 2023 | Warm Up | Vacuum was soft after an exchange. Warm and pump. |
June 23, 2022 | Cool Down | Cooled after emergency shutdown. Filter LowOH1 put in place of W, too. |
May 20, 2022 | Warm Up | Warm up to swap W filter for LowOH1 and to remove WIRCam from cage for emergency shutdown. |
January 25, 2022 | Cool Down | Cooldown after swapping BrGamma in for the CO filter. |
January 13, 2022 | Warm Up | Warm up to swap filters. |
October 26, 2021 | Cool Down | Cool down after swapping out cold head with spare. |
October 7, 2021 | Warm Up | Warm up to swap cold head. Cold head had loss of cooling power. |
July 1, 2021 | Cool Down | Cool down after filter change from Kcont to CO |
June 2, 2021 | Warm Up | Warm up to change filters |
January 8, 2021 | Cool Down | Cool down after shutdown |
October 8, 2020 | Warm Up | Warm up in preparation for shutdown |
September 22, 2020 | Cool Down | Cool down after filter change from CO to Kcont; shutdown delayed. |
August 25, 2020 | Warm Up | Warm up in preparation for PM mirror coating shutdown. |
August 13, 2020 | Cool Down | Cool down after Covid-19 warming. |
April 7, 2020 | Warm Up | Warm up to save power during Covid-19 induced shutdown of many operations. |
August 12, 2019 | Cool Down | Cool down after access to summit restored. |
July 31, 2019 | Warm Up | Warm up as precaution since summit inaccessible due to protests. |
April 17, 2019 | Cool Down | Cool down after leak check and gauge replacement. |
March 29, 2019 | Warm Up | Warm up to leak check (seemed like a small leak was possible) and to replace vacuum gauge 1. Further troubleshooting after vacuum gauge was replaced was needed since both were still not working properly. |
December 17, 2018 | Cool Down | Cool down after replacing the cold head. Pumped for ~1 week with heat on the getter. |
November 29, 2018 | Warm Up | Warm up to replace cold head. Preventative maintenance. |
September 19, 2017 | Cool Down | Cool down after repair of filter wheel. Cryovessel only pumped for about 2 days after being open for 1 week. |
September 5, 2017 | Warm Up | Warm up to repair the filter wheel. Instrument completely disassembled. |
August 16, 2017 | Cool Down | Cool down after shutdown. Cryovessel never open. |
July 14, 2017 | Warm Up | Warm up and remove from cage in preparation for shutdown. Not opened and pumping in the compressor room. |
March 7, 2017 | Cool Down | Cool down, but dewar never fully warmed. Dewar never opened, only pumped while partially warming. |
March 2, 2017 | Warm Up (partial) | Partial warmup to around 208K in response to a prolonged power outage and loss of monitoring associated with the outage. |
April 19, 2016 | Cool Down | Cool down after less than 24 hours of pumping in order to get on the telescope. |
April 15, 2016 | Warm Up | Emergency warm up to fix a small vacuum leak that prevented the cryostat from staying at a reasonable pressure. Found the leak to be in the emergency blow-off valve. The valve was rebuilt and leak checks showed no leaks. The cryo-cooler was also losing power (though still functional), so it was also replaced since it was due to be replaced in August 2016 anyway. |
August 10, 2015 | Cool Down | Replaced one vacuum gauge and replaced the narrow CO and CH4Off filters with the new CO filter and the new W filter. Checked and regreased accessible o-rings. Found one o-ring badly degraded. |
July 8, 2015 | Warm Up | Replace vacuum gauges and filter change. |
April 24, 2015 | Cool Down | Replaced both vacuum gauges, put more thermal grease on the cryo-cooler, and replaced the Kcont and LowOH2 filters with the CH4On and new CO filter that is too narrow (the proper filter is still being fabricated, but is very late). |
April 9, 2015 | Warm Up | Replace vacuum gauges and filter change. |
September 2, 2014 | Cool Down | Cool down after replacing cryo-cooler and fixing leak in slave feedthrough. |
August 4, 2014 | Warm Up | Replace cryo-cooler (preventative), leak check. |
January 23, 2014 | Cool Down | Cool down after filter change. >1 week pumping. |
December 27, 2013 | Warm Up | CH4Off in for BrG and pumping. |
January 28, 2013 | Cool Down | Cool down after power restored. Short pumping time. |
January 20, 2013 | Warm Up | Warm up due to loss of building power. |
September 28, 2012 | Cool Down | Filter LowOH2 in place of LowOH1 and cryo-cooler replaced. |
September 24, 2012 | Partial Cool Down | Replaced cryo-cooler. Tested system to make sure contaminated lines not the issue. Detectors never below 273 K. |
September 23, 2012 | Partial Cool Down | Cryo-cooler failed after ~30 minutes. Detectors never below 273 K. |
September 7, 2012 | Warm Up | In preparation for filter change. |
February 11, 2012 | Cool Down | Filter LowOH1 is in, LowOH2 is removed. |
January 30, 2012 | Warm Up | In preparation for filter change. |
January 12, 2012 | Cool Down | Cooled after connector repaired. |
January 5, 2012 | Warm Up | A precaution in case the hermetic connector for the Master SDSU was breached during repair attempt. |
August 31, 2011 | Cool Down | Filter LowOH2 is in, LowOH1 is removed. |
August 16, 2011 | Warm Up | In preparation for filter change, and for alignment after shutdown. |
April 6, 2011 | Cool Down | Filter LowOH1 is back in. CH4On is removed. |
March 24, 2011 | Warm Up | In preparation for filter change. |
September 9 2010 | Cool Down | Vacuum good again. Cryo-cooler replaced. |
September 1 2010 | Warm Up | Cryo-cooler head contamination/failure. |
December 13 2009 | Cool Down | Vaccum good again. |
November 17 2009 | Warm Up | To improve the vaccum by pumping. |
December 28 2008 | Cool Down | Filter Kcont is back
in. CH4Off is removed. |
November 17 2008 | Warm Up | In prevision of filter
change as well as need to open the cage for secondary alignment after
aluminizing. |
August 23 2007 | Cool Down | After filters exchange
and observatory shutdown. |
August 15 2007 | Filter Change | CH4On and CH4Off in,
LowOH1 and Kcont out. |
August 10 2007 | Warm Up | In prevision of filter change
and observatory shutdown for aluminizing the mirrors. |
January 17 2007 | Cool Down | |
January 9-10 2007 | Warm Up | To remove filters CH4On
and CH4Off and put filters Kcont and Y in. |
October 19 2006 | Cool Down | |
October 15 2006 | Warm Up | 6.7 earthquake plus power
outage. |
October 10 2006 | Cool Down | |
October 4 2006 | Warm Up | Cryo-cooler head
contamination. |
September 25 2006 | Cool Down | |
September 19 2006 | Filter Change | Y filter removed,
BrG filter installed. |
September 15 2006 | Warm Up | Vaccum less than perfect
and coming filter change. |
April 5 2006 | Cool Down | |
April 3 2006 | Warm Up | Summit power outage. |
June 17 2005 | Cool Down | For 1st engineering run
with 4 science chips which occured June 26th. |
Early June 2005 | Warm Up | To install 4 science-grade
detectors. |
May 13 2005 | Cool Down | For 3rd engineering run with
E-chips. |
Early May 2005 | Warm Up | To put in new Lyot mask to
align with spider arms. |
April 16 2005 | Semi Warm Up | Fault somewhere - reason
unknown. Warmed up to 130K then was pumped and cooled back down. |
April 8-10 2005 | Cool Down | For 2nd engineering run
with E-chips. |
April 1-5 2005 | Warm Up | To fix problem with a Hall
sensor inside the dewar. |
March 18 2005 | Cool Down | For first engineering run
with 2 E-chips. |
Telescope Related Changes
Date | Action | Comment |
October 1 2007 | Mirror C02 Cleaning | Don't expect ZP
change. |
August 20-24 2007 | Mirror Aluminizing | Expect a zero
point change. Megacam got increases of 0.20 in u, 0.12 in g, 0.05 in r and 0.10
in z. |
December 7 2006 | Primary Mirror Wash | No noticeable
zero point change on WIRCam data. |
December 15 2005 | Primary Mirror Wash | No obvious
zero point jump in i,z Megacam data. |
|
|