\documentstyle[12pt]{article}

\topmargin=-15mm
\textheight=233mm
\textwidth=155mm
\hoffset=-13mm

\begin{document}

\title{CCD School Notes: Near-IR Observing}

\author{Paul Francis, University of Melbourne}

\date{\today}

\maketitle

\section{Background}

The near-IR, for the purposes of these notes, starts at wavelengths of
around 1$\mu$m, and goes out as far as about 5$\mu$m in wavelength.
In older books, you will often here the wavelength range $0.75 \mu$m to
$1 \mu$m referred to as near-IR (that is why the $I$ band at $\sim 0.88 \mu$m
is called the $I$ band). However, normal silicon CCD detectors work
fine at these wavelengths, so the observational techniques are the same
as those in the optical. Beyond about $5 \mu$m, IR observations have
to be made from space, or at least high flying aircraft; this defines
the mid-IR. All these definitions are notoriously flexible.

The near-IR has many scientific advantages over the optical; in particular
its ability to penetrate dust. Until recently, these were offset by the
extreme difficulty of making sensitive near-IR observations. However,
near-IR detector technology has made a series of breakthroughs in the
last ten years, and now is almost as good as optical detector technology.

\subsection{Why the Near-IR is Different} 

Near-IR observational techniques differ from optical techniques due to two 
atmospheric problems: absorption and sky emission.

The near-IR wavelength range has many molecular transitions. This is great
for probing astrophysics, but unfortunately the photons have to make it
through the atmosphere, which is also full of molecules. In particular,
water vapor absorbs strongly at many near-IR wavelengths.

What this means in practice is that at most near-IR wavelengths, the
atmosphere is effectively opaque. If, for example, you wished to observe
at $1.8 \mu$m; tough! The water vapor in the atmosphere completely absorbs
light at this wavelength. IR observations are thus restricted to a set
of wavelengths that are free of strong molecular absorption. The major
bands are listed in Table~1. Note that these numbers are approximations;
you should look up exact filter curves in the relevant instrument manuals
if precise wavelength ranges are important. Most near-IR observations are 
made through one of these atmospheric windows.

\begin{table}

\begin{small}

\begin{tabular}{ccccc} \hline \hline & \\
Band & Central  & Width   & Flux of Zero & Sky Brightness \\ 
  & Wavelength ($\mu$m) & ($\mu$m) & Mag Star (${\rm W\ m}^{-2}{\rm nm}^{-1}$) 
& (Mag per square arcsec) \\ \\ \hline & \\
J & 1.3 & 0.24 & $2.8 \times 10^{-12}$ & 15.0 \\
H & 1.6 & 0.24 & $1.3 \times 10^{-12}$ & 13.7 \\
K & 2.1 & 0.35 & $4.6 \times 10^{-13}$ & 12.5 \\
Kn & 2.15 & 0.25 & $4.6 \times 10^{-13}$ & 13.7 \\
 \\ \hline \hline
\end{tabular}

\caption{Table 1: Major Near-IR Wavebands. Kn (narrow K) is a variant
of the K band designed to minimise night sky emission. $K^{\prime}$ is
similar. Use of Kn and/or $K^{\prime}$ is recommended at Siding Spring, and
at other low altitude sites.}

\end{small}

\end{table}



The second major problem with near-IR emission is skyglow. Throughout
the near-IR, the upper atmosphere emits radiation due to a host of
molecular transitions. At wavelengths of $\sim 2 \mu$m and longer, thermal
black-body radiation from the sky and the telescope itself are also
important. As Table~1 shows, this sky glow is quite intense.

The brightness of the near-IR sky has a whole host of consequences.
Firstly, the sky glows so brightly that having the moon up makes
hardly any difference. Indeed, at longer wavelengths, having the sun up
makes very little difference. Thus IR telescopes work just as well
during full moon, and even work respectably during the day (though
in practice sunlight on the telescope causes enormous thermal problems,
and most observatories do not allow daylight operation, except in very
special circumstances). This is why IR astronomy traditionally takes
place during bright time.

Secondly, all exposures with near-IR cameras have to be short ones, to
avoid saturating the detector. You won't get away with the hour-long
exposures possible in the optical; many IR detectors will saturate in
10--20 seconds, or even less at wavelengths longer than $2 \mu$m. After
a typical night of IR observing, you may go home with over 1000 data
frames, each only a few seconds long.

Thirdly, nearly all observations will be of objects much fainter than
the sky. This means that accurate flat fielding and sky subtraction
are quite vital. Indeed, unlike in the optical, you typically can't
see a thing in raw near-IR observations.

\subsection{Near-IR detectors}

CCDs do not work at wavelengths longer than about $1 \mu$m, as the
photon energies are smaller than the band-gap of silicon. IR detectors
therefore use different and exotic materials, typically either
HgCdTe (Mercury-Cadmium-Telluride) or InSb (Indium Antimonide). These
materials were mainly developed by US defense researchers for use in
missiles, and nearly all the world's IR detector chips are
purchased from an handful of US companies, for very large prices ($\sim$
A\$ 100,000). The technology is evolving very rapidly, and detectors
even a couple of years old are often considered obsolete.

In near-IR detector chips, the incoming photons create a voltage
pattern in an array of pixels. Each pixel is bonded onto a circuit
network, which enables the control electronics to read the voltage
of any individual pixel at any time, without altering it. This is quite
different from a CCD chip, where measuring the charge pattern requires
reading out the whole chip. What this means in practice is that
IR arrays can be read out extremely quickly; typically in less than
a second.

As compared to CCDs, IR arrays have reasonable quantum efficiencies
(around 50\%), but high read-out noise and dark currents. However,
the latter two hardly matter for imaging applications, as the brightness of
the sky swamps all other sources of noise. Currently HgCdTe chips have
better quantum efficiency, but they only work out to wavelengths of
$\sim 2.3 \mu$m. InSb chips are a little less efficient, but work out to
$5 \mu$m.

There are currently two near-IR cameras in use in Australia:

\begin{itemize}

\item IRIS, on the AAT. This is a rather elderly HgCdTe array detector,
used both for imaging and low resolution spectroscopy. The detector is
128$\times$128 in size, with 60$\mu$m square pixels. The camera is
very flexible, with an amazing number of observing modes. It can be
operated as an imager with pixel scales of between 1.94 arcsec and
0.24 arcsec, and for spectroscopy with resolution 300 or 400. Being
HgCdTe, it only operates out to the K-band. Details are on-line at the
AAO web site, or contact Stuart Lumsden.

\item CASPIR, on the Siding Spring 2.3-m. This is a much more modern
InSb camera, used both for imaging and spectroscopy. It has a larger
detector (256$\times$256), which can be set to have imaging scales of
$0.5$ or $0.25$ arcsec per pixel. A variety of spectral modes are
being implemented, giving resolutions of 500 to 1100. Being more modern
than IRIS, its quantum efficiency is higher, and its sensitivity is
comparable to IRIS, despite the smaller aperture of the 2.3-m. Being
InSb, it works out to wavelengths of $5 \mu$m. Details (including
an excellent manual) are on-line at the Mt Stromlo web site, or contact
Peter McGregor.

\end{itemize}

Experience suggests that for imaging purposes, the superior quantum
efficiency of CASPIR cancels the mirror size advantage of IRIS and
both have comparable sensitivity. This means that CASPIR (with its
larger field of view) is to be preferred for most purposes. The AAO are
currently planning to build a much larger and more powerful IR camera,
with a 1k chip, in which case the balance of IR power will shift down the
mountain once more!

Overseas telescopes with modern HgCdTe detectors, on cold, dry high altitude
sites, can do substantially better.

\section{Observing}

\subsection{Applying for time}

As a rule of thumb, CASPIR and IRIS have can image to a magnitude limit
of $K \sim 20$ in an hour, or $K \sim 18.7$ in 20 minutes, given typical
Siding Spring seeing (for a point source). Spectroscopy requires that objects 
be brighter that $K \sim 15$. More detailed calculations can be made using
the night sky brightnesses in Table~1, and sensitivities listed in the
manuals. Note that nearly all near-IR observations will be sky limited.

\subsection{At the Telescope}

IR observations start a minute after sunset and end a minute before
sunrise (brutal in mid winter). It is worth obtaining bias and dark
frames. Opinions differ as to the utility of dome flats. They are generally
recommended with IRIS, but I find them to be little use with CASPIR. They
will be necessary for spectroscopy.

\subsubsection{Imaging}

Due to the extreme brightness of the near-IR sky, long exposures are
impossible, as they would saturate the chip. In the $Kn$ band, exposures of 
more than about 30 seconds are at risk
of saturating. You can get away with a bit longer in $J$ and $H$. I
typically take 10 second exposures to be on the safe side.

For standard stars, a few 10 second exposures are adequate. A list of
good IR standard stars can be found in the back of the CASPIR manual
(available on line). I generally choose standards with $K \sim 9$, and
take $0.5$ second exposures. Due to the short exposures, there is
seldom any need to autoguide the telescope.

For science exposures, many individual frames will normally have to be
taken, and added together during reduction, to get a deep enough image.
Both CASPIR and IRIS have an observing option to take a set of images
and average them before writing to disk. This cuts the disk-space and
exabyte requirements dramatically; I typically get the system to average
every six frames and write them to disk, so I only get one image a minute.

The use of large numbers of very short images has a useful side effect.
It is often possible to get good quality data through patchy cloud.
It is not atypical at Siding Spring to have patchy cirrus around. The
target will keep disappearing behind the clouds and the emerging for
a bit, before the next lump of cirrus drifts into the way. If you set 
a long series of short exposures of your target going, when reduction
time comes around it is possible to pick out those frames that were
taken while the target was exposed, and through away those taken
while it was hidden by the clouds.

For most targets, the sky brightness is far in excess of the target
brightness. This means that even tiny errors in sky subtraction can
completely mess up an image. The only way to get sufficient accuracy
is usually to take sky exposures with the same instrumental setup
and the same exposure time, and use these to flat field.

Unfortunately, the brightness of the IR sky glow changes rapidly during
the night. This is probably caused by changing temperatures and winds in
the upper atmosphere. This means that the sky frames have to be taken
close in time to the data frames.

Until recently, this was done by alternating exposures on target with
images of blank bits of sky nearby. This wastes half the observing time,
and finding blank bits of the sky can be hard. Fortunately, for most
projects there is a better way; shift and add imaging.

\subsubsection{Dithering}

Rather than taking all the images with the telescope pointing at the same
place, shift the telescope pointing between different sets of images. The
procedure is as follows:

\begin{enumerate}

\item Take a set of exposures at one position, and get the software
to add them up and write them to disk.

\item Shift the pointing of the telescope by a few arcseconds.

\item Take another set of exposures.

\item Shift the pointing again.

\item etc\ldots

\end{enumerate}

One might typically shift the pointing ten or twenty times during the
observation of a single target. The software it set up to let you
do this automatically, rastering the pointing along some grid pattern.
 This is called `dithering', or `shift and add', and is rapidly
gaining followers as an imaging technique in the optical as well as the 
near-IR.  

When you come to reduce the data, you first median all the images.
This will get rid of the objects in your field; because you moved the
telescope pointing, the objects appear in different places in each of
the individual images and hence disappear in the median. This medianed
frame is a perfect sky flat!

You flat-field the individual images using this medianed sky frame,
and then shift them until the stars line up. You can now sum them
to produce a nice finished image, very accurately sky flat fielded without
wasting telescope time.

This procedure has other advantages too. Bad pixels, fringing and
many other chip defects can be removed by this technique. The disadvantage
is an increased workload for the poor sod who has to reduce
the data, and a signal-to-noise ratio that varies from place to place in the
final image. Generally the advantages far outweigh the disadvantages.
 
You should choose a dither step than is larger than the images you wish to
study (typically a few times the seeing for point sources). This will ensure that they median out when you produce the flat field. I typically use
10 arcsec, and do a grid of 4 by 5 pointings.

Problems arise when you are studying a very crowded field, or looking 
at a large object such as a near-by galaxy. In this case, medianing may
not remove all objects from the field. In this case you will have to
take sky flats before or after your exposure, by finding a nice uncrowded 
part of the sky, and taking a set of dithered exposures there. With IRIS,
apparently, dome flats also work well.

One caveat; when the time comes to reduced dithered data, the images
are lined up by measuring the position of an object in each. This
means that you have to be able to see a particular object in each
individual image. This is not a problem if you have a bright point
source in your field, but if not, it is quite possible that you could
see nothing in any of the individual frames on disk. 

The solution to this problem is to take an adequate number of exposures
with the telescope pointing at each place before moving the pointing.
I typically expose for at least a minute (six summed exposures, each of 
ten seconds) before each move of the telescope, to make sure I see something
that I can use in the reduction for the shifting. In $K$ with CASPIR,
a one minute exposure virtually guarantees that you see some random star 
usable for aligning the images, wherever in the sky you are pointing.

\subsubsection{Spectroscopy}

Spectroscopy with an IR camera is less unusual, as the sky brightness
is dispersed. Exposures of several minutes are possible, depending on
the dispersion, and the observing procedure is very similar to that
for optical spectroscopy. Accurate sky subtraction is needed!

\section{Data Reduction}

Well, nobody ever said astrophysics was easy\ldots

In this section, I discuss the steps needed to reduce near-IR data.
I also list, in small print, the technical details of how I do the reduction
using IRAF. 

\begin{small}

The IRAF examples listed here use a very basic version of IRAF, and should
be workable almost anywhere. More modern versions of IRAF have some
powerful new tasks which make the job easier. Peter McGregor at Stromlo
has also written an add-on package to IRAF which makes the reduction of
CASPIR data easy, if you have it (it is described in the CASPIR manual
and on-line). MIDAS and FIGARO also have near-IR reduction algorithms.

\end{small}

\subsection{Bias and Dark Subtraction}

Bias and dark frames are prepared and used in exactly the same way as they are
in optical CCD imaging (ie. median all the individual bias and dark
frames, and subtract the resultant frames from each data frame). There is
no overscan on IR detectors. The removal of bad pixels is also standard.

\medskip

\begin{small}
In IRAF, use IMCOMBINE to produce your bias and dark frames, and CCDPROC
to subtract them from the data and correct for bad pixels.
\end{small}

\subsection{Linearity Correction}

At high charge levels, near-IR detectors goes non-linear. If you have high
count levels in your data, you should
correct for this before proceeding any further with the reduction.
For CASPIR, the non-linearity has been modelled, and can be corrected
at present by a polynomial fit as follows:

\[
{\rm Linear} = {\rm Raw} + 6.4 \times 10^{-6} ({\rm Raw})^2
\]

Check an up-to-date manual, as the recommended correction may change.

\medskip

\begin{small}
In IRAF, I do this as follows. I use the FILES command to create a list of
all IRAF images.
\begin{verbatim}
files *%.imh%% >allfiles
\end{verbatim}
I then square all the files, multiply by $6.4 \times 10^{-6}$, and add the
result to the original files to get the the linearly corrected images, as
follows (using IRAF list-handling procedures and adding sufficies to the
various intermediate stages in the processing).

\begin{verbatim}
imfunction @allfiles @allfiles//a square
imarith @allfiles//a * 6.4e-6 @allfiles//b
imarith @allfiles//b + @allfiles @allfiles//c
imdel @allfiles
imdel @allfiles//a
imdel @allfiles//b
imrename @allfiles//c @allfiles
\end{verbatim}

\end{small}

\subsection{Non-Dithered Data}

If you didn't dither your data (ie. if taking spectra), the data reduction
is identical to optical CCD reduction. Just be careful in your sky subtraction!

\subsection{Flat Fielding Dithered Data}

With CASPIR, I find that the best flat fields are those obtained by
medianing actual data frames, and this is the procedure I describe
below. This is also standard practice at a number of other observatories
around the world. For IRIS however, dome flats seem to do a better job,
which makes flatfielding straightforward.

If you have a set of ten or more data frames, all taken with the
telescope pointed somewhere different, medianing them should produce a
nice, star-free flat field. 
This can go wrong if there is an extended object in the field; perhaps
a large galaxy or the scattered light from a very bright star. Also, very 
crowded fields may give problems. In these cases, you 
construct a flat field frame either from observations of some less
troublesome region of the sky observed at roughly the same time during the
night. 

You should use the same procedure for flat fielding standard star frames;
use the medianed sky flat of the image taken before or after the standard.

Even if you had crowded fields and/or extended sources, and you forgot
to take suitable sky flats at the telescope, the wonderful power of the
median can save you; if you median enough even bad flat fields, they often
produce an acceptable one. If not, try surface fitting to remove large
scale gradients, or minmax rejection of stars.

If your data was taken in iffy weather (not unusual at Siding Spring), you
will have to make sure that only images taken through gaps in the clouds
are used to construct the flat field, otherwise you will start finding
funny patterns in your reduced data. The problem is working out which frames
were taken while a gap in the clouds lay over the target. Generally
when you are looking at clouds, the sky brightness increases, probably
due to reflected thermal radiation from the ground. If you measure
the sky brightness in all your images, you will find that it tends to
plateau at a low value when no clouds are in the way, rising in irregular
peaks when clouds pass over. If you take only the data frames with the 
sky value at the lower plateau, you should be OK. Before combining
the flat fielded data frames, however, you should double check the fluxes
of the stars within them to ensure that no significant cloud was there.
This may seem like a lot of work (and it is), but you'd be amazed at the
quality of data you can achieve in quite foul conditions!

\medskip

\begin{small}
In IRAF, medianing of data is done by setting the `combine' option in the
IMCOMBINE task to `median'. IMSTAT can be used to spot clouds in a run
of data frames. Once you have a flat field, use CCDPROC to correct the
data frames.
\end{small}

\subsection{Shifting and Adding}

So, you've finally produced bias subtracted, flat-fielded images. If
you were doing optical data reduction, you'd be finished by now.
Unfortunately, each image is only a one minute or so exposure; you may have
to add dozens of different frames up to produce a final, sensitive
image of your target. And you can't just add the frames together
as they are; you have to shift them so that all the stars line up.

Experience shows that, at least with Australian telescopes, the process
of working out by how much the images need to be shifted to line them up
needs to be partially manual. The telescopes do not offset precisely
enough to allow you to align the images automatically (though Bruce
Peterson is still trying\ldots). You will have to look at all the
individual images, and use a cursor to track the position of some star
in them all. You can then feed this into some software package which
will work out accurate offsets, shift the images, and combine the
shifted images. The details depend on the software package in use; I
describe the procedure I use in IRAF below.

This sounds (and is) very time consuming; you have to look at each individual
image and measure the position of a star within it. However, experience
suggests that this is very valuable for picking up stray problems; funny
bias patterns, satellite trails, reflected moonlight. If a few of your
frames are bad, just leave them out of the final sum.

One puzzle is what to do with the outer regions of your image. Depending on
the dithering pattern used, there will be a central region of your image
which was seen in each of the individual frames. As you move further out,
only some of the individual frames will have looked at a given bit of the
sky. This means the signal-to-noise ratio will decrease towards the edges of 
the image; something to be careful about when analysing your data later.

When you have your individual frames lined up, you have to average them.
Doing a straightforward average maximises the signal-to-noise ratio, but
is very sensitive to dud pixels etc. I usually median the frames instead.
This reduces the signal-to-noise ratio slightly, but it also removes all the
spurious features very nicely. You might want to try both and see which 
you prefer.

\begin{small}

The following IRAF technique is messy and cumbersome; IRAF wasn't really
designed with IR reduction in mind. Recent IRAF releases have much more
powerful ways of doing this. However, this method works on even very old
versions of IRAF, and it does an excellent job. With practice I can
reduce a 20-frame image in about 25 minutes.

You start with a whole series of processed images, dithered around on the
sky in some pattern. Make up an ASCII file listing all the image names
in a column, list.d. Bring up an image display (SAOIMAGE or ximtool), and
use the imexamine command to look at all the images in succession.

\begin{verbatim}
imexamine @list.d
\end{verbatim}

Choose one of the images as a reference image (usually the first). In 
imexamine, use the `a'
key to measure the centroid of all reasonably bright point sources in
the field, and enter their x and y coordinates (in pixels) into an ascii
file, one pair of x and y numbers per line. These stars will be used by
the alignment software to line up the images.

Now choose one star, preferably a nice bright isolated one, that can be
seen in all the individual frames. Measure its x and y coordinates in the
reference image, and then use the `n' key in imexamine to bring up the
next image from the list. Measure the position of the same star in the
next image, using the `a' key again. Subtract the new position from the 
reference star position to generate the offset, and enter this into
an ascii file for each frame (it will be `0. 0.' for the reference frame,
of course). Do this for all the frames that make up an individual
image.

You are now ready to use the imalign task to line up all the images.
Edit the parameters of this task (using epar) to tell it the file in which
you listed the offsets, and the star positions in the reference image.
This program then measures accurate offsets, using your values as a starting
point, and shifts the individual images until they are aligned. You then
finish the job with the imcombine command.

One complication; imalign trims off the edge of the images; the bit that
doesn't overlap with the reference image. This missing outer bit
generally has a low signal-to-noise ratio, as only a few individual
frames are contributing to it, but despite that it is often useful. Including
this extra bit takes a little more work.

Create a set of images twice the size of your individual data frames
using the magnify command. Set their pixels to a large negative number
everywhere. Now copy (using imcopy) each of your individual data frames
into the central region of one of these new larger images.  These new
larger images will now have your data in the middle of each, surrounded
by a margin willed with large negative pixel values.

You can now align and combine these, as described above. Set imcombine's
lower threshold to be above the large negative value, and when you do the
final combining all these margin regions will be ignored, leaving you with 
the full image, properly calculated. 

\end{small}

\end{document}