CANICA data reduction with IRAF

Reduction of near infrared (NIR) data is laborious due to the amount of frames that are necessary to handle in order to produce an image of a single object. Typically around 10 frames would have to be handled in order to produce an image of net exposure time of 5 minutes in the K-band. The best way to handle these multiple exposure frames is through a pipeline. This manual describes such a pipeline, written for the reduction of data taken with CANICA, the CAnanea Near Infrared CAmera. The pipeline is a script based on various tasks under the IRAF environment.

What are the basic steps in NIR data Reduction?

Bias subtraction
Sky frame preparation and subtraction
Flat fielding
Image alignment and combining
Stellar photometry

Data Acquisition with CANICA and File name Nomenclature

CANICA acquires images under Linux environment. Images are acquired by sending an observing sequence containing RA-Dec coordinates and relative shifts through a Java script. Useful observational parameters, such as RA-Dec coordinates, filter, exposure time etc. of each image are written into image headers. The image names are formed from a root name by concatenating a three digit number, which is incremented sequentially until the sequence ends. Typical image names are of the type root_name001_I.fits, root_name002_I.fits, ...... root_name0nn_I.fits. Images taken using the Java sequence until now (June 2003) start with number 002 (001 is reserved for a "training" image). Bias subtraction is done during the acquisition. However for each image, a bias image is also stored. Bias images can be identified by their termination "_Z.fits".

Multiple exposure co-adds are not necessary with CANICA, given that the combination of parameters (image scale, Full well capacity and the sky background) allows single exposures of duration greater than 60 seconds in the K-band.

Basic function of the script

The most important step in the reduction of NIR images is the sky pattern subraction. sky in NIR images is a combination of the real sky background and a local background originated due to the emission of the warm optics surrounding the detector. The former is expected to be uniform, whereas the latter contribution can be non-uniform over the detector. Relative contribution of the two backgrounds depends on the wavelength. In addition, any vignetting gives a structure to the background. Hence it is necessary to subtract a sky frame from each object frame.

The pipeline starts by first identifying the sky frames in a given sequence. The RA-DEC coordinates are used to separate sky frames from object frames. Sky frames are then combined (imcombine combine=median) to prepare a star-free sky image.

The next step is to flat field the images using a previously prepared flat field frame. The thermal emission of the warm optics makes the task of getting a good flat field image somewhat difficult. Ideally flat field image should be prepared by subtracting a mean night sky image from a mean dawn/dusk image in the same filter, both exposed to same duration.

Centroid positions of a reference star are then obtained in each sky subtracted, flat-fielded image. These centroids are used to find shifts of each frame with respect to the reference frame. Images are aligned using the calculated shifts and then combined (imcombine combine=average, reject=minmax).

Photometry of the reference star (or any other object) is then carried out using the technique of aperture photometry. The combined image is then transformed into the normal astronomical coordinate convention (North to the top, east to the left).

CANICA script names and their usage

The script which does all the reduction is called canobj. For an error-free reduction canobj should be run twice. The first run is a trial run, basically to have a first-hand look at the images. In this run, an aproximate sky frame is prepared and subtracted from all the frames of the sequence. The resulting frames are displayed one by one, so as to enable a quick look at each frame. The display also helps to identify at least one star common to all the object frames. Before running canobj a second time, a script called canref is then run, which reads header parameters, and substitutes suitable values for the parameters of canobj. Now canobj can be re-run without the need to edit any of its parameters. Here is a summary of all the scripts:

canobj: the main reduction script for canica images.
canref: reads the object coordinates from image headers and estimates the position of the reference star in all the frames of a sequence. This script also sets all the parameters that are necessary for a successful run of canobj.
cansky: form a sky using the given frames.

Description of parameters of canobj

cl> epar canobj

                    Image Reduction and Analysis Facility
PACKAGE = user
   TASK = canobj
    
root    =                as23k  Root Image Name
imbeg   =                    1  First Image Number
imend   =                    6  Last Image Number
(sequenc=         as23k002.seq) Sequence file [file.seq/preview]
(skyfram=                stack) Sky frame [stack/name]
(flattyp=                 null) Flat-fielding strategy [skynew/skyold/flat/null]
(flatim =             ../flatk) Flat field frame Name
(refimag=          as23j002ref) Reference Image name
(iref   =                    2) ref star image number
(skyedit=                   no) Clean the sky using imedit?
(disp   =                  yes) Display before imcombine? 
(submed =                   no) Remove the horizontal band?
(immos  =                  yes) Mosaic the images?
(dophot =                  yes) Do photometry?
(xshift =                   2.) X-shift[=Xref-Xin] wrt refimag
(yshift =                 -15.) Y-shift[=Yref-Yin] wrt refimag
(skyout =                 600.) Minimum Distance of the off-field sky
(skyin  =                 150.) Minimum Distance of the on-field sky
(skysect=    [551:575,551:575]) Section for skyvalue estimation
(flatsec=    [551:575,551:575]) Section for flat normalization
(boxsize=                  15.) imalign boxsize
(bigbox =                  21.) imalign bigbox
(crval1 =            180.73875) RA of the header (in degrees)
(crval2 =      4.1288055555556) Dec of the header (in degrees)
(crpix1 =                512.5) X-Pixel corresponding to the RA-Dec
(crpix2 =                512.5) Y-Pixel corresponding to the RA-Dec
(cdelt1 =   8.8888888888889E-5) Pixel scale in RA (in degrees)
(cdelt2 =   8.8888888888889E-5) Pixel scale in Dec (in degrees)
(epoch  =                2000.) Epoch of RA-Dec
(dowcs  =                  yes) Transform the image to N_up E_left?
(xymag  =                   2.) XY reduction factor if dowcs=yes
(jhkzp  =    20.60 20.40 20.20) JHK Zeropoints
(mode   =                   ql)

Parameter	Default/acceptable values	Short description
root		root file name (e.g. as23k for files of the type as23k002_I.fits, as23k003_I.fits, as23k004_I.fits etc)
imbeg		the first number of the sequence (2 if the first image is as23k002_I.fits)
imend		the last number of the sequence (7 if the last image is as23k007_I.fits)
sequenc	preview/file.seq	A filename (e.g. file.seq) containing the positions of a reference star or preview. The default option preview allows a trial run for the sequence. When sequenc=preview, the only parameters that are used are the first three parameters (root, imbeg and imend). So cl> canobj sequenc=preview can be run for any sequence without any prior knowledge of the kind of object it contains. The second run of canobj should have sequenc=file.seq, in which case many parameters of canobj should have been to set to proper values. The complementary script canref helps in setting these parameters. Read the help on canref. The file containing the positions of the reference star is automatically prepared when running canref. The file is a simple text file containing xy pixel coordinates of the chosen common star. In case, telescope pointing is wrong, it may be required to measure the position of the star in each frame, and then correct the entries in the sequence file. Any frame with coordinates outside [1,1024] pixel range implies that the frame is a sky frame. The file should contain exactly the same number of lines as the number of frames in the sequence.
skyfram	stack/name	Algorithm for determining the sky frames. With the option stack, sky frames are guessed based on the information in the sequence file sequenc, and then combined (imcombine combine=median zero=median) to prepare a star-free sky frame. Optionally the frame can be edited interactively using imedit (if skyedit=yes). all, sky frame is prepared using all the frames (suitable when the object is stellar, and observations are done with dithering technique).> The option name allows the user to give the name of an already prepared sky frame.
flattyp	null / flat / skynew / skyold	Flat-fielding strategies. With the option null flat fielding is not done. With flat, flat fielding is done using the flat image contained in the parameter flatim. With skynew, the normalized sky frame is used as a flat field image. In this case, sky value (not frame) is subtracted. With skyold, the sky image contained in the parameter flatim is used as a flat field image. Suitable when a good flat field is obtained in the current observing run using night-sky observations. When flattyp=skyold, sky frame formed using the images in the current sequence is used only to estimate the sky value, based on the parameter biassec. Effectively, this option reduces to the optical type of data reduction (i.e. flat fielding first, and then subtract an average sky value). It is not necessary to pre-normalize the flat fields, as the program normalizes the flat image using the average in the parameter flatsec. Warning: flattyp=skynew and skyold are meant for test purposes. Hence values null or flat should be preferred for routine reductions.
flatim		Flat field frame Name (when flattyp=flat) or the name of the previously prepared sky image (when flattyp=skyold) obtained in the current observing run using night-sky observations. In the former case, flat field image should have been prepared by subtracting a mean night sky image from a mean dawn/dusk image in the same filter, both exposed to same duration.
refimag	as23j002ref	Root name for files that contain the coordinate information of a common star in the reference frame. e.g. for refimag=as23j002ref, the working directory should contain files as23j002ref.coo (xy coordinates of the reference star), as23j002ref.fits (image containing the reference star at the reference position) and as23j002ref.xy (xy coordinates of stars for aperture photometry). These three files are automatically created during the run of canref. Otherwise should be created by the user. It is not necessary that the image refimag .fits be part of the current sequence. For example, it could be an image of the same field, but in another filter. See help on xshift,yshift.
iref	2	Reference star image number. To which image number, the reference position corresponds (e.g. 2 if the reference coordinates are measured in the image as23k002)? Its value is setup automatically during the run of canref. Otherwise should be entered by the user. It is an error if iref > imend or iref < imbeg.
skyedit	yes/no	Clean the sky using imedit? If skyedit=no, script uses the prepared sky (without displaying it) and continues with the rest of canobj. If yes, combined sky frame is displayed with the IRAF command imedit. Enter "q" to quit imedit and continue with canobj. Sky editing is required in only exceptional cases and hence normally enter "q" when the cursor is in the image window. Useful imedit keystroke commands are: a Background replacement (rectangle) b Background replacement (aperture) d Constant value substitution (rectangle) e Constant value substitution (aperture) u Undo last change (see also 'i', 'j', and 'k') + Increase radius by one - Decrease radius by one Statistics q Quit and save changes Q Quit without saving changes Read help on imedit for further details.
disp	yes/no	Display before imcombine? With disp=yes, bad images can be interactively rejected from the list of images to be aligned and averaged (combined). With disp=no, program runs in non-interactive mode.
submed	no/yes	Remove the horizontal band using the wrap technique? (default option "no" should be good for majority of the sequences, and need to be changed to "yes" in very exceptional cases).
immos	yes/no	Mosaic the images?
dophot	yes/no	Do photometry?
xshift,yshift	0,0	X-shift[=Xref-Xin] and Y-shift[=Yref-Yin] with respect to the reference image. Especially useful when all the images of a sequence need to be aligned to an image not included in the sequence (e.g. reference image is in the J-band, and we want all the K-images aligned to the J-band reference image). In such a case, mean shift in pixels between the two sets should be entered here. However, if coordinates of the reference star in each image are measured and entered into the "sequenc" file, then XY shifts should be left at their default values of zero.
skyout	600	Minimum shift of the off-field sky frame. If a frame in the sequence is shifted more than skyout pixels with respect to the reference frame, then it is assumed that the frame does not contain the object of interest, and that frame is used only as a sky frame.
skyin	150.	Minimum shift of the on-field sky frame. While observing point sources, it is not necessary to go completely out of the object field to obtain good sky frames. It is a practice to center the point source in different quadrants for different exposures of a single sequence. The parameter skyin is the minimum shift of a frame with respect to the reference, in order to consider that frame suitable for sky extraction. So if the shift of a frame (wrt to the reference) is less than skyin, then it is not used for sky extraction, and if the shift is more than skyout, then it is not used for object extraction. All frames with shifts greater than skyin are used as sky frames. Frames in a sequence with shifts greater than skyin, but less than skyout, are used both for sky and object extraction. Warning:: For off-field sky frames (shifts > skyout), make sure that each off-field sky in a sequence is distinct from the rest. Also when point sources are moved from one quandrant to the other, and shifts are intermediate between skyin and skyout, make sure that the position of the point source is not repeated in a single sequence. In both these cases, it is recommended to use cansky to get a good sky frame and then give it in the parameter skyfram.
skysect	[551:575,551:575]	Section for skyvalue estimation. This parameter is used only when flattyp=skynew. The section is used to estimate the average sky counts, which will be subtracted from all images.
flatsec	[551:575,551:575]	Section for flat normalization. Flat field frame is normalized to the average of this section.
boxsize	11	Imalign boxsize. The size in pixels of the box to use for the final centering, during which all the sources in refimag.coo are recentered in each image using the initial estimate of the relative shift for each image. Care should be taken to choose an appropriate value for this parameter, since it is highly data dependent. See imalign.
bigbox	15	Imalign bigbox. The size in pixels of the box to use for coarse centering. The coarse pass through the centering algorithm is made with the box centered at the nominal position of the first source in the coordinate list. Coarse centering is performed only if the shifts file is undefined. Care should be taken to choose an appropriate value for this parameter, since it is highly data dependent. Large values should be suspect until the final results are checked to see that the centering did not converge on the wrong coordinates, although the usual result for an inappropriate bigbox size is that the algorithm fails to converge and the task aborts. See imalign. What to do if the task aborts? The values of the parameters xshift,xshift,boxsize,bigbox are critical for a successful alignment of all the images in a sequence. If the telescope pointing and guiding is good, then imalign might be successful with the default values. However, if the centering algorithm fails, then task aborts, and canobj is terminated without completion. In such a case, you should rerun canobj after editing one or more of these four parameters. However it might take a lot of experience to tune the values of these parameters. Alternatively, sequenc=file.seq can be edited to put the exact centroid of the reference star in each image of a sequence. This can be achieved by displaying each sky subtracted image in the sequence and obtaining the centroid of reference star (with imexamin "a"). The latter option, though involves some interaction, is foolproof and is highly recommended whenever the task aborts.
crval1, crval2	0,0	Right Ascension and Declination (both in degrees) corresponding to the pixel (crpix1,crpix2). When canref is run, the header RA and Dec values of the reference frame are substituted here. The image header of the combined final image is updated with these keywords.
crpix1, crpix2	512.5,512.5	Pixel corresponding to the crval1, crval2 values. When canref is run, these parameters take the values for the center of the chip (canref.crpix=center) or the pixel value of the reference star (canref.crpix=refstar). The image header of the combined final image is updated with these keywords.
cdelt1, cdelt2	8.888889E-5, 8.888889E-5	Pixel scale in RA and Dec (in degrees/pixel). The default values correspond to 0.32 arcsec/pixel. It is assumed that the detector had been aligned before the observations, such that RA and Dec axes coincide with X-axis, and Y-axis respectively. Both the RA and Declination increase in the same direction as the pixels, i.e. North is up, and East is to the right.
dowcs	yes	Transform the image to N_up E_left? If yes, rotates the X-axis by 180 degrees, so that RA increases to the left (i.e. East is to the left).
xymag	2.	XY reduction factor. If dowcs=yes, the image is compressed by the factor xymag, in addition to rotating the X-axis by 180 degrees.
jhkzp	20.60 20.40 20.20	JHK Zeropoints to be used when doing photometry of the objects in the file refimag.xy. The script automatically detects the filters and uses the corresponding Zeropoint. It is assumed that the the image header keyword FILTER2 = 1,2,3 for J,H and K bands respectively. For any other value of FILTER2, a zeropoint of 20 is used.

Description of parameters of canref

cl> epar canref

                    Image Reduction and Analysis Facility
PACKAGE = user
   TASK = canref

imref   =                    2  No of the ref image (2 for as40k002 etc.)
refcoo  =          as23j002ref  file (ref.coo) with ref star position or as40k00
(crpix  =               center) [center/refstar] XY pixels of RA-Dec
(mode   =                   ql)

Parameter	Default/acceptable values	Short description
imref	2	No of the ref image (2 for as40k002 etc.)
refcoo	as23j002ref	file (ref.coo) with ref star position or as40k002ref. File "ref.coo" should contain the position (pixels) of the reference star. Name of the file can be anything that ends with extension ".coo". If the file name does not end with ".coo", then it is assumed that you are giving the value suitable for canobj.refimag. Purpose of this option is to align images in a second filter (e.g. K) to previously analysed filter image (e.g. J). In this special case, files refimag.coo, refimag.xy and refimag.fits should have been created manually or through an earlier run of canref.
crpix	center/refstar	[center/refstar] XY pixels corresponding to the header RA-Dec. Default option corresponds to the center of the image [512.5,512.5]. With "refstar" header RA-Dec values are asigned to the reference star.

Guidelines and examples for successful running of canobj and canref

cl> ls as23*I.fits
as23h001_I.fits  as23h002_I.fits as23h003_I.fits 
as23h004_I.fits  as23h005_I.fits as23h006_I.fits  
as23j001_I.fits  as23j002_I.fits as23j003_I.fits
as23j004_I.fits  as23j005_I.fits as23j006_I.fits  
as23k001_I.fits  as23k002_I.fits as23k003_I.fits
as23k004_I.fits  as23k005_I.fits as23k006_I.fits
cl> unlearn canobj
cl> canobj as23j 1 6 sequenc=preview

... when images are displayed one by one, identify one star (preferably in image "as23j002_I.fits"), and find its centroid by pressing "a" when the cursor is the image window. Once the canobj trial run is over, edit (with any editor of your preference) a file called "ref.coo" and paste the xy pixel coordiantes of the identified star. You are now ready to run canref.

cl> canref 2 ref.coo

...Hit "Return" for all the querries. You will note that the querried parameters are basically the first 3 parameters of canobj. When the task ends, the RA and Dec of each image in the sequence, as well as the shifts of images with respect to the reference image, appears on the screen. Canref passes all the parameters, that are necessary to align the images, to the parent task canobj. You are now ready to run canobj for the second time. If you have a good flat field, then change the canobj.flattyp=flat and canobj.flatim="flatname". Otherwise keep them at their default values.

cl> canobj

...Hit "Return" for all the querries. Quit "imedit" and "imexamin" tasks that canobj invokes, with 'q", and if all the images look good, keep hitting "Return". A series of messages would appear on the text window. If the script ends successfully, the final rotated image (as23j_2a.fits) is displayed with the coordinate grid superimposed. On text screen, you will see magnitudes of the reference star in each image, followed by its magnitude in as23j_2.fits. Make sure that the coordinates and magnitudes of the star in different images of the sequence, do not differ by more than 2 pixels and 1 mag (approximately), respectively.

Input/Output filenames

Filename	Examples	Short description
root	as23k	An example for the root file name
iref	2	Reference image number. 2===> as23k002_I.fits is the reference image
rootiii_I.fits	as23k002_I.fits, as23k003_I.fits, as23k004_I.fits etc	Input file name with raw data (bias already subtracted during data acquisition)
rootiiisf.fits with iii = imbeg to imend	as23k002sf.fits, as23k003sf.fits, as23k004sf.fits etc	Sky subtracted, and flat-fielded Images. Saved only when the the script aborts due to error in centering stars.
rgrootiiisf.fits with iii = imbeg to imend	rgas23k002sf.fits, rgas23k003sf.fits, rgas23k004sf.fits etc	Sky subtracted, and flat-fielded and Aligned Images.
root_iref.fits	as23k_2.fits	The final image resulting from combining (imcombine combine=average reject=minmax) all rgrootiiisf.fits.
root_irefa.fits	as23k_2a.fits	X-axis of Image root_iref.fits rotated by 180 deg and compressed by xymag (generated only when dowcs=yes).
root_iref_sky.fits	as23k_2_sky.fits	Combined (and edited) sky frame (imcombine combine=median zero=median)
refimag.fits	as23k002ref.fits	Sky subtracted Image containing the reference star at the reference position.
root00iref.seq	as23k002.seq	Text file containing the xy pixels of the reference star in each of the sequence images.
refimag.coo	as23k002ref.coo	Text file containing the xy pixels of the reference star(s) in the reference image.
refimag.xy	as23k002ref.xy	Text file containing the xy pixels of all stars for which a quick photometry is to be carried out.
refimag.mag	as23k002ref.mag	Text file containing the photometric results of the first four stars in refimag.xy. Output of txdump.
rgrootiiisf.mag.1 with iii = imbeg to imend	rgas23k002sf.mag.1, rgas23k003sf.mag.1, rgas23k004sf.mag.1 etc.	Unedited output of photometry with phot.
root-canica.log	as23k-canica.log	Log file containing the details of reduction with canobj.
imexam.log		Log file containing the results of imexamin task

Frequently Encountered Problems and Troubleshooting

What do I do if the task canobj aborts in the second run?
See the help of canobj.bigbox above.
What do I do if the task canref aborts?
The only reason this could happen is if the header keywords "AR" or "DEC" are empty for at least one image in the sequence. Mostly this error is due to the header of the first image in the sequence, which is probably the "training" image. In such a case, you should edit the header parameters "AR" or "DEC" using "hedit", and put the RA and Dec values corresponding to the next image in the sequence. As of now (Dec 2003), the first two images should have the same coordinates. Even when canref does not abort, if you find that the shifts printed out is different for the first two images, then also, you should correct the header keywords "AR" or "DEC", and re-run canref.
What do I do if I don't have a good flat field?
Skyframes have too many "black structures" or "stellar residuals".
None of the images in the sequence are "off-source". So how do I obtain sky?
I want my JHK bands to be aligned to each other? How can I do that?
How reliable are the astrometric parameters putin by canobj?

Frequently asked questions (FAQs)

Is it correct to refer near infrared images as CCD images?
Technically speaking No. They should be called near infrared array images. Unlike CCDs, NIR array detectors do not use Charge transfer to collect photoelectrons. Instead the amount of charge (or number of photoelectrons) at each pixel is sensed using a capacitor.
What is the basic difference between a NIR observation and a CCD observation?
Observations in the optical and the NIR differ basically due to the differing levels of background values. At optical wavelengths, the background comes mainly from scattering of city lights by atmospheric particles, where as at infrared wavelengths, the thermal emission from the atmosphere and/or the detector/telescope, competes (and even exceedes in the K-band), the emission due to scattering. Observations in the NIR bands are usually background-limited, and not limited by the strength of the source of interest, as the case in the optical. Thus total exposure time required to get the desired S/N ratio should be achieved through multiple exposures in the NIR, where as single exposure could do that in the optical wavelengths.
Is there any difference in the bias frames for NIR array and CCD observations?
What about flat-fielding in the NIR wavelengths?
Why subtract sky frame, and not a value in NIR observations?
How does the photometric accuracy of CCDs compare with that of NIR arrays?
What are the interesting objects for NIR observations?
What is the trade-off between sensitivity, and background, for highly reddened objects? (What is the wavelength dependence of sensitivity, background and extinction law in the JHK bands?)

Written by Divakara Mayya (ydm@inaoep.mx). Last updated: 20 Jan 2004