How To

Input Files

Input files are the survey images, passed to the pipeline in one directory:

$ ssos path/to/images

The images must have a .fits file ending to be recognized by the script.

If weight images shall be passed to the SExtractor runs, specify the directory containing the weight images using the WEIGHT_IMAGES parameter in the default.ssos. The weight images need to have the same filename as the exposures they shall be applied on, but with a _SCI_EXTENSION.weight file extension replacing the .fits, where SCI_EXTENSION has to be replaced by the appropriate extension integer. If no science extension is specified (equivalent to setting SCI_EXTENSION to all), simply replace the .fits extension with .weight.

Output Files

The script creates several directories in the target directory (current working directory unless specified with the -t option). The cats directory contains the SExtractor and SCAMP catalogues and the final output ssos_$CURRENTDATETIME.csv. For every SExtractor catalogue, there is also one .ahead file with the same filename. This file contains the observation date as MJD-OBS keyword, which is required for the subsequent run of SCAMP. SCAMP looks for extensions of catalogue headers in .ahead files.

target directory
│
└───cats
│      exposure1.cat
│      exposure1.head
│      exposure1.ahead  *optional*
│      exposure2.cat
│      exposure2.head
│      exposure2.ahead  *optional*
│      ..
│      full_1.cat
│      merged_1.cat
│      scamp.xml
│      skybot_all.csv
│      ssos.csv
│
└───checkplots
│      skybot_residuals.png
│
└───cutouts
│      SOURCE1_CATALOG1.fits
│      SOURCE1_CATALOG2.fits
│      SOURCE2_CATALOG1.fits
│      ..
│
└───logs
│      ssos_$DATETIME.log
│
└───weights
│      exposure1.weight  *optional*
│      expsoure2.weight  *optional*
│      ..

The cutouts directory contains the cutouts made by SWarp of the SSO detections. In the logs directory, you can find the log file of the pipeline, following the naming scheme ssos_$CURRENTDATETIME.log, carrying the same DATETIME appendix as the output csv file. The skybot directory stores the SkyBoT query results.

To judge the results of the pipeline, first go into the logfile. It looks like this:

$ cat /tmp/logs/sso_20180906142433.log

    --- SSO Recovery Pipeline ---       --- 14:24:33 2018/09/06 ---

SEX_CONFIG          semp/ssos.sex
SEX_PARAMS          semp/ssos.param
SEX_NNW             semp/default.nnw
SEX_FILTER          semp/gauss_2.5_5x5.conv
SCI_EXTENSION       0
SCAMP_CONFIG        semp/ssos.scamp
SWARP_CONFIG        semp/ssos.swarp
FILTER_DETEC        True
DETECTIONS          1,2,3
FILTER_PM           True
PM_LOW              0
PM_UP               225
PM_SNR              20
FILTER_PIXEL        True
DELTA_PIXEL         2
FILTER_MOTION       True
IDENTIFY_OUTLIER    True
OUTLIER_THRESHOLD   2
R_SQU_M             0.95
FILTER_TRAIL        False
RATIO               0.25
FILTER_T_DIST       False
SIGMA               2
FILTER_STAR_REGIONS True
DISTANCE            300
HYGCAT              semp/hygdata_v3.csv
CROSSMATCH_SKYBOT   True
CROSSMATCH_RADIUS   10
OBSERVATORY_CODE    500
FOV_DIMENSIONS      1.5x1.5
EXTRACT_CUTOUTS     True
FIXED_APER_MAGS     False
REFERENCE_FILTER    gSDSS
WEIGHT_IMAGES       False
CUTOUT_SIZE         256

21 Exposures    |   epoch1 |   35.92deg Ecliptic Latitude


Running SExtractor..
Running SCAMP.. Done.

 --- Starting Filter pipeline ---

All Sources          15323
FILTER_DETEC         7744
FILTER_PM            30
FILTER_PIXEL         30
FILTER_MOTION        8
FILTER_STAR_REGIONS  7

Cross-matching SSO candidates with SkyBoT.. 2 matches found.

Extracting cutouts with SWARP.. Done.

All done!   |   7 SSO candidates found  |   The analysis ran in 4 seconds

Output File: /tmp/cats/ssos.csv
Log File:   /tmp/logs/sso_20180906142433.log

In case an SSO was detected, you should look at the cats/ssos.csv file and the cutouts to verify the detection. In cats/ssos.csv, you can also find basic SkyBoT parameters, if the object was successfully matched. For more detailed information on the possible match, look into the SkyBoT queries in skybot/.

The final database contains the following columns

SOURCE_NUMBER - SCAMP groups detections into sources by giving them the same SOURCE_NUMBER
CATALOG_NUMBER - Number of SExtractor catalog containing this source detections
RA - Right Asecension of source in degree
DEC - Declincation of source in degree
EPOCH - Beginning of observation in decimalyear
MAG - Magnitude
MAGERR - Error of magnitude as derived by SExtractor
PM - Proper motion of source in "/h
PMERR - Error on proper motion
PMRA - Proper motion in RA in "/h
PMRA_ERR - Error
PMDEC - Proper motion in Dec in "/h
PMDEC_ERR - Error
MID_EXP_MJD - Mid-exposure time in MJD
DATE-OBS_EXP - Beginning of observation in ISOT format
EXPTIME_EXP - Exposure time
OBJECT_EXP - Object ID of J_VAR field
FILTER_EXP - Name of filter that the field was imaged in
RA_EXP - Center right ascension coordinate of exposure
DEC_EXP - Center declination coordinate of exposure
FILENAME_EXP - Filename of input image that the SSO was detected in
MATCHED - Boolean value, true if SSO was matched to known object
SKYBOT_NAME - If matched, name of matching SSO
SKYBOT_CLASS - Class of SkyBoT match
SKYBOT_MAG - Predicted magnitude
SKYBOT_RA - Predicted RA
SKYBOT_DEC - Predicted Dec
SKYBOT_PMRA - Predicted PM in RA
SKYBOT_PMDEC - Predicted PM in Dec
SKYBOT_NUMBER - SkyBoT match number
SKYBOT_DELTARA - Residual of observed and predicted RA in arcsecond
SKYBOT_DELTADEC - Residual of observed and predicted Dec in arcsecond
X_IMAGE - position of source in exposure in X in pixel
Y_IMAGE - position of source in exposure in Y in pixel
AWIN_IMAGE - Semi-major axis of fitted source ellipse in pixel
ERRA_IMAGE - Error
BWIN_IMAGE - Semi-minor axis of fitted source ellipse in pixel
ERRB_IMAGE - Error
THETAWIN_IMAGE - Angle of source, see SExtractor guide
ERRTHETA_IMAGE - Error
ERRA_WORLD - Error of AWIN in degree
ERRB_WORLD - Error of BWIN in degree
ERRTHETA_WORLD - Error of THETAWIN in degree
ERRX2_WORLD - Variance of RA in degree
ERRY2_WORLD - Variance of Dec in degree
ERRXY_WORLD - Covariance of RA/Dec in degree
FLAGS_EXTRACTION - SCAMP parameter
FLAGS_SCAMP - SCAMP parameter
FLAGS_IMA - SCAMP parameter
FLAS_SSOS - Flags added by ssos pipeline

More information on these parameters can be found in the SExtractor and SCAMP manuals.

Command-Line API

Again, the command-line API is heavily inspired by the SExtractor and SCAMP softwares. The following help is printed when the pipeline is called without arguments or with the -h or --help flag set:

$ ssos --help
usage: ssos [-h] [-c SET_UP] [-t TARGET] [-l LOG] [-q] [--sex] [--scamp]
        [--swarp] [--skybot] [-FILTER_DETEC bool] [-FILTER_PM bool]
        [-FILTER_PIXEL bool] [-FILTER_MOTION bool]
        [-IDENTIFY_OUTLIERS bool] [-FILTER_TRAIL bool]
        [-FILTER_T_DIST bool] [-FILTER_STAR_REGIONS bool]
        [-CROSSMATCH_SKYBOT bool] [-EXTRACT_CUTOUTS bool]
        [-FIXED_APER_MAGS bool] [-SEX_CONFIG value] [-SEX_PARAMS value]
        [-SEX_FILTER value] [-SEX_NNW value] [-SCAMP_CONFIG value]
        [-SWARP_CONFIG value] [-SCI_EXTENSION value]
        [-WEIGHT_IMAGESDETECTIONS value] [-PM_LOW value] [-PM_UP value]
        [-PM_SNR value] [-DELTA_PIXEL value] [-OUTLIER_THRESHOLD value]
        [-R_SQU_M value] [-R_SQU_T value] [-SIGMA value] [-DISTANCE value]
        [-HYGCAT value] [-CROSSMATCH_RADIUS value] [-CUTOUT_SIZE value]
        [-REFERENCE_FILTER value] [-OBSERVATORY_CODE value]
        [-FOV_DIMENSIONS value]
        fields [fields ...]

Pipeline to search for Solar System objects in wide-field imaging surveys

positional arguments:
  fields                Path to directory of field to be searched

optional arguments:
  -h, --help            show this help message and exit
  -c SET_UP, --config SET_UP
                        Path to configuration file
  -t TARGET, --target TARGET
                        Target directory to save fits files. If no target
                        given, writing to CWD
  -l LOG, --log LOG     Set the logging level. Valid arguments are DEBUG,
                        INFO, WARNING, ERROR, CRITICAl.
  -q, --quiet           Suppress logging to console
  --sex                 Force SExtractor runs
  --scamp               Force SCAMP runs
  --swarp               Force SWARP runs
  --skybot              Force SkyBoT query

Filter Settings:
  -FILTER_DETEC bool    Override FILTER_DETEC setting. Must be True or False.
  -FILTER_PM bool       Override FILTER_PM setting. Must be True or False.
  -FILTER_PIXEL bool    Override FILTER_PIXEL setting. Must be True or False.
  -FILTER_MOTION bool   Override FILTER_MOTION setting. Must be True or False.
  -IDENTIFY_OUTLIERS bool
                        Override IDENTIFY_OUTLIERS setting. Must be True or
                        False.
  -FILTER_TRAIL bool    Override FILTER_TRAIL setting. Must be True or False.
  -FILTER_T_DIST bool   Override FILTER_T_DIST setting. Must be True or False.
  -FILTER_STAR_REGIONS bool
                        Override FILTER_STAR_REGIONS setting. Must be True or
                        False.
  -CROSSMATCH_SKYBOT bool
                        Override CROSSMATCH_SKYBOT setting. Must be True or
                        False.
  -EXTRACT_CUTOUTS bool
                        Override EXTRACT_CUTOUTS setting. Must be True or
                        False.
  -FIXED_APER_MAGS bool
                        Override FIXED_APER_MAGS setting. Must be True or
                        False.
  -SEX_CONFIG value     Override SEX_CONFIG setting.
  -SEX_PARAMS value     Override SEX_PARAMS setting.
  -SEX_FILTER value     Override SEX_FILTER setting.
  -SEX_NNW value        Override SEX_NNW setting.
  -SCAMP_CONFIG value   Override SCAMP_CONFIG setting.
  -SWARP_CONFIG value   Override SWARP_CONFIG setting.
  -SCI_EXTENSION value  Override SCI_EXTENSION setting.
  -WEIGHT_IMAGESDETECTIONS value
                        Override WEIGHT_IMAGESDETECTIONS setting.
  -PM_LOW value         Override PM_LOW setting.
  -PM_UP value          Override PM_UP setting.
  -PM_SNR value         Override PM_SNR setting.
  -DELTA_PIXEL value    Override DELTA_PIXEL setting.
  -OUTLIER_THRESHOLD value
                        Override OUTLIER_THRESHOLD setting.
  -R_SQU_M value        Override R_SQU_M setting.
  -R_SQU_T value        Override R_SQU_T setting.
  -SIGMA value          Override SIGMA setting.
  -DISTANCE value       Override DISTANCE setting.
  -HYGCAT value         Override HYGCAT setting.
  -CROSSMATCH_RADIUS value
                        Override CROSSMATCH_RADIUS setting.
  -CUTOUT_SIZE value    Override CUTOUT_SIZE setting.
  -REFERENCE_FILTER value
                        Override REFERENCE_FILTER setting.
  -OBSERVATORY_CODE value
                        Override OBSERVATORY_CODE setting.
  -FOV_DIMENSIONS value
                        Override FOV_DIMENSIONS setting.
[1]In case weight maps are provided, the pipeline overrides the WEIGHT_TYPE parameter in the SExtractor config file with MAP_WEIGHT