ftcalc - Calculate values for a table column.


ftcalc infile[ext] outfile column expression


'ftcalc' is a general table calculator utility that computes values for a table column based on an arbitrarily complex expression, which may be a function of the value of other columns or keywords in the table. The computed values are written into the specified column in an output copy of the input file. A new column will be created and appended to the table if the specified column does not already exist.

The expression is evaluated for each row of the table and the result is written to the specified column. The expression may evaluate to a numeric value (or a vector of values) or it may evaluate to a character string, depending on the data type of the output column.

See the 'calc_express' help file for a full description of the allowed calculator expression syntax. In general, the calculator expression may contain any of the following elements:

Column and keyword names:
The value of the column (in the current row) or the keyword will be substituted into the expression. Precede keyword names with a single pound sign, '#', if necessary to distinguish it from a column with the same name. If the column or keyword name contains a space or a character that could be interpreted as a mathematical operator, enclose the name in dollar sign characters, as in the column $MAX-PHA$, or the keyword #$DATE-OBS$. NOTE: column names that begin with the letters 'b', 'o', or 'h' followed by numeric digits must be enclosed in '$' characters, otherwise they will be interpreted as numeric constants in binary, octal, or hex notation, respectively.

To use a table entry in a row other than the current one in the calculator expression, follow the column's name with a row offset within curly braces. For example, 'PHA{-3}' will evaluate to the value of column PHA, 3 rows above the row currently being processed. One cannot specify an absolute row number, only a relative offset. Rows that fall outside the table will be treated as undefined, or NULL.

To use a single element of a vector column within a calculator expression, follow the column's name with a element number (beginning with 1 for the first element) within square brackets, as in 'PHAS[1] * 10'. Note that this expression evaluates to a scalar quantity (a single number), whereas the expression 'PHAS * 10' evaluates to a vector that is created by multiplying each element of the PHAS vector by 10.

Mathematical operators:
+, -, *, /, ** or ^ (exponentiation)

Boolean operators in C or Fortran-type notation:
.eq., ==, .ne., !=, .lt., <, .le., <=, .gt., >, .ge., >=, .or., ||, .and., &&, .not., !, and ~ (approximately equal, to within 1E-07)

Math library functions:
abs(x), cos(x), sin(x), tan(x), arccos(x), arcsin(x), arctan(x), arctan2(y,x), cosh(x), sinh(x), tanh(x), round(x), floor(x), ceil(x) exp(x), sqrt(x), log(x), log10(x), x%y (modulus), random() (returns random number >= 0 and < 1), randomn() (returns Gaussian distribution with zero mean and unit standard deviation), randomp(x) (returns a Poisson random distribution whose expected number of counts is X. X may be any positive real number of expected counts, including fractional values, but the return value is an integer.) min(x,y), max(x,y), accum(x), seqdiff(x), angsep(ra1, dec1, ra2, dec2) (all in degrees).

Numerical constants:
Numeric values are assumed to be in decimal notation unless preceded by a 'b', 'o', or 'h' for binary, octal, or hexidecimal notation, respectively (e.g., 'b100111', 'o72', or 'hFFC0'. (Note that some common column names such as 'H2' or 'B10' may be interpreted as numeric constants unless they are enclosed in '$' characters, as explained previously). The following predefined constants may also be used: #pi (3.1415...), #deg (#pi/180), #e (2.7182...), #row (substitutes the current row number into the expression). Two special constants, #null and #snull, can be used for testing if the expression value has an undefined numeric value or an undefined string value, respectively.

String constants:
enclose string values in quotes, e.g., 'Crab', 'M101'

Datatype casts to convert reals to integers or integers to reals:
(int) x, (float) i

Conditional expressions:
'b?x:y' where expression 'x' is evaluated if 'b' is TRUE (not equal to zero), otherwise expression 'y' is evaluated.

Test for near equality:
near(value1, value2, tolerance) returns 0 if value1 and value2 differ by more than tolerance.

Bit masks:
The 'x' character represents a wild card: b11x001 (binary), o447x1 (octal), h0FxD (hex)

Good time interval test:
This function returns 1 if the time value lies within one of the good time intervals, otherwise it returns 0. Specifying 'gtifilter()' is equivalent to 'gtifilter("", TIME, "*START*", "*STOP*")' and uses the GTI extension in the current FITS file to filter the TIME column using the START and STOP columns in the GTI extension.

Spatial region test:
This function returns 1 if the spatial position associated with that row of the table is located within the region defined by the specified region file. Specifying 'regfilter("region.reg", xpos, ypos)' uses the xpos and ypos table columns (and any associated World Coordinate System keywords) to determine the position, and the ASCII region file named 'region.reg'.

Vector column operators:
These funcions operate on a vector and return a scalar result: MIN(V), MAX(V), AVERAGE(V), MEDIAN(V), SUM(V), STDDEV(V), NELEM, and NVALID(V) (number of non-null values). See the 'calc_express' help file for more information.


infile [filename]
File name and optional extension name or number enclosed in square brackets of the input table to be operated on (e.g., 'file.fits[events]'). If an explicit extension is not specified then the first table extension in the file that is not a GTI (Good Time Interval) extension will be used. Additional filter specifiers can be appended to the file name, also enclosed in square brackets, to create a virtual input table as shown in some of the examples. The input file is not modified and instead, the calculations are performed on the output file copy.

outfile [filename]
Name of the output file that will contain the modified copy of the input file. To overwrite a preexisting file with the same name, prefix the name with an exclamation point '!' (or '\!' on the Unix command line), or set the 'clobber' parameter = YES.

column [string]
Name of the column to which the calculated values will be written. If the column does not already exist, a new column with this name will be appended to the output table. If a vector column is specified, and if the calculation evaluates to a scalar quantity (not a vector), then the 'element' paramenter (described below) may be used to specify which element of the vector should be used to store the result.

expression [string]
The expression used to calculate the output column's values. The expression can evaluate to any of the following data types: boolean, integer, double, string, or bit. If a vector column is used in the calculation expression, then the result will also be a vector with the same length. A text file containing the expression can be specified by preceding the filename with '@', such as '@file.txt'. The expression in the file can be arbitrarily complex and extend over multiple lines of the file. Lines that begin with 2 slash characters ('//') will be ignored and may be used to add comments to the file.

(rows = "-") [string]
A comma-separated list of rows or row ranges that will be operated on (e.g., "10-20, 25, 30-40"). By default, all the rows in a table are calculated.

(element = 0) [integer]
If the calculation evaluates to a scalar quantity (not a vector), the result may be written to a single element of an existing vector column by specifying the name of the column with the 'column' parameter and the element number (beginning with 1 for the first element) with this 'element' parameter. In this case the 'rows' parameter is ignored, and the calculation is performed on every row of the table. In all other cases 'element' should be set to the default value of 0.

(tform = "") [string]
Optional value string for the TFORM keyword which defines the datatype of the output column, if a new column is being created. If 'tform' is blank, then a default datatype as determined by the input expression will be used. The specified tform value must conform to the rules specified in the FITS standard for the TFORMn keyword, which depend on whether the table being operated on is a binary FITS table or an ASCII FITS table. For binary tables, legal TFORMn datatype codes are
               L - Logical
               X - Bit
               B - Byte
               I - 16-bit Integer
               J - 32-bit Integer
               A - Character string
               E - Single precision floating point
               D - Double precision floating point
The datatype code may be preceded by an integer vector length (usually 1).

Examples of legal TFORMn values for an ASCII table (where the first number represents the column width, and the second number represents the number of numeric digits after the decimal place) are:

               A10 - Character string
               I6  - Integer
               F12.2 - Fixed decimal floating point
               E12.5 - Exponential notation floating point
               D16.9 - Double precision exponential notation
(clobber = NO) [boolean]
If outfile already exists, then setting "clobber = yes" will cause it to be overwritten by the new file.

(chatter = 1) [integer, 0 - 5]
Controls the amount of informative text written to standard output. Setting chatter = 5 will produce detailed diagnostic output, otherwise this task normally does not write any output.

(history = NO) [boolean]
If history = YES, then a set of HISTORY keywords will be written to the header of the output table to record the value of all the ftcalc task parameters that were used to produce the output file.


Note that when commands are issued on the Unix command line, strings containing special characters such as '[' or ']' must be enclosed in single or double quotes.

1. Calculate the values for a column named AREA in the first table extension of a the input file by taking the product of the 'Xlen' and 'Ylen' column values in each row. Write the result to the new file 'outfile.fits'.

      ftcalc input.fits outfile.fits AREA Xlen*Ylen

2. Compute a 3 point running mean of the values in the 'Rate' column and write result to a single precision floating point column called 'Smooth'. A double precision column would have been created by default if the 'tform' parameter had not been specified.

      ftcalc input.fits outfile.fits Smooth '(Rate{-1}+Rate+Rate{1})/3.'

3. First, create a virtual input file that contains only the X and Y columns in the table, then compute the distance between a point whose X and Y coordinates are defined by the XCENTER and YCENTER keywords, and the positions defined by the X and Y column values in each row of the table. Store the result in a column called 'Radius'.

      ftcalc 'input.fits[col X;Y]'  outfile.fits  Radius 
        'sqrt( (X - #XCENTER)**2 + (Y - #YCENTER)**2 )'

4. Conditionally compute the 'Rate' column. If 'Counts' is greater than zero then Rate is calculated by dividing the 'Counts' column value by the 'EXPOSURE' keyword value. If the 'Counts' value is less than or equal to zero, then the Rate value is set = 0.

      ftcalc input.fits outfile.fits Rate 'Counts>0?Counts/#exposure:0.'


calc_express, ftcopy, ftselect, ftsort, filenames, colfilter, rowfilter, binfilter

fv, the interactive FITS file editor, can also be used to calculate values for table columns.

The design of this task is based on fcalc in the ftools package and on the CXC dmtcalc task.


March 2002