DDECal

Calibrate direction dependent gains.

inputs

type

type:

string

doc:

Case-insensitive step type; must be ‘ddecal’.

antenna_averaging_factors

type:

list of ints

doc:

Averaging factors per antenna to allow antenna-dependent solution intervals. This allows using different intervals for different antennas. This is done by averaging solutions the same antenna but from different sub-intervals together. This feature works together with the solutions_per_direction setting. An antenna averaging factor is not allowed to exceed the maximum number of direction subsolutions.

Ionospheric effects vary more strongly in time and frequency for more distant stations (relative to a central reference station). Thus, by using shorter time intervals and less frequency smoothing for more distant groups of stations, these fast variations can be captured without adding too much additional free parameters for more central stations. Antenna intervals and smoothing depend on the expected variation per group of antenna. For LOFAR, CS can typically have at least 8 times longer intervals compared to RS. If only antenna-dependent intervals are desired, the solutions_per_direction setting should be set to the maximum antenna averaging factor. E.g., with antenna_averaging_factors = [a:3, b:6, c:1], the solutions_per_direction should be set to 6 for all directions. This setting will use a time interval of 1/2, 1 and 1/6 times the full solution interval (solint) for the antennas a, b and c, respectively. Pattern-matching can be used to set multiple antennas at once, e.g. [RS*:5, CS*:2, [CS001_HBA001, CS002_HBA001]:3] would set all stations starting with RS to 5, with CS to 2 except CS001_HBA001 and CS002_HBA001 which are set to 1. Directional and antenna dependence can be combined, in which case the two work cummulatively. That is, with a setting of antenna_averaging_factors = [a:3, b:6, c:1] and solutions_per_direction = [6, 6, 1, 1], the solver will divide the solution interval into 6 subsolutions for direction 0 and 1. For antenna a, every 3 of those will be averaged together, and antenna b will be averaged by a factor of 6 (i.e. its solution time equals solint). Antenna c will not be averaged. For directions 2 and 3, there is only 1 subsolution available, so no averaging is performed for any of the antennas.

default:

[]

antenna_smoothness_factors

type:

list of doubles

doc:

A list of extra smoothing factors per antenna in the same format as antenna_averaging_factors, e.g. [RS*:6.5,CS*:2.5]. Values above one cause stronger smoothing, values lower than one cause less smoothing. When smoothnessrefdistance > 0, these factors will be multiplied by the distance-based smoothing factors, so both affect the per-antenna smoothness setting in this case.

default

antennaconstraint

type:

list

default:

[]

doc:

A list of lists specifying groups of antennas that are to be constrained to have the same solution. Example: “[ [CS002HBA0,CS002HBA1],[CS003HBA0,CS003HBA1] ]” will keep the solutions of CS002HBA0 and 1 the same, and the same for CS003.

applycal.*

doc:

ApplyCal sub-step, same as in Predict step. One can pass an h5parm with as many directions as set in “directions” and each direction model is corrupted accordingly.

approxchunksize

type:

int

default:

0

doc:

Size of fitted chunksize during approximation stage in nr of channels. With approxchunksize=1 the constraint is disabled during the approx stage (so channels are solved for independently). Once converged, the solutions are constrained and more iterations are performed until that has converged too. The default is approxchunksize=0, which calculates the chunksize from the bandwidth (resulting in 10 chunks per octave of bandwidth).

approximatetec

type:

bool

default:

False

doc:

Uses an approximation stage in which the phases are constrained with the piece-wise fitter, to solve local minima problems. Only effective when mode=tec or mode=tecandphase.

approxtolerance

type:

double

default:

tolerance*10

doc:

Tolerance at which the approximating first stage is considered to be converged and the second full-constraining stage is started. The second stage convergences when the tolerance set by the ‘tolerance’ keyword is reached. Setting approxtolerance to lower values will cause more approximating iterations. Since tolerance is by default 1e-5, approxtolerance is by default 1e-4.

coreconstraint

type:

double

default:

0

doc:

Distance in meters. When unequal to 0, all stations within the given distance from the reference station (0) will be constrained to have the same solution.

datause

type:

string

default:

full

doc:

Selects what part of the data to use: full, dual or single. When full is selected, the full visibility matrix is used (e.g. XX, XY, YX, YY). With dual, only the diagonal entries (e.g. XX and YY) of the visibility matrix are used, off-diagonal values (e.g. YX and XY) are not used. This makes the solver faster and decreases its memory requirements, and can be used in situations where the off-diagonal visibility values do not contribute much. Dual visibility solving is currently only implemented for the iterative algorithm (scalar or diagonal). Nomenclature of duo and dual is used for this feature to distinguish from the solution form (e.g. diagonal, full-Jones). With single, the diagonal is averaged together to a single (Stokes I) visibility. This is only implemented for the scalar iterative solver.

detectstalling

type:

bool

doc:

Stop iterating when no improvement is measured anymore (after a minimum of 30 iterations).

default:

True

directions

type:

list

default:

[]

doc:

List of directions to calibrate on. Every element of this list should be a list of facets. Default: every facet is a direction.

faradaydiagonalmode

type:

string

default:

rotation

doc:

In Faraday-rotation mode, this parameter specifies how to constrain the diagonal. It supports the same options as rotationdiagonalmodel. If set to ‘rotation’ (the default), only Faraday rotation is fitted: after rotation, the diagonal will have unit gains.

faradaylimit

type:

double

default:

0.0

doc:

In Faraday-rotation mode, this limits the range of the differential Faraday rotation value, which means that the fitter will only go through the possible wraps for the given range. This is to make the solver faster. The searched range is [-faradaylimit, faradaylimit], and the faradaylimit is given in units of radians per meter. If set to zero, DP3 will search the entire range of possible Faraday rotation values. For some sets the default is sufficiently fast, but for some observations (e.g. LOFAR LBA with many channels) this may be too slow, and a faradaylimit of for example 5.0 can make the fitter considerably faster.

flagdivergedonly

type:

bool

default:

False

doc:

Flag only the unconverged solutions for which divergence was detected. At the moment, this option is effective only for rotation+diagonal solves, where divergence is detected when the amplitudes of any station are found to be more than a factor of 5 from the mean amplitude over all stations. If divergence for any one station is detected, all stations are flagged for that solution interval. Only effective when flagunconverged=true and mode=rotation+diagonal.

flagunconverged

type:

bool

default:

False

doc:

Flag unconverged solutions (i.e., those from solves that did not converge within maxiter iterations).

grouppredict

type:

bool

default:

False

doc:

Alternative predict step for BDA data, which groups the input data into similarly averaged baselines and uses a regular Predict step for each group. Due to the overhead of having different groups, the normal predict step processes BDA data faster. This step is therefore mainly used for benchmarking.

h5parm

type:

string

doc:

Filename of output H5Parm. If empty, defaults to instrument.h5 within the measurement set.

idg.buffersize

type:

int

default:

Based on memory

doc:

Set the amount of timesteps that are to be used for each IDG buffer.

idg.images

type:

list

doc:

Filename of .fits model images, one per frequency term. The terms are defined as for a polynomial source spectra (not logarithmic), e.g. see this WSClean page. The frequency in the metadata of the fits files is used as nu₀ in the polynomial evaluation.

default:

[]

idg.regions

type:

string

default:

""

doc:

DS9 regions file describing the facets for IDG prediction.

initialsolutions.gaintype

type:

enum?

doc:

If using H5Parm, specifies what type of jones parameters are stored. Will be deduced from soltab if not provided.

symbols:

gain, tec, clock, rotationangle, rotation, scalarphase, scalaramplitude, rotationmeasure, fulljones

initialsolutions.h5parm

type:

string

doc:

Filename of the H5Parm containing initial solutions, which the solver will use as initial values, if provided.

initialsolutions.interpolation

type:

enum

default:

nearest

doc:

If using H5Parm, the type of interpolation (in time and frequency) to use.

symbols:

nearest, linear

initialsolutions.missingantennabehavior

type:

enum

defeault:

error

doc:

What to do if an antenna is not in the H5Parm. Possible values are error (stop the program), flag (flag all data involving this antenna) and unit (treat the solutions for this antenna as 1, e.g., no correction for this antenna).

symbols:

error, flag, unit

initialsolutions.soltab

type:

string[]?

default:

[amplitude000, phase000]

doc:

The name or names of the H5Parm soltab(s).

keep_host_buffers

type:

bool

default:

False

doc:

Setting this to true will cause the host buffers to be kept during the execution if used in combination with the GPU solver. This can cause an higher consumption of memory but saves execution time.

keepmodel

type:

bool

default:

False

doc:

If true, pass the corrected model data to the next step. DDECal stores the model data using its step name plus the (first) direction name. For example, if steps=[ddecal1] and ddecal1.directions=[[center,dec_off],[ra_off]], the model data names are ddecal1.center and ddecal1.ra_off. If false, delete all model data, including the model data from reusemodel.

llssolver

type:

string

default:

qr

doc:

Select which linear least-squares solver to use for the direction-solving algorithm. When solveralgorithm is not directionsolve, this parameter has no effect. Supported least-squares solvers are qr, svd and normalequations.

maxapproxiter

type:

int

default:

maxiter/2

doc:

Maximum number of iterations during approximating stage.

maxiter

type:

int

default:

50

doc:

Maximum number of iterations.

minvisratio

type:

double

default:

0

doc:

Minimum number of visibilities within a solution interval, e.g. 0.6 for at least 60% unflagged vis. Intervals with fewer vis will be flagged.

mode

type:

string

default:

diagonal

doc:

Solve type. Options are scalar, scalarphase, scalaramplitude, diagonal, diagonalphase, diagonalamplitude, fulljones, tec, tecandphase, rotation, rotation+diagonal and faradayrotation.

model_weighted_constraints

type:

bool

default:

False

doc:

Setting this to true will use (direction-dependent) weights in the constraints that are calculated from the model data: solutions with large predicted model data values are upweighted. This is currently an experimental feature. It may be useful for the smoothness constraint when dealing with sources that go through the beam null at certain frequencies. In this case, the direction-dependent weights would make sure that the solutions where the model is predicted to be zero will not skew the smoothing. At present, only the smoothness constraint is affected by this setting.

modeldatacolumns

type:

list

default:

[]

doc:

Use model data from the measurement set. All directions from this parameter will be pointing to the measurement set metadata.

modelnextsteps

type:

list

default:

[]

doc:

Defines the steps that will be applied to the model in every direction before using the output for calibration.

modelnextsteps.<direction>

type:

string

doc:

Allows steps to be defined for a specific direction. These directions, (given in the <direction> placeholder) can be specified by their model column name, the name of a DS9 region, or the first patch name in the direction if using a sky model. For example: modelnextsteps.MODEL_DATA_2=[applyextrabeam], applyextrabeam.type=applybeam.

nchan

type:

int

doc:

Number of channels in each channel block, for which the solution is assumed to be constant. The default is 1, meaning one solution per channel (or in the case of constraints, fitting the constraint over all channels individually). 0 means one solution for the whole channel range. If the total number of channels is not divisable by nchan, some channelblocks will become slightly larger. When using BDA, the value of nchan should be bigger than or equal to the maximum number of averaged frequency channels.

default:

1

onlypredict

type:

bool

default:

False

doc:

Instead of solving, output the predicted visibilities instead. This is useful for testing, although when doing faceted prediction with IDG, it might be fast for certain cases.

phasereference

type:

bool

default:

True

doc:

Use phase referencing. This setting is only effective when mode=tec or mode=tecandphase.

propagateconvergedonly

type:

bool

default:

False

doc:

Propagate solutions of the previous time slot only if the solve converged. Only effective when propagatesolutions=true.

propagatesolutions

type:

bool

default:

False

doc:

Initialize solver with the solutions of the previous time slot.

reusemodel

type:

list

default:

[]

doc:

List of model data names to reuse from previous step(s). The list should contain the model data names as set by a previous step, prefixed with the name of that step. Example: [ddecal1.MODEL_DATA, ddecal1.center]. The list may contain expressions with ?, which matches any single character, and *, which matches zero or more characters. For example, [*] matches all model data names.

rotationdiagonalmode

type:

string

default:

diagonal

doc:

In rotation+diagonal mode, this parameter specifies how to constrain the diagonal. Options are: diagonal, diagonalamplitude, diagonalphase, scalar, scalaramplitude, scalarphase. If for example ‘scalar’ is selected, a rotation is fitted together with a (complex) scalar value, i.e., a diagonal with two equal complex values. This choice affects the structure of the h5parm file: e.g. in scalar mode, the amplitude and phase tables will not have a “pol” (polarization) dimension.

rotationreference

type:

bool

default:

False

doc:

Use rotation referencing with the first antenna as the reference antenna. When enabled, the first antenna gets a rotation angle of 0. The rotation angle of the other antennas will be relative to the first antenna. If the first antenna has no valid data, the second antenna will be the reference antenna, and so on. This setting is only effective when mode=rotation+diagonal.

savefacets

type:

bool

default:

False

doc:

Write out each facet as a fits file (named facet<N>.fits). Only useful using idg.

smoothness_dd_factors

type:

list

default:

empty

doc:

If non-empty, the provided factors scale the smoothing kernel per direction. This allows e.g. to smooth high S/N-ratio directions by a smaller amount. Higher values mean stronger smoothing. Currently, each value should be larger than zero and at most one, i.e. they can only make the kernel smaller. Therefore, smoothnessconstraint should be set to the largest desired smoothing value, and the factors in this list can shrink the kernel as needed. The factors in this list work cummulative with other smoothness modifiers. If this solve uses direction-dependent intervals, all intervals toward some directions will use the same value.

smoothness_kernel_truncation

type:

bool

default:

True

doc:

Whether to trim the Gaussian kernel that is used for smoothing at 3 sigma. Truncation makes calibration faster but causes a slightly less smooth result. For analyses that require highly smooth gain solutions, such as high-redshift 21-cm experiments, the truncation can be turned off.

smoothnessconstraint

type:

double

default:

0.0

doc:

Kernel size in Hz. When unequal to 0, will constrain the solutions to be smooth over frequency by convolving the solutions with a kernel of the given size (bandwidth). The default kernel is a Gaussian kernel, and the kernel size parameter is the 3 sigma point from the centre where the kernel is cut off. For example, with a kernel size of 2 MHz and with 1 MHz channels, the kernel will cover 2 channels on both side from the centre, and therefore 5 channels in total.

smoothnessrefdistance

type:

double

default:

0.0

doc:

An optional reference distance (in meters) for the smoothness constraint. When unequal to 0, the size of the smoothing kernel will vary, such that antennas (stations for LOFAR) closer to antenna 0 will be smoothed stronger, i.e. with a larger kernel size..

smoothnessreffrequency

type:

double

default:

0.0

doc:

An optional reference frequency (in Hz) for the smoothness constraint. When unequal to 0, the size of the smoothing kernel will vary over frequency by a factor of “(smoothnessreffrequency / frequency) ^ smoothnessspectralexponent”. Normally, the exponent < 0 and the kernel will be smaller for lower frequencies. This compensates for the fact that the ionosphere causes larger fluctuations at lower frequencies.

smoothnessspectralexponent

type:

double

default:

-1.0

doc:

The exponent term that scales the kernel size in relation to the frequency. See smoothnessreffrequency: smoothnessspectralexponent only has an effect when smoothnessreffrequency is set to a nonzero value. A value of -1 assumes a TEC 1/nu relationship..

solint

type:

int

doc:

Solution interval in timesteps. If a value of 0 is specified, the solver will use a single solution interval that covers the entire observation. When using BDA, the solint should be bigger than or equal to the maximum averaged time interval.

default:

1

solutions_per_direction

type:

list

default:

[1, 1, ..., 1]

doc:

List of number of subsolutions per solution interval per direction. Defaults to a single solution per solution interval per direction. Entries in provided list should be integer divisors of solint. If not all directions are specified, missing directions are solved with one solution per solution interval. Direction-dependent intervals are only supported when solveralgorithm is set to directioniterative.

solveralgorithm

type:

string

doc:

Select the algorithm used for the solving. Options are directionsolve, directioniterative, lbfgs, and hybrid. For 50 directions, the direction-iterating algorithm can be an order of magnitude faster than the direction-solving algorithm. The downside might be that it is less stable and/or less accurate, which is still to be tested. All algorithms support all solver modes (scalar, diagonal, full Jones, etc). The use of direction-dependent solution intervals (see solutions_per_direction) is only supported for the directioniterative method. The hybrid algorithm is the best of both. It runs 1/6th of the maximum iterations with the direction-solving algorithm, and then switches to the direction-iterating.

default:

directionsolve

solverlbfgs.dof

type:

double

default:

200.0

doc:

The degrees of freedom of the robust noise model.

solverlbfgs.history

type:

int

doc:

LBFGS solver history size. This is the memory (as a multiple of the number of parameters) used to store information pertaining to previous iterations.

default:

10

solverlbfgs.iter

type:

int

default:

4

doc:

Iterations per each minibatch of data. This is the inner iterations and maxiter is the outer iterations. Therefore, the total number iterations is the product of these two values.

solverlbfgs.max_solution

type:

double

default:

0.0

doc:

Restrict the solutions (real, imaginary parts) to fit within the range solverlbfgs.min_solution to solverlbfgs.max_solution.. A valid range is needed to enable this, if both lower and upper limits are zero, not enabled.

solverlbfgs.min_solution

type:

double

default:

0.0

doc:

Restrict the solutions (real, imaginary parts) to fit within the range solverlbfgs.min_solution to solverlbfgs.max_solution.. A valid range is needed to enable this, if both lower and upper limits are zero, not enabled.

solverlbfgs.minibatches

type:

int

default:

1

doc:

Divide the data into minibatches equal to this number at each iteration. If this value is greater than one, less data is used at each iteration (without losing the final solution accuracy) making the iterations faster.

sourcedb

type:

string

doc:

Sourcedb (created with makesourcedb) with the sky model to calibrate on. When the path ends with .skymodel or .txt DP3 expects a skymodel file as used by makesourcedb. This makes it possible to directly use a skymodel file without using makesourcedb to convert the file.

statfilename

type:

string

doc:

File to write the step-sizes to. Form of the file is: “<iterationnr> <normalized-stepsize> <unnormalized-stepsize>”, and all solution intervals are concatenated. File is not written when this parameter is empty.

stepsigma

type:

double

default:

0.1

doc:

Threshold for stalling detection, stop iterating when the running mean of the step sizes is less than stepsigma times their running standard deviation, i.e., the step size is just noise.

stepsize

type:

double

default:

0.2

doc:

stepsize between iterations.

storebuffer

type:

bool

doc:

Setting this to true will store the solution of DDECal into the buffer, allowing the usage of this solution in a later step. For example, a pipeline with DDECal -> OneApplyCal would be able to apply solutions to the data without requiring an intermediate format to be stored to disk. Note that it currently only works for single-direction solutions.

default:

False

subtract

type:

bool

default:

False

doc:

Subtracts the corrected model from the data. NOTE This may not work when you apply a uv-cut.

tolerance

type:

double

default:

1e-5

doc:

Controls the accuracy to be reached: when the normalized solutions move less than this value, the solutions are considered to be converged and the algorithm finishes. Lower values will cause more iterations to be performed.

usebeammodel

type:

bool

default:

False

doc:

use the beam model. All beam-related options of the Predict step are also valid.

usegpu

type:

bool

default:

False

doc:

Use GPU solver. This is an experimental feature only available for the iterative diagonal solver and requires DP3 to be built with BUILD_WITH_CUDA=1.

uvlambdamin

type:

double

default:

0

doc:

Ignore baselines / channels with UV < uvlambdamin wavelengths. Note: also all other variants of uv flagging described in UVWFlagger (uvmmin, uvmrange, uvlambdarange, etc) are supported.