gwcelery.tasks.orchestrator module

This module implements the alert orchestrator, which responsible for the vetting and annotation workflow to produce preliminary, initial, and update alerts for gravitational-wave event candidates.

The orchestrator consists of two IGWN Alert message handlers:

  • handle_superevent() is called for each superevent. It waits for a short duration of orchestrator_timeout seconds for the selection of the superevent by the superevent manager to stabilize, then performs data quality checks. If the data quality checks pass, then it calls earlywarning_preliminary_alert() to copy annotations from the preferred event and send the preliminary notice.

  • handle_cbc_event() is called for each CBC event. It performs some CBC-specific annotations that depend closely on the CBC matched-filter parameters estimates and that might influence selection of the preferred event: rapid sky localization with BAYESTAR and rapid source classification.

    Note that there is no equivalent of this task for burst events because both burst searches (cWB, LIB) have integrated source localization and have no other annotations.

Preliminary Alerts

The flow chart below illustrates the operation of these two tasks.

digraph earlywarning_preliminary_alert { compound = true nodesep = 0.1 ranksep = 0.1 node [ fillcolor = white shape = box style = filled target = "_top" ] graph [ labeljust = "left" style = filled target = "_top" ] superevent [ label = "IGWN Alert message\nfor new\nsuperevent" style = rounded ] subgraph cluster_handle_superevent { href = "../gwcelery.tasks.orchestrator.html#gwcelery.tasks.orchestrator.handle_superevent" label = <<B><FONT face="monospace">handle_superevent</FONT></B>> orchestrator_timeout [ href = "../gwcelery.conf.html#gwcelery.conf.orchestrator_timeout" label = <Wait<BR/><B><FONT face="monospace">orchestrator_timeout</FONT></B><BR/>seconds> ] get_preferred_event [ label = "Get preferred event" ] check_vectors [ href = "../gwcelery.tasks.detchar.html#gwcelery.tasks.detchar.check_vectors" label = "Check state vectors" ] offline_event [ label = "Offline event\n?" shape = diamond ] far_threshold [ label = "N_trials * FAR \n < threshold?" shape = diamond ] dqv [ label = "Vetoed by\nstate vectors?" shape = diamond ] subgraph cluster_preliminary_alert { href = "../gwcelery.tasks.orchestrator.html#gwcelery.tasks.orchestrator.earlywarning_preliminary_alert" label = <<B><FONT face="monospace">earlywarning_preliminary_alert</FONT></B>> copy_from_preferred_event [ label = "Copy classification\n(if CBC) and\nsky map from\npreferred event" ] annotate_skymaps [ label = "Make sky\nmap plots" ] send_notice [ label = "Send preliminary\nnotice" ] circular [ label = "Create GCN\ncircular draft" shape = diamond ] } } superevent -> orchestrator_timeout [lhead = cluster_handle_superevent] orchestrator_timeout -> get_preferred_event -> check_vectors -> offline_event offline_event -> far_threshold [label = No, lhead = prelim_gcn_checks] far_threshold -> dqv [label = Yes, lhead = prelim_gcn_checks] dqv -> copy_from_preferred_event [label = No, lhead = cluster_preliminary_alert] copy_from_preferred_event -> annotate_skymaps -> send_notice -> circular cbc_event [ label = "IGWN Alert for\nfile added\nto CBC event" style = rounded ] subgraph cluster_handle_cbc_event { href = "../gwcelery.tasks.orchestrator.html#gwcelery.tasks.orchestrator.handle_cbc_event" label = <<B><FONT face="monospace">handle_cbc_event</FONT></B>> { rank = same which_file [ label = "What is the\nfilename?" shape = diamond ] download_psd [ label = <Download<BR/><FONT face="monospace">psd.xml.gz</FONT>> ] } download_ranking_data [ label = <Download<BR/><FONT face="monospace">ranking_data<BR/>.xml.gz</FONT>> ] download_coinc_psd [ label = <Download<BR/><FONT face="monospace">coinc.xml</FONT>> ] download_coinc_ranking_data [ label = <Download<BR/><FONT face="monospace">coinc.xml</FONT>> ] bayestar [ href = "../gwcelery.tasks.bayestar.html#gwcelery.tasks.bayestar.localize" label = <Create<BR/><FONT face="monospace">bayestar<BR/>.fits.gz</FONT>> ] em_bright [ href = "../gwcelery.tasks.em_bright.html#gwcelery.tasks.em_bright.classifier" label = <Create<BR/><FONT face="monospace">source_<BR/>classi<BR/>fication<BR/>.json</FONT>> ] p_astro [ href = "../gwcelery.tasks.p_astro_gstlal.html#gwcelery.tasks.p_astro_gstlal.compute_p_astro" label = <Create<BR/><FONT face="monospace">p_astro<BR/>_gstlal.json</FONT>> ] } cbc_event -> which_file [lhead = cluster_handle_cbc_event] which_file -> download_psd [ fontname = monospace label = "psd\n.xml\n.gz" ] which_file -> download_ranking_data [ fontname = monospace label = "ranking_data.xml.gz" ] download_psd -> download_coinc_psd -> bayestar -> em_bright download_ranking_data -> download_coinc_ranking_data -> p_astro }

Initial and Update Alerts

The initial_alert() and update_alert() tasks create Initial and Update alerts respectively. At the moment, there is no handler or user interface to trigger these tasks, and they must be invoked manually (see Command-Line Tools). A flow chart for the initial alerts is shown below; the flow chart for update alerts is the same.

digraph initial_alert { compound = true nodesep = 0.1 ranksep = 0.1 node [ fillcolor = white shape = box style = filled target = "_top" ] graph [ labeljust = "left" style = filled target = "_top" ] subgraph cluster_initial_alert { href = "../gwcelery.tasks.orchestrator.html#gwcelery.tasks.orchestrator.initial_alert" label = <<B><FONT face="monospace">initial_alert</FONT></B>> annotate_skymaps [ label = "If sky map provided,\nthen make sky map plots" ] send_notice [ label = "Send\nnotice" ] } annotate_skymaps -> send_notice }

Retraction Alerts

Likewise, the retraction_alert() task creates Retraction alerts, and at the moment must be invoked manually. A flow chart is shown below.

digraph retraction_alert { compound = true nodesep = 0.1 ranksep = 0.1 node [ fillcolor = white shape = box style = filled target = "_top" ] graph [ labeljust = "left" style = filled target = "_top" ] subgraph cluster_initial_alert { href = "../gwcelery.tasks.orchestrator.html#gwcelery.tasks.orchestrator.retraction_alert" label = <<B><FONT face="monospace">retraction_alert</FONT></B>> send_notice [ label = "Send\nnotice" ] } }

Tasks

Tasks that comprise the alert orchestrator.

The orchestrator is responsible for the vetting and annotation workflow to produce preliminary, initial, and update alerts for gravitational-wave event candidates.

(task)gwcelery.tasks.orchestrator.handle_superevent(alert)[source]

Schedule annotations for new superevents.

After waiting for a time specified by the orchestrator_timeout configuration variable for the choice of preferred event to settle down, this task performs data quality checks with gwcelery.tasks.detchar.check_vectors() and calls earlywarning_preliminary_alert() to send a preliminary notice.

(task)gwcelery.tasks.orchestrator.handle_cbc_event(alert)[source]

Perform annotations for CBC events that depend on pipeline-specific matched-filter parameter estimates.

Notes

This IGWN alert message handler is triggered by a new upload or by updates that include the file pipeline.p_astro.json. If also generates pipeline.p_astro.json information for pipelines that do not provide such information.

The table below lists which files are created as a result of a new upload, and which tasks generate them.

File

Task

bayestar.multiorder.fits

gwcelery.tasks.bayestar.localize()

em_bright.json

gwcelery.tasks.em_bright.source_properties()

pipeline.p_astro.json

gwcelery.tasks.p_astro.compute_p_astro()

(task)gwcelery.tasks.orchestrator.handle_burst_event(alert)[source]

Perform annotations for burst events that depend on pipeline-specific

(task)gwcelery.tasks.orchestrator.handle_posterior_samples(alert)[source]

Generate multi-resolution and flat-resolution FITS files and skymaps from an uploaded HDF5 file containing posterior samples.

(task)gwcelery.tasks.orchestrator._create_mattermost_channel(superevent_id)[source]

Creates a mattermost channel when ADVREQ label is applied and posts a cooresponding gracedb link of that event in the channel

Channel name : O4 RRT {superevent_id}

Parameters:

superevent_id: str

The superevent id

(task)gwcelery.tasks.orchestrator._set_pipeline_preferred_events(pipeline_event, superevent_id)[source]

Return group for setting pipeline preferred event using gracedb.add_pipeline_preferred_event().

Parameters:
  • pipeline_event (dict) – {pipeline: event_dict} key value pairs, returned by superevents.select_pipeline_preferred_event().

  • superevent_id (str) – The superevent id

(task)gwcelery.tasks.orchestrator._update_if_dqok(event, superevent_id)[source]

Update preferred_event of superevent_id to event_id if DQOK label has been applied.

(task)gwcelery.tasks.orchestrator._get_preferred_event(superevent_id)[source]

Determine preferred event for a superevent by querying GraceDB.

This works just like gwcelery.tasks.gracedb.get_superevent(), except that it returns only the preferred event, and not the entire GraceDB JSON response.

(task)gwcelery.tasks.orchestrator._create_voevent(classification, *args, **kwargs)[source]

Create a VOEvent record from an EM bright JSON file.

Parameters:
Returns:

The filename of the newly created VOEvent.

Return type:

str

(task)gwcelery.tasks.orchestrator._create_label_and_return_filename(filename, label, graceid)[source]

Proxy that evaluates object once.

Proxy will evaluate the object each time, while the promise will only evaluate it once.

(task)gwcelery.tasks.orchestrator._leave_log_message_and_return_event_dict(event, superevent_id, message, **kwargs)[source]

Wrapper around gracedb.upload() that returns the event dictionary.

(task)gwcelery.tasks.orchestrator._update_superevent_and_return_event_dict(event, superevent_id)[source]

Wrapper around gracedb.update_superevent() that returns the event dictionary.

(task)gwcelery.tasks.orchestrator._proceed_if_not_blocked_by(files, superevent_id, block_by)[source]

Return files in case the superevent does not have labels block_by

Parameters:
  • files (tuple) – List of files

  • superevent_id (str) – The superevent id corresponding to files

  • block_by (set) – Set of blocking labels. E.g. {‘ADVOK’, ‘ADVNO’}

(task)gwcelery.tasks.orchestrator._revise_and_send_second_less_significant_alert(alert, query, superevent_id)[source]

Proxy that evaluates object once.

Proxy will evaluate the object each time, while the promise will only evaluate it once.

(task)gwcelery.tasks.orchestrator._annotate_fits_and_return_input(input_list, superevent_id)[source]

Unpack the output of the skymap, embright, p-astro download group in the beginning of the earlywarning_preliminary_alert() canvas and call annotate_fits().

Parameters:
  • input_list (list) – The output of the group that downloads the skymap, embright, and p-astro files. This list is in the form [skymap, skymap_filename], [em_bright, em_bright_filename], [p_astro_dict, p_astro_filename], though the em-bright and p-astro lists can be populated by Nones

  • superevent_id (str) – A list of the sky map, em_bright, and p_astro filenames.

(task)gwcelery.tasks.orchestrator._unpack_args_and_send_earlywarning_preliminary_alert(input_list, alert, alert_type)[source]

Unpack the output of the skymap, embright, p-astro download group in the beginning of the earlywarning_preliminary_alert() canvas and call gwcelery.tasks.orchestrator.earlywarning_preliminary_initial_update_alert().

Parameters:
  • input_list (list) – The output of the group that downloads the skymap, embright, and p-astro files. This list is in the form [skymap, skymap_filename], [em_bright, em_bright_filename], [p_astro_dict, p_astro_filename], though the em-bright and p-astro lists can be populated by Nones

  • alert (dict) – IGWN-Alert dictionary

  • alert_type (str) – alert_type passed to earlywarning_preliminary_initial_update_alert()

(task)gwcelery.tasks.orchestrator.earlywarning_preliminary_alert(event, alert, alert_type='preliminary', initiate_voevent=True)[source]

Produce a preliminary alert by copying any sky maps.

This consists of the following steps:

  1. Copy any sky maps and source classification from the preferred event to the superevent.

  2. Create standard annotations for sky maps including all-sky plots by calling gwcelery.tasks.skymaps.annotate_fits().

  3. Create a preliminary VOEvent.

  4. Send the VOEvent to GCN and notices to SCiMMA and GCN.

  5. Apply the GCN_PRELIM_SENT or LOW_SIGNIF_PRELIM_SENT depending on the significant or less-significant alert respectively.

  6. Create and upload a GCN Circular draft.

(task)gwcelery.tasks.orchestrator._get_cbc_lowest_far(superevent_id)[source]

Obtain the lowest FAR of the CBC events in the target superevent.

(task)gwcelery.tasks.orchestrator.parameter_estimation(far_event, superevent_id, pe_pipeline)[source]

Parameter Estimation with Bilby and RapidPE-RIFT. Parameter estimation runs are triggered for CBC triggers which pass the FAR threshold and are not mock uploads. For those which do not pass these criteria, this task uploads messages explaining why parameter estimation is not started.

(task)gwcelery.tasks.orchestrator.earlywarning_preliminary_initial_update_alert(filenames, superevent, alert_type, filecontents=None)[source]

Create a canvas that sends an earlywarning, preliminary, initial, or update notice.

Parameters:
  • filenames (tuple) – A list of the sky map, em_bright, and p_astro filenames.

  • superevent (dict) – The superevent dictionary, typically obtained from an IGWN Alert or from querying GraceDB.

  • alert_type ({'less-significant', 'earlywarning', 'preliminary', 'initial', 'update'} # noqa: E501) – The alert type.

Notes

Tasks that call this function should be decorated with gwcelery.tasks.gracedb.task rather than gwcelery.app.task so that a synchronous call to gwcelery.tasks.gracedb.get_log() is retried in the event of GraceDB API failures. If EM_COINC is in labels will create a RAVEN circular.

(task)gwcelery.tasks.orchestrator._update_rapidpe_pastro(input_list, em_bright=None, pipeline_pastro=None)[source]

If p_terr from rapidpe is different from the p_terr from the most recent preferred event, replaces rapidpe’s p_terr with pipeline p_terr. Returns a tuple of em_bright, rapidpe pastro and a boolean(rapidpe_pastro_updated) indicating if rapidpe pastro has been updated. If p_terr in rapidpe has been updated, the return list contains the updated pastro and the rapidpe_pastro_updated is True. Else, the return list contains the rapidpe pastro from the input_list and rapidpe_pastro_updated is False.

(task)gwcelery.tasks.orchestrator._update_rapidpe_pastro_shouldnt_run()[source]

Proxy that evaluates object once.

Proxy will evaluate the object each time, while the promise will only evaluate it once.

(task)gwcelery.tasks.orchestrator._upload_rapidpe_pastro_json(input_list, superevent_id, rapidpe_pastro_filename)[source]

Add public tag to RapidPE_RIFT.p_astro.json if p_terr from the preferred event is same as the p_terr in RapidPE_RIFT.p_astro.json. Else, uploads an updated version of RapidPE_RIFT.p_astro.json with file content from the task update_rapidpe_pastro.

(task)gwcelery.tasks.orchestrator._check_pastro_and_send_alert(input_classification, skymap, em_bright, superevent, alert_type, raven_coinc=False)[source]

Wrapper for send() meant to take a potentially new p-astro as input from the preceding task.

(task)gwcelery.tasks.orchestrator.initial_alert(filenames, alert)[source]

Produce an initial alert.

This does nothing more than call earlywarning_preliminary_initial_update_alert() with alert_type='initial'.

Parameters:
  • filenames (tuple) – A list of the sky map, em_bright, and p_astro filenames.

  • alert (dict) – IGWN-Alert dictionary

Notes

This function is decorated with gwcelery.tasks.gracedb.task rather than gwcelery.app.task so that a synchronous call to gwcelery.tasks.gracedb.get_log() is retried in the event of GraceDB API failures.

(task)gwcelery.tasks.orchestrator.update_alert(filenames, superevent_id)[source]

Produce an update alert.

This does nothing more than call earlywarning_preliminary_initial_update_alert() with alert_type='update'.

Parameters:
  • filenames (tuple) – A list of the sky map, em_bright, and p_astro filenames.

  • superevent_id (str) – The superevent ID.

Notes

This function is decorated with gwcelery.tasks.gracedb.task rather than gwcelery.app.task so that a synchronous call to gwcelery.tasks.gracedb.get_log() is retried in the event of GraceDB API failures.

(task)gwcelery.tasks.orchestrator.retraction_alert(alert)[source]

Produce a retraction alert.