ESO Recipe Execution Tool  3.13
Functions
Python Interfacing Functions

Functions

cpl_error_code er_python_load_modules (er_stringarray_t *paths)
 Loads relevant information for Python modules that contain plugins. More...
 
cpl_error_code er_python_select_module (const char *module_name)
 Selects a specific Python module to use for execution. More...
 
int er_python_get_plugin_list (cpl_pluginlist *list)
 Fills a list with all available plugins for a selected Python module. More...
 
void er_python_cleanup (void)
 Resets loaded Python module information. More...
 

Detailed Description

This module provides functions needed to interface with a Python interpreter running as an external process that will execute a Python based plugin. The CPL plugin API is preserved as much as possible, both on the EsoRex side and within the Python plugin code. Thus, the EsoRex core code will invoke the Python based plugin through the usual calls to the cpl_plugin class (more specifically, derived classes such as cpl_recipe). This module provides the mechanism to load Python modules, identify the Python based plugins, select the desired module, and then actually perform the necessary communication with the Python interpreter to execute the plugin.

The exported functions of this interface deal with the differences in loading Python module plugins versus C/C++ based shared library plugins. Only the loading and unloading needs special treatment from EsoRex, since the actual invocation of the plugin is through the usual cpl_plugin API. Thus, starting/stopping the Python interpreter and the communication protocol is all hidden behind the interface from EsoRex's point of view inside the private functions of this module.

Loading of a Python module involves starting the Python interpreter, loading the named Python module file and finding all Python classes that derive from a class called CplPlugin. This is accomplished with the function er_python_load_modules(). All such classes are registered and can be subsequently selected with er_python_select_module(). Selecting the desired module is necessary, so that er_python_get_plugin_list() knows for which Python module to return the list of available plugins. The function er_python_get_plugin_list() can be considered as the equivalent of the cpl_plugin_get_info() entry point function that must be implemented by a C/C++ based shared library plugin. Once er_python_get_plugin_list() has successfully returned a list of CPL plugin objects, there is no difference in the invocation procedure to execute the plugin between a Python module or shared library based plugin. Once the invocation of the plugin is complete, one must cleanup by calling the er_python_cleanup() function. A simple code example for the call sequence to invoke a Python based plugin will look as follows (error handling is ignored for simplicity):

cpl_pluginlist * list = cpl_pluginlist_new();
cpl_plugin_func plugin_init, plugin_exec, plugin_deinit;
// Python specific module initialisation.
er_stringarray_append(modules, "./recipe.py");
er_python_select_module("./recipe.py");
// Common plugin invocation sequence.
cpl_recipe * recipe = (cpl_recipe *) cpl_calloc(1, sizeof(cpl_recipe));
recipe->frames = cpl_frameset_new();
plugin_init = cpl_plugin_get_init(&recipe->interface);
if (plugin_init != NULL) {
plugin_init(&recipe->interface);
}
plugin_exec = cpl_plugin_get_exec(&recipe->interface);
if (plugin_exec != NULL) {
plugin_exec(&recipe->interface);
}
plugin_deinit = cpl_plugin_get_deinit(&recipe->interface);
if (plugin_deinit != NULL) {
plugin_deinit(&recipe->interface);
}
cpl_frameset_delete(recipe->frames);
cpl_plugin_delete(&recipe->interface);
// Python specific cleanup.

Under the hood, communication with the Python interpreter is handled by the execute function, which is registered with the cpl_plugin objects returned by er_python_get_plugin_list(). The communication protocol uses JSON as the exchange format that is sent over Unix pipes. The execute function is a stub that will implement the following steps:

  1. Start an external Python interpreter process with the python shell command and give it the initial Python stub code to execute.
  2. Encode the input cpl_plugin structure as JSON format.
  3. Send the JSON text to the Python interpreter over a Unix pipe.
  4. Close the input pipe to indicate end of input.
  5. Read the output from a second pipe, to which the Python process will write the results in JSON format.
  6. Wait for the output pipe to be closed, which indicates end of output.
  7. Decode the output JSON text back into a cpl_plugin structure and update the cpl_plugin argument of the execute function.
  8. Wait for the Python interpreter to terminate (join with the process).
  9. Return the plugin's return code received from the Python plugin.

The other end of the protocol is implemented in the initial Python stub code passed to the Python interpreter to execute. The steps it will perform are:

  1. Import the desired Python module containing the plugin.
  2. Instantiate the selected plugin object (a class that derives from CplPlugin).
  3. Read the JSON input from the Unix pipe.
  4. Wait for the input pipe to be closed, which indicates the end of input.
  5. Decode the JSON into a Python object.
  6. Call the plugin object's execute method, passing the decoded Python object as the method's argument.
  7. Once the Python execute method completes, the updated argument and return value are encoded as JSON again.
  8. The encoded JSON result is send over the output pipe.
  9. The output pipe is closed to indicate the end of output.
  10. The Python interpreter ends execution and terminates.

Note that the JSON is not send over the stdin and stdout pipes, but rather two new dedicated pipes are created to handle the interprocess communication. This allows the stdin and stdout pipes to behave as one would normally expect. For example, they will be attached to the same terminal or file as the parent process (i.e. EsoRex) was attached to. This means that print statements in the Python code will write the output to the same location as one would expect if the Python code was executed manually in the Python interpreter. Separation of the EsoRex to Python communication protocol from stdin/stdout avoids unexpected interference or having stdin/stdout redirected in an unexpected manner.

Developers of Python based plugins must implement a plugin class that has the following requirements:

The following is a simple example of a Python class that implements the required interface:

1 class CplPlugin(object):
2  name = "test"
3  version = 102030
4  synopsis = "short description"
5  description = "long description"
6  author = "Some Author"
7  email = "someone@example.org"
8  copyright = "copyright 2017"
9  parameters = [
10  {
11  'class': 'value',
12  'name': 'test.par1',
13  'description': 'parameter 1',
14  'context': 'test',
15  'default': 3
16  }
17  ]
18 
19  def execute(self, plugin):
20  # Fetch the parameter value.
21  parameters = plugin['parameters']
22  par1 = None
23  for par in parameters:
24  if par['name'] == 'par1':
25  par1 = par['value']
26 
27  # Fetch the first frame's file name.
28  frames = plugin['frames']
29  filename = frames[0]['filename']
30 
31  try:
32  # ... process frames ...
33  except:
34  # Set the error message and indicate an error.
35  self.error_message = "Recipe failed"
36  return 1
37 
38  # Append new frames as output.
39  frames.append(
40  {
41  'filename': 'output.fits',
42  'tag': 'PROD',
43  'type': 2,
44  'group': 3,
45  'level': 3,
46  }
47  )
48 
49  # Indicate success.
50  return 0

Function Documentation

void er_python_cleanup ( void  )

Resets loaded Python module information.

This function should be called when one no longer needs to deal with any Python based plugins.

int er_python_get_plugin_list ( cpl_pluginlist *  list)

Fills a list with all available plugins for a selected Python module.

This function is the equivalent of the cpl_plugin_get_info() function found in a compiled plugin library. It will fill a list of CPL plugin objects for all plugins found in the currently selected Python module. er_python_select_module() must be called before this function is invoked.

Parameters
listThe plugin list that will be filled with CPL plugin objects.
Returns
0 is returned on success and an appropriate CPL error code otherwise.
cpl_error_code er_python_load_modules ( er_stringarray_t paths)

Loads relevant information for Python modules that contain plugins.

This function will go through a list of module paths that point to Python code files and attempt to load these. If the module can be loaded successfully and it contains any classes that derive from a base class called CplPlugin (including CplPlugin itself) then these are recorded, such that er_python_select_module() can then be used to select a particular Python module to run. Only classes directly within the Python module's namespace are considered, i.e. secondary imported modules will not have their classes added unless they are imported into the namespace with a "from ... import ..." Python statement.

Parameters
[in,out]pathsThe list of module paths that need to be checked if they can be loaded. Any paths that do not contain loadable plugins will be removed from the list by this function.
Returns
CPL_ERROR_NONE on success and an appropriate error code if a severe error occurred.
Note
This function will still succeed if a Python module could not be loaded. All such modules are simply ignored. However, the user may get error messages printed by the Python process on stderr.
cpl_error_code er_python_select_module ( const char *  module_name)

Selects a specific Python module to use for execution.

This function is used to select the Python module that will be used when the er_python_get_plugin_list() function is invoked and attempts to load all available plugin classes.

Parameters
module_nameThe full path name of the python module to use.
Returns
CPL_ERROR_NONE on success or an appropriate error code if a severe error occurs.