User Guide¶

vmt-report provides two handlers for the arbiter Python package, enabling Turbonomic data to be easily utilized in custom data workflows, reports, and data handling tasks

arbiter¶

vmt-report is an extension to arbiter, providing interfaces to Turbonomic. A basic understanding of arbiter is necessary to be successful with vmt-report, though one does not need to have mastered the dependency by any means.

The heart of arbiter is the arbiter.Process class, which consumes a configuration file defining handlers and directives. Basically a config telling the processor what data to input, and what to output, along with some possible notifications to send. A minimum working example for converting a CSV file to a JSON file may look like this:

example.py¶

import arbiter

if __name__ == '__main__':
  report = arbiter.Process('report.conf')
  report.run()

report.conf¶

{
  "sources": [
    {
      "handler": "CSV",
      "resource": "file:./input.csv"
    }
  ],
  "outputs": [
    {
      "handler": "JSON",
      "resource": "file:./output.json",
      "keepfile": true
    }
  ],
  "logging": {
    "mode": "DEBUG"
  }
}

Handlers are free to define their own configuration parameters, and the documentation for the specific handler being used should be consulted for such information. The two handlers vmt-report defines, GroupedData and Connection, are documented.

Cluster & Group Based Reports¶

The GroupedData provides basic data processing capabilities based on groups (including clusters) within Turbonomic. The handler utilizes a list of groups, and a list of field definitions. Optionally a third list of sorting parameters may be included. The resultant data is a mapped dataset represented by an collections.OrderedDict.

Example cluster based report config¶

{
  "sources": [
    {
      "handler": "VmtGroupedData",
      "resource": "https://hostname",
      "authentication": {
        "type": "env",
        "username": "VMT_USERNAME",
        "password": "VMT_PASSWORD"
      },
      "options": {
        "groups": {
          "type": "cluster"
        },
        "fields": [
          {"id": "target", "type": "property", "value": "source:displayName", "label": "vCenter"},
          {"id": "cluster", "type": "property", "value": "displayName", "label": "Cluster Name"},
          {"id": "numpm", "type": "commodity", "value": "numHosts:value", "label": "Num Hosts"},
          {"id": "numvm", "type": "commodity", "value": "numVMs:value", "label": "Num VMs"},
          {"id": "cpuutil", "type": "computed", "value": "round($cpuused/$cpucap * 100, 2)", "label": "CPU Util"},
          {"id": "memutil", "type": "computed", "value": "round($memused/$memcap * 100, 2)", "label": "Mem Util"},
          {"id": "cpucap", "type": "commodity", "value": "CPU:capacity:total"},
          {"id": "cpuused", "type": "commodity", "value": "CPU:values:total"},
          {"id": "memcap", "type": "commodity", "value": "Mem:capacity:total"},
          {"id": "memused", "type": "commodity", "value": "Mem:values:total"}
        ],
        "sortby": ["target", "cluster"]
      }
    }
  ],
  "outputs": [
    {
      "handler": "CSV",
      "resource": "file:/tmp/{timestamp}-clusters.csv",
      "options": {
        "keepfile": true
      }
    }
  ],
  "logging": {
    "mode": "DEBUG"
  }
}

Groups¶

Groups define the scope within Turbonomic to limit data gathering to. Two types of groups are supported: cluster and group. Clusters are distinguished because Turbonomic treats these separate from all other entities internally. When using the cluster type, the default behavior is to use all clusters unless a names filter is provided; when using the type group a names filter must be provided. Groups are defined in the groups sub-block of the configuration.

Parameters:

type: Specifies if the intended scope objects are Turbo groups or clusters.
names: List of names / uuids to limit the scope to.
stop_on_error: If True, non-existent or groups that produce errors will be raised as an exception. By default errors are ignored. Default: False

Fields¶

Fields provide vmt-report with detailed information regarding which data to gather, and how to make use of it in the final output. All field definitions must contain a unique id, a valid type, and a valid value. The label parameter controls if the field is included in final output, thus you may declare fields as required for use in calculations, that are then excluded from the final report. This is a critical feature for computed fields, as they themselves cannot directly reference commodities or properties on entities.

Computed fields permit custom calculations upon report data, storing the data into the field for use by other computed fields, or for display in the report. The value may contain most any valid python expression that evaluates to a value; and explicitly excludes the use of assignment operators. Functions may also be called provided they do not require external input and return a value. Other fields may be referenced by prefixing the field id with a $. All other field data is populated prior to the processing of computed fields, thus field order is not typically an issue for computed fields in general. However, if a computed field references another computed field, the referenced field must appear earlier in the list than the field referencing it so that the referenced field is computed before its value is required. Computed fields are evaluated in sequential order starting at the top of the list.

Warning

vmt-report does not check for field order dependency or circular dependency errors.

Fields are defined in the fields sub-block of the configuration as a list of dictionaries with the following parameters.

Parameters:

id: Unique, user supplied internal identifier for the id. This is used when referencing other fields in calculated fields, and the field must contain only alphanumeric characters.
type: One of the the defined FieldTypes, as a string literal: commodity, computed, property, or string.
value: The value. This differs by type, and is discussed in detail below.
label: Optional. The column header to use in the final output.

Values by type:

commodity - colon separated path to the statistical value, prefixed by the commodity name.
computed - the expression to evaluate
property - colon separated path to the entity property
string - string literal value which is not interpreted further. An empty string is valid.

Sorting¶

Final output may be sorted using the sortby option, by supplying a list of field ids. Multi-level sorting is supported, and proceeds left-to-right in the field list. Fields are sorted in ascending order by default, and any field may independently be reversed to descending order by prefixing the id with a - dash.

Advanced Use Cases¶

For use cases that go beyond group level data, or that require special processing of the data gathering, vmt-report supports full customization of both the handler and worker components.

Connection Handler¶

The base handler vmt-report provides is the Connection. The handler is responsible for establishing a connection to Turbonomic using the vmt-connect module, and returns a vmtconnect.Connection object. Coupled with a customer worker process, detailed below, permits complete access to the Turbonomic API. The handler requires only an authentication directive and supports any that return either a username and password, or an auth string, including the basic, auth, env, and vmt-report supplied credstore.

Credstore Authentication¶

vmt-report includes an additional authentication handler for accepting vmt-connect style credentials.

Connection handler & credstore authentication¶

{
  "sources": [
    {
      "handler": "vmtconnect",
      "resource": "https://localhost",
      "authentication": {
        "type": "credstore",
        "credential": "./turbo.cred",
        "keyfile": "./turbo.keyfile"
      }
    }
  ],
  "logging": {
    "mode": "DEBUG"
  }
}

Custom Workers¶

A custom worker definition is required to make use of a Connection handler. arbiter natively supports custom worker definitions by passing a function reference to the arbiter.Process constructor. All workers are run as separate processes, and so the function must return data in a mergable format. If the data requires special attention to merge, the user is responsible for providing the proper arbiter.Process.merge_results() overloaded method.

Custom worker example.¶

# Dump selected fields from all market actions
import arbiter
import vmtreport
import umsg

# source worker sub-process must accept 3 parameters
def actions(source, config, logger):
    umsg.log(f"Retrieving data from {config['resource']}", logger=logger)

    # source.connect() gives us the vmt-connect.Connection instance returned by
    # vmt-report; here we use the get_actions() method inline.
    # Another use would be the standard vmt-connect idiom:
    #   vmt = source.connection()
    #   res = vmt.get_action()
    res = source.connect().get_actions()

    fields = ['createTime', 'actionType', 'details']
    return [{x: res[x]} for x in res if x in fields]

if __name__ == '__main__':
  report = arbiter.Process('report.config', actions)
  report.run()