'Pivot and Aggregate' Dataflow link

Important: The 'Pivot and Aggregrate' link type is only available in version 16.5 Update 7 and later.

For the 'Default' link creation, see Create a Dataflow link.

The key differences between the link types are explained in the following table. The security permissions for 'Pivot and Aggregate' have a significantly more complex configuration.

  Default link type Pivot and Aggregate link type
Hierarchy structure Dataflow and Economics hierarchy structures are the same. One-to-one matching of nodes and documents. Dataflow documents will be re-grouped to a different hierarchy structure based on selected Text variables. Aggregation of several Dataflow documents into single Economics projects is based on the same grouping criteria.
Documents transferred All Dataflow documents based on the Entity Type, Path, and Filter settings. Dataflow documents based on the Entity Type and Path settings, and further restricted by the document access permissions of the user account that is refreshing the link (i.e., only those documents that the user has access to in Dataflow).
Permissions in Economics Permissions attached to the Dataflow documents are synchronized to the Economics project permissions. It is not possible to modify the permissions within Economics.

Permissions for the Economics folders and projects must be configured in Economics. Upon creation of the link, only the workgroup specified by the Link Manager Workgroup setting will be granted Full Access to all of the folders and projects created by the link. It will be the responsibility of any member of that workgroup to revise the folder and project permissions as needed. During a refresh, permissions on pre-existing entities will not be modified but any new entity created as a result of the refresh operation will have Full Access granted to the Link Manager Workgroup and then the permissions must be modified as needed.

For version 16.5 Update 31 and later: Additional permissions are applied automatically for newly-created projects and folders after link creation or link refresh, based on inherited permissions from the parent node (i.e., the same behavior as when nodes are created manually in a hierarchy).

To link a project to Dataflow using 'Pivot and Aggregate' use the following steps:

  1. In the Hierarchy viewer, click the folder to be linked.
  2. Note: The folder must contain a regime, and it must not contain any existing sub-folders or projects.

  3. Click the right arrow > and an action menu window will appear:
  4. link to Dataflow

  5. Click Link Dataflow Version to open the Link Dataflow Version dialog:
  6. Link Dataflow Version

    Specify the settings you want to use, as described in the following table.

    Setting Description
    Version Specifies the Dataflow version to link to.
    Revision Tag Specify a tag to determine which document revision will be loaded. Check Show inactive tags to make inactive tags available for selection. Select <Latest> to use the latest document revision.
    Link Type Set this to Pivot and Aggregate. This link type can only support Manual refresh (hence the selector is disabled).
    Point Forward Date Data in linked Dataflow documents can be truncated at a specified Point Forward Date to allow for point forward analysis. Check the Enable Point Forward Date box and enter a date. If this option is disabled, the default full cycle analysis will be set up. This option is only available in version 16.5 Update 7 and later.
  7. Click OK to close the dialog.
  8. The Dataflow Integration tab will open in the main workspace panel. Note: the link configuration has some differences for the Default or Pivot and Aggregate link types.
  9. Dataflow Integration

    Projects will be created for the Entity Types that are specified. The Path setting allows you to specify the section of the Dataflow version that you wish to link to. Note that the Entity Types you select must be at least one hierarchy level below the location you specify in Path.

    Note that the variables that are synced to projects in Economics are determined by the Economics Configuration in the Dataflow template that is associated with each entity type. For more information, see Economics configuration. If a working interest variable has been defined as part of the Economics configuration, the working Interest information will be synced from the Dataflow documents to the Economics project. For more information, see Working interest shares.

    For version 16.5 Update 17 and later:

    • The list of entity types displayed in the Entity Types dialog will only contain entities for which the currrent user has been granted 'Read' permission.
    • The Filter option can be used to create projects only for Dataflow documents that meet some specific criteria. When setting up a Filter the list of variable names can be searched by substring matching against text typed into the variable name box.
  10. The Document Group option defines the node structure of the hierarchy in Economics based on a set of scalar text variables in the Dataflow Documents. Drag the available variables onto the right column and arrange them to configure the required Economics hierarchy. Click the ellipsis button to open the dialog:
  11. PivotAndAggregate-DocumentGroup-dialog

  12. The Link Manager Workgroup specifies one workgroup that will have permission to change the configuration of the link or to refresh (synchronize) it. In an active Dataflow link the Last Refreshed by field will show the user account that refreshed the link the last and the Modified field will show the timestamp for the last modification.
  13. The Currency, Unit System, Unit Size and Real/Nominal selectors specify how data from the aggregated Dataflow documents will be displayed in the projects that will be created in the Economics hierarchy.
  14. For version 16.5 Update 25 and later:

    • The Unit System and Unit Size settings have been removed from the link configuration. They are no longer required because the unit transformations between Dataflow and Economics are handled automatically.
    • The optional Inflation Date setting can be used by checking the Enable Inflation Date box and selecting a date. Otherwise, as in previous versions, the project start date will be used at the inflation date.
  15. The next step is to map the Dataflow scenario tags to the project scenarios required in the Economics projects. Add a new row by clicking Click here to add new item (underneath the 'Dataflow Scenario Tag' column heading). The Select Scenario Tag dialog will open.
  16. Select Scenario Tag window

    Select the scenario you want to use and click OK.

  17. A new row will be added. An Economics copy of this scenario will automatically be created and linked to the specified Dataflow scenario tag.
  18. Dataflow Integration page

    You can also customize the Scenario Weighting for the project(s), either by entering a value, or clicking the Expression button to open the Edit Expression window, where you can build an expression to determine the applied weighting. For more information, see Expression Editor.

  19. Once you have added the scenario(s) you want to use, click the Apply Link button in the Dataflow Integration tab on the ribbon to create the link. An amber-coloured banner at the top of the hierarchy view will indicate that the projects are being synced.
  20. Project links being created between Dataflow and Economics

  21. Once successfully linked, a small Dataflow icon will be displayed next to the linked project folder name. The banner at the top of the hierarchy will indicate the link status (the gray color denotes a Manual link).
  22. link banner

  23. If an error occurs during a sync, the banner will turn red with status Sync Error. To open the Sync Log Viewer, click the right arrow > inside the banner to open a link actions menu, and click View Sync Log.

The linked project(s) are now available for use in Economics. A summary of your hierarchy links can be found on the Manage Hierarchies screen, accessed by clicking the Manage Hierarchies button in the Home tab on the ribbon.

If you want to replace the regime of a linked project, select the regime and click the Replace Regime button Replace Regime button in the Home tab on the ribbon.

Note: Only values of unmapped variables can be edited within Economics. To change mapped values, you must edit the project in Dataflow.

Temporary data cleanup

Temporary data sets are created to store transformed and aggregated Dataflow data. These are not accessible to users but they do consume storage in the tenant database. In version 16.5 Update 8 and later an automatic scheduled cleanup process was added, and by default this runs weekly (every Sunday at 00:01; all temporary data which are older than 5 days are deleted). The cleanup job is controlled by the Dataflow application settings 'Aggregation Data Cleanup Schedule' and 'Remove Aggregation Data Older Than' which can be modified by the IPS Administrator. The cleanup job should be set to run more frequently if the temporary data are found to be causing storage issues for the tenant database.

Troubleshooting a Dataflow link

Linked projects can only be created in the Economics hierarchy if the following conditions are met:

  • The corresponding document in Dataflow is one of the entity types specified in the Dataflow Integration tab.
  • The corresponding document in Dataflow must have at least one of the Dataflow scenario tags specified in the Dataflow Integration tab, applied to one of its scenarios.
  • If a specific revision tag has been applied in the Dataflow Integration tab (instead of the Latest revision), it must be applied to one of the revisions in the corresponding Dataflow document.
  • The relevant Dataflow template must have at least one variable mapping defined on the Input tab in the Economics configuration.
  • Any filtering criteria applied in the Dataflow Integration tab do not exclude the project.

Folder paths are only created if they contain at least one project. If no projects are created due to any combination of the above reasons, then the status of the hierarchy sync banner will be red (Sync Error).

For version 16.5 Update 17 and later: When a configured Revision Tag is not referenced for any Dataflow document in a synchronization then a fatal Sync Error was triggered in earlier versions. This has been changed so that such events cause warnings only.

Use the Sync Log Viewer to check the errors in the sync log.