Creating Usage Data Export Schedules - m3ter Documentation

Documentation Index

Fetch the complete documentation index at: /llms.txt

Use this file to discover all available pages before exploring further.

When you have created some Export Destinations using S3 buckets in your AWS account or using buckets in your Google Cloud Storage account, you can set up Export Schedules to perform usage data exports to those destinations from your m3ter Organization. You can specify an hourly or daily frequency for these exports to run. This topic explains how to work with and create Export Schedules for exporting usage data from your Organization at regular intervals:

• Working with Usage Data Export Schedules
• Creating Usage Data Export Schedules

Note and Tips:

• Start in Usage Data Explorer? You might want to first review the usage data returned for a query you plan to use for your Export Schedule. If so, you can start in the Usage Data Explorer and build and perform the query. If the data the query returns is the data you want to export, you can then select Create export schedule. See Using Queries for Data Export for more details.
• Export jobs for Export schedules? When you have created a data Export schedule, an export job is created for each time the data export is due to run under the schedule. You can review and check the status of data export jobs on the Export jobs page - see Reviewing Export Jobs.
• Usage data (legacy format) Export Schedules? If you were an early customer of Data Exports you might also still have access to the legacy usage format for data Export Schedules. See Creating Legacy Usage Data Export Schedules. However, the legacy usage format is NOT available for new customers.

Working with Usage Data Export Schedules

This section explains how the Data Export service will compile and run usage data exports according to the various options available to you when setting up your usage data Export Schedules:

• Data Export Service - Timezone Used
• When Export Schedules are Run
• Export File Format and Composition
• Maximum Results and Output File Sizes

Important! Please review this section before setting up any usage data Export schedules to understand the various options you can use to ensure the exports contain the correct usage data to serve your specific needs.

Data Export Service - Timezone Used

The Data Export service uses UTC and this is not affected by the Timezone you’ve set for your Organization.

When Export Schedules are Run

When you’ve completed the configuration for an Export Schedule and select to create it:

• An Export job is scheduled to run the data export immediately with the Data Export service.
• Export jobs for the next and subsequent in the series of exports are then scheduled to run with the Data Export Service according to the frequency you’ve configured for the Schedule - at a specified number of minutes, hours, or days intervals. You can also define an offset: for hourly scheduling an offset in minutes from start of hour; for daily scheduling an offset in hours from midnight.
• For example, if you select to create an Export schedule at 10:00 a.m. on June 1st, 2024 UTC and the Export schedule is configured for a frequency of every 2 days, an Export job is scheduled to run with the Data Export Service immediately. An Export job for the next data export will then be scheduled to run at 10:00 a.m. on June 3rd 2024 UTC, and so on.
• Exported files are retained in m3ter for 24 hours after the Export job completes, after which they are removed.

Export File Format and Composition

The range of usage data and the way it is compiled will depend on several options which allow you to control closely what the export file will contain.

File Format

The available data exports file formats are .CSV or JSONL.

Time Period

You can define a Time period when creating a usage data Export schedule to control the range of data you want the data export to contain each time it runs according to the schedule. Options:

• Today. Data collected for the current day up until the time the export is scheduled to run. For example, if the Export schedule is configured to run with a frequency of every 3 hours and is created initially at 10:00 a.m. UTC, the first export scheduled to run will contain the usage data collected up until 10 a.m. UTC of the current day, the second export will be scheduled to run at 1 p.m. UTC that day and will contain all the usage data collected up until 1 p.m. for that day, and so on. ( Default setting)
• Yesterday. Data collected for the day before the export runs under the schedule - that is, the 24 hour period from midnight to midnight of the day before.
• Week, Month, Year to date. Data collected for the period covering the current week, month, or year period. For example, for Week to date setting weeks run Monday to Monday - if the Export is scheduled to run at 10 a.m. UTC on October 16th 2024, which is a Wednesday, it will contain all usage data collected starting Monday October 14th 2024 through to the Wednesday at 10 a.m. UTC of the current week.
• Previous week, month, quarter, year. Data collected for the previous full week, month, quarter, or year period. For example, weeks run Monday to Monday - if the export is scheduled to run on June 12th 2024, which is a Wednesday, the export will contain data for the period running from Monday, June 3rd to midnight on Sunday, June 9th 2024.
• Last 12 hours. Data collected for the twelve hour period up to the start of the hour in which the export is scheduled to run.
• Last 7 days, 30 days, 35 days, 90 days, 120 days, year. Data collected for the selected period prior to the date the export is scheduled to run. For example, if Last 30 days and the export is scheduled to run for any time on June 15th 2024, it will contain usage data collected for the previous 30 days - starting May 16th 2024 through to midnight on June 14th 2024.

Meters and Accounts

You can select Meters and Accounts in your Organization for the export. The usage data export file will then contain only the usage data collected for those Meters and Accounts:

• If you do not select to aggregate the usage data, then the export will contain the raw usage data measurements ingested by the selected Meters and selected Accounts. The usage data measurement values included in the export for non-aggregated usage data exports are:

• For all Data Fields types - both String and Numeric - and any Derived Fields on the selected Meters.

  • The values will be the post-enrichment ingest process values. See Usage Data Ingest Process for more details.

• Default option is for all Meters and all Accounts.

Dimension Filters

You can filter the usage data returned for export by adding specific values of non-numeric Dimension data fields on included Meters. Only the data collected for the values you’ve added for the selected Dimension fields will be included in the export.

Aggregations

You can apply Aggregation functions to the usage data returned for export.

Group By

If you’ve applied Aggregation functions to the usage data for export, you can then select to group the data returned for export by:

• Account
• Time
• Dimension

Important: Applying Aggregations and Grouping for Usage Data Exports! If you intend to apply Aggregations to usage data returned for export and apply grouping to the aggregated data, please ensure you review the Applying Aggregations for Queries section first. The functionality used by the Data Export service when you set up a usage data export and aggregate the data is the same as used by the Usage Data Explorer (v2) for querying usage data.

Maximum Results and Output File Sizes

Data Exports can contain up to 500,000,000 records, and the file size limit is determined by the file size limit imposed by your destination.

Creating Usage Data Export Schedules

You can quickly set up Export schedules for data exports containing usage data collected for your Organization. Usage data exports contain the usage data measurements ingested into m3ter for the selected Meters and Accounts covering the specified time period.To create a Usage Data Export Schedule:

1. Select Data exports:

The Data exports page opens and lists any existing Export schedules:

2. Select Create export schedule. The Create page opens.
3. Enter the Export schedule details:

• Name. Descriptive name for the schedule.

  • Code. Unique identifier for the schedule.

• Note that when you’ve entered a Name and clicked in the Code field, a Code is automatically entered based on the Name. You can then edit the auto-generated Code as required.

  • Destinations. Use the drop-down to select the Export destinations to be used each time the data export runs for the schedule. Notes:

• Select a destination.

  • If No options shows in the drop-down, you must first create at least one destination before you can set up an Export Schedule - see Creating and Managing Export Destinations.
  • Alternatively, if you simply want to download the data export file locally, do not select a Destination. See Downloading Data Export Files Locally for more details.
  • Export file format. Select the format you require for the usage data export files:

• CSV. ( Default).

  • JSONL.

4. Set up the Export schedule:

• You can use the default Simple scheduling mode to do this:

• Frequency. Define the frequency for the schedule in Hours, Days, or Minutes.

  • Offset. If you use an hourly or daily frequency, then you can define an offset for when the export will run relative to the hour or day:

• For hourly scheduling, an offset in minutes from start of hour. - For daily scheduling an offset in hours from midnight.

  • Alternatively, select the Advanced scheduling mode and define a cron expression for when you want the export to run:

The data export will be scheduled to run immediately when you create it and then at the defined frequency.

Warning! Limits are imposed:

• Lowest frequency is every 3 days.
  • Highest frequency is every 15 minutes.
  • Errors will show if you try to set a frequency outside of these limits.

5. Enter the Export data:

• Source type. For a usage data Export schedule, leave this as the default selection for Usage data.

  • Time period. Define a time period you want the data export to cover each time it runs according to the schedule. This setting controls what usage data collected for the specified Meters/Accounts the export will contain. See above: Time Period for details on each of the options. Today is the Default.

• Meters. Add Meters for the data export:

• Select the plus icon. A popup will show listing all Meters in your Organization and you can search the list.

  • Check the Meters whose usage data you want to include in the export and select Add.
  • For the current example, we’ve used the Search field on the dialog to filter the list.
  • Alternatively, you can enter the ID of a Meter and then Add it.
  • The export will contain the usage data collected by the Meters you add for the specified Time period. ( Default is All meters)

• Accounts. Add Accounts for the data export:

• Select the plus icon. A popup will show listing all Accounts in your Organization and you can search the list.

  • Check the Accounts whose usage data you want to include in the export and select Add.
  • For the current example, we’ve used the Search field on the dialog to filter the list.
  • Alternatively, you can enter the ID of an Account and then Add it.
  • The export will contain the usage data collected by the Meters and Accounts you’ve added for the specified Time period. ( Default is All Accounts)

6. If you want to use Dimensions filters for the usage data included in the export, select Add dimension filter. A menu appears and lists any Dimension fields for Meters selected for the export:

• In the current example, a single Meter has two Dimension fields configured.

7. Select the Meter Dimension field you want to use to filter the data returned for the export. For the current example, we’ll select RevGraph Meter 1 > Customer Name. A dialog replaces the menu, which lists values used for this field when usage submissions were made for the Meter within the selected Time period.
8. Check the values used for the selected Meter Dimension field you want to use to filter the data returned for the export and then select Add filter:

The dialog closes and the select Dimension field values are added under Dimension filters:

9. If you want to apply Aggregations to the usage data returned for export, under Aggregations, select Add aggregation. A menu listing available aggregation methods appears:

10. Select the Aggregation method you want to apply. For example, select Sum. A dialog replaces the menu and shows the fields on the Meters selected for the export which can have this aggregation method applied:

11. Select the Meter data field you want to apply the Aggregation method to. The selection is added under Aggregations.

12. Repeat steps to apply any other Aggregations for the usage data export: For the current example, we’ve applied a Sum Aggregation to both of the numeric data fields on the selected Meter.

13. If you’ve applied Aggregations to the returned usage data for export and you want to group the data, under Aggregations, select the grouped by plus icon. A menu appears:

In the current example, we select to group by a Meter Dimension. The menu is replaced by a dialog showing the available Dimensions on the Meter:

14. Select the Dimension you want to group the data by. The dialog closes and your grouped by selection is added under Aggregations.
15. When you have entered all details, select Create export schedule. The Export schedule details page opens:

• In this example, we’ve selected a single Meter and Account for which filtered and aggregated usage data grouped by a dimension will be included in the data exports ran every three days for the Export Schedule.

16. Return to the Data exports page where the new schedule is listed on the Export schedules panel: