Creating Legacy Usage Data Export Schedules - m3ter Documentation

Documentation Index

Fetch the complete documentation index at: /llms.txt

Use this file to discover all available pages before exploring further.

When you have created some Export Destinations using S3 buckets in your AWS account or using buckets in your Google Cloud Storage account, you can set up Export Schedules to perform usage data exports into those destinations from your m3ter Organization. You can specify an hourly or daily frequency for these exports to run. This topic explains how to work with and create Export Schedules for exporting usage data from your Organization at regular intervals:

• Working with Usage Data Export Schedules
• Creating Legacy Usage Data Export Schedules

Notes:

• Usage data (legacy format) Export Schedules? If you were an early customer of Data Exports you might also still have access to the legacy usage format for data Export Schedules. However, the legacy usage format is NOT available for new customers - please see Creating Usage Data Export Schedules.
• Export jobs for Export schedules. When you have created a data Export schedule, an export job is created for each time the data export is due to run under the schedule. You can review and check the status of data export jobs on the Export jobs page - see Reviewing Export Jobs.

Working with Usage Data Export Schedules

This section explains how the Data Export service will compile and run usage data exports according to the various options available to you when setting up your usage data Export Schedules:

• Data Export Service - Timezone Used
• When Export Schedules are Run
• Export File Format and Composition
• Export File Size Limit

Important! Please review this section before setting up any usage data Export schedules to understand the various options you can use to ensure the exports contain the correct usage data to serve your specific needs.

Data Export Service - Timezone Used

The Data Export service uses UTC and this is not affected by the Timezone you’ve set for your Organization.

When Export Schedules are Run

When you’ve completed the configuration for an Export Schedule and select to create it:

• An Export job is scheduled to run the data export immediately with the Data Export service.
• Export jobs for the next and subsequent in the series of exports are then scheduled to run with the Data Export Service according to the hourly or daily frequency you’ve configured for the Schedule.
• For example, if you select to create an Export schedule at 10:00 a.m. on June 1st, 2024 UTC and the Export schedule is configured for a frequency of every 5 days, an Export job is scheduled to run with the Data Export Service immediately. An Export job for the next data export will then be scheduled to run at 10:00 a.m. on June 6th 2024 UTC, and so on.
• Exported files are retained in m3ter for 24 hours after the job completes, after which they are removed.

Export File Format and Composition

The range of usage data and the way it is compiled will depend on several options which allow you to control closely what the export file will contain.

File Format

The available data exports file formats are .CSV or JSONL.

Time Period

You can define a Time period when creating a usage data Export schedule to control the range of data you want the data export to contain each time it runs according to the schedule. Options:

• Today. Data collected for the current day up until the time the export is scheduled to run. For example, if the Export schedule is configured to run with a frequency of every 3 hours and is created initially at 10:00 a.m. UTC, the first export scheduled to run will contain the usage data collected up until 10 a.m. UTC of the current day, the second export will be scheduled to run at 1 p.m. UTC that day and will contain all the usage data collected up until 1 p.m. for that day, and so on. (Default setting)
• Yesterday. Data collected for the day before the export runs under the schedule - that is, the 24 hour period from midnight to midnight of the day before.
• Week, Month, Year to date. Data collected for the period covering the current week, month, or year period. For example, for Week to date setting weeks run Monday to Monday - if the Export is scheduled to run at 10 a.m. UTC on October 16th 2024, which is a Wednesday, it will contain all usage data collected starting Monday October 14th 2024 through to the Wednesday at 10 a.m. UTC of the current week.
• Previous week, month, quarter, year. Data collected for the previous full week, month, quarter, or year period. For example, weeks run Monday to Monday - if the export is scheduled to run on June 12th 2024, which is a Wednesday, the export will contain data for the period running from Monday, June 3rd to midnight on Sunday, June 9th 2024.
• Last 12 hours. Data collected for the twelve hour period up to the start of the hour in which the export is scheduled to run.
• Last 7 days, 30 days, 35 days, 90 days, 120 days, year. Data collected for the selected period prior to the date the export is scheduled to run. For example, if Last 30 days and the export is scheduled to run for any time on June 15th 2024, it will contain usage data collected for the previous 30 days - starting May 16th 2024 through to midnight on June 14th 2024.

Meters and Accounts

You can select Meters and Accounts in your Organization for the export. The usage data export file will then contain only the usage data collected for those Meters and Accounts:

• If you do not select to aggregate the usage data, then the export will contain the raw usage data measurements ingested by the selected Meters and selected Accounts. The usage data measurement values included in the export for non-aggregated usage data exports are:

• For all Data Fields types - both String and Numeric - and any Derived Fields on the selected Meters.

  • The values will be the post-enrichment ingest process values. See Usage Data Ingest Process for more details.

• Default option is for all Meters and all Accounts.

Aggregated Usage

You can select to aggregate the usage data measurement values collected for the selected Meters and Accounts:

• Aggregation Frequency. Select a time period for aggregation - Hourly, Daily, Weekly, or Monthly:

• The default setting is Original, which means the usage data is not aggregated for export. See the previous section for what is included when the usage data export is for non-aggregated usage.

• Aggregation. Select a function for aggregating raw usage data measurement values over the selected aggregation frequency time period:

• Sum. Adds the values.

  • Minimum. Uses the minimum value.
  • Maximum. Uses the maximum value.
  • Count. Counts the number of values.
  • Latest. Uses the most recent value, based on the timestamp - ts - values of ingested usage data measurements.
  • Mean. Uses the arithmetic mean of the values.

If you select to aggregate usage data measurements for data export, the usage data values contained in the data export will only include:

• The usages data measurements collected for any numeric Data Fields on selected Meters for Accounts - Data Fields of type MEASURE, INCOME, or COST - and aggregated in accordance with the aggregation frequency and function you’ve selected.
• For example:
  • Suppose you select a single Meter for the data export that has two Data Fields:
    • A Data Field of type MEASURE, a Numeric field which is used to collect usage data for file submissions by size of file in megabytes.
    • A Data Field of type WHAT, a String field which is used to collect usage data on the file format.
  • You then implement aggregation for a daily sum of usage data values for the export.
  • The aggregated usage data export will only contain the values for the daily sums of file submission size from the MEASURE Data Field.

Export File Size Limit

We perform gzip compression for usage data files for export. The file size limit for usage data exports is constrained by the limit imposed by your S3 bucket destination, which we understand to be 5TB maximum size after gzip compression.

Creating Legacy Usage Data Export Schedules

You can quickly set up Export schedules for data exports containing usage data collected for your Organization. Usage data exports contain the usage data measurements ingested into m3ter for the selected Meters and Accounts covering the specified time period.To create a Legacy Usage Data Export Schedule:

1. Select Data exports:

The Data exports page opens and lists any existing Export schedules.

2. Select Create export schedule. The Create page opens.
3. Enter the Export schedule details:

• Name. Descriptive name for the schedule.

  • Code. Unique identifier for the schedule.

• Note that when you’ve entered a Name and clicked in the Code field, a Code is automatically entered based on the Name. You can then edit the auto-generated Code as required.

  • Destinations. Use the drop-down to select the Export destinations to be used each time the data export runs for the schedule. Notes:

• Select at least one destination.

  • If No options shows in the drop-down, you must first create at least one destination before you can set up an Export Schedule - see Creating and Managing Export Destinations.
  • Alternatively, if you simply want to download the data export file locally, do not select a Destination. See Downloading Data Export Files Locally for more details.
  • Export file format. Select the format you require for the usage data export files:

• CSV. (Default).

  • JSONL.
  • Frequency. Define the frequency for the schedule in Hours or Days. Highest frequency is for exports to run every hour.

• Note that the data export will be scheduled to run immediately when you create it and then at the defined frequency. See above Working with Usage Data Export Schedules for details.

  • Source type. Select Usage data (legacy format). The Export schedule details panel adjusts.
  • Time period. Define a time period you want the data export to cover each time it runs according to the schedule. This setting controls what usage data collected for the specified Meters/Accounts the export will contain. See above: Time Period for details on each of the eight options. Today is the Default.

• Aggregation frequency and Aggregation. If you want to aggregate ingested values for your usage data export, then use these options in combination:

• Aggregation frequency. Select the frequency for aggregation - five options:

• Original. No aggregation performed. (Default) - Hourly. - Daily. - Weekly. - Monthly.

  • Aggregation. Select the type of Aggregation you want to apply to the data values for the selected frequency - six options:

• Sum. Adds the values. - Minimum. Uses the minimum value. - Maximum. Uses the maximum value. - Count. Counts the number of values. - Latest. Uses the most recent value, based on the timestamp - ts values of ingested usage data measurements. - Mean. Uses the arithmetic mean of the values.

Aggregation for Usage Data exports - notes:

• Aggregation type is required! If you select an Aggregation frequency other than the default of Original (which means the data will not be aggregated), then you must select an Aggregation type. If you don’t select a type, then you won’t be able to create the Export schedule and an error will show.

  • Aggregation vs. non-aggregated usage data for export? If you choose to aggregate usage data for an export schedule, this will affect what data is included in the export. For details, please see the Working with Usage Data Export Schedules section above.

• Meters. Add Meters for the data export:

• Select the plus icon. A popup will show listing all Meters in your Organization and you can search the list.

  • Check the Meters whose usage data you want to include in the export and select Add.
  • The export will contain the usage data collected by the Meters you add for the specified Time period. (Default is All meters)

• Accounts. Add Accounts for the data export:

• Select the plus icon. A popup will show listing all Accounts in your Organization and you can search the list.

  • Check the Accounts whose usage data you want to include in the export and select Add.
  • The export will contain the usage data collected by the Meters you’ve added and for the Accounts you add for the specified Time period. (Default is All Accounts)

4. When you have entered all details, select Create export schedule:

The Schedule is saved and the Export schedule details page opens:

• In this example, we’ve selected two Meters and two Accounts for which non-aggregated usage data will be included in the data exports ran every day for the Export Schedule.

5. Return to the Data exports page where the new schedule is listed on the Export schedules panel: