Creating and Managing Export Destinations - m3ter Documentation

Documentation Index

Fetch the complete documentation index at: /llms.txt

Use this file to discover all available pages before exploring further.

You can create data Export Destinations using:

• S3 buckets on your AWS account.
• Buckets in your Google Cloud Storage account.

You’ll need to create Export Destinations before you can create data Export Schedules or run ad-hoc data Exports using the Data Export service:

• Preparing to Create Export Destinations
• Creating AWS S3 Bucket Export Destinations
• Creating Google Cloud Storage Bucket Export Destinations
• Reviewing and Managing Export Destinations

Preparing to Create Export Destinations

AWS S3 Bucket

For a data upload into your S3 bucket to succeed, the m3ter Data Export service has to assume an IAM role with PutObject permission for the bucket. For this reason, you’ll need to go into your AWS account and set up an IAM role for this access level. You then must enter the IAM role ARN when you create the Export Destination.

Tip: Copy Permission and Trust Policies from m3ter Console! To assist you in setting up the required IAM role, the Create export destination page in the Console shows sample Permission and Trust policies, which you can copy, complete for your specific bucket details, and then use to create the required IAM role. See the following section.

Google Cloud Storage Bucket

For a data upload into your Google Cloud Storage (GCS) bucket to succeed, the m3ter Data Export service has to authenticate with your Google Cloud Platform Project using Workload Identity Federation. You can grant the storage.objectCreator role to the data exports principal directly or using service account impersonation.

Tip: Copy gcloud commands from m3ter Console! To assist you in setting up the required authentication with your Google Cloud Platform project, we’ve created sample gcloud commands for you to create a workload identity pool, identity provider and policy binding. You can copy these commands to use with your specific project details. See the following section.

Creating AWS S3 Bucket Export Destinations

You can create Export Destinations for your AWS S3 buckets in the m3ter Console.To create an AWS S3 Export Destination:

1. Select Data exports:

The Data export page opens and lists any existing Export destinations:

2. On the Export destinations panel, select Create export destination. The Create page opens with AWS S3 selected by default as Export destination type.
3. Select Next. The page adjusts for Export destination details.
4. Enter the details of the new Export destination:

• Name. Descriptive name for the destination.

  • Code. Unique identifier for the destination.

• Note that when you’ve entered a Name and clicked in the Code field, a Code is automatically entered based on the Name. You can then edit the auto-generated Code as required.

  • Bucket name. Name of the S3 bucket for the destination.
  • Prefix. Location in bucket for the export. Leave this blank if you want to use the root of the bucket.
  • IAM role ARN. You must use the correct format:

• The IAM role ARN must begin with “arn:aws:iam”.

  • The general format required is: “arn:aws:iam::<aws account id>:role/<role name>”. For example: arn:aws:iam::922609978421:role/IAMRole636.
  • If the IAM role ARN you enter doesn’t comply with this format, then you won’t be able to save the Export Destination and an error message will show.
  • Partition data by. Required. Specify how you want the file path to be structured in your bucket destination - by Time first (Default) or Type first.

• Type is dependent on whether the data export is for usage or operational data: - Usage. Type is measurements. - Operational. Type is one of the entities for which operational data exports are available, such as account, commitment, meter, and so on.

  • Here’s an example for usage data export:
    • Time first:

• {bucketName}/{prefix}/orgId={orgId}/date=2025-01-27/hour=10/type=measurements/b9a317a6-860a-40f9-9bf4-e65c44c72c94_measurements.csv.gz - Type first:

• {bucketName}/{prefix}/orgId={orgId}/type=measurements/date=2025-01-27/hour=10/b9a317a6-860a-40f9-9bf4-e65c44c72c94_measurements.csv.gz

Notes:

• Sample Permission and Trust polices are provided on the Create form and you can Copy these directly to your clipboard.
  • For the Trust policy, your m3ter Organization ID is used as the ExternalId. Please ensure this is not changed. This is what the Data Export service will use when assuming the role and if you change it, the export will not work and security will be compromised.

5. Select Create export destination. The new Export destination is saved and is listed under Export destinations on the Data export page. You can read-off the Code, Bucket name, and Prefix settings, and also Export destination type.

Note: The service validates that it can assume the defined role prior to saving.

Creating Google Cloud Storage Bucket Export Destinations

You can create Export Destinations for your Google Cloud Storage (GCS) buckets in the m3ter Console.To create an Google Cloud Storage Export Destination:

1. Select Data exports:

The Data export page opens and lists any existing Export destinations.

2. On the Export destinations panel, select Create export destination. The Create page opens with AWS S3 selected by default as Export destination type.
3. Select Google Cloud Storage.
4. Select Next. The page adjusts for Export destination details.
5. Enter the details of the new Export destination:

• Name. Descriptive name for the destination.

  • Code. Unique identifier for the destination.

• Note that when you’ve entered a Name and clicked in the Code field, a Code is automatically entered based on the Name. You can then edit the auto-generated Code as required.

  • Bucket name. Name of the GCS bucket for the destination.
  • Prefix. Location in bucket for the export. Leave this blank if you want to use the root of the bucket.
  • Workload identity pool ID. ( Required)
  • Identity Provider ID. ( Required)
  • GCP Project Number. ( Required)
  • Service account email. ( Optional) If you want m3ter to authenticate using service account impersonation, the email of the service account.
  • Partition data by. Required. Specify how you want the file path to be structured in your bucket destination - by Time first (Default) or Type first.

• {bucketName}/{prefix}/orgId={orgId}/date=2025-01-27/hour=10/type=measurements/b9a317a6-860a-40f9-9bf4-e65c44c72c94_measurements.csv.gz - Type first:

• {bucketName}/{prefix}/orgId={orgId}/type=measurements/date=2025-01-27/hour=10/b9a317a6-860a-40f9-9bf4-e65c44c72c94_measurements.csv.gz

Note: Sample gcloud commands. You can copy sample gcloud commands to create a workload identity pool, identity provider and policy binding. You can grant the ‘storage.objectCreator’ role to the data exports principal directly or using service account impersonation. For further help in performing this setup, please consult your Google Cloud Platform documentation, such as this Workload Identity Federation page.

6. Select Create export destination. The new Export destination is saved and is listed under Export destinations on the Data export page. You can read-off the Code, Bucket name, and Prefix settings, and also Export destination type.

Note: The service validates that it can assume the defined role prior to saving.

Reviewing and Managing Export Destinations

From the Data export page you can manage your export Destinations to review or edit a Destination’s details and delete a Destination if you no longer want to export data to it.To view details, edit, or delete an Export Destination:

1. Select Data exports. The Data export page opens and lists any existing Export destinations.
2. If you want to review details of a Destination, on the Export destinations panel, select the name hotlink text.

The Export destination details page opens: You can review details and copy directly to your clipboard the:

• Destination ID.
• In the example of an AWS S3 bucket Destination, the IAM role ARN.

3. If you want edit a Destination, on the Data export page under Export destinations select the Edit icon:

• Alternatively, if you’ve opened the Export destination details page, you can Edit the Destination directly from there.

The Edit page opens and you can update the Export destination details.

4. When you have completed your editing changes, select Update export destination.
5. If you want to delete a Destination, select the Delete icon:

A confirmation dialog appears.

6. Select Yes to confirm the delete action.

Tip: Alternatively, you can use the Edit and Delete buttons at top right of the Export destination details page.

Warning: Destination currently in use! If the Export Destination is currently being used for an Export Schedule, you won’t be able to delete it and an error message will show.