This page describes how to use platform logs generated by AML AI
as part of [Cloud Logging](https://docs.cloud.google.com/logging/docs.md.txt). AML AI uses the Logging
API service name `financialservices.googleapis.com` to log the activity of:

- Engine config creation (tuning)
- Model creation (training)
- Backtest operations
- Prediction operations

## Before you begin

To view and manage logs, make sure you have the correct
[IAM permissions and roles](https://docs.cloud.google.com/logging/docs/access-control.md.txt#permissions_and_roles).

## Platform logging activation status

The platform logs for AML AI are always active (can't be turned
off).

## Log severity

AML AI log entries use three severity levels:

- `NOTICE` for entries sent when operations start or succeed
- `ERROR` for entries about the end of failed operations
- `INFO` for entries about operation progress

## View platform logs

To view platform logs, follow the instructions below:

### Console

To view platform logs in the Google Cloud console:

1. Navigate to the Logs Explorer:

   [Go to the Logs Explorer](https://console.cloud.google.com/logs/query)
2. Select the appropriate Google Cloud project.

3. In the **Query** field, enter the following query command:

   ```
   logName=("projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fbacktest" OR 
   "projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fengine_config_creation" OR 
   "projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fmodel_creation" OR 
   "projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fprediction")
   ```

   where:

   `PROJECT_ID` is the ID of the project you want to debug or monitor. For example, `my-project`.
4. Click **Run query**.

For more information about the Logs Explorer, see
[Logs Explorer Overview](https://docs.cloud.google.com/logging/docs/view/logs-explorer-summary) and [Using the Logs Explorer](https://docs.cloud.google.com/logging/docs/view/logs-explorer-interface.md.txt).

### gcloud

The gcloud command-line tool provides a command-line interface to Cloud Logging.

To view the logs for your project, run the following command:

```
gcloud logging read 'logName=("projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fbacktest" OR 
"projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fengine_config_creation" OR 
"projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fmodel_creation" OR 
"projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fprediction")' --project=PROJECT_ID
```

where `PROJECT_ID` is the ID for your Google Cloud project.

For more information about using the gcloud tool with Cloud Logging, see
[`gcloud logging`](https://docs.cloud.google.com/sdk/gcloud/reference/logging.md.txt).

## Understand platform logs

This section describes how to understand specific platform logs for
AML AI.

### START log

Logs with `eventKind=START` are generated when the operation starts its execution.

The following is an example of a log of starting a prediction run.

```
jsonPayload:
  '@type': type.googleapis.com/google.cloud.financialservices.logging.v1.PredictionLog
  engineVersion: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/engineVersions/ENGINE_VERSION_ID
  eventKind: START
  predictionResult:
    dataset: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/datasets/DATASET_ID
    endTime: '2023-05-31T00:00:00Z'
    model: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/models/MODEL_ID
    outputs:
      explainabilityDestination:
        tableUri: bq://PROJECT_ID.DATASET_ID.EXPLAINABILITY_TABLE_ID
        writeDisposition: WRITE_EMPTY
      predictionDestination:
        tableUri: bq://PROJECT_ID.DATASET_ID.PREDICTION_TABLE_ID
        writeDisposition: WRITE_EMPTY
logName: projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fprediction
operation:
  first: true
  id: projects/PROJECT_NUMBER/locations/REGION_ID/operations/OPERATION_ID
  producer: financialservices.googleapis.com
receiveTimestamp: '2023-06-07T12:30:48.417285528Z'
resource:
  labels:
    instance_id: INSTANCE_ID
    location: REGION_ID
    prediction_result_id: PREDICTION_ID
    resource_container: projects/PROJECT_NUMBER
  type: financialservices.googleapis.com/PredictionResult
```

You can add additional commands in the Logs Explorer **Query** field to narrow down the displayed logs.

Add the following command to display all started prediction runs on a selected dataset:

```
logName="projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fprediction" AND
jsonPayload.predictionResult.dataset="projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/datasets/DATASET_ID" AND
jsonPayload.eventKind="START"
```

### PROGRESS log

Logs with `eventKind=PROGRESS` are informing about the progress of an operation.

The following is an example of a log of creating a model. The `completedTaskCount` vs `taskCount`
can be used to estimate the progress that was made on the model training.

```
jsonPayload:
  '@type': type.googleapis.com/google.cloud.financialservices.logging.v1.ModelCreationLog
  completedTaskCount: 11
  engineVersion: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/engineVersions/ENGINE_VERSION_ID
  eventKind: PROGRESS
  model:
    endTime: '2023-05-31T00:00:00Z'
    engineConfig: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/engineConfigs/ENGINE_CONFIG_ID
    engineVersion: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/engineVersions/ENGINE_VERSION_ID
    lineOfBusiness: RETAIL
    primaryDataset: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/datasets/DATASET_ID
    state: CREATING
  partyCount: '9246'
  taskCount: 16
logName: projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fmodel_creation
operation:
  id: projects/PROJECT_NUMBER/locations/REGION_ID/operations/OPERATION_ID
  producer: financialservices.googleapis.com
receiveTimestamp: '2023-06-07T13:57:00.454668648Z'
resource:
  labels:
    instance_id: INSTANCE_ID
    location: REGION_ID
    model_id: MODEL_ID
    resource_container: projects/PROJECT_NUMBER
  type: financialservices.googleapis.com/Model
severity: INFO
timestamp: '2023-06-07T13:56:59.772973055Z'
```

### END log

Logs with `eventKind=END` are produced when operation ends.

The following is an example of a log of a failed engine config creation.
It contains the error about wrong data in the provided dataset.

```
jsonPayload:
  '@type': type.googleapis.com/google.cloud.financialservices.logging.v1.EngineConfigCreationLog
  completedTaskCount: 3
  engineConfig:
    engineVersion: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/engineVersions/ENGINE_VERSION_ID
    lineOfBusiness: RETAIL
    performanceTarget:
      partyInvestigationsPerPeriodHint: '100'
    state: CREATING
    tuning:
      endTime: '2019-04-30T00:00:00Z'
      primaryDataset: projects/PROJECT_ID/locations/REGION_ID/instances/INSTANCE_ID/datasets/DATASET_ID
  eventKind: END
  operationStatus:
    code: 9
    details:
    - '@type': type.googleapis.com/google.rpc.ErrorInfo
      domain: financialservices.googleapis.com
      metadata:
        count: '15'
        data_field: party_id, validity_start_time
        data_table: party
        description: There is a duplicate primary key value in the database resulting
          in unique key violation. Note that for tables with validity_start_time,
          the primary key includes validity_start_time
        test: GROUP BY party_id, validity_start_time HAVING count(1) > 1
      reason: DUPLICATE_PRIMARY_KEY
    message: Dataset validation failed with 1 error. See error details for individual
      violations.
  partyCount: '9246'
  taskCount: 16
logName: projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fengine_config_creation
operation:
  id: projects/PROJECT_NUMBER/locations/REGION_ID/operations/OPERATION_ID
  last: true
  producer: financialservices.googleapis.com
receiveTimestamp: '2023-06-07T14:26:30.214382295Z'
resource:
  labels:
    engine_config_id: ENGINE_CONFIG_ID
    instance_id: INSTANCE_ID
    location: REGION_ID
    resource_container: projects/PROJECT_NUMBER
  type: financialservices.googleapis.com/EngineConfig
severity: ERROR
timestamp: '2023-06-07T14:26:29.670913895Z'
```

To see all the create engine config error logs, use the filter below:

```
logName="projects/PROJECT_ID/logs/financialservices.googleapis.com%2Fengine_config_creation" AND
severity>=ERROR
```