Skip to main content

Data conversion

This guide provides an overview of data handling using a Data Converter on the Temporal Platform.

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster. It is used by the Temporal SDK framework to serialize/deserialize data such as input and output of Activities and Workflows that need to be sent over the wire to the Temporal Cluster.

Data Converter encodes and decodes data

Data Converter encodes and decodes data

The Data Converter encodes data from your application to a PayloadLink preview iconWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more before it is sent to the Temporal Cluster in the Client call. When the Temporal Server sends the encoded data back to the Worker, the Data Converter decodes it for processing within your application. This technique ensures that all your sensitive data exists in its original format only on hosts that you control.

The main pieces of data that run through the Data Converter are arguments and return values:

  • The Client:
    • Encodes Workflow, Signal, and Query arguments.
    • Decodes Workflow and Query return values.
  • The Worker:
    • Decodes Workflow, Signal, and Query arguments.
    • Encodes Workflow and Query return values.
    • Decodes and encodes Activity arguments and return values.

Each piece of data (like a single argument or return value) is encoded as a PayloadLink preview iconWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more Protobuf message, which consists of binary data and key-value metadata.

For details, see the API references:

Payload​

A Payload represents binary data such as input and output from Activities and Workflows. Payloads contain metadata that describe the binary data, such as its data type or other arbitrary values for use by custom encoders/converters.

When processed through the SDK, the default Data ConverterLink preview iconWhat is a default Data Converter?

The default Data Converter is used by the Temporal SDK to convert objects into bytes using a series of Payload Converters.

Learn more serializes your data/value to a Payload before sending it to the Temporal Server. The default Data Converter processes supported type values to Payloads, and you can create a custom Payload ConverterLink preview iconWhat is a Payload Converter?

A Payload Converter serializes data, converting objects or values to bytes and back.

Learn more to convert your custom object types.

You can additionally apply custom codecsLink preview iconWhat is a Payload Codec?

A Payload Codec transforms an array of Payloads into another array of Payloads.

Learn more (such as for encryption or compression) on your Payloads to wrap them into new encoded Payloads.

Default Data Converter​

Each Temporal SDK includes and uses a default Data Converter. The default Data Converter converts objects to bytes using a series of Payload Converters and supports binary, Protobufs, and JSON formats. It encodes values in the following order:

  • Null
  • Byte array
  • Protobuf JSON
  • JSON

For example:

  • If a value is an instance of a Protobuf message, it is encoded with proto3 JSON.
  • If a value isn't null, binary, or a Protobuf, it is encoded as JSON. If any part of it is not serializable as JSON, an error is thrown.

Custom Data Converter​

A custom Data Converter extends the default Data ConverterLink preview iconWhat is a Data Converter?

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster.

Learn more with custom logic for PayloadLink preview iconWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more conversion or encoding.

You can create a custom Data Converter to alter formats (for example, using MessagePack instead of JSON) or add compression and encryption.

You can customize the default Data Converter behavior in two ways:

Custom Data Converters are not applied to all data; for example, Search AttributesLink preview iconWhat is a Search Attribute?

A Search Attribute is an indexed name used in List Filters to filter a list of Workflow Executions that have the Search Attribute in their metadata.

Learn more are simple values and persisted unencoded so they can be indexed for searching.

A customized Data Converter can have the following three components:

For details on how to implement custom Payload Converters in your SDK, see Custom Payload conversion for the following SDKs:

For details on how to implement custom encryption and compression in your SDK, see Data Encryption.

Payload Converter​

A Payload Converter serializes data, converting values to bytes and back.

When you request a Workflow Execution through your Client and pass a data input, the input is serialized using a Data Converter that runs it through a set of Payload Converters. When your Workflow Execution starts, this data input is deserialized and passed as input to your Workflow.

For more information, see the API references.

For supported values, see default Data Converter.

Custom payload conversion​

If you use custom objects or types that are not supported by the Payload Converters provided in the SDKs, you can create a custom Payload Converter and configure the Data Converter with it to run the specific conversions.

You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

For details on how to use the Payload Converter for custom data types, see Custom Payload Conversion in the following langauages:

Failure Converter​

A Failure Converter converts error objects to proto Failures and back. The default Failure Converter copies error messages and stack traces as plain text.

For details, see the API references.

You can make a custom Failure Converter, but if you use multiple SDKs, you must implement the same logic in each. Creating a custom Failure Converter is not yet supported in Java.

Failure messages and stack traces are not encoded as codec-capable Payloads by default; you must explicitly enable encoding these common attributes on failures. If your errors might contain sensitive information, you can encrypt the message and stack trace by configuring the default Failure Converter to use your encoded attributes, in which case it moves your message and stack_trace fields to a Payload that's run through your codecLink preview iconWhat is a Payload Codec?

A Payload Codec transforms an array of Payloads into another array of Payloads.

Learn more.

Payload Codec​

A Payload Codec transforms an array of PayloadsLink preview iconWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more (for example, a list of Workflow arguments) into another array of Payloads.

The Payload Codec is an optional step that happens between the wire and the Payload ConverterLink preview iconWhat is a Payload Converter?

A Payload Converter serializes data, converting objects or values to bytes and back.

Learn more:

User code <--> Payload Converter <--> Payload Codec <--> Wire <--> Temporal Server

When serializing to Payloads, the Payload Converter is applied first to convert your objects to bytes, followed by codecs that convert bytes to bytes. When deserializing from Payloads, codecs are applied first to last to reverse the effect, followed by the Payload Converter.

For details, see the API references.

Use a custom Payload Codec to transform your Payloads; for example, implementing compression and/or encryption on your Workflow Execution data.

Encryption​

Using end-to-end encryption in your custom Data Converter ensures that sensitive application data is secure when handled by the Temporal Server.

Apply your encryption logic in a custom Payload Codec and use it locally to encrypt data. You maintain all the encryption keys, and the Temporal Server sees only encrypted data. Your data exists unencrypted only on the Client and the Worker process that is executing the Workflows and Activities, on hosts that you control.

For details, see Data encryption.

The following samples use encryption (AES GCM with 256-bit key) in a custom Data Converter:

Remote data encoding​

Remote data encoding is exposing your Payload Codec via HTTP endpoints to support remote encoding and decoding.

Running your encoding remotely allows you to use it with tctl to encode/decode data for several commands including tctl workflow start and with Temporal Web UI to encode and decode data in your Workflow Execution details view.

To run data encoding/decoding remotely, use a Codec ServerLink preview iconWhat is a Codec Server?

A Codec Server is an HTTP server that uses your custom Payload Codec to encode and decode your data remotely through endpoints.

Learn more. A Codec Server is an HTTP server that is configured to use your custom Payload Codec.

Before you use a remote data encoder to encode/decode your data, ensure that you consider all the security implications of running codecs remotely. For example, codecs that perform encryption may need to be secured to prevent decryption by untrusted callers.

Encoding data on the Web UI and tctl​

You can perform some operations on your Workflow Execution using tctl and the Web UI, such as starting or sending a Signal to an active Workflow Execution using tctl or canceling a Workflow Execution from the Web UI, which might require inputs that contain sensitive data.

To encode this data, specify your Codec Server endpoints with the tctl command and configure your Web UI to use the Codec Server endpoints.

Decoding data on the Web UI and tctl​

If you use custom encoding in your custom Data Converter, Payload data handled by the Temporal Cluster is encoded. Since the Web UI uses the VisibilityLink preview iconWhat is Visibility?

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Cluster.

Learn more database to show events and data stored on the Temporal Server, all data in the Workflow Execution History in your Web UI or tctl shows in the encoded format.

To see the original format of data in your Web UI and tctl, create a Codec ServerLink preview iconWhat is a Codec Server?

A Codec Server is an HTTP server that uses your custom Payload Codec to encode and decode your data remotely through endpoints.

Learn more with a remote data encoder and use the Payload Codec to decode your data locally.

Note that a remote data encoder is a separate system with access to your encryption keys and exposes APIs to encode and decode any data with the Payload Codec used. Evaluate and ensure that your remote data encoder endpoints are secured and only authorized users have access to them.

Samples:

Codec Server​

A Codec Server is an HTTP/HTTPS server that uses a custom Payload Codec to decode your data remotely through endpoints.

A Codec Server follows the Temporal Codec Server Protocol. It implements two endpoints:

  • /encode
  • /decode

Each endpoint receives and responds with a JSON body that has a payloads property with an array of PayloadsLink preview iconWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more. The endpoints run the Payloads through a Payload CodecLink preview iconWhat is a Data Converter?

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster.

Learn more before returning them.

Most SDKs provide example Codec Server implementation samples, listed here:

Usage​

When you apply custom encoding with encryption or compression on your Workflow data, it is stored in the encrypted/compressed format on the Temporal Server. For details on what data is encoded, see Data encryption.

To see decoded data when using the CLI or Web UI to perform some operations on a Workflow Execution, configure the Codec Server endpoint in the Web UI and CLI. When you configure the Codec Server endpoints, the CLI and Web UI send the encoded data to the Codec Server, and display the decoded data received from the Codec Server.

For details on creating your Codec Server, see Codec Server Setup.

After you start your Codec Server, configure your Codec Server endpoints.

Codec Server setup​

Use a Codec Server to decode your encoded payloadsLink preview iconWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more and integrate it with the Temporal Web UI and CLI commands when debugging your Workflows.

A Codec Server is an HTTP or HTTPS Server that you create and host. It must be configured to use a Payload CodecLink preview iconWhat is a Payload Codec?

A Payload Codec transforms an array of Payloads into another array of Payloads.

Learn more with the required decode logic and encryption keys.

The Codec Server is independent of the Temporal Server and decodes your encrypted payloads through endpoints. When you set the codec endpoint in the Temporal Web UI, the Web UI uses the remote endpoint to send encoded payloads to the Codec Server and receive decoded payloads from the Codec Server. See API contract requirements. Decoded payloads are then displayed in the Workflow Execution Event History on the Web UI.

Note that when you use a Codec Server, the decoded payloads are visible only to you on the Web UI; payloads on the Temporal Server (whether on Temporal Cloud or a self-hosted Temporal Cluster) remain encrypted.

Because you create, operate, and manage access to your Codec Server in your controlled environment, ensure that you consider the following:

  • When you set your codec endpoint with your Web UI, expect your Codec Server to receive a large number of requests per Workflow Execution from the Web UI.
  • Ensure that you secure access to the decrypted data from your Codec Server.
  • The Temporal Web UI only displays the decoded payloads received from your Codec Server in real-time; it does not store or send the data back to the Temporal Server (whether on Cloud or self-hosted Temporal Cluster).
  • You might have latencies introduced in the Web UI when sending and receiving payloads to the Codec Server.

To create a Codec Server, you need the following components:

  • A Payload CodecLink preview iconWhat is a Payload Codec?

    A Payload Codec transforms an array of Payloads into another array of Payloads.

    Learn more with the requisite keys and logic to decode your encrypted payloads. You can use the Payload Codec that you applied with your Data Converter to encode your Payloads and configure it with your Codec Server. However, if you are writing your Codec Server in a different SDK from the one that applies the Data Converter, ensure that your logic and keys are correctly replicated.
  • Key management infrastructure or plan for sharing your encryption keys between the Workers and your Codec Server.
  • CORS configuration on the HTTP endpoints in your Codec Server for sending and receiving requests from the Temporal Web UI.

For examples on how to create your Codec Server, see following Codec Server implementation samples:

API contract specifications​

When you create your Codec Server to handle requests from the Web UI, the following requirements must be met.

Endpoints​

The Web UI/CLI calls the POST method with the /decode endpoint.

In your Codec Server, create a /decode path and pass the incoming payload to the decode method in your Payload Codec.

You can also add a verification step to check whether the incoming request has the required authorization to access the decode logic in your Payload Codec.

Headers​

Each request from the Web UI to your Codec Server includes the following headers:

  • Content-Type: application/json: Ensure that your Codec Server can accommodate this MIME type.

  • X-Namespace: {namespace}: This is a custom HTTP Header. Ensure that the CORS configuration in your Codec Server includes this header.

  • [Optional] Authorization: <credentials>: Include this in your CORS configuration when enabling authorization with your Codec Server.

For details on setting up authorization, see Authorization.

Request body​

The general specification for the POST request body contains payloads. By default, all field values in your payload are base64 encoded, regardless of whether they are encrypted by your custom codec implementation.

The following example shows a sample POST request body with base64 encoding.

{
  "payloads": [{
    "metadata": {
      "encoding": <base64EncodedEncodingHint>
    },
    "data": <encryptedPayloadData>
  }, ...]
}

CORS​

Enable Cross-Origin Resource Sharing (CORS) requests on your Codec Server to receive HTTP requests from the Temporal Web UI.

At a minimum, enable the following responses from your Codec Server to allow requests coming from the Temporal Web UI:

  • Access-Control-Allow-Origin
  • Access-Control-Allow-Methods
  • Access-Control-Allow-Headers

For example, for Temporal Cloud Web UI hosted at https://cloud.temporal.io, enable the following in your Codec Server:

  • Access-Control-Allow-Origin: https://cloud.temporal.io
  • Access-Control-Allow-Methods: POST, GET, OPTIONS
  • Access-Control-Allow-Headers: X-Namespace, Content-Type

For details on what a sample request/response looks like from the Temporal Web UI, see Sample Request/Response. If setting authorization, include Authorization in your Access-Control-Allow-Headers. For details on setting up authorization, see Authorization.

Authorization​

To enable authorization from the Web UI, your Codec Server must be an HTTPS Server.

Temporal Cloud

Temporal Cloud uses Auth0 to authenticate access. The Temporal Cloud UI provides an option to pass access tokens (JWT) to your Codec Server endpoints. Use the access tokens to validate access and then return decoded payloads from the Codec Server.

You can enable this by selecting Pass access token in your Codec Server endpoint interface where you add your endpoint. Enabling this option in the Temporal Cloud UI adds an authorization header to each request sent to the Codec Server endpoint that you set.

In your Codec Server implementation, verify the signature on this access token (in your authorization header) against the JWKS endpoint provided to you.

The token provided from Temporal Cloud UI contains the email identifier of the person requesting access to the payloads. Based on the permissions you have provided to the user in your access control systems, set conditions in your Codec Server whether to return decoded payloads or just return the original encoded payloads.

Self-hosted Temporal Cluster

On self-hosted Temporal Clusters, configure authorization in the Web UI configurationLink preview iconTemporal Web UI configuration reference

The Temporal Web UI Server uses a configuration file for many of the UI's settings.

Learn more in your Temporal Cluster setup.

With this enabled, you can pass access tokens to your Codec Server and validate the requests from the Web UI to the Codec Server endpoints that you set.

Note that with self-hosted Temporal Clusters, you must explicitly configure authorization specifications for the Web UI and CLI.

The following sample provides implementation examples for applying authentication on your Codec Server using the Go SDK.

Sample request/response​

Consider the following sample request/response when creating and hosting a Codec Server with the following specifications:

  • Scheme: https
  • Host: dev.mydomain.com/codec
  • Path: /decode
HTTP/1.1 POST /decode
Host: https://dev.mydomain.com/codec
Content-Type: application/json
X-Namespace: myapp-dev.acctid123
Authorization: Bearer <token>

{"payloads":[{"metadata":{"encoding":"anNvbi9wcm90b2J1Zg==","messageType":"dGVtcG9yYWxfc2hvcC5vcmNoZXN0cmF0aW9ucy52MS5TdGFydFNob3BwaW5nQ2FydFJlcXVlc3Q="},"data":"eyJjYXJ0SWQiOiJleGFtcGxlLWNhcnQiLCJzaG9wcGVySWQiOiJ5b3VyLXNob3BwZXItaWQtZXhhbXBsZSIsImVtYWlsIjoieW91ci1lbWFpbEBkb21haW4uY29tIn0"}]}

200 OK
Content-Type: application/json

{
  "payloads": [{
    "metadata":{
      "encoding": "json/protobuf",
      "messageType": "temporal_shop.orchestrations.v1.StartShoppingCartRequest"
    },
    "data":{
      "cartId":"example-cart",
      "shopperId":"your-shopper-id-example",
      "email":"your-email@domain.com"
    }}]
}

Hosting your Codec Server​

Your Codec Server can be hosted at an organization level or locally.

Organization-level hosting​

Hosting the Codec Server for your organization simplifies both key management used for decryption and versioning the codec itself. Consider the following details for a multi-tenant approach to setting up your Codec Server:

  • Ingress: Your server will require ingress configuration for your users to access the server.
  • Authorization: You must set explicit authorization checks to validate requests to your Codec Server.

Local hosting​

Locally hosting the Codec Server is simpler to get started. However, consider the following before choosing to do so:

  • A single URL configuration is accepted for the Cloud account. This means some agreed-upon policy on the URL must be made for everyone using the Namespaces in this account. For example, if you configure your remote codec endpoint to be http://localhost:8080/codec, every developer must host your Codec Server locally at that port.

    Alternatively, you can use the local hosts file to allow each developer to choose where to host. For example, configure the remote codec endpoint as http://codec.server and allow each developer to control what it maps to locally.

  • Distributing encryption keys that can decrypt the payloads at your organization can be a security risk.

Set your Codec Server endpoints with Web UI and CLI​

After you create your Codec Server and expose the requisite endpoints, set the endpoints in your Web UI and CLI.

CLI​

After the Codec Server is started, provide the exposed endpoint to CLI using the --codec_endpoint command option.

For example, if you are running your Codec Server locally and expose port 8888 as your endpoint, run the following command to set the codec endpoint globally.

temporal env set --codec-endpoint "http://localhost:8888"

If your codec endpoint is not set globally, use the --codec-endpoint option with your CLI commands. For example, to see the decoded output of the Workflow Execution "yourWorkflow" in the Namespace "yourNamespace", run the following command.

temporal --codec-endpoint "http://localhost:8888" --namespace "yourNamespace" workflow show --workflow-id "yourWorkflow"  --run-id "<yourRunId>" --output "table"

For details, see the CLI reference.

Web UI​

On Temporal Cloud and self-hosted Temporal Clusters, you can set the codec endpoints in the Web UI.

Codec Server endpoint setting

Codec Server endpoint setting

In the top-right corner on the Web UI, select Configure Codec Server. In the codec endpoint dialog, enter the URL and port number for your codec endpoint. This sets the codec endpoint on the currently selected Namespace. Refresh your Workflow Execution page to see encoded/decoded data.

In self-hosted Temporal Clusters where you set up your UI Server, you can also set the codec endpoint in the UI server configuration file. Specify the codec endpoint in the UI server configuration fileLink preview iconTemporal Web UI configuration reference

The Temporal Web UI Server uses a configuration file for many of the UI's settings.

Learn more as shown in the following example.

codec:
    endpoint: {{ default .Env.TEMPORAL_CODEC_ENDPOINT "{namespace}"}}

Start the UI server to use this endpoint on the Web UI for decoding data in Workflow Executions in the specified Namespace.