This document lists CRD configurations available for Pulsar source connectors. The source CRD configurations consist of source connector configurations and the common CRD configurations.
This table lists source configurations.
|The connector name is a string of up to
|The class name of a source connector.
|The tenant of a source connector.
|The Pulsar namespace of a source connector.
|The Pulsar cluster of a source connector.
|The number of instances that you want to run for this source connector. If no value is set, the system will set it to
|The minimum number of instances that you want to run for this source connector. If no value is set, the system will set it to
1. When HPA auto-scaling is enabled, the HPA controller scales the Pods up / down based on the values of the
maxReplicas options. The number of the Pods should be greater than the value of the
minReplicas and be smaller than the value of the
|The image for installing the init container that is used to download packages or functions from Pulsar if the download path is specified.
|The maximum number of instances that you want to run for this source connector. When the value of the
maxReplicas parameter is greater than the value of
replicas, it indicates that the source controller automatically scales the source connector based on the CPU usage. By default,
maxReplicas is set to 0, which indicates that auto-scaling is disabled.
|The source connector configurations in YAML format.
|The processing guarantees (delivery semantics) applied to the source connector. Available values:
|Configure whether to pass message properties to a target topic.
|The batch source configurations in YAML format. You can configure the following properties.
discoveryTriggererClassName: the class that is used for triggering the discovery process.
discoveryTriggererConfig: the configurations that are required for initiating the discovery Triggerer.
In Kubernetes, an annotation defines an unstructured Key Value Map (KVM) that can be set by external tools to store and retrieve metadata.
annotations must be a map of string keys and string values. Annotation values must pass Kubernetes annotations validation. For details, see Kubernetes documentation on Annotations.
This example shows how to use an annotation to make an object unmanaged. Therefore, the Controller will skip reconciling unmanaged objects in reconciliation loop.
This section describes image options available for Pulsar source CRDs.
The base runner is an image base for other runners. The base runner is located at
./pulsar-functions-base-runner. The base runner image contains basic tool-chains like
/pulsar/lib to ensure that the
pulsar-admin CLI tool works properly to support Apache Pulsar Packages.
Function Mesh uses runner images as images of Pulsar connectors. Each runner image only contains necessary tool-chains and libraries for specified runtime.
Pulsar connectors support using the Java runner images as their images. The Java runner is based on the base runner and contains the Java function instance to run Java functions or connectors. The
streamnative/pulsar-functions-java-runner Java runner is stored at the Docker Hub and is automatically updated to align with Apache Pulsar release.
Image pull policies
When the Function Mesh Operator creates a container, it uses the
imagePullPolicy option to determine whether the image should be pulled prior to starting the container. There are three possible values for the
|Always pull the image.
|Never pull the image.
|Only pull the image if the image does not already exist locally.
Function Mesh provides the following fields for stateful configurations in the CRD definition.
|The state storage configuration for the source connector.
|The service URL that points to the state storage service. By default, the state storage service is the BookKeeper table service.
|(Optional) If you want to overwrite the default configuration, you can use the state storage configuration for the Java runtime. For example, you can change it to other backend services other than the BookKeeper table service.
|The Java class name of the state storage provider implementation. The class must implement the
org.apache.pulsar.functions.instance.state.StateStoreProvider interface. If not,
org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl will be used.
|The configurations that are passed to the state storage provider.
The output topics of a Pulsar Function. This table lists options available for the
|The output topic of a Pulsar Function (If none is specified, no output is written).
|The map of output topics to SerDe class names (as a JSON string).
|The built-in schema type or custom schema class name to be used for messages sent by the function.
|The producer specifications. Available options:
maxPendingMessages: the maximum number of pending messages.
maxPendingMessagesAcrossPartitions: the maximum number of pending messages across partitions.
useThreadLocalProducers: configure whether the producer uses a thread.
cryptoConfig: cryptography configurations of the producer.
batchBuilder: support key-based batcher.
|The map of output topics to Schema class names (as a JSON string).
When you specify a function or connector, you can optionally specify how much of each resource they need. The resources available to specify are CPU and memory (RAM).
If the node where a Pod is running has enough of a resource available, it's possible (and allowed) for a Pod to use more resources than its
request for that resource. However, a Pod is not allowed to use more than its resource
Function Mesh provides the
secretsMap field for Function, Source, and Sink in the CRD definition. You can refer to the created secrets under the same namespace and the controller can include those referred secrets. The secrets are provide by
EnvironmentBasedSecretsProvider, which can be used by
context.getSecret() in Pulsar functions and connectors.
secretsMap field is defined as a
Map struct with
String keys and
SecretReference values. The key indicates the environment value in the container, and the
SecretReference is defined as below.
|The name of the secret in the Pod's namespace to select from.
|The key of the secret to select from. It must be a valid secret key.
Suppose that there is a Kubernetes Secret named
credential-secret defined as below:
To use it in Pulsar Functions in a secure way, you can define the
secretsMap in the Custom Resource:
Then, in the Pulsar Functions and Connectors, you can call
context.getSecret("username") to get the secret value (
Function Mesh provides the
authConfig fields for Function, Source, and Sink in the CRD definition. You can configure TLS encryption, TLS authentication, and OAuth2 authentication using the following configurations.
The TLS configurations and TLS Secret configurations are exclusive. If you configure TLS configurations, TLS Secret configurations will not take effect.
Allow insecure TLS connection.
The Secret key.
The Secret name.
Enable TLS configurations.
Enable hostname verification.
Allow insecure TLS connection. By default, it is set to
Enable hostname verification. By default, it is set to
The path of the TLS trust certificate file.
The client authentication plugin.
The client authentication parameters.
Currently, only OAuth2 authentication configurations are supported. For other authentication methods, you can configure them using the Authentication Secret.
Field Description OAuth2 authentication
The OAuth2 "resource server" identifier for the Pulsar cluster.
The URL of the authentication provider that allows a Pulsar client to obtain an access token.
The scope of an access request. For more information, see access token scope.
The name of the Kubernetes secret.
The key of the Kubernetes secret, which contains the content of the OAuth2 private key.
Function Mesh supports running Pulsar connectors in Java.
|The path to the JAR file for the connector.
|It specifies JVM options to better configure JVM behaviors, including
exitOnOOMError, Garbage Collection logs, Garbage Collection tuning, and so on.
|It specifies the dependent directory for the JAR package.
In Function Mesh, the Pulsar cluster is defined through a ConfigMap. Pods can consume ConfigMaps as environment variables in a volume. The Pulsar cluster ConfigMap defines the Pulsar cluster URLs.
|The Web service URL of the Pulsar cluster.
|The broker service URL of the Pulsar cluster.
Function Mesh supports customizing the Pod running connectors. This table lists sub-fields available for the
|Specify labels attached to a Pod.
|Specify a map of key-value pairs. For a Pod running on a node, the node must have each of the indicated key-value pairs as labels.
|Specify the scheduling constraints of a Pod.
|Specify the tolerations of a Pod.
|Specify the annotations attached to a Pod.
|Specify the security context for a Pod.
|The amount of time that Kubernetes gives for a Pod before terminating it.
|A list of volumes that can be mounted by containers belonging to a Pod.
|An optional list of references to secrets in the same namespace for pulling any of the images used by a Pod.
|Specify the name of the service account that is used to run Pulsar Functions or connectors.
|The initialization containers belonging to a Pod. A typical use case could be using an initialization container to download a remote JAR to a local path.
|Sidecar containers run together with the main function container in a Pod.
|Specify the built-in autoscaling rules.
If you configure the
|Specify how to scale based on customized metrics defined in connectors. For details, see MetricSpec v2 autoscaling.
|Configure the scaling behavior of the target in both up and down directions (
scaleDown fields respectively). If not specified, the default Kubernetes scaling behaviors are adopted. For details, see HorizontalPodAutoscalerBehavior v2 autoscaling.
|Specify the environment variables to expose on the containers. It is a key/value map. You can either use the
value option to specify a particular value for the environment variable or use the
valueFrom option to specify the source for the environment variable's value, as shown below.