Version: 0.1.4

Run Function Mesh

Function Mesh is a collection of functions collaborated together to process data with clearly-defined stages on Kubernetes environments.

This document describes how to run Function Mesh through a .yaml file and submit Pulsar connectors and Pulsar functions through the Function Mesh CRD.

Prerequisites

Steps

To create a Function Mesh, follow these steps.

  1. Define a Function Mesh named functionmesh-sample by using a YAML file and save the YAML file functionmesh-sample.yaml.

    apiVersion: compute.functionmesh.io/v1alpha1
    kind: FunctionMesh
    metadata:
    name: functionmesh-sample
    spec:
    sinks:
    name: sink-sample
    image: streamnative/pulsar-io-elastic-search:2.7.1 # using connector image here
    className: org.apache.pulsar.io.elasticsearch.ElasticSearchSink
    replicas: 1
    maxReplicas: 1
    input:
    topics:
    - persistent://public/default/input
    typeClassName: "[B"
    sinkConfig:
    elasticSearchUrl: "http://quickstart-es-http.default.svc.cluster.local:9200"
    indexName: "my_index"
    typeName: "doc"
    username: "elastic"
    password: "wJ757TmoXEd941kXm07Z2GW3"
    pulsar:
    pulsarConfig: "test-sink"
    resources:
    limits:
    cpu: "0.2"
    memory: 1.1G
    requests:
    cpu: "0.1"
    memory: 1G
    java:
    jar: connectors/pulsar-io-elastic-search-2.7.1.nar # the NAR location in image
    jarLocation: "" # leave empty since we will not download package from Pulsar Packages
    clusterName: test-pulsar
    autoAck: true
    functions:
    name: ex1
    className: exclamation_function.ExclamationFunction
    replicas: 1
    maxReplicas: 1
    logTopic: persistent://public/default/logging-function-log
    input:
    topics:
    - persistent://public/default/functionmesh-input-topic
    typeClassName: java.lang.String
    output:
    topic: persistent://public/default/mid-topic
    typeClassName: java.lang.String
    pulsar:
    pulsarConfig: "mesh-test-pulsar"
    java:
    jar: pulsar-functions-api-examples.jar
    jarLocation: public/default/nlu-test-functionmesh-ex1
    # following value must be provided if no auto-filling is enabled
    forwardSourceMessageProperty: true
    autoAck: true
    resources:
    requests:
    cpu: "0.1"
    memory: 1G
    limits:
    cpu: "0.2"
    memory: 1.1G
    clusterName: test-pulsar
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: mesh-test-pulsar
    data:
    webServiceURL: http://test-pulsar-broker.default.svc.cluster.local:8080
    brokerServiceURL: pulsar://test-pulsar-broker.default.svc.cluster.local:6650
  2. Apply the YAML file to create the Function Mesh.

    kubectl apply -f /path/to/functionmesh-sample.yaml
  3. Check whether the Function Mesh is created successfully.

    kubectl get all