Version: 0.10.0

Package Go Functions

After developing and testing your Pulsar function, you need to package it so that the function can be submitted to a Pulsar cluster. This document describes how to package a Go function to an external package or an image.

Function packages

This section describes how to package a Go function and upload it to the Pulsar package management service.

Prerequisites

Apache Pulsar 2.8.0 or higher
Function Mesh v0.1.3 or higher

Steps

To package a function in Go, follow these steps.

Write a Go function.

Currently, Go functions can be implemented using SDK only and the interface of the function is exposed in the form of SDK. Before using a Go function, you need to import github.com/apache/pulsar/pulsar-function-go/pf.

import (
  "context"
  "fmt"

  "github.com/apache/pulsar/pulsar-function-go/pf"
)

func HandleRequest(ctx context.Context, input []byte) error {
  fmt.Println(string(input) + "!")
  return nil
}

func main() {
  pf.Start(HandleRequest)
}

When writing a Go function, remember that

In main(), you only need to register the function name to Start(). Only one function name is received in Start().

The Go function uses Go reflection, which is based on the received function name, to verify whether the parameter list and returned value list are correct. The parameter list and returned value list must be one of the following sample functions:

 func ()
 func () error
 func (input) error
 func () (output, error)
 func (input) (output, error)
 func (context.Context) error
 func (context.Context, input) error
 func (context.Context) (output, error)
 func (context.Context, input) (output, error)

You can use the context to connect to the Go function.

if fc, ok := pf.FromContext(ctx); ok {
  fmt.Printf("function ID is:%s, ", fc.GetFuncID())
  fmt.Printf("function version is:%s\n", fc.GetFuncVersion())
}

Build the Go function.

go build <your Go Function filename>.go 

Upload Go function packages

Use the pulsar-admin CLI tool to upload the package to the Pulsar package management service.

Note
Before uploading the package to the Pulsar package management service, you need to enable the package management service in the broker.config file.

This example shows how to upload the package of the my-function@0.1 function to the Pulsar package management service.

bin/pulsar-admin packages upload function://my-tenant/my-ns/my-function@0.1 --path "/path/to/package-file" --description PACKAGE_DESCRIPTION

Then, you can define a function CRD by specifying the uploaded Python function package.

Docker images

This section describes how to package a Go function to a Docker image.

Prerequisites

Apache Pulsar 2.7.0 or higher
Function Mesh v0.1.3 or higher
Install Docker. Download the Community edition and follow the instructions for your OS.
Install kubectl.

Build a Docker image

To build a Docker image, follow these steps.

Package your Go function. For details, see function packages.

Define a Dockerfile.

This example shows how to define a Dockerfile with a JAR package (example-function.jar) of the Go function.

# Use pulsar-functions-go-runner since we pack Go function
FROM streamnative/pulsar-functions-go-runner:2.7.1
# Copy function JAR package into /pulsar directory  
COPY example-function.jar /pulsar/

Then, you can push the Docker image to an image registry (such as Docker Hub, or any private registry) and use the Docker image to configure and submit the function to a Pulsar cluster.

Self-built images

This section describes how to create a Go function image by using Buildpacks.

User flow

To create a Go function image, you need to go through the following steps:

Create a Build image.
Create a Run image.
Create a buildpack.
Create a Builder.

Prerequisites

Apache Pulsar 2.7.0 or higher
Function Mesh v0.1.3 or higher
Install the Pack CLI tools.
Install Docker. Download the Community edition and follow the instructions for your OS.
Install kubectl.

Create a Build image

This example shows how to create a Build image.

Define a Dockerfile.

This example defines a Dockerfile with the Stack ID (io.functionmesh.stack).

FROM ubuntu:20.04

ARG pulsar_uid=10000
ARG pulsar_gid=10001
ARG stack_id="io.functionmesh.stack"

RUN apt-get update && \
  apt-get install -y xz-utils ca-certificates git wget jq gcc && \
  rm -rf /var/lib/apt/lists/* && \
  wget -O /usr/local/bin/yj https://github.com/bruceadams/yj/releases/download/v1.2.2/yj.linux.x86_64 && \
  chmod +x /usr/local/bin/yj

LABEL io.buildpacks.stack.id=${stack_id}

RUN groupadd pulsar --gid ${pulsar_gid} && \
  useradd --uid ${pulsar_uid} --gid ${pulsar_gid} -m -s /bin/bash pulsar

ENV CNB_USER_ID=${pulsar_uid}
ENV CNB_GROUP_ID=${pulsar_gid}
ENV CNB_STACK_ID=${stack_id}

USER ${CNB_USER_ID}:${CNB_GROUP_ID}

Run the following command to create the Build image.

docker build -t fm-stack-build:v1 -f ./stack.build.Dockerfile .

Create a Run image

This example shows how to create a Run image.

Define a Dockerfile.

This example uses the streamnative/pulsar-functions-go-runner:2.9.2.23 runner image as the base image. You can specify a specific base image based on your requirements.

FROM streamnative/pulsar-functions-go-runner:2.9.2.23

ARG pulsar_uid=10000
ARG pulsar_gid=10001
ARG stack_id="io.functionmesh.stack"
LABEL io.buildpacks.stack.id=${stack_id}

ENV CNB_USER_ID=${pulsar_uid}
ENV CNB_GROUP_ID=${pulsar_gid}
ENV CNB_STACK_ID=${stack_id}

Run the following command to create the Run image.

docker build -t fm-stack-go-runner-run:v1 -f ./stack.go-runner.run.Dockerfile .

Create a buildpack

This example shows how to create a buildpack with the buildpack ID functionmesh/golang.

pack buildpack new functionmesh/golang \
  --api 0.7 \
  --path golang \
  --version 0.0.1 \
  --stacks io.functionmesh.stack

Then, a buildpack directory named golang is created.

`-- golang
  |-- bin
  |   |-- build
  |   `-- detect
  `-- buildpack.toml

Update the content of the three files with the following content.

buildpack.toml

api = "0.7"
[buildpack]
  id = "functionmesh/golang"
  version = "0.0.1"
[[stacks]]
  id = "io.functionmesh.stack"

bin/detect

#!/usr/bin/env bash
set -eo pipefail
go_num=$(find . -maxdepth 1 -name "*.go" | wc -l)
if [[ ${go_num} -eq 0 ]]; then
  echo "no Go files found"
  exit 100
fi
go_mod_num=$(find . -maxdepth 1 -name "go.mod" | wc -l)
if [[ ${go_mod_num} -eq 0 ]]; then
  echo "no go.mod found"
  exit 100
fi
exit 0

bin/build

#!/usr/bin/env bash
set -euo pipefail
layers_dir="$1"
env_dir="$2/env"
plan_path="$3"
# LOAD USER-PROVIDED BUILD-TIME ENVIRONMENT VARIABLES
if compgen -G "${env_dir}/*" > /dev/null; then
  for var in ${env_dir}/*; do
    declare "$(basename ${var})=$(<${var})"
  done
fi
# FUNCTION_TARGET=<function name>
if [[ -z ${FUNCTION_TARGET} ]]; then
  echo "need to set FUNCTION_TARGET"
  exit 100
fi
# Download Go runtime, default version is 1.18
go_url="https://go.dev/dl/go1.18.5.linux-amd64.tar.gz"
go_version="1.18.5"
cached_go_url=""
go_layer_dir=${layers_dir}/go
if [[ -f ${go_layer_dir}.toml ]]; then
  cached_go_url=$(cat "${go_layer_dir}.toml" | yj -t | jq -r .metadata.url 2>/dev/null || echo 'GO TOML parsing failed')
fi
if [[ ${go_url} != ${cached_go_url} ]] ; then
  rm -rf "$layers_dir"/go
  mkdir -p "$layers_dir"/go/env
  wget -q -O - "$go_url" | tar pxz -C "${go_layer_dir}" --strip-components=1
  export PATH=$PATH:${go_layer_dir}/bin
  # here we use the function-runner image as the run image, so we set the `launch = false`
  cat > "${go_layer_dir}.toml" << EOF
[types]
launch = false
build = true
cache = true
[metadata]
version = "${go_version}"
url = "${go_url}"
EOF
  echo "${go_layer_dir}" > "$layers_dir"/go/env/GOROOT
fi
# Set env variables to make jdk accessible
for var in "$layers_dir"/go/env/*; do
  declare "$(basename "$var")=$(<"$var")"
done
export PATH=${go_layer_dir}/bin:$PATH
# build golang binary
mkdir -p target
go mod tidy
go build -o target/go_function "${FUNCTION_TARGET}"
# clear source
target_dir="target"
ls | grep -v ${target_dir} | xargs rm -rf
mv ${target_dir}/go_function .
rm -rf ${target_dir}
exit 0

Create a Builder image

This section describes how to create a Builder image.

Define a Builder configuration file (builder.toml) with the following content.

# Buildpacks to include in builder
[[buildpacks]]
uri = "../../buildpacks/golang"
# Order used for detection
[[order]]
  # This buildpack will display build-time information (as a dependency)
  [[order.group]]
  id = "functionmesh/golang"
  version = "0.0.1"
# Stack that will be used by the builder
[stack]
id = "io.functionmesh.stack"
# This image is used at runtime
run-image = "fm-stack-go-runner-run:v1"
# This image is used at build-time
build-image = "fm-stack-build:v1"

Run the following command to create the Builder image.

pack builder create fm-golang-builder:v1 \
  --config ./builder.toml \
  --pull-policy if-not-present

Create a Go function image

After creating the Build image (fm-stack-build:v1), the Run image (fm-stack-go-runner-run:v1), and the Builder image (fm-golang-builder:v1), you can build a Go function image and then use the Go function image to run a Go function.

The package directory structure is similar to:

.
|-- go.mod
`-- exclamation_function.go

Prepare a Go function file.

This example writes a Go function named exclamation_function.go.

package main
import (
  "context"
  "fmt"
  "github.com/apache/pulsar/pulsar-function-go/pf"
)
func HandleRequest(ctx context.Context, input []byte) error {
  fmt.Println(string(input) + "!")
  return nil
}
func main() {
  pf.Start(HandleRequest)
}

Run the following command to build the Go function image in the current directory.

pack build golang-exclamation-function:v1 \
  --builder fm-golang-builder:v1 \
  --workspace /pulsar \
  --pull-policy if-not-present \
  --env FUNCTION_TARGET="exclamation_function.go"

The output is similar to:

===> ANALYZING
[analyzer] Restoring data for SBOM from previous image
===> DETECTING
[detector] functionmesh/golang 0.0.1
===> RESTORING
[restorer] Restoring metadata for "functionmesh/golang:go" from cache
[restorer] Restoring data for "functionmesh/golang:go" from cache
===> BUILDING
[builder] go: downloading github.com/apache/pulsar/pulsar-function-go v0.0.0-20220823033039-423ab7542521
[builder] go: downloading github.com/apache/pulsar-client-go v0.8.1
[builder] go: downloading github.com/golang/protobuf v1.5.2
[builder] go: downloading github.com/prometheus/client_golang v1.11.1
[builder] go: downloading github.com/prometheus/client_model v0.2.0
...
...
...
[builder] go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c
[builder] go: downloading github.com/keybase/go-keychain v0.0.0-20190712205309-48d3d31d256d
[builder] go: downloading github.com/mitchellh/go-homedir v1.1.0
[builder] go: downloading github.com/mtibben/percent v0.2.1
[builder] go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
[builder] go: downloading google.golang.org/appengine v1.6.7
[builder] go: downloading github.com/nxadm/tail v1.4.4
[builder] go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
[builder] go: downloading github.com/fsnotify/fsnotify v1.4.9
[builder] go: downloading github.com/ardielle/ardielle-go v1.5.2
[builder] go: downloading github.com/dimfeld/httptreemux v5.0.1+incompatible
===> EXPORTING
[exporter] Reusing layer 'launch.sbom'
[exporter] Reusing 1/1 app layer(s)
[exporter] Reusing layer 'launcher'
[exporter] Reusing layer 'config'
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] no default process type
[exporter] Saving golang-exclamation-function:v1...
[exporter] *** Images (2d1c29ce58dd):
[exporter]       golang-exclamation-function:v1
[exporter] Reusing cache layer 'functionmesh/golang:go'
Successfully built image golang-exclamation-function:v1

Next step

Run Go Functions

Function packages​

Prerequisites​

Steps​

Upload Go function packages​

Docker images​

Prerequisites​

Build a Docker image​

Self-built images​

User flow​

Prerequisites​

Create a Build image​

Create a Run image​

Create a buildpack​

Create a Builder image​

Create a Go function image​

Next step​

Function packages

Prerequisites

Steps

Upload Go function packages

Docker images

Prerequisites

Build a Docker image

Self-built images

User flow

Prerequisites

Create a Build image

Create a Run image

Create a buildpack

Create a Builder image

Create a Go function image

Next step