Package Java Functions

After developing and testing your Pulsar function, you need to package it so that the function can be submitted to a Pulsar cluster. This document describes how to package a Java function to a NAR/JAR package or an image.

Function packages

This section describes how to package a Java function and upload it to the Pulsar package management service.

Prerequisites

• Apache Pulsar 2.8.0 or higher
• Function Mesh v0.1.3 or higher

Steps

To package a function in Java, follow these steps.

1. Create a new Maven project with a pom.xml file.

   In the following code sample, the value of mainClass is your package name.

   <?xml version="1.0" encoding="UTF-8"?>
   <project xmlns="http://maven.apache.org/POM/4.0.0"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
   
       <groupId>java-function</groupId>
       <artifactId>java-function</artifactId>
       <version>1.0-SNAPSHOT</version>
   
       <dependencies>
           <dependency>
               <groupId>org.apache.pulsar</groupId>
               <artifactId>pulsar-functions-api</artifactId>
               <version>2.6.0</version>
           </dependency>
       </dependencies>
   
       <build>
           <plugins>
               <plugin>
                   <artifactId>maven-assembly-plugin</artifactId>
                   <configuration>
                       <appendAssemblyId>false</appendAssemblyId>
                       <descriptorRefs>
                           <descriptorRef>jar-with-dependencies</descriptorRef>
                       </descriptorRefs>
                       <archive>
                       <manifest>
                           <mainClass>org.example.test.ExclamationFunction</mainClass>
                       </manifest>
                   </archive>
                   </configuration>
                   <executions>
                       <execution>
                           <id>make-assembly</id>
                           <phase>package</phase>
                           <goals>
                               <goal>assembly</goal>
                           </goals>
                       </execution>
                   </executions>
               </plugin>
               <plugin>
                   <groupId>org.apache.maven.plugins</groupId>
                   <artifactId>maven-compiler-plugin</artifactId>
                   <configuration>
                       <source>8</source>
                       <target>8</target>
                   </configuration>
               </plugin>
           </plugins>
       </build>
   
   </project>

2. Write a Java function.

   package org.example.test;
   
   import java.util.function.Function;
   
   public class ExclamationFunction implements Function<String, String> {
     @Override
     public String apply(String s) {
         return "This is my function!";
     }
   }

3. Package the Java function.

   mvn package

After the Java function is packaged, a target directory is created automatically. Open the target directory to check if there is a JAR package similar to java-function-1.0-SNAPSHOT.jar.

Upload a Java function package

Use the pulsar-admin CLI tool to upload the package to the Pulsar package management service.

Note

Before uploading the package to the Pulsar package management service, you need to enable the package management service in the broker.config file.

This example shows how to upload the package of the my-function@0.1 function to the Pulsar package management service.

bin/pulsar-admin packages upload function://my-tenant/my-ns/my-function@0.1 --path "/path/to/package-file" --description PACKAGE_DESCRIPTION

Then, you can define a function CRD by specifying the uploaded Java function package.

Docker images

This section describes how to package a Pulsar function to a Docker image.

Prerequisites

• Apache Pulsar 2.7.0 or higher
• Function Mesh v0.1.3 or higher
• Install Docker. Download the Community edition and follow the instructions for your OS.
• Install kubectl.

Build a Docker image

To build a Docker image, follow these steps.

1. Package your Pulsar function. For details, see function packages.

2. Define a Dockerfile.

   This example shows how to define a Dockerfile with a JAR package (example-function.jar) of the Java function.

   # Use pulsar-functions-java-runner since we pack Java function
   FROM streamnative/pulsar-functions-java-runner:2.7.1
   # Copy function JAR package into /pulsar directory  
   COPY example-function.jar /pulsar/

Then, you can push the Docker image to an image registry (such as Docker Hub, or any private registry) and use the Docker image to configure and submit the function to a Pulsar cluster.

Self-built images

This section describes how to create a Java function image by using Buildpacks.

User flow

To create a Java function image, you need to go through the following steps:

1. Create a Build image.
2. Create a Run image.
3. Create a buildpack.
4. Create a Builder.

Prerequisites

• Apache Pulsar 2.7.0 or higher
• Function Mesh v0.1.3 or higher
• Install the Pack CLI tools.
• Install Docker. Download the Community edition and follow the instructions for your OS.
• Install kubectl.

Create a Build image

This example shows how to create a Build image.

1. Define a Dockerfile.

   This example defines a Dockerfile with the Stack ID (io.functionmesh.stack).

   FROM ubuntu:20.04
   
   ARG pulsar_uid=10000
   ARG pulsar_gid=10001
   ARG stack_id="io.functionmesh.stack"
   
   RUN apt-get update && \
     apt-get install -y xz-utils ca-certificates git wget jq gcc && \
     rm -rf /var/lib/apt/lists/* && \
     wget -O /usr/local/bin/yj https://github.com/bruceadams/yj/releases/download/v1.2.2/yj.linux.x86_64 && \
     chmod +x /usr/local/bin/yj
   
   LABEL io.buildpacks.stack.id=${stack_id}
   
   RUN groupadd pulsar --gid ${pulsar_gid} && \
     useradd --uid ${pulsar_uid} --gid ${pulsar_gid} -m -s /bin/bash pulsar
   
   ENV CNB_USER_ID=${pulsar_uid}
   ENV CNB_GROUP_ID=${pulsar_gid}
   ENV CNB_STACK_ID=${stack_id}
   
   USER ${CNB_USER_ID}:${CNB_GROUP_ID}

2. Run the following command to create the Build image.

   docker build -t fm-stack-build:v1 -f ./stack.build.Dockerfile .

Create a Run image

This example shows how to create a Run image.

1. Define a Dockerfile.

   This example uses the streamnative/pulsar-functions-java-runner:2.9.2.23 runner image as the base image. You can specify a specific base image based on your requirements.

   FROM streamnative/pulsar-functions-java-runner:2.9.2.23
   
   ARG pulsar_uid=10000
   ARG pulsar_gid=10001
   ARG stack_id="io.functionmesh.stack"
   LABEL io.buildpacks.stack.id=${stack_id}
   
   ENV CNB_USER_ID=${pulsar_uid}
   ENV CNB_GROUP_ID=${pulsar_gid}
   ENV CNB_STACK_ID=${stack_id}

2. Run the following command to create the Run image.

   docker build -t fm-stack-java-runner-run:v1 -f ./stack.java-runner.run.Dockerfile .

Create a buildpack

This example shows how to create a buildpack with the buildpack ID functionmesh/java-maven.

pack buildpack new functionmesh/java-maven \
  --api 0.7 \
  --path java-maven \
  --version 0.0.1 \
  --stacks io.functionmesh.stack

Then, a buildpack directory named java-maven is created.

`-- java-maven
    |-- bin
    |   |-- build
    |   `-- detect
    `-- buildpack.toml

Update the content of the three files with the following content.

• buildpack.toml

  api = "0.7"
  
  [buildpack]
    id = "functionmesh/java-maven"
    version = "0.0.1"
  
  [[stacks]]
    id = "io.functionmesh.stack"

• bin/detect

  #!/usr/bin/env bash
  
  set -eo pipefail
  
  # Detect the presence of Java files here
  java_num=$(find . -name "*.java" | wc -l)
  if [[ ${java_num} -eq 0 ]]; then
    echo "no java files found"
    exit 100
  fi
  
  # Detect the presence of required items here
  requires=("pom.xml")
  require_met=false
  for r in ${requires[*]}; do 
    ls ${r}
    if [ $? -eq 0 ]; then
        require_met=true
        break
    fi
  done
  
  if [[ !${require_met} ]]; then
    echo "no required items found"
    exit 100
  fi
  
  exit 0

• bin/build

  #!/usr/bin/env bash
  
  set -eo pipefail
  
  layers_dir="$1"
  env_dir="$2/env"
  plan_path="$3"
  
  # LOAD USER-PROVIDED BUILD-TIME ENVIRONMENT VARIABLES
  if compgen -G "${env_dir}/*" > /dev/null; then
    for var in ${env_dir}/*; do
      declare "$(basename ${var})=$(<${var})"
    done
  fi
  
  # Download Java runtime, default version is jdk 17
  jdk_url="https://cdn.azul.com/zulu/bin/zulu17.36.13-ca-jdk17.0.4-linux_x64.tar.gz"
  jdk_version="1.17.0_4"
  
  maven_url="https://dlcdn.apache.org/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz"
  maven_version="3.8.6"
  
  cached_jdk_url=""
  jdk_layer_dir=${layers_dir}/jdk
  if [[ -f ${jdk_layer_dir}.toml ]]; then
    cached_jdk_url=$(cat "${jdk_layer_dir}.toml" | yj -t | jq -r .metadata.url 2>/dev/null || echo 'JDK TOML parsing failed')
  fi
  
  if [[ ${jdk_url} != ${cached_jdk_url} ]] ; then
    rm -rf "$layers_dir"/jdk
    mkdir -p "$layers_dir"/jdk/env
    wget -q -O - "$jdk_url" | tar pxz -C "${jdk_layer_dir}" --strip-components=1
  
    # here we use the function-runner image as the run image, so we set the `launch = false`
    cat > "${jdk_layer_dir}.toml" << EOF
  [types]
  launch = false
  build = true
  cache = true
  [metadata]
  version = "${jdk_version}"
  url = "${jdk_url}"
  EOF
  
      echo "$layers_dir"/jdk > "$layers_dir"/jdk/env/JAVA_HOME
      if [[ -z ${LD_LIBRARY_PATH} ]]; then
          echo "${JAVA_HOME}/jre/lib/amd64/server" > ${jdk_layer_dir}/env/LD_LIBRARY_PATH
      else
          echo "${JAVA_HOME}/jre/lib/amd64/server:${LD_LIBRARY_PATH}" > ${jdk_layer_dir}/env/LD_LIBRARY_PATH
      fi
  
      mkdir -p ${jdk_layer_dir}/profile.d
      cat > "${jdk_layer_dir}/profile.d/jdk.sh" << EOF
  export JAVA_HOME=${jdk_layer_dir}
  if [[ -z \$LD_LIBRARY_PATH ]]; then
      export LD_LIBRARY_PATH="\$JAVA_HOME/jre/lib/amd64/server"
  else
      export LD_LIBRARY_PATH="\$JAVA_HOME/jre/lib/amd64/server:${LD_LIBRARY_PATH}"
  fi
  EOF
  fi
  
  # Set env variables to make jdk accessible
  for var in "$layers_dir"/jdk/env/*; do
      declare "$(basename "$var")=$(<"$var")"
  done
  export PATH=${jdk_layer_dir}/bin:$PATH
  
  # MAKE MAVEN M2 CACHE LAYER
  m2_layer_dir="${layers_dir}/maven_m2"
  if [[ ! -d ${m2_layer_dir} ]]; then
      mkdir -p ${m2_layer_dir}
      cat > "${m2_layer_dir}.toml" << EOF
  [types]
  cache = true
  EOF
  
  fi
  ln -s ${m2_layer_dir} $HOME/.m2
  
  # RUN BUILD
  MAVEN_OPTS="${MAVEN_OPTS:-"-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"}"
  
  if [[ -x mvnw ]]; then
    echo "---> Running Maven Wrapper"
    ./mvnw clean install -B -DskipTests
  else
    maven_layer_dir=${layers_dir}/maven
    if [[ -f ${layers_dir}/maven.toml ]]; then
        cached_maven_url=$(cat "${maven_layer_dir}.toml" | yj -t | jq -r .metadata.url 2>/dev/null || echo 'Maven TOML parsing failed')
    fi
    if [[ ${maven_url} != ${cached_maven_url} ]] ; then
        echo "---> Installing Maven"
        rm -rf "${maven_layer_dir}"
        mkdir -p "${maven_layer_dir}"
        wget -q -O - "${maven_url}" | tar pxz -C "${maven_layer_dir}" --strip-components=1
        cat > "${maven_layer_dir}.toml" << EOF
  [types]
  launch = false
  build = true
  cache = true
  [metadata]
  version = "${maven_version}"
  url = "${maven_url}"
  EOF
    fi
    export PATH="${PATH}:${layers_dir}/maven/bin"
  
    echo "---> Running Maven"
    mvn clean install -B -DskipTests
  fi
  
  # clear source
  target_dir="target"
  ls | grep -v ${target_dir} | xargs rm -rf
  for jar_file in $(find "$target_dir" -maxdepth 1 -name "*.jar" -type f); do
      mv ${jar_file} .
  done
  rm -rf ${target_dir}
  
  exit 0

Create a Builder image

This section describes how to create a Builder image.

1. Define a Builder configuration file (builder.toml) with the following content.

   # Buildpacks to include in builder
   [[buildpacks]]
   uri = "../../buildpacks/java-maven"
   
   # Order used for detection
   [[order]]
     # This buildpack will display build-time information (as a dependency)
     [[order.group]]
     id = "functionmesh/java-maven"
     version = "0.0.1"
   
   # Stack that will be used by the builder
   [stack]
   id = "io.functionmesh.stack"
   # This image is used at runtime
   run-image = "fm-stack-java-runner-run:v1"
   # This image is used at build-time
   build-image = "fm-stack-build:v1"

2. Run the following command to create the Builder image.

   pack builder create fm-java-maven-builder:v1 \
     --config ./builder.toml \
     --pull-policy if-not-present

Create a Java function image

After creating the Build image ( fm-stack-build:v1), the Run image (fm-stack-java-runner-run:v1), and the Builder image (fm-java-maven-builder:v1), you can build a Java function image and then use the Java function image to run a Java function.

The package directory structure is similar to:

.
|-- pom.xml
`-- src/
    `-- main/
        `-- java/
            `-- org.example/
                `-- ExclamationFunction.java

1. Prepare a Java function file.

   This example writes a Java function named java-exclamation-function.

   package org.example;
   
   import org.apache.pulsar.functions.api.Context;
   import org.apache.pulsar.functions.api.Function;
   import org.slf4j.Logger;
   
   public class ExclamationFunction implements Function<String, String> {
     @Override
     public String process(String input, Context context) {
       Logger LOG = context.getLogger();
       LOG.debug("My exclamation function");
       return String.format("%s!", input);
     }
   }

2. Run the following command to build the Java function image in the current directory.

   pack build java-exclamation-function:v1 \
     --builder fm-java-maven-builder:v1 \
     --workspace /pulsar \
     --pull-policy if-not-present

   The output is similar to:

   ===> ANALYZING
   [analyzer] Previous image with name "java-exclamation-function:v1" not found
   ===> DETECTING
   [detector] functionmesh/java-maven 0.0.1
   ===> RESTORING
   ===> BUILDING
   [builder] ---> Installing Maven
   [builder] ---> Running Maven
   [builder] [INFO] Scanning for projects...
   [builder] [WARNING]
   [builder] [WARNING] Some problems were encountered while building the effective model for java-function:java-function:jar:1.0-SNAPSHOT
   [builder] [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 44, column 15
   [builder] [WARNING]
   [builder] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
   [builder] [WARNING]
   [builder] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
   [builder] [WARNING]
   [builder] [INFO]
   [builder] [INFO] --------------------< java-function:java-function >---------------------
   [builder] [INFO] Building java-function 1.0-SNAPSHOT
   [builder] [INFO] --------------------------------[ jar ]---------------------------------
   [builder] [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom
   [builder] [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom (3.9 kB at 3.7 kB/s)
   [builder] [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/22/maven-plugins-22.pom
   [builder] [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/22/maven-plugins-22.pom (13 kB at 53 kB/s)
   [builder] [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/21/maven-parent-21.pom
   ...
   ...
   ...
   [builder] [INFO] Installing /pulsar/target/java-function-1.0-SNAPSHOT.jar to /home/pulsar/.m2/repository/java-function/java-function/1.0-SNAPSHOT/java-function-1.0-SNAPSHOT.jar
   [builder] [INFO] Installing /pulsar/pom.xml to /home/pulsar/.m2/repository/java-function/java-function/1.0-SNAPSHOT/java-function-1.0-SNAPSHOT.pom
   [builder] [INFO] ------------------------------------------------------------------------
   [builder] [INFO] BUILD SUCCESS
   [builder] [INFO] ------------------------------------------------------------------------
   [builder] [INFO] Total time:  01:15 min
   [builder] [INFO] Finished at: 2022-08-22T12:21:48Z
   [builder] [INFO] ------------------------------------------------------------------------
   ===> EXPORTING
   [exporter] Adding layer 'launch.sbom'
   [exporter] Adding 1/1 app layer(s)
   [exporter] Adding layer 'launcher'
   [exporter] Adding layer 'config'
   [exporter] Adding label 'io.buildpacks.lifecycle.metadata'
   [exporter] Adding label 'io.buildpacks.build.metadata'
   [exporter] Adding label 'io.buildpacks.project.metadata'
   [exporter] no default process type
   [exporter] Saving java-exclamation-function:v1...
   [exporter] *** Images (f9f06af32790):
   [exporter]       java-exclamation-function:v1
   [exporter] Adding cache layer 'functionmesh/java-maven:jdk'
   [exporter] Adding cache layer 'functionmesh/java-maven:maven'
   [exporter] Adding cache layer 'functionmesh/java-maven:maven_m2'
   Successfully built image java-exclamation-function:v1

Next step

• Run Java Functions