Deploying with the HTTP Service

In this tutorial we will deploy a dataflow(ffmpeg dataflow) which converts a video to gif over an HTTP service. We’ll also see how to deploy the same in a docker container. Finally in Serving the DataFlow we’ll setup another HTTP service which waits on GitHub webhooks to rebuilt and deploy the ffmpeg dataflow.

Note

All the code for this example is located under the examples/ffmpeg directory of the DFFML source code.

We’ll be using additional plugins from dffml, dffml-config-yaml and dffml-http-service.

$ pip install dffml-config-yaml dffml-http-service

Writing the function

We’ll first write the function to convert videos files to gif by calling ffmpeg (Make sure you download and install it) in operation.py. The function accepts bytes (of the video), converts it to gif and outputs it as bytes.

operations.py

import asyncio
import tempfile


async def convert_to_gif(input_file: bytes, resolution: int) -> bytes:
    temp_input_file = tempfile.NamedTemporaryFile(prefix="ffmpeg-")
    temp_input_file.write(input_file)
    temp_input_file.seek(0)
    proc = await asyncio.create_subprocess_exec(
        "ffmpeg",
        "-ss",
        "0.3",
        "-t",
        "10",
        "-i",
        temp_input_file.name,
        "-y",
        "-vf",
        f"fps=10,scale={resolution}:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse",
        "-loop",
        "0",
        "-f",
        "gif",
        "pipe:1",
        stdout=asyncio.subprocess.PIPE,
    )
    out, error = await proc.communicate()
    temp_input_file.close()
    return {"output_file": out}

Dataflow and Config files

DFFML can create a dataflow out of our python function.

$ dffml dataflow create -config yaml operations:convert_to_gif get_single \
    -inputs 480=operations.convert_to_gif.inputs.resolution \
    "['operations.convert_to_gif.outputs.result']=get_single_spec" \
     > deploy/df/ffmpeg.yaml

Here through the inputs argument, we set the default resolution to 480 and output of the dataflow to result of convert_to_gif.

Create the config file for the HTTP service in deploy/mc/http/ffmpeg.yaml

$ cat > ./deploy/mc/http/ffmpeg.yaml <<EOF
path: /ffmpeg
input_mode: bytes:operations.convert_to_gif.inputs.input_file
output_mode: bytes:image/gif:post_input."operations.convert_to_gif.outputs.result".output_file
EOF

  • input_mode

    • bytes:operations.convert_to_gif.inputs.input_file

      • We want the input from the request to be treated as bytes with definition operations.convert_to_gif.inputs.input_file.

  • output_mode

    • bytes:image/gif:post_input."operations.convert_to_gif.outputs.result".output_file

      • We want the response (content_type = image/gif) to be bytes taken from results["post_input"]["operations.convert_to_gif.outputs.result"][output_file], where results is the output of the dataflow.

For more details see HttpChannelConfig .

Serving the DataFlow

Serving the dataflow on port 8080

$ dffml service http server -insecure -mc-config deploy -port 8080

Warning

The -insecure flag is only being used here to speed up this tutorial. See documentation on HTTP API Security for more information.

Now from another terminal, we can send post requests to the dataflow running at this port.

$ curl -v --request POST --data-binary @input.mp4 http://localhost:8080/ffmpeg -o output.gif

You should replace input.mp4 with path to your video file and output.gif to where you want the converted gif to be output to. An example video is available here .

Deploying via container

Dockerfile

# Usage
# docker build -t $USER/ffmpeg .
# docker run --rm -ti -p 8080:8080 $USER/ffmpeg -mc-config deploy -insecure -log debug
#
# curl -v --request POST --data-binary @input.mp4 http://localhost:8080/ffmpeg -o output.gif
FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
  apt-get install -y \
    gcc \
    python3-dev \
    python3-pip \
    python3 \
    ca-certificates \
    ffmpeg && \
  python3 -m pip install -U pip && \
  python3 -m pip install dffml-service-http && \
  python3 -m pip install dffml-config-yaml && \
  apt-get purge -y \
    gcc \
    python3-dev && \
  rm -rf /var/lib/apt/lists/*

WORKDIR /usr/src/app
COPY . /usr/src/app

ENTRYPOINT ["python3", "-m", "dffml", "service", "http", "server", "-addr", "0.0.0.0"]
CMD ["-mc-config", "deploy"]

Note

The run command in the comment section of the Dockerfile will be used to execute the container after receving webhooks, so make sure you change it to your usecase.

For this tutorial we will set it to

# docker run --rm -ti -p 8080:8080 $USER/ffmpeg -mc-config deploy -insecure -log debug

Note

The image built after pulling the contaier will be taged USERNAME/REPONAME, where USERNAME and REPONAME are gathered from the github html url, received in the webhook.

We can run the container and sent a post request to verify that the container is working.

$ docker build -t $USER/ffmpeg .
$ docker run --rm -ti -p 8080:8080 $USER/ffmpeg -mc-config deploy -insecure -log debug

Now in another terminal

$ curl -v --request POST --data-binary @input.mp4 http://localhost:8080/ffmpeg -o output.gif

Now in Redeploying on receiving GitHub webhook we’ll setup this container to be automatically redeployed whenever we push to the Git repo containing this code.