API

Here’s an example of using the API from JavaScript.

Here’s an example of using the API from Python (with requests).

Service

Various APIs specific to this service rather than DFFML are under this path.

Upload

  • /service/upload/{filename}

Upload a file to the -upload-dir specified when starting the server. You can make the filename include a path, but the server won’t create any directories that don’t already exist.

{"error": null}

If no -upload-dir was given on the command line, an error will be returned. The HTTP status code will be 501, Not Implemented.

{"error": "Uploads not allowed"}

If you attempt to use ../../ or anything which would cause a path outside of the given -upload-dir this endpoint will return an error. The HTTP status code will be 400, Bad Request.

{"error": "Attempted path traversal"}

If your multipart data doesn’t have a field with the name of file, you will get an error. The HTTP status code will be 400, Bad Request.

{"error": "Missing 'file' field"}

Files

  • /service/files

List files in the -upload-dir specified when starting the server. The file listing will include the size of the file in bytes.

[
  {
    "filename": ".gitignore",
    "size": 199
  },
  {
    "filename": ".dockerignore",
    "size": 18
  },
  {
    "filename": "setup.py",
    "size": 2337
  },
  {
    "filename": ".coveragerc",
    "size": 170
  }
]

If no -upload-dir was given on the command line, an error will be returned. The HTTP status code will be 501, Not Implemented.

{"error": "File listing not allowed"}

List

  • /list/{plugin_type}

List APIs return JSON objects where the keys are the names of the loadable classes for a given type of DFFML plugin. The values are that plugin’s configuration options.

Current supported DFFML plugins are as follows.

  • sources

  • models

To list available plugins, send a GET request to the endpoint.

The following is an example response body for a request to list available sources.

  • /list/sources

{
  "csv": {
    "source": {
      "plugin": null,
      "config": {
        "csv": {
          "plugin": null,
          "config": {
            "filename": {
              "plugin": {},
              "config": {}
            },
            "readwrite": {
              "plugin": {
                "type": "bool",
                "action": "store_true",
                "default": false
              },
              "config": {}
            },
            "allowempty": {
              "plugin": {
                "type": "bool",
                "action": "store_true",
                "default": false
              },
              "config": {}
            },
            "label": {
              "plugin": {
                "type": "str",
                "default": "unlabeled"
              },
              "config": {}
            },
            "key": {
              "plugin": {
                "type": "str",
                "default": null
              },
              "config": {}
            }
          }
        }
      }
    }
  }
}

Configure

  • /configure/{dffml plugin type}/{plugin name}/{label}

The configure API allows for creation of instances of DFFML plugin types. Callers supply the type of plugin to instantiate, the name of that plugin, and then label it will be assigned when using it.

Configuration options can be found in the docs for the various plugins or via the List endpoint.

To configure a plugin, send a POST request to the endpoint containing only the JSON object to be used as the configuration of the requested plugin.

On successful creation and configuration the server will return null for error.

{"error": null}

If the plugin name requested is not loadable the server will return a HTTP status code of 404, Not Found.

{"error": "source non-existant not found"}

If there is a problem with configuration the server will tell the client. The HTTP status code will be 400, Bad Request.

{"error": "CSVSource missing 'filename' from source.mydataset"}

Source

The following is an example request body to configure the csv source. The URL this POST request is sent to is.

  • /configure/source/csv/mydataset

{
  "source": {
    "plugin": null,
    "config": {
      "filename": {
        "plugin": [
          "dataset.csv"
        ],
        "config": {}
      },
      "readwrite": {
        "plugin": [
          true
        ],
        "config": {}
      }
    }
  }
}

Model

The following is an example request body to configure a model. The URL this POST request is sent to is.

  • /configure/model/fake/mymodel

{
  "model": {
    "plugin": null,
    "config": {
      "location": {
        "plugin": [
          "/home/user/modeldirs/mymodel"
        ],
        "config": {}
      },
      "features": {
        "plugin": [
          {
            "name": "Years",
            "dtype": "int",
            "length": 1
          },
          {
            "name": "Expertise",
            "dtype": "int",
            "length": 1
          },
          {
            "name": "Trust",
            "dtype": "float",
            "length": 1
          }
        ],
        "config": {}
      }
    }
  }
}

Context

After a plugin has been configured, a context must be created. The context label will be used in all requests for that plugin type, to reference which context the respective methods should be called on.

  • /context/{plugin_type}/{label}/{ctx_label}

To create a context, send a GET or POST request to the endpoint containing the JSON object to be used as the configuration parameters of the requested plugin context type.

On successful creation of a context the server will return null for error.

{"error": null}

If there is no configured plugin for the given label the server will return a HTTP status code of 404, Not Found.

{"error": "mydataset source not found"}

Source

The following is an example request body to create a source context. The URL this GET request is sent to is.

  • /context/source/mydataset/ctx_mydataset

Model

The following is an example request body to create a model context. The URL this GET request is sent to is.

  • /context/model/mymodel/ctx_mymodel

Source

  • /source/{ctx_label}/{source context method}/{...}

The source endpoint exposes all of the methods you’d find in dffml.source.BaseSourceContext. The ctx_label parameter in the URL is the label of the source context that was configured via the Configure and then the Context APIs.

If the ctx_label provided does not exist, for instance the configure and context APIs were not used prior to calling a source method, the server will return a 404, Not Found response.

{"error": "Source not loaded"}

Record

Access a record by it’s unique key. The response will be the JSON representation of the record. Here’s an example response for a GET request.

  • /source/{ctx_label}/record/{key}

{
  "key": "myrecord",
  "features": {
    "myfeature": "somevalue"
  }
}

Just as with DFFML, you’ll still get a record even if the record doesn’t exist within the source. However, it will only contain the key.

Update

Update a record by it’s unique key. POST data in the same format received from record.

  • /source/{ctx_label}/update/{key}

{
  "key": "myrecord",
  "features": {
    "myfeature": "somevalue"
  }
}

Unless something goes wrong within the source, you’ll get a null error response.

{"error": null}

Records

Initially, client makes a GET request to the API with the chunk_size for the first iteration. chunk_size is the number of records to return in one iteration. The response object will have two properties, iterkey and records.

records is a key value mapping of record key’s to their JSON serialized record object.

iterkey will be null if there are no more records in the source. If iterkey is not null then there are more records to iterate over. The API should be called using the response’s iterkey value until the response contains an iterkey value of null.

Sample response where chunk_size is 1 and there are more records to iterate over. We continue making GET requests until iterkey is null.

  • /source/{ctx_label}/records/{chunk_size}

  • /source/{ctx_label}/records/{iterkey}/{chunk_size}

{
  "iterkey": "1a164836c6d8a27fdf9cd12688440aaa16a852fd1814b170c924a89fba4e084c8ea7522c34f9f5a539803d6237238e90",
  "records": {
    "myrecord": {
      "key": "myrecord",
      "features": {
        "myfeature": "somevalue"
      }
    }
  }
}

Sample response where the end of iteration has been reached.

{
  "iterkey": null,
  "records": {
    "anotherrecord": {
      "key": "anotherrecord",
      "features": {
        "myfeature": "othervalue"
      }
    }
  }
}

Model

  • /model/{ctx_label}/{model context method}/{...}

The model endpoint exposes all of the methods you’d find in dffml.model.ModelContext. The ctx_label parameter in the URL is the label of the model context that was configured via the Configure and then the Context APIs.

If the ctx_label provided does not exist, for instance the configure and context APIs were not used prior to calling a model method, the server will return a 404, Not Found response.

{"error": "Model not loaded"}

Train

Send a POST request with the JSON body being a list of source context labels to use as training data.

  • /model/{ctx_label}/train

[
  "my_training_dataset"
]

Unless something goes wrong within the model, you’ll get a null error response.

{"error": null}

Accuracy

Send a POST request with the JSON body being a list of source context labels to use as test data.

  • /scorer/{ctx_label}/{mctx_label}/score

[
  "my_test_dataset"
]

The response will be a JSON object containing the accuracy as a float value.

{"accuracy": 0.42}

Unless something goes wrong within the model, you’ll get a null error response.

{"error": null}

Predict

To use a model for prediction, send a POST request to the following URL with the body being a JSON object mapping key of the record to the JSON representation of dffml.record.Record as received by the source record endpoint.

  • /model/{ctx_label}/predict/0

{
  "42": {
    "features": {
      "by_ten": 420
    }
  }
}

Sample response.

{
  "iterkey": null,
  "records": {
    "42": {
      "key": "42",
      "features": {
        "by_ten": 420
      },
      "prediction": {
        "confidence": 42,
        "value": 4200
      },
      "last_updated": "2019-10-15T08:19:41Z",
      "extra": {}
    }
  }
}