Working with Multi-Output models¶

In this notebook, we’ll be performing multi-output tasks with the help of DFFML. DFFML makes it so that users can use a range of different models to perform multi-output tasks with the usual worklow, but instantiating models with a list of targets/labels!

Import Packages

Let us import dffml and other packages that we might need.

[1]:

import asyncio
import nest_asyncio

import numpy as np
from sklearn.datasets import load_linnerud
from sklearn.datasets import make_multilabel_classification

from dffml import *

To use asyncio in a notebook, we need to use nest_asycio.apply()

[2]:

nest_asyncio.apply()

Multi-Output Regression¶

To perform multi-output regression tasks, we can select any scikit regression model and pass our multi-output dataset onto the model.

Build our Dataset¶

For our dataset, we will be generating a random multi-output dataset

[3]:

X, y = load_linnerud(return_X_y=True)

data = np.concatenate((X, y), axis=1)

print(X.shape, y.shape)
print(data)

(20, 3) (20, 3)
[[  5. 162.  60. 191.  36.  50.]
 [  2. 110.  60. 189.  37.  52.]
 [ 12. 101. 101. 193.  38.  58.]
 [ 12. 105.  37. 162.  35.  62.]
 [ 13. 155.  58. 189.  35.  46.]
 [  4. 101.  42. 182.  36.  56.]
 [  8. 101.  38. 211.  38.  56.]
 [  6. 125.  40. 167.  34.  60.]
 [ 15. 200.  40. 176.  31.  74.]
 [ 17. 251. 250. 154.  33.  56.]
 [ 17. 120.  38. 169.  34.  50.]
 [ 13. 210. 115. 166.  33.  52.]
 [ 14. 215. 105. 154.  34.  64.]
 [  1.  50.  50. 247.  46.  50.]
 [  6.  70.  31. 193.  36.  46.]
 [ 12. 210. 120. 202.  37.  62.]
 [  4.  60.  25. 176.  37.  54.]
 [ 11. 230.  80. 157.  32.  52.]
 [ 15. 225.  73. 156.  33.  54.]
 [  2. 110.  43. 138.  33.  68.]]

[4]:

data = [
    {"x1": row[0], "x2": row[1], "x3": row[2], "y1": row[3], "y2": row[4], "y3": row[5]}
    for row in data
]
data

[4]:

[{'x1': 5.0, 'x2': 162.0, 'x3': 60.0, 'y1': 191.0, 'y2': 36.0, 'y3': 50.0},
 {'x1': 2.0, 'x2': 110.0, 'x3': 60.0, 'y1': 189.0, 'y2': 37.0, 'y3': 52.0},
 {'x1': 12.0, 'x2': 101.0, 'x3': 101.0, 'y1': 193.0, 'y2': 38.0, 'y3': 58.0},
 {'x1': 12.0, 'x2': 105.0, 'x3': 37.0, 'y1': 162.0, 'y2': 35.0, 'y3': 62.0},
 {'x1': 13.0, 'x2': 155.0, 'x3': 58.0, 'y1': 189.0, 'y2': 35.0, 'y3': 46.0},
 {'x1': 4.0, 'x2': 101.0, 'x3': 42.0, 'y1': 182.0, 'y2': 36.0, 'y3': 56.0},
 {'x1': 8.0, 'x2': 101.0, 'x3': 38.0, 'y1': 211.0, 'y2': 38.0, 'y3': 56.0},
 {'x1': 6.0, 'x2': 125.0, 'x3': 40.0, 'y1': 167.0, 'y2': 34.0, 'y3': 60.0},
 {'x1': 15.0, 'x2': 200.0, 'x3': 40.0, 'y1': 176.0, 'y2': 31.0, 'y3': 74.0},
 {'x1': 17.0, 'x2': 251.0, 'x3': 250.0, 'y1': 154.0, 'y2': 33.0, 'y3': 56.0},
 {'x1': 17.0, 'x2': 120.0, 'x3': 38.0, 'y1': 169.0, 'y2': 34.0, 'y3': 50.0},
 {'x1': 13.0, 'x2': 210.0, 'x3': 115.0, 'y1': 166.0, 'y2': 33.0, 'y3': 52.0},
 {'x1': 14.0, 'x2': 215.0, 'x3': 105.0, 'y1': 154.0, 'y2': 34.0, 'y3': 64.0},
 {'x1': 1.0, 'x2': 50.0, 'x3': 50.0, 'y1': 247.0, 'y2': 46.0, 'y3': 50.0},
 {'x1': 6.0, 'x2': 70.0, 'x3': 31.0, 'y1': 193.0, 'y2': 36.0, 'y3': 46.0},
 {'x1': 12.0, 'x2': 210.0, 'x3': 120.0, 'y1': 202.0, 'y2': 37.0, 'y3': 62.0},
 {'x1': 4.0, 'x2': 60.0, 'x3': 25.0, 'y1': 176.0, 'y2': 37.0, 'y3': 54.0},
 {'x1': 11.0, 'x2': 230.0, 'x3': 80.0, 'y1': 157.0, 'y2': 32.0, 'y3': 52.0},
 {'x1': 15.0, 'x2': 225.0, 'x3': 73.0, 'y1': 156.0, 'y2': 33.0, 'y3': 54.0},
 {'x1': 2.0, 'x2': 110.0, 'x3': 43.0, 'y1': 138.0, 'y2': 33.0, 'y3': 68.0}]

Instantiate our Model with parameters¶

Dffml makes it quite easy to load multiple models dynamically using the Model.load() function. All the entrypoints for models available in DFFML can be found at the Model Plugins Page.

Note: All(and only) Scikit Regressors and Classifiers support multioutput.

After that, you just have to parameterize the loaded model and it is ready to train!

Note that we will set predict in the model config to Features, a list of features ie. the targets, instead of a single feature. This sets the output targets in the model causing it to work as a Multi-Output model.

[5]:

ScikitRidgeModel = Model.load("scikitridge")

features = Features(
    Feature("x1", int, 1), Feature("x2", int, 1), Feature("x3", int, 1),
)
predict_features = Features(
    Feature("y1", int, 1), Feature("y2", int, 1), Feature("y3", int, 1),
)

multi_reg_model = ScikitRidgeModel(
    features=features, predict=predict_features, location="scikitridgemulti",
)
print(type(predict_features))

<class 'dffml.feature.feature.Features'>

Train our Model¶

Finally, our model is ready to be trained using the high-level API. Let’s make sure to pass each record as a parameter by simply using the unpacking operator(*).

[6]:

await train(multi_reg_model, *data)

Test our Models¶

To test our model, we’ll use the score() function in the high-level API.

Just like models, the scorers can be dynamically loaded in a similar fashion.

We will evaluate our model using the Mean Squared Error Scorer by passing "meansqrerr" to AccuracyScorer.load(), which is an entrypoint for a scikit scorer.

All the available entrypoints for scorers can be found at the Scorers Plugin Page.

[7]:

MeanSquaredError = AccuracyScorer.load("meansqrerr")

scorer = MeanSquaredError()

print("Score:", await score(multi_reg_model, scorer, predict_features, *data))

Score: 158.02450547009664

Make predictions using our Model.¶

Let’s make predictions and see what they look like for each model using the predict function in the high-level API. Note that the predict function returns an asynciterator of a Record object that contains a tuple of record.key, features and predictions.

[8]:

async for i, features, prediction in predict(multi_reg_model, *data):
    print(prediction)

{'y1': {'confidence': nan, 'value': 176.16484295610107}, 'y2': {'confidence': nan, 'value': 35.054840701238405}, 'y3': {'confidence': nan, 'value': 57.09000135700629}}
{'y1': {'confidence': nan, 'value': 188.91063061276134}, 'y2': {'confidence': nan, 'value': 37.56282467492615}, 'y3': {'confidence': nan, 'value': 54.90134303600152}}
{'y1': {'confidence': nan, 'value': 189.9534236494583}, 'y2': {'confidence': nan, 'value': 37.709031672727995}, 'y3': {'confidence': nan, 'value': 53.32601213195409}}
{'y1': {'confidence': nan, 'value': 183.12565674452532}, 'y2': {'confidence': nan, 'value': 35.757565130397566}, 'y3': {'confidence': nan, 'value': 55.37951699532168}}
{'y1': {'confidence': nan, 'value': 173.71664101067168}, 'y2': {'confidence': nan, 'value': 34.190412250345105}, 'y3': {'confidence': nan, 'value': 56.86332544362222}}
{'y1': {'confidence': nan, 'value': 188.24891042818894}, 'y2': {'confidence': nan, 'value': 37.1498878097515}, 'y3': {'confidence': nan, 'value': 55.05551171914713}}
{'y1': {'confidence': nan, 'value': 185.9832838429502}, 'y2': {'confidence': nan, 'value': 36.49249789985963}, 'y3': {'confidence': nan, 'value': 55.177648143397576}}
{'y1': {'confidence': nan, 'value': 181.88881353329703}, 'y2': {'confidence': nan, 'value': 35.85249351395624}, 'y3': {'confidence': nan, 'value': 56.125241718843995}}
{'y1': {'confidence': nan, 'value': 161.29353272098604}, 'y2': {'confidence': nan, 'value': 31.59790186825947}, 'y3': {'confidence': nan, 'value': 59.28698314880402}}
{'y1': {'confidence': nan, 'value': 168.78382349070674}, 'y2': {'confidence': nan, 'value': 35.14015097438437}, 'y3': {'confidence': nan, 'value': 55.246088161502314}}
{'y1': {'confidence': nan, 'value': 177.58499810260847}, 'y2': {'confidence': nan, 'value': 34.49820471614462}, 'y3': {'confidence': nan, 'value': 55.985845402357114}}
{'y1': {'confidence': nan, 'value': 167.04250441212406}, 'y2': {'confidence': nan, 'value': 33.56470984175445}, 'y3': {'confidence': nan, 'value': 57.49566250529646}}
{'y1': {'confidence': nan, 'value': 164.5494400870963}, 'y2': {'confidence': nan, 'value': 32.946828909890485}, 'y3': {'confidence': nan, 'value': 58.00146809453469}}
{'y1': {'confidence': nan, 'value': 201.5214637228238}, 'y2': {'confidence': nan, 'value': 39.84126139242746}, 'y3': {'confidence': nan, 'value': 52.67320665194464}}
{'y1': {'confidence': nan, 'value': 193.03036486026033}, 'y2': {'confidence': nan, 'value': 37.82070841168854}, 'y3': {'confidence': nan, 'value': 54.07885873309051}}
{'y1': {'confidence': nan, 'value': 167.98119561913504}, 'y2': {'confidence': nan, 'value': 33.84093335998922}, 'y3': {'confidence': nan, 'value': 57.34729123889391}}
{'y1': {'confidence': nan, 'value': 195.59664386575733}, 'y2': {'confidence': nan, 'value': 38.32927601046475}, 'y3': {'confidence': nan, 'value': 53.833189096823645}}
{'y1': {'confidence': nan, 'value': 160.37561584969586}, 'y2': {'confidence': nan, 'value': 32.05130196894592}, 'y3': {'confidence': nan, 'value': 59.36513951595868}}
{'y1': {'confidence': nan, 'value': 158.91979326107125}, 'y2': {'confidence': nan, 'value': 31.511817391159646}, 'y3': {'confidence': nan, 'value': 59.36551593805333}}
{'y1': {'confidence': nan, 'value': 187.3284212297808}, 'y2': {'confidence': nan, 'value': 37.08735150168848}, 'y3': {'confidence': nan, 'value': 55.402150967446254}}

Multi-Output Classifier¶

To perform multi-output classification tasks, we can select any scikit classification model and pass our multi-output dataset onto the model.

Build our Dataset¶

We again utilize sklearn to build our multi-output dataset for our classification task.

[9]:

X, y = make_multilabel_classification(n_classes=3, n_features=3, random_state=1)
data = np.concatenate((X, y), axis=1)
data = [
    {"x1": row[0], "x2": row[1], "x3": row[2], "y1": row[3], "y2": row[4], "y3": row[5]}
    for row in data
]
data[0:5]

[9]:

[{'x1': 12.0, 'x2': 11.0, 'x3': 20.0, 'y1': 1.0, 'y2': 1.0, 'y3': 0.0},
 {'x1': 21.0, 'x2': 13.0, 'x3': 17.0, 'y1': 0.0, 'y2': 0.0, 'y3': 0.0},
 {'x1': 22.0, 'x2': 13.0, 'x3': 19.0, 'y1': 1.0, 'y2': 0.0, 'y3': 0.0},
 {'x1': 23.0, 'x2': 12.0, 'x3': 20.0, 'y1': 0.0, 'y2': 0.0, 'y3': 0.0},
 {'x1': 7.0, 'x2': 18.0, 'x3': 33.0, 'y1': 1.0, 'y2': 1.0, 'y3': 0.0}]

Instantiate our Model with parameters¶

Dffml makes it quite easy to load multiple models dynamically using the Model.load() function. All the entrypoints for models available in DFFML can be found at the Model Plugins Page. After that, you just have to parameterize the loaded model and it is ready to train!

Note that we will set predict in the model config to Features, a list of features ie. the targets, instead of a single feature. This sets the output targets in the model causing it to work as a Multi-Output model.

[10]:

ScikitETCModel = Model.load("scikitetc")

features = Features(
    Feature("x1", int, 1), Feature("x2", int, 1), Feature("x3", int, 1),
)
predict_features = Features(
    Feature("y1", int, 1), Feature("y2", int, 1), Feature("y3", int, 1),
)

multi_classif_model = ScikitETCModel(
    features=features, predict=predict_features, location="scikitetcmulti",
)

Train our Model¶

Finally, our model is ready to be trained using the high-level API. Let’s make sure to pass each record as a parameter by simply using the unpacking operator(*).

[11]:

await train(multi_classif_model, *data)

Test our Models¶

To test our model, we’ll again use the score() function in the high-level API.

This time, we evaluate our model using the Accuracy Scorer by passing "acscore" to AccuracyScorer.load(), which is also an entrypoint for a scikit scorer.

All the available entrypoints for scorers can be found at the Scorers Plugin Page.

[12]:

Accuracy = AccuracyScorer.load("acscore")

scorer = Accuracy()

print("Accuracy:", await score(multi_classif_model, scorer, predict_features, *data))

Accuracy: 1.0

Make predictions using our Model.¶

Let’s make predictions and see what they look like using the predict function in the high-level API. Note that the predict function returns an asynciterator of a Record object that contains a tuple of record.key, features and predictions.

[13]:

async for i, features, prediction in predict(multi_classif_model, *data):
    print(prediction)

{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 0.0}, 'y2': {'confidence': nan, 'value': 0.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 0.0}}
{'y1': {'confidence': nan, 'value': 1.0}, 'y2': {'confidence': nan, 'value': 1.0}, 'y3': {'confidence': nan, 'value': 1.0}}