<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>immuneML: Selection Details</title>
    <style>
        {{{css_style}}}
    </style>
    <meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
    <div class="container">
        <h2>Selection for assessment split {{assessment_split}} - label {{label}}</h2>
        <p>
            In the table below, the performance for each hyperparameter setting (combination of preprocessing (if any), encoding and ML model)
            on each of validation split is shown along with the average performance for the optimization metric (if applicable).
            The hyperparameter setting chosen as optimal (the best average performance) is shown in bold.
        </p>
        <div class="padded">
            <div class="table-container">
                <table>
                    <thead>
                        <tr>
                            <th rowspan="2">Hyperparameter settings (preprocessing, encoding, ML method)</th>
                            <th colspan="{{split_count}}">Performance ({{optimization_metric}})</th>
                            {{#show_average}}
                            <th rowspan="2">Average</th>
                            {{/show_average}}
                        </tr>
                        <tr>
                            {{#splits}}
                            <th>Split {{split_index}}</th>
                            {{/splits}}
                        </tr>
                    </thead>
                    <tbody>
                        {{#hp_settings}}
                        <tr>
                            <td>
                                {{#optimal}}<b>{{/optimal}}
                                {{hp_setting}}
                                {{#optimal}}</b>{{/optimal}}
                            </td>
                            {{#hp_splits}}
                            <td>{{optimization_metric_val}}</td>
                            {{/hp_splits}}
                            {{#show_average}}
                            <td>
                                {{#optimal}}<b>{{/optimal}}
                                {{average}}
                                {{#optimal}}</b>{{/optimal}}
                            </td>
                            {{/show_average}}
                        </tr>
                        {{/hp_settings}}
                    </tbody>
                </table>
            </div>
        </div>

        <p class="padded-top">
            If the selection strategy was set to random (randomly split to training and validation datasets) and training percentage was set
            to 1 (meaning that all data will be used for training), the performance in the table above will not be computed, since no validation data
            will be available.
        </p>
        {{#has_other_metrics}}
        <p>The tables below show the performance for each hyperparameter setting across selection splits for different metrics which were not
        used for optimization.</p>
        {{#metrics}}
        <h4>{{metric}}</h4>
        <div class="table-container padded">
            {{{performance}}}
        </div>
        {{/metrics}}
        {{/has_other_metrics}}
        {{#has_data_split_reports}}
        <h2>Reports on training and validation datasets</h2>
        <p>The reports below are obtained from the datasets after they were split to training and validation datasets for each of the splits,
        before any preprocessing, encoding or model training.</p>
        <div class="padded">
            {{#data_split_reports}}
            <h3>Split {{split_index}}</h3>
            {{#train}}
            <p>Report {{name}} (training): {{info}}</p>
            <div class="grid-container">
                {{#output_figures}}
                <div class="">
                    {{#is_embed}}
                    <embed src="{{path}}">
                    {{/is_embed}}
                    {{^is_embed}}
                    <img src="{{path}}">
                    {{/is_embed}}
                    {{#name}}<p>{{name}} (training data)</p>{{/name}}
                </div>
                {{/output_figures}}
            </div>
            <ul>
            {{#output_tables}}
                <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}} (training data)</a></li>
            {{/output_tables}}
            {{#output_text}}
                <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}} (training data)</a></li>
            {{/output_text}}
            </ul>

            {{/train}}
            {{#test}}
            <p>Report {{name}} (validation): {{info}}</p>
            <div class="grid-container">
                {{#output_figures}}
                <div class="">
                    {{#is_embed}}
                    <embed src="{{path}}">
                    {{/is_embed}}
                    {{^is_embed}}
                    <img src="{{path}}">
                    {{/is_embed}}
                    {{#name}}<p>{{name}} (validation data)</p>{{/name}}
                </div>
                {{/output_figures}}
            </div>
            <ul>
                {{#output_tables}}
                    <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}} (validation data)</a></li>
                {{/output_tables}}
                {{#output_text}}
                    <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}} (validation data)</a></li>
                {{/output_text}}
            </ul>
            {{/test}}
            {{/data_split_reports}}
        </div>
        {{/has_data_split_reports}}

        {{#has_reports_per_setting}}
        <h2>Reports per hyperparameter setting</h2>
        <p>The reports below are obtained for each split to training and validation datasets separately, after encoding or training a machine learning model,
        as noted below.</p>
        {{#reports_per_setting}}
        <div class="padded">
            <h3>Hyperparameter setting: {{hp_setting}}</h3>
            <hr>
            {{#reports}}
            <h4>Split {{split_index}}</h4>
                {{#has_encoding_train_reports}}
                <h5>Encoded training data reports</h5>
                {{/has_encoding_train_reports}}
                {{#encoding_train_reports}}
                <p>Report {{name}} (training): {{info}}</p>
                {{#output_figures}}
                <div class="padded center">
                    {{#is_embed}}
                    <embed src="{{path}}">
                    {{/is_embed}}
                    {{^is_embed}}
                    <img src="{{path}}">
                    {{/is_embed}}
                    <p>{{name}}</p>
                </div>
                {{/output_figures}}
                <ul>
                {{#output_tables}}
                <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}} (encoded training data)</a></li>
                {{/output_tables}}
                {{#output_text}}
                <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}} (encoded training data)</a></li>
                {{/output_text}}
                </ul>
                {{/encoding_train_reports}}

                {{#has_encoding_test_reports}}
                <h5>Encoded validation data reports</h5>
                {{/has_encoding_test_reports}}
                {{#encoding_test_reports}}
                <p>Report {{name}} (validation): {{info}}</p>
                {{#output_figures}}
                <div class="padded center">
                    {{#is_embed}}
                    <embed src="{{path}}">
                    {{/is_embed}}
                    {{^is_embed}}
                    <img src="{{path}}">
                    {{/is_embed}}
                    <p>{{name}}</p>
                </div>
                {{/output_figures}}
                <ul>
                {{#output_tables}}
                <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}} (encoded validation data)</a></li>
                {{/output_tables}}
                {{#output_text}}
                <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}} (encoded validation data)</a></li>
                {{/output_text}}
                </ul>
                {{/encoding_test_reports}}

                {{#has_ml_reports}}
                <h5>ML model reports</h5>
                {{/has_ml_reports}}
                {{#ml_reports}}
                <p>Report {{name}}: {{info}}</p>
                <div class="grid-container">
                {{#output_figures}}
                <div class="">
                    {{#is_embed}}
                    <embed src="{{path}}">
                    {{/is_embed}}
                    {{^is_embed}}
                    <img src="{{path}}">
                    {{/is_embed}}
                    <p>{{name}}</p>
                </div>
                {{/output_figures}}
                </div>
                <ul>
                {{#output_tables}}
                <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}}</a></li>
                {{/output_tables}}
                {{#output_text}}
                <li><a href="{{path}}">Download {{#name}}{{name}} report{{/name}}</a></li>
                {{/output_text}}
                </ul>
                {{/ml_reports}}

            {{/reports}}
        </div>
        {{/reports_per_setting}}
        {{/has_reports_per_setting}}
    </div>
</body>
</html>