How do I create a multi-class Support Vector Machine

It is impossable to create a Support Vector Machine with more than one output value. This is inherent in the way that SVM’s are defined. Unlike neural networks, which have multiple output neurons, a SVM always has one single output. However, this does not mean that you cannot do multi-class classification with a SVM. Multi-class classification is the usual reason for having multiple output neurons in a neural network.

To do this with an SVM your input must still be the regular normalized input values that an SVM typically needs. Input should always be normalized to 0 to 1. However, your output is totally un-normalized. Think of your output as a class number. Zero is your first class, one is your second, up to however many classes you actually have. Do not use decimal numbers. You cannot have class 1.5. Because the output of a SVM is a double number you must encode your class numbers as integer doubles, i.e. 0.0, 1.0, 2.0, 3.0 etc… as many as you need.

The following program is an example of this in C#.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Encog.ML.SVM;
using Encog.ML.Data;
using Encog.ML.Data.Basic;
using Encog.ML.Train;
using Encog.ML.SVM.Training;

namespace MultiClassSVM
{
    class Program
    {
        /// 

        /// Input for function, normalized to 0 to 1.
        /// 

        public static double[][] ClassificationInput = {
            new[] {0.0, 0.0},
            new[] {0.1, 0.0},
            new[] {0.2, 0.0},
            new[] {0.3, 0.0},
            new[] {0.4, 0.5},
            new[] {0.5, 0.5},
            new[] {0.6, 0.5},
            new[] {0.7, 0.5},
            new[] {0.8, 0.5},
            new[] {0.9, 0.5}
            };

        /// 

        /// Ideal output, these are class numbers, a total of four classes here (0,1,2,3).
        /// DO NOT USE FRACTIONAL CLASSES (i.e. there is no class 1.5)
        /// 

        public static double[][] ClassificationIdeal = {
            new[] {0.0},
            new[] {0.0},
            new[] {0.0},
            new[] {0.0},
            new[] {1.0},
            new[] {1.0},
            new[] {2.0},
            new[] {2.0},
            new[] {3.0},
            new[] {3.0}
        };

        static void Main(string[] args)
        {
            // create a neural network, without using a factory
            var svm = new SupportVectorMachine(2, false); // 2 input, & false for classification

            // create training data
            IMLDataSet trainingSet = new BasicMLDataSet(ClassificationInput, ClassificationIdeal);

            // train the SVM
            IMLTrain train = new SVMSearchTrain(svm, trainingSet);

            int epoch = 1;

            do
            {
                train.Iteration();
                Console.WriteLine(@"Epoch #" + epoch + @" Error:" + train.Error);
                epoch++;
            } while (train.Error > 0.01);

            // test the SVM
            Console.WriteLine(@"SVM Results:");
            foreach (IMLDataPair pair in trainingSet)
            {
                IMLData output = svm.Compute(pair.Input);
                Console.WriteLine(pair.Input[0]
                                  + @", actual=" + output[0] + @",ideal=" + pair.Ideal[0]);
            }

            Console.WriteLine("Done");
        }
    }
}

This will produce the following output.

1
2
3
4
5
6
7
8
9
10
11
12
Epoch #1 Error:0
SVM Results:
0, actual=0,ideal=0
0,1, actual=0,ideal=0
0,2, actual=0,ideal=0
0,3, actual=0,ideal=0
0,4, actual=1,ideal=1
0,5, actual=1,ideal=1
0,6, actual=2,ideal=2
0,7, actual=2,ideal=2
0,8, actual=3,ideal=3
0,9, actual=3,ideal=3