triplet loss article (#27)

* triplet loss article

* add main file

ARTICLE_CHECLLIST.md

@@ -7,6 +7,10 @@ What to prepare before publishing a new article?
 * [ ] - Small preview icon - similar to [this](./qdrant-landing/static/articles_data/neural-search-tutorial/tutorial.svg)
     * transparent BG
     * White color only
+    * Find icon - https://www.flaticon.com/
+    * Download png (for free)
+    * Convert with - https://www.freeconvert.com/png-to-svg/
+    * Change color to white, background to transparent
 
 
 ## Publish to

qdrant-landing/content/articles/filtrable-hnsw.md

@@ -5,7 +5,7 @@ description: How to make ANN search with custom filtering? Search in selected su
 external_link: https://blog.vasnetsov.com/posts/categorical-hnsw/
 preview_image: /articles_data/filtrable-hnsw/preview.png
 small_preview_image: /articles_data/filtrable-hnsw/global-network.svg
-weight: 30
+weight: 60
 author: Andrei Vasnetsov
 author_link: https://blog.vasnetsov.com/
 ---

qdrant-landing/content/articles/neural-search-tutorial.md

@@ -5,7 +5,7 @@ description: Our step-by-step guide on how to build a neural search service with
 external_link: https://blog.qdrant.tech/neural-search-tutorial-3f034ab13adc
 preview_image: /articles_data/neural-search-tutorial/preview.png
 small_preview_image: /articles_data/neural-search-tutorial/tutorial.svg
-weight: 10
+weight: 50
 author: Andrei Vasnetsov
 author_link: https://blog.vasnetsov.com/
 ---

qdrant-landing/content/articles/triplet-loss.md

@@ -0,0 +1,299 @@
+---
+title: Triplet Loss - Advanced Intro
+short_description: "What are the advantages of Triplet Loss and how to efficiently implement it?"
+description: "What are the advantages of Triplet Loss over Contrastive loss and how to efficiently implement it?"
+preview_image: /articles_data/triplet-loss/preview.png
+small_preview_image: /articles_data/triplet-loss/icon.svg
+weight: 10
+author: Yusuf Sarıgöz
+author_link: https://medium.com/@yusufsarigoz
+date: 2022-03-24T15:12:00+03:00
+---
+
+## What is Triplet Loss?
+
+Triplet Loss was first introduced in [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/abs/1503.03832) in 2015,
+and it has been one of the most popular loss functions for supervised similarity or metric learning ever since.
+In its simplest explanation, Triplet Loss encourages that dissimilar pairs be distant from any similar pairs by at least a certain margin value.
+Mathematically, the loss value can be calculated as
+$L=max(d(a,p) - d(a,n) + m, 0)$, where:
+
+- $p$, i.e., positive, is a sample that has the same label as $a$, i.e., anchor,
+- $n$, i.e., negative, is another sample that has a label different from $a$,
+- $d$ is a function to measure the distance between these three samples,
+- and $m$ is a margin value to keep negative samples far apart.
+
+The paper uses Euclidean distance, but it is equally valid to use any other distance metric, e.g., cosine distance.
+
+The function has a learning objective that can be visualized as in the following:
+
+{{< figure src=/articles_data/triplet-loss/loss_objective.png caption="Triplet Loss learning objective" >}}
+
+Notice that Triplet Loss does not have a side effect of urging to encode anchor and positive samples into the same point
+in the vector space as in Contrastive Loss.
+This lets Triplet Loss tolerate some intra-class variance, unlike Contrastive Loss,
+as the latter forces the distance between an anchor and any positive essentially to $0$.
+In other terms, Triplet Loss allows to stretch clusters in such a way as to include outliers
+while still ensuring a margin between samples from different clusters, e.g., negative pairs.
+
+Additionally, Triplet Loss is less greedy. Unlike Contrastive Loss,
+it is already satisfied when different samples are easily distinguishable from similar ones. It does not change the distances in a positive cluster if
+there is no interference from negative examples.
+This is due to the fact that Triplet Loss tries to ensure a margin between distances of negative pairs and distances of positive pairs.
+However, Contrastive Loss takes into account the margin value only when comparing dissimilar pairs,
+and it does not care at all where similar pairs are at that moment.
+This means that Contrastive Loss may reach a local minimum earlier,
+while Triplet Loss may continue to organize the vector space in a better state.
+
+Let's demonstrate how two loss functions organize the vector space by animations.
+For simpler visualization, the vectors are represented by points in a 2-dimensional space,
+and they are selected randomly from a normal distribution.
+
+{{< figure src=/articles_data/triplet-loss/contrastive.gif caption="Animation that shows how Contrastive Loss moves points in the course of training." >}}
+
+{{< figure src=/articles_data/triplet-loss/triplet.gif caption="Animation that shows how Triplet Loss moves points in the course of training." >}}
+
+
+From mathematical interpretations of the two-loss functions, it is clear that Triplet Loss is theoretically stronger,
+but Triplet Loss has additional tricks that help it work better.
+Most importantly, Triplet Loss introduce online triplet mining strategies, e.g., automatically forming the most useful triplets.
+
+## Why triplet mining matters?
+
+The formulation of Triplet Loss demonstrates that it works on three objects at a time:
+
+- `anchor`,
+- `positive` - a sample that has the same label as the anchor,
+- and `negative` - a sample with a different label from the anchor and the positive.
+
+In a naive implementation, we could form such triplets of samples at the beginning of each epoch
+and then feed batches of such triplets to the model throughout that epoch. This is called "offline strategy."
+However, this would not be so efficient for several reasons:
+- It needs to pass $3n$ samples to get a loss value of $n$ triplets.
+- Not all these triplets will be useful for the model to learn anything, e.g., yielding a positive loss value.
+- Even if we form "useful" triplets at the beginning of each epoch with one of the methods that I will be implementing in this series,
+they may become "useless" at some point in the epoch as the model weights will be constantly updated.
+
+Instead, we can get a batch of $n$ samples and their associated labels,
+and form triplets on the fly. That is called "online strategy." Normally, this gives
+$n^3$ possible triplets, but only a subset of such possible triplets will be actually valid. Even in this case,
+we will have a loss value calculated from much more triplets than the offline strategy.
+
+Given a triplet of `(a, p, n)`, it is valid only if:
+
+- `a` and `p` has the same label,
+- `a` and `p` are distinct samples,
+- and `n` has a different label from `a` and `p`.
+
+These constraints may seem to be requiring expensive computation with nested loops,
+but it can be efficiently implemented with tricks such as distance matrix, masking, and broadcasting.
+The rest of this series will focus on the implementation of these tricks.
+
+
+## Distance matrix
+
+A distance matrix is a matrix of shape $(n, n)$ to hold distance values between all possible
+pairs made from items in two $n$-sized collections.
+This matrix can be used to vectorize calculations that would need inefficient loops otherwise.
+Its calculation can be optimized as well, and we will implement [Euclidean Distance Matrix Trick (PDF)](https://www.robots.ox.ac.uk/~albanie/notes/Euclidean_distance_trick.pdf)
+explained by Samuel Albanie. You may want to read this three-page document for
+the full intuition of the trick, but a brief explanation is as follows:
+
+- Calculate the dot product of two collections of vectors, e.g., embeddings in our case.
+- Extract the diagonal from this matrix that holds the squared Euclidean norm of each embedding.
+- Calculate the squared Euclidean distance matrix based on the following equation: $||a - b||^2 = ||a||^2  - 2 ⟨a, b⟩ + ||b||^2$
+- Get the square root of this matrix for non-squared distances.
+
+We will implement it in PyTorch, so let's start with imports.
+
+
+```python
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+eps = 1e-8 # an arbitrary small value to be used for numerical stability tricks
+```
+
+---
+
+```python
+def euclidean_distance_matrix(x):
+  """Efficient computation of Euclidean distance matrix
+
+  Args:
+    x: Input tensor of shape (batch_size, embedding_dim)
+    
+  Returns:
+    Distance matrix of shape (batch_size, batch_size)
+  """
+  # step 1 - compute the dot product
+
+  # shape: (batch_size, batch_size)
+  dot_product = torch.mm(x, x.t())
+
+  # step 2 - extract the squared Euclidean norm from the diagonal
+
+  # shape: (batch_size,)
+  squared_norm = torch.diag(dot_product)
+
+  # step 3 - compute squared Euclidean distances
+
+  # shape: (batch_size, batch_size)
+  distance_matrix = squared_norm.unsqueeze(0) - 2 * dot_product + squared_norm.unsqueeze(1)
+
+  # get rid of negative distances due to numerical instabilities
+  distance_matrix = F.relu(distance_matrix)
+
+  # step 4 - compute the non-squared distances
+  
+  # handle numerical stability
+  # derivative of the square root operation applied to 0 is infinite
+  # we need to handle by setting any 0 to eps
+  mask = (distance_matrix == 0.0).float()
+
+  # use this mask to set indices with a value of 0 to eps
+  distance_matrix += mask * eps
+
+  # now it is safe to get the square root
+  distance_matrix = torch.sqrt(distance_matrix)
+
+  # undo the trick for numerical stability
+  distance_matrix *= (1.0 - mask)
+
+  return distance_matrix
+```
+
+## Invalid triplet masking
+
+Now that we can compute a distance matrix for all possible pairs of embeddings in a batch,
+we can apply broadcasting to enumerate distance differences for all possible triplets and represent them in a tensor of shape `(batch_size, batch_size, batch_size)`.
+However, only a subset of these $n^3$ triplets are actually valid as I mentioned earlier,
+and we need a corresponding mask to compute the loss value correctly.
+We will implement such a helper function in three steps:
+
+- Compute a mask for distinct indices, e.g., `(i != j and j != k)`.
+- Compute a mask for valid anchor-positive-negative triplets, e.g., `labels[i] == labels[j] and labels[j] != labels[k]`.
+- Combine two masks.
+
+
+```python
+def get_triplet_mask(labels):
+  """compute a mask for valid triplets
+
+  Args:
+    labels: Batch of integer labels. shape: (batch_size,)
+
+  Returns:
+    Mask tensor to indicate which triplets are actually valid. Shape: (batch_size, batch_size, batch_size)
+    A triplet is valid if:
+    `labels[i] == labels[j] and labels[i] != labels[k]`
+    and `i`, `j`, `k` are different.
+  """
+  # step 1 - get a mask for distinct indices
+
+  # shape: (batch_size, batch_size)
+  indices_equal = torch.eye(labels.size()[0], dtype=torch.bool, device=labels.device)
+  indices_not_equal = torch.logical_not(indices_equal)
+  # shape: (batch_size, batch_size, 1)
+  i_not_equal_j = indices_not_equal.unsqueeze(2)
+  # shape: (batch_size, 1, batch_size)
+  i_not_equal_k = indices_not_equal.unsqueeze(1)
+  # shape: (1, batch_size, batch_size)
+  j_not_equal_k = indices_not_equal.unsqueeze(0)
+  # Shape: (batch_size, batch_size, batch_size)
+  distinct_indices = torch.logical_and(torch.logical_and(i_not_equal_j, i_not_equal_k), j_not_equal_k)
+
+  # step 2 - get a mask for valid anchor-positive-negative triplets
+
+  # shape: (batch_size, batch_size)
+  labels_equal = labels.unsqueeze(0) == labels.unsqueeze(1)
+  # shape: (batch_size, batch_size, 1)
+  i_equal_j = labels_equal.unsqueeze(2)
+  # shape: (batch_size, 1, batch_size)
+  i_equal_k = labels_equal.unsqueeze(1)
+  # shape: (batch_size, batch_size, batch_size)
+  valid_indices = torch.logical_and(i_equal_j, torch.logical_not(i_equal_k))
+
+  # step 3 - combine two masks
+  mask = torch.logical_and(distinct_indices, valid_indices)
+
+  return mask
+```
+
+## Batch-all strategy for online triplet mining
+
+Now we are ready for actually implementing Triplet Loss itself.
+Triplet Loss involves several strategies to form or select triplets, and the simplest one is
+to use all valid triplets that can be formed from samples in a batch.
+This can be achieved in four easy steps thanks to utility functions we've already implemented:
+
+- Get a distance matrix of all possible pairs that can be formed from embeddings in a batch.
+- Apply broadcasting to this matrix to compute loss values for all possible triplets.
+- Set loss values of invalid or easy triplets to $0$.
+- Average the remaining positive values to return a scalar loss.
+
+I will start by implementing this strategy, and more complex ones will follow as separate posts.
+
+
+```python
+class BatchAllTtripletLoss(nn.Module):
+  """Uses all valid triplets to compute Triplet loss
+
+  Args:
+    margin: Margin value in the Triplet Loss equation
+  """
+  def __init__(self, margin=1.):
+    super().__init__()
+    self.margin = margin
+    
+  def forward(self, embeddings, labels):
+    """computes loss value.
+
+    Args:
+      embeddings: Batch of embeddings, e.g., output of the encoder. shape: (batch_size, embedding_dim)
+      labels: Batch of integer labels associated with embeddings. shape: (batch_size,)
+
+    Returns:
+      Scalar loss value.
+    """
+    # step 1 - get distance matrix
+    # shape: (batch_size, batch_size)
+    distance_matrix = euclidean_distance_matrix(embeddings)
+
+    # step 2 - compute loss values for all triplets by applying broadcasting to distance matrix
+
+    # shape: (batch_size, batch_size, 1)
+    anchor_positive_dists = distance_matrix.unsqueeze(2)
+    # shape: (batch_size, 1, batch_size)
+    anchor_negative_dists = distance_matrix.unsqueeze(1)
+    # get loss values for all possible n^3 triplets
+    # shape: (batch_size, batch_size, batch_size)
+    triplet_loss = anchor_positive_dists - anchor_negative_dists + self.margin
+
+    # step 3 - filter out invalid or easy triplets by setting their loss values to 0
+
+    # shape: (batch_size, batch_size, batch_size)
+    mask = get_triplet_mask(labels)
+    triplet_loss *= mask
+    # easy triplets have negative loss values
+    triplet_loss = F.relu(triplet_loss)
+
+    # step 4 - compute scalar loss value by averaging positive losses
+    num_positive_losses = (triplet_loss > eps).float().sum()
+    triplet_loss = triplet_loss.sum() / (num_positive_losses + eps)
+
+    return triplet_loss
+```
+
+## Conclusion
+
+I mentioned that Triplet Loss is different from Contrastive Loss not only mathematically but also in its sample selection strategies, and I implemented the batch-all strategy for online triplet mining in this post
+efficiently by using several tricks.
+
+There are other more complicated strategies such as batch-hard and batch-semihard mining,
+but their implementations, and discussions of the tricks I used for efficiency in this post,
+are worth separate posts of their own.
+
+The future posts will cover such topics and additional discussions on some tricks
+to avoid vector collapsing and control intra-class and inter-class variance.

qdrant-landing/static/articles_data/triplet-loss/triplet-loss.svg

@@ -0,0 +1,416 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="99.286148mm"
+   height="30.70155mm"
+   viewBox="0 0 99.28615 30.70155"
+   version="1.1"
+   id="svg5"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   sodipodi:docname="triplet-loss.svg"
+   inkscape:export-filename="/home/generall/projects/vector_search/landing_page/qdrant-landing/static/articles_data/triplet-loss/triplet-loss.png"
+   inkscape:export-xdpi="250"
+   inkscape:export-ydpi="250"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg">
+  <sodipodi:namedview
+     id="namedview7"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="1"
+     inkscape:pagecheckerboard="0"
+     inkscape:document-units="mm"
+     showgrid="false"
+     inkscape:snap-midpoints="true"
+     inkscape:snap-object-midpoints="true"
+     inkscape:snap-smooth-nodes="true"
+     inkscape:object-paths="true"
+     fit-margin-left="3"
+     fit-margin-top="3"
+     fit-margin-right="3"
+     fit-margin-bottom="3"
+     inkscape:zoom="3.073513"
+     inkscape:cx="172.44111"
+     inkscape:cy="51.244292"
+     inkscape:window-width="1920"
+     inkscape:window-height="1048"
+     inkscape:window-x="0"
+     inkscape:window-y="32"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="layer1" />
+  <defs
+     id="defs2">
+    <marker
+       style="overflow:visible"
+       id="DotM"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="DotM"
+       inkscape:isstock="true">
+      <path
+         transform="matrix(0.4,0,0,0.4,2.96,0.4)"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
+         d="m -2.5,-1 c 0,2.76 -2.24,5 -5,5 -2.76,0 -5,-2.24 -5,-5 0,-2.76 2.24,-5 5,-5 2.76,0 5,2.24 5,5 z"
+         id="path2194" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="Arrow2Lend"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Lend"
+       inkscape:isstock="true">
+      <path
+         transform="matrix(-1.1,0,0,-1.1,-1.1,0)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:0.625;stroke-linejoin:round"
+         id="path2151" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="Arrow1Lend"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow1Lend"
+       inkscape:isstock="true">
+      <path
+         transform="matrix(-0.8,0,0,-0.8,-10,0)"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         id="path2133" />
+    </marker>
+    <inkscape:path-effect
+       effect="bspline"
+       id="path-effect17235"
+       is_visible="true"
+       lpeversion="1"
+       weight="33.333333"
+       steps="2"
+       helper_size="0"
+       apply_no_weight="true"
+       apply_with_weight="true"
+       only_selected="false" />
+    <marker
+       style="overflow:visible"
+       id="Arrow2Mstart"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Mstart"
+       inkscape:isstock="true">
+      <path
+         transform="scale(0.6)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:0.625;stroke-linejoin:round"
+         id="path2154" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="Arrow2Mend"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Mend"
+       inkscape:isstock="true">
+      <path
+         transform="scale(-0.6)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:0.625;stroke-linejoin:round"
+         id="path2157" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="Arrow1Mend"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow1Mend"
+       inkscape:isstock="true">
+      <path
+         transform="matrix(-0.4,0,0,-0.4,-4,0)"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         id="path2139" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="Arrow2Mend-7"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Mend"
+       inkscape:isstock="true">
+      <path
+         transform="scale(-0.6)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:0.625;stroke-linejoin:round"
+         id="path2157-0" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="Arrow2Mend-3"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Mend"
+       inkscape:isstock="true">
+      <path
+         transform="scale(-0.6)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:0.625;stroke-linejoin:round"
+         id="path2157-6" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="Arrow2Mend-3-6"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Mend"
+       inkscape:isstock="true">
+      <path
+         transform="scale(-0.6)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:0.625;stroke-linejoin:round"
+         id="path2157-6-2" />
+    </marker>
+  </defs>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-28.092545,-8.1742327)">
+    <path
+       style="fill:none;stroke:#808080;stroke-width:0.2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend)"
+       d="M 34.853943,29.01652 55.021498,21.158923"
+       id="path2048"
+       sodipodi:nodetypes="cc" />
+    <path
+       style="fill:none;stroke:#808080;stroke-width:0.2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend-3)"
+       d="M 82.794838,25.534269 118.14165,21.676582"
+       id="path2048-0"
+       sodipodi:nodetypes="cc" />
+    <path
+       style="fill:none;stroke:#808080;stroke-width:0.2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend-3-6)"
+       d="m 82.377168,25.523264 21.042492,4.940634"
+       id="path2048-0-6"
+       sodipodi:nodetypes="cc" />
+    <path
+       style="fill:none;stroke:#808080;stroke-width:0.2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#Arrow2Mend-7)"
+       d="m 35.393858,29.134453 35.100587,2.766881"
+       id="path2048-9"
+       sodipodi:nodetypes="cc" />
+    <path
+       style="fill:#ffffff;stroke:#182b3a;stroke-width:2;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;paint-order:stroke fill markers"
+       id="path1552"
+       sodipodi:type="arc"
+       sodipodi:cx="34.853943"
+       sodipodi:cy="29.01652"
+       sodipodi:rx="0.70527649"
+       sodipodi:ry="0.70527649"
+       sodipodi:start="0"
+       sodipodi:end="6.2808117"
+       sodipodi:open="true"
+       sodipodi:arc-type="chord"
+       d="m 35.559219,29.01652 a 0.70527649,0.70527649 0 0 1 -0.704858,0.705276 0.70527649,0.70527649 0 0 1 -0.705694,-0.704439 0.70527649,0.70527649 0 0 1 0.70402,-0.706113 0.70527649,0.70527649 0 0 1 0.70653,0.703601 z" />
+    <path
+       style="fill:#ffffff;stroke:#800000;stroke-width:2;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;paint-order:stroke fill markers"
+       id="path1552-6"
+       sodipodi:type="arc"
+       sodipodi:cx="56.485584"
+       sodipodi:cy="20.525818"
+       sodipodi:rx="0.70527649"
+       sodipodi:ry="0.70527649"
+       sodipodi:start="0"
+       sodipodi:end="6.2808117"
+       sodipodi:open="true"
+       sodipodi:arc-type="chord"
+       d="m 57.190861,20.525818 a 0.70527649,0.70527649 0 0 1 -0.704858,0.705276 0.70527649,0.70527649 0 0 1 -0.705695,-0.704439 0.70527649,0.70527649 0 0 1 0.704021,-0.706113 0.70527649,0.70527649 0 0 1 0.70653,0.703602 z" />
+    <path
+       style="fill:#ffffff;stroke:#008000;stroke-width:2;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;paint-order:stroke fill markers"
+       id="path1552-2"
+       sodipodi:type="arc"
+       sodipodi:cx="72.245514"
+       sodipodi:cy="32.06588"
+       sodipodi:rx="0.70527649"
+       sodipodi:ry="0.70527649"
+       sodipodi:start="0"
+       sodipodi:end="6.2808117"
+       sodipodi:open="true"
+       sodipodi:arc-type="chord"
+       d="m 72.95079,32.06588 a 0.70527649,0.70527649 0 0 1 -0.704858,0.705276 0.70527649,0.70527649 0 0 1 -0.705694,-0.704439 0.70527649,0.70527649 0 0 1 0.70402,-0.706113 0.70527649,0.70527649 0 0 1 0.70653,0.703602 z" />
+    <path
+       style="fill:#ffffff;stroke:#182b3a;stroke-width:2;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;paint-order:stroke fill markers"
+       id="path1552-9"
+       sodipodi:type="arc"
+       sodipodi:cx="82.377167"
+       sodipodi:cy="25.523264"
+       sodipodi:rx="0.70527649"
+       sodipodi:ry="0.70527649"
+       sodipodi:start="0"
+       sodipodi:end="6.2808117"
+       sodipodi:open="true"
+       sodipodi:arc-type="chord"
+       d="m 83.082443,25.523264 a 0.70527649,0.70527649 0 0 1 -0.704858,0.705276 0.70527649,0.70527649 0 0 1 -0.705694,-0.704439 0.70527649,0.70527649 0 0 1 0.70402,-0.706112 0.70527649,0.70527649 0 0 1 0.70653,0.703601 z" />
+    <path
+       style="fill:#ffffff;stroke:#800000;stroke-width:2;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;paint-order:stroke fill markers"
+       id="path1552-6-1"
+       sodipodi:type="arc"
+       sodipodi:cx="119.84409"
+       sodipodi:cy="21.496456"
+       sodipodi:rx="0.70527649"
+       sodipodi:ry="0.70527649"
+       sodipodi:start="0"
+       sodipodi:end="6.2808117"
+       sodipodi:open="true"
+       sodipodi:arc-type="chord"
+       d="m 120.54937,21.496456 a 0.70527649,0.70527649 0 0 1 -0.70486,0.705277 0.70527649,0.70527649 0 0 1 -0.70569,-0.70444 0.70527649,0.70527649 0 0 1 0.70402,-0.706112 0.70527649,0.70527649 0 0 1 0.70653,0.703601 z" />
+    <path
+       style="fill:#ffffff;stroke:#008000;stroke-width:2;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;paint-order:stroke fill markers"
+       id="path1552-2-2"
+       sodipodi:type="arc"
+       sodipodi:cx="105.04742"
+       sodipodi:cy="30.971603"
+       sodipodi:rx="0.70527649"
+       sodipodi:ry="0.70527649"
+       sodipodi:start="0"
+       sodipodi:end="6.2808117"
+       sodipodi:open="true"
+       sodipodi:arc-type="chord"
+       d="m 105.75269,30.971603 a 0.70527649,0.70527649 0 0 1 -0.70485,0.705277 0.70527649,0.70527649 0 0 1 -0.7057,-0.70444 0.70527649,0.70527649 0 0 1 0.70402,-0.706112 0.70527649,0.70527649 0 0 1 0.70653,0.703601 z" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:0.1;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:0.8, 0.4;stroke-dashoffset:0;stroke-opacity:1;paint-order:stroke fill markers"
+       id="path4637"
+       sodipodi:type="arc"
+       sodipodi:cx="82.377167"
+       sodipodi:cy="25.523264"
+       sodipodi:rx="23.377106"
+       sodipodi:ry="23.377106"
+       sodipodi:start="6.125034"
+       sodipodi:end="0.32924291"
+       sodipodi:arc-type="arc"
+       d="m 105.46253,21.841537 a 23.377106,23.377106 0 0 1 -0.9639,11.240169"
+       sodipodi:open="true" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:0.1;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:0.8, 0.4;stroke-dashoffset:0;stroke-opacity:1;paint-order:stroke fill markers"
+       id="path4637-1"
+       sodipodi:type="arc"
+       sodipodi:cx="82.377167"
+       sodipodi:cy="25.523264"
+       sodipodi:rx="37.716255"
+       sodipodi:ry="37.716255"
+       sodipodi:start="6.1251063"
+       sodipodi:end="0.26179939"
+       sodipodi:arc-type="arc"
+       sodipodi:open="true"
+       d="m 119.62316,19.585916 a 37.716255,37.716255 0 0 1 -0.81489,15.699033" />
+    <path
+       style="fill:none;stroke:#999999;stroke-width:0.165;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:0.66, 0.165;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#Arrow2Mstart);marker-end:url(#Arrow2Mend-3)"
+       d="m 120.05841,27.148055 -14.3259,-0.616266"
+       id="path5302"
+       sodipodi:nodetypes="cc" />
+    <text
+       xml:space="preserve"
+       style="font-size:2.38125px;line-height:1;font-family:Roboto;-inkscape-font-specification:Roboto;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+       x="111.22878"
+       y="30.002396"
+       id="text8664"><tspan
+         sodipodi:role="line"
+         id="tspan8662"
+         style="font-size:2.38125px;fill:#182b3a;fill-opacity:1;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+         x="111.22878"
+         y="30.002396">Margin</tspan></text>
+    <text
+       xml:space="preserve"
+       style="font-size:2.38125px;line-height:1;font-family:Roboto;-inkscape-font-specification:Roboto;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+       x="31.058826"
+       y="33.078125"
+       id="text8664-8"><tspan
+         sodipodi:role="line"
+         id="tspan8662-7"
+         style="font-size:2.38125px;fill:#182b3a;fill-opacity:1;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+         x="31.058826"
+         y="33.078125">Anchor</tspan></text>
+    <text
+       xml:space="preserve"
+       style="font-size:2.38125px;line-height:1;font-family:Roboto;-inkscape-font-specification:Roboto;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+       x="78.582047"
+       y="29.303215"
+       id="text8664-8-5"><tspan
+         sodipodi:role="line"
+         id="tspan8662-7-9"
+         style="font-size:2.38125px;fill:#182b3a;fill-opacity:1;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+         x="78.582047"
+         y="29.303215">Anchor</tspan></text>
+    <text
+       xml:space="preserve"
+       style="font-size:2.38125px;line-height:1;font-family:Roboto;-inkscape-font-specification:Roboto;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+       x="51.754478"
+       y="24.634073"
+       id="text8664-8-9"><tspan
+         sodipodi:role="line"
+         id="tspan8662-7-2"
+         style="font-size:2.38125px;fill:#182b3a;fill-opacity:1;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+         x="51.754478"
+         y="24.634073">Negative</tspan></text>
+    <text
+       xml:space="preserve"
+       style="font-size:2.38125px;line-height:1;font-family:Roboto;-inkscape-font-specification:Roboto;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+       x="115.11298"
+       y="18.607304"
+       id="text8664-8-9-8"><tspan
+         sodipodi:role="line"
+         id="tspan8662-7-2-9"
+         style="font-size:2.38125px;fill:#182b3a;fill-opacity:1;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+         x="115.11298"
+         y="18.607304">Negative</tspan></text>
+    <text
+       xml:space="preserve"
+       style="font-size:2.38125px;line-height:1;font-family:Roboto;-inkscape-font-specification:Roboto;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+       x="67.984726"
+       y="35.852528"
+       id="text8664-8-9-0"><tspan
+         sodipodi:role="line"
+         id="tspan8662-7-2-2"
+         style="font-size:2.38125px;fill:#182b3a;fill-opacity:1;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+         x="67.984726"
+         y="35.852528">Positive</tspan></text>
+    <text
+       xml:space="preserve"
+       style="font-size:2.38125px;line-height:1;font-family:Roboto;-inkscape-font-specification:Roboto;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+       x="100.78663"
+       y="34.937653"
+       id="text8664-8-9-0-2"><tspan
+         sodipodi:role="line"
+         id="tspan8662-7-2-2-2"
+         style="font-size:2.38125px;fill:#182b3a;fill-opacity:1;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+         x="100.78663"
+         y="34.937653">Positive</tspan></text>
+    <text
+       xml:space="preserve"
+       style="font-size:2.38125px;line-height:1;font-family:Roboto;-inkscape-font-specification:Roboto;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+       x="70.18811"
+       y="15.524105"
+       id="text8664-8-9-0-3"><tspan
+         sodipodi:role="line"
+         id="tspan8662-7-2-2-7"
+         style="font-size:2.38125px;fill:#182b3a;fill-opacity:1;stroke-width:0.265;stroke-miterlimit:4;stroke-dasharray:none"
+         x="70.18811"
+         y="15.524105">LEARNING</tspan></text>
+    <path
+       style="fill:none;stroke:#182b3a;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#DotM);marker-end:url(#Arrow2Lend)"
+       d="m 64.508198,17.570871 c 0,-2.16772 0,-4.33566 3.673611,-5.429083 3.673611,-1.093422 11.020136,-1.111975 14.693483,-0.02783 3.673346,1.084145 3.673346,3.27064 3.673346,5.456912"
+       id="path17233"
+       inkscape:path-effect="#path-effect17235"
+       inkscape:original-d="m 64.508198,17.570871 c 2.65e-4,-2.16772 2.65e-4,-4.33566 0,-6.503951 7.347223,-0.01829 14.693748,-0.03684 22.04044,-0.05566 2.64e-4,2.186844 2.64e-4,4.373339 0,6.559611"
+       sodipodi:nodetypes="cccc" />
+  </g>
+</svg>