From 251487b39e9d418ec70d0b49eddf01ac13d2e41b Mon Sep 17 00:00:00 2001 From: George Date: Fri, 17 Jun 2022 23:47:54 +0300 Subject: [PATCH] FAQ demo (#36) * new: update content * review * refactoring: speak on behalf of a person, not a company * new: add more links * new: add baseline results * review n2 * allow preview * fix: fix code snippet * add author link, description and short description * new: add dataset-wise metrics for trained model * fix: review fixes * new: replace process_results with configure_metrics, add evaluators * refactoring: replace metric model with similarity model * fix: call train, remove redundant imports * add preview and icon images Co-authored-by: Andrey Vasnetsov --- .../articles/faq-question-answering.md | 646 +++++++++--------- .../faq-question-answering/icon.svg | 8 + .../faq-question-answering/preview.png | Bin 0 -> 25226 bytes 3 files changed, 319 insertions(+), 335 deletions(-) create mode 100644 qdrant-landing/static/articles_data/faq-question-answering/icon.svg create mode 100644 qdrant-landing/static/articles_data/faq-question-answering/preview.png diff --git a/qdrant-landing/content/articles/faq-question-answering.md b/qdrant-landing/content/articles/faq-question-answering.md index 0be0ffcfb..f2ad4b8cd 100644 --- a/qdrant-landing/content/articles/faq-question-answering.md +++ b/qdrant-landing/content/articles/faq-question-answering.md @@ -1,140 +1,133 @@ --- -sitemapExclude: True +title: Q&A with Similarity Learning +short_description: A complete guide to building a Q&A system with similarity learning. +description: A complete guide to building a Q&A system using Quaterion and SentenceTransformers. +preview_image: /articles_data/faq-question-answering/preview.png +small_preview_image: /articles_data/faq-question-answering/icon.svg +weight: 9 +author: George Panchuk +author_link: https://medium.com/@george.panchuk +date: 2022-06-28T08:57:07.604Z --- -# Question-answering system with Metric learning and Quaterion - -There are vast amount of tasks in modern machine learning which are being decided as classification -tasks. Some of them are naturally classification tasks and some of them were converged to such, it -is not a rare case. Seeing a new problem, the most of us start with an endeavour to solve it with -an approach we already applied to other tasks? It is a well-known way, it probably might be -implemented with less effort and so on. The main drawback of such an approach is that it is a part -of supervised learning. - -Supervised learning assumes that in our dataset besides data objects themselves we also have labels -determining to which class our objects belong to. There are more unlabelled data in the wild, -rather labelled one and data labelling might be a really time-consuming and expensive task, since -modern models requires huge amount of data. - -Well, today we are going to solve a problem without a classification approach. We will build a -question-answering system which makes its decisions based on distances between questions and -answers. A technique which solves problems by analysing distances between objects is metric -learning. - -Metric learning might be used both in supervised and unsupervised learning. Today we are going to -work with the latter one. Our dataset contains questions and answers downloaded from F.A.Q. pages -of popular cloud providers, such as AWS, GCP, etc. - -It's a common practice in NLP to represent texts as a collection of high-dimensional vectors, also -called _embeddings_. And to measure distance between embeddings one can just compute cosine or -euclidean distances. Then nothing prevents us from training a model which is capable of placing -embeddings from the same question-answer pair close in space and embeddings from different pairs -far from each other. Then the nearest to question embedding will be it's right answer. -And that's it! Also, we don't need any classes and might be able to add new questions and answers -without re-training. - -We haven't discussed yet how to obtain embeddings and how to train a model. - -As embeddings is not a novel technique in NLP, there are already plenty of algorithms and models -which can provide suitable representations. There are two general ways to train a model with -embeddings. One is when you use them as features: apply some model, get embeddings and use them -as input for your model with a task-specific architecture to emit whatever output you want. This is -so called _feature-based_ approach, and some of well-known algorithms used here are Word2Vec, GloVe, -ELMo. - -Quaterion, however, encourages another approach – _fine-tuning_. In fine-tuning you take a -model pre-trained on another task (e.g. to achieve a general language understanding), apply a -couple of layers on top of it and tune parameters which you need during training. In such an -approach you don't need to generate embeddings explicitly, model's hidden states could be used as -ones. You also don't need to build a task-specific architecture. In our case we usually just apply -some linear layers and train model with a metric learning loss to be able to measure distance -between model's outputs. - -One of the most known and easy to understand losses is a Contrastive Loss. -Contrastive loss formula is straightforward: - -![contrastive_loss_formula](https://gist.githubusercontent.com/joein/f8318cac8ea1d205ac3533c7d313f459/raw/3f6e754f53fc0b544b36b9f4359b0edcdc0411dd/contrastive_loss_formula.png) -Contrastive loss (http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf) - -It makes similar objects to be close to each other (left part) and pushes away objects marked as -dissimilar (right part). When distance between dissimilar objects is greater than (m)argin, loss -stops to push them away, and focuses on similar objects. On the picture you can see how it works. -After training, the green circles were attracted to each other, the blue one remained at the same -distance, and the orange circle repelled from the green ones. - -![contrastive_loss_image](https://gist.githubusercontent.com/joein/f8318cac8ea1d205ac3533c7d313f459/raw/f3e8049ef5180c8a8fc391edf79641005672be67/contrastive_loss_image.png) -Contrastive loss example - -With contrastive loss you need to set which objects are similar and which are not. But in our case -we have only positive pairs: question and its answer, why would we need to label negative examples? -There is a loss, called multiple negative ranking loss, and it treats all other sentences in a -batch as negative examples, and it is exactly what we need here. - -Having all this theory in mind, let's start practising! - -During this tutorial we will need a couple of _utils_ scripts. They can be found in the repository. - -- `train_val_split.py` - to split data into train and val parts. - -- `download_data.sh` - to download data from storage, create data folder and apply train_val_split. - -- `config.py` - for universal path usage. - -Let's begin with data acquisition. To download and split data you need to -launch `download_data.sh` (don't forget to make it executable with `chmod +x download_data.sh` ). - -After data downloading you should see 3 files: `cloud_faq_dataset.jsonl`, -`train_cloud_faq_dataset.jsonl`, `val_cloud_faq.jsonl`. -Our dataset is pretty imbalanced, but according to experiments, it is not a severe problem for us. -Let's take a look at a pie plot of our data. +# Question-answering system with Similarity Learning and Quaterion -![dataset_pie_plot](https://gist.githubusercontent.com/joein/f8318cac8ea1d205ac3533c7d313f459/raw/f3e8049ef5180c8a8fc391edf79641005672be67/dataset_pie_plot.png) -Cloud providers F.A.Q dataset distribution +Many problems in modern machine learning are approached as classification tasks. +Some are the classification tasks by design, but others are artificially transformed into such. +And when you try to apply an approach, which does not naturally fit your problem, you risk coming up with over-complicated or bulky solutions. +In some cases, you would even get worse performance. -A regular record from our datasets will be a JSON object contains four fields: +Imagine that you got a new task and decided to solve it with a good old classification approach. +Firstly, you will need labeled data. +If it came on a plate with the task, you're lucky, but if it didn't, you might need to label it manually. +And I guess you are already familiar with how painful it might be. -- source - name of cloud provider to which this particular record belongs to (might be useful in -experiments, won't be used in this tutorial) +Assuming you somehow labeled all required data and trained a model. +It shows good performance - well done! +But a day later, your manager told you about a bunch of new data with new classes, which your model has to handle. +You repeat your pipeline. +Then, two days later, you've been reached out one more time. +You need to update the model again, and again, and again. +Sounds tedious and expensive for me, does not it for you? + +## Automating customer support -- filename - if F.A.Q is divided into several sections, it is a section name (also won't be used in -this tutorial) -- question - question in lower case -- answer - answer in lower case +Let's now take a look at the concrete example. There is a pressing problem with automating customer support. +The service should be capable of answering user questions and retrieving relevant articles from the documentation without any human involvement. -Notices about questions and answers: -may contain unicode characters -joint length of question and its answer does not exceed 384 (just a feature of the data) +With the classification approach, you need to build a hierarchy of classification models to determine the question's topic. +You have to collect and label a whole custom dataset of your private documentation topics to train that. +And then, each time you have a new topic in your documentation, you have to re-train the whole pile of classifiers with additionally labeled data. +Can we make it easier? + +## Similarity option -As Quaterion suggests usage of fine-tuning approach, a common model built via it will consist of an -encoder(s) and a head attached upon it. Let's make our one then. +One of the possible alternatives is Similarity Learning, which we are going to discuss in this article. +It suggests getting rid of the classes and making decisions based on the similarity between objects instead. +To do it quickly, we would need some intermediate representation - embeddings. +Embeddings are high-dimensional vectors with semantic information accumulated in them. -The main entity in Quaterion is `TrainableModel`. It is a class to make model building process fast -and convenient. +As embeddings are vectors, one can apply a simple function to calculate the similarity score between them, for example, cosine or euclidean distance. +So with similarity learning, all we need to do is provide pairs of correct questions and answers. +And then, the model will learn to distinguish proper answers by the similarity of embeddings. -`TrainableModel` is a wrapper around `pytorch_lightning.LightningModule`. Lightning handles all the -training process complexities, like training loop, device managing, etc. and saves user from a -necessity to implement all this routine manually. Also Lightning's modularity is worth to be -mentioned. Modularity improves separation of responsibilities, makes code more readable, robust and -easy to write. And these are exactly the features we tried to provide in our metric learning -framework. +>If you want to learn more about similarity learning and applications, check out this [article](https://blog.qdrant.tech/neural-search-tutorial-3f034ab13adc) which might be an asset. -First, we need to create a package to store our scripts. Let's call it `faq`. -Then we should take a closer look at `TrainableModel`. +## Let's build -`TrainableModel` is an abstract class whose methods need to be overridden to perform training. +Similarity learning approach seems a lot simpler than classification in this case, and if you have some +doubts on your mind, let me dispel them. + +As I have no any resource with exhaustive F.A.Q. which might serve as a dataset, I've scrapped it from sites of popular cloud providers. +The dataset consists of just 8.5k pairs of question and answers, you can take a closer look at it [here](https://github.com/qdrant/demo-cloud-faq). + +Once we have data, we need to obtain embeddings for it. +It is not a novel technique in NLP to represent texts as embeddings. +There are plenty of algorithms and models to calculate them. +You could have heard of Word2Vec, GloVe, ELMo, BERT, all these models can provide text embeddings. + +However, it is better to produce embeddings with a model trained for semantic similarity tasks. +For instance, we can find such models at [sentence-transformers](https://www.sbert.net/docs/pretrained_models.html). +Authors claim that `all-mpnet-base-v2` provides the best quality, but let's pick `all-MiniLM-L6-v2` for our tutorial +as it is 5x faster and still offers good results. + +Having all this, we can test our approach. We won't take all our dataset at the moment, but only +a part of it. To measure model's performance we will use two metrics - +[mean reciprocal rank](https://en.wikipedia.org/wiki/Mean_reciprocal_rank) and +[precision@1](https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Precision_at_k). +We have a [ready script](https://github.com/qdrant/demo-cloud-faq/blob/experiments/faq/baseline.py) +for this experiment, let's just launch it now. + + +| precision@1 | reciprocal_rank | +|-------------|-----------------| +| 0.564 | 0.663 | + +That's already quite decent quality, but maybe we can do better? + +## Improving results with fine-tuning + +Actually, we can! Model we used has a good natural language understanding, but it has never seen +our data. An approach called `fine-tuning` might be helpful to overcome this issue. With +fine-tuning you don't need to design a task-specific architecture, but take a model pre-trained on +another task, apply a couple of layers on top and train its parameters. + +Sounds good, but as similarity learning is not as common as classification, it might be a bit inconvenient to fine-tune a model with traditional tools. +For this reason we will use [Quaterion](https://github.com/qdrant/quaterion) - a framework for fine-tuning similarity learning models. +Let's see how we can train models with it + +First, create our project and call it `faq`. + +> All project dependencies, utils scripts not covered in the tutorial can be found in the +> [repository](https://github.com/qdrant/demo-cloud-faq/tree/tutorial). + +### Configure training + +The main entity in Quaterion is [TrainableModel](https://quaterion.qdrant.tech/quaterion.train.trainable_model.html). +This class makes model's building process fast and convenient. + +`TrainableModel` is a wrapper around [pytorch_lightning.LightningModule](https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html). + +[Lightning](https://www.pytorchlightning.ai/) handles all the training process complexities, like training loop, device managing, etc. and saves user from a necessity to implement all this routine manually. +Also Lightning's modularity is worth to be mentioned. +It improves separation of responsibilities, makes code more readable, robust and easy to write. +All these features make Pytorch Lightning a perfect training backend for Quaterion. + +To use `TrainableModel` you need to inherit your model class from it. +The same way you would use `LightningModule` in pure `pytorch_lightning`. Mandatory methods are `configure_loss`, `configure_encoders`, `configure_head`, `configure_optimizers`. -The majority of these methods are quite easy to realise, you'll probably need a couple of imports -to do that. But `configure_encoders` requires some code:) -Let's create a `model.py` with model's template and a blank space instead of `configure_encoders` at -the moment. +The majority of mentioned methods are quite easy to implement, you'll probably just need a couple of +imports to do that. But `configure_encoders` requires some code:) + +Let's create a `model.py` with model's template and a placeholder for `configure_encoders` +for the moment. ```python -from typing import Union, Dict, Optional, Any +from typing import Union, Dict, Optional -from torch import Tensor from torch.optim import Adam from quaterion import TrainableModel @@ -145,7 +138,7 @@ from quaterion_models.heads.skip_connection_head import SkipConnectionHead class FAQModel(TrainableModel): - def __init__(self, lr=10e-2, *args, **kwargs): + def __init__(self, lr=10e-5, *args, **kwargs): self.lr = lr super().__init__(*args, **kwargs) @@ -156,54 +149,39 @@ class FAQModel(TrainableModel): return MultipleNegativesRankingLoss(symmetric=True) def configure_encoders(self) -> Union[Encoder, Dict[str, Encoder]]: - ... + ... # ToDo def configure_head(self, input_embedding_size: int) -> EncoderHead: return SkipConnectionHead(input_embedding_size) ``` -P.S. all required dependencies can be found in `pyproject.toml` in the repository. If you are not -familiar with poetry, you can just run this script in your project root: - -```shell -#!/bin/bash -python3 -m venv venv # creates new virtual environment -source venv/bin/activate # activates virtual environment -pip3 install poetry # install poetry -poetry install # install dependencies listed in pyproject.toml -``` - -- `configure_optimizers` is a method provided by Lightning. An eagle-eye of you could notice mysterious -`self.model` in our implementation, it is actually a `quaterion_models.MetricModel` instance. We will -cover it later. -- `configure_loss` is a loss function to be used during training. You can choose a ready loss class -from already implemented in Quaterion. +- `configure_optimizers` is a method provided by Lightning. An eagle-eye of you could notice +mysterious `self.model`, it is actually a [SimilarityModel](https://quaterion-models.qdrant.tech/quaterion_models.model.html) instance. We will cover it later. +- `configure_loss` is a loss function to be used during training. You can choose a ready-made implementation from Quaterion. However, since Quaterion's purpose is not to cover all possible losses, or other entities and -features of metric learning, but to provide a convenient framework to build and use such models, -there might not be a desired loss. In this case it is possible to use `PytorchMetricLearningWrapper` -to bring required loss from `pytorch-metric-learning` library, which has more rich collection of -losses. Also, you can implement a desired loss yourself. -By default, `MultipleNegativesRankingLoss` use cosine to measure distance under the hood, but it is -a configurable parameter. Quaterion provides several objects to calculate similarities or -distances. You can find available ones at `quaterion.distances`. +features of similarity learning, but to provide a convenient framework to build and use such models, +there might not be a desired loss. In this case it is possible to use [PytorchMetricLearningWrapper](https://quaterion.qdrant.tech/quaterion.loss.extras.pytorch_metric_learning_wrapper.html) +to bring required loss from [pytorch-metric-learning](https://kevinmusgrave.github.io/pytorch-metric-learning/) library, which has a rich collection of losses. +You can also implement a custom loss yourself. - `configure_head` - model built via Quaterion is a combination of encoders and a top layer - head. -As with losses, some ready head implementations are provided. They can be found at -`quaterion_models.heads`. +As with losses, some head implementations are provided. They can be found at [quaterion_models.heads](https://quaterion-models.qdrant.tech/quaterion_models.heads.html). -Having our methods described, we can return to `configure_encoders`:) +At our example we use [MultipleNegativesRankingLoss](https://quaterion.qdrant.tech/quaterion.loss.multiple_negatives_ranking_loss.html). +This loss is especially good for training retrieval tasks. +It assumes that we pass only positive pairs (similar objects) and considers all other objects as negative examples. -There is a base class for encoders in quaterion_models - `Encoder`. -Similar to `TrainableModel`, `Encoder` has some methods and properties which have to be -implemented. Two properties - `trainable` and `embedding_size` and one method - `forward` are -required. The other methods have no strict requirement to be overridden, but it's up to you and -depends on particular task. +`MultipleNegativesRankingLoss` use cosine to measure distance under the hood, but it is a configurable parameter. +Quaterion provides implementation for other distances as well. You can find available ones at [quaterion.distances](https://quaterion.qdrant.tech/quaterion.distances.html). -Our encoder should have noticeable general language understanding to provide embeddings for our -questions and answers. A nice choice might be to use one of presented in `sentence-transformers`. -In our case we will use `all-MiniLM-L6-v2`, as it was trained on a dataset combined from different -sources with contrastive objective, so it might work well for the task. +Now we can come back to `configure_encoders`:) -Having all this, we can create our first encoder in `encoder.py`: +### Configure Encoder + +The encoder task is to convert objects into embeddings. +They usually take advantage of some pre-trained models, in our case `all-MiniLM-L6-v2` from `sentence-transformers`. +In order to use it in Quaterion, we need to create a wrapper inherited from the [Encoder](https://quaterion-models.qdrant.tech/quaterion_models.encoders.encoder.html) class. + +Let's create our encoder in `encoder.py` ```python import os @@ -215,7 +193,6 @@ from quaterion_models.encoders import Encoder from quaterion_models.types import TensorInterchange, CollateFnType - class FAQEncoder(Encoder): def __init__(self, transformer, pooling): super().__init__() @@ -225,6 +202,7 @@ class FAQEncoder(Encoder): @property def trainable(self) -> bool: + # Defines if we want to train encoder itself, or head layer only return False @property @@ -262,27 +240,27 @@ class FAQEncoder(Encoder): As you can notice, there are more methods implemented, then we've already discussed. Let's go through them now! -- In `__init__` we need to register our layers, as `Encoder` is actually `torch.nn.Module` heir. +- In `__init__` we register our pre-trained layers, similar as you do in [torch.nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html) descendant. -- `trainable` defines whether current `Encoder` layers should change during training or not. If -trainable is False, then all layers will be frozen. Quaterion also can cache frozen encoders -output. +- `trainable` defines whether current `Encoder` layers should be updated during training or not. If `trainable=False`, then all layers will be frozen. -- `embedding_size` is a size of encoder's output, it is required for proper head configuration. +- `embedding_size` is a size of encoder's output, it is required for proper `head` configuration. -- `get_collate_fn` is a tricky one. Here you should return a method which prepares a batch of raw data -into encoder suitable input. If `get_collate_fn` is not overridden, then default implementation -will be used. The default one is `default_collate` from `torch.utils.data._utils.collate`. +- `get_collate_fn` is a tricky one. Here you should return a method which prepares a batch of raw +data into the input, suitable for the encoder. If `get_collate_fn` is not overridden, then the [default_collate](https://pytorch.org/docs/stable/data.html#torch.utils.data.default_collate) will be used. -The remaining methods are considered self-describing:) -As our encoder is ready, we now are able to fill `configure_encoders`. Just insert the following code -into `model.py`: +The remaining methods are considered self-describing. + +As our encoder is ready, we now are able to fill `configure_encoders`. +Just insert the following code into `model.py`: + ```python ... from sentence_transformers import SentenceTransformer from sentence_transformers.models import Transformer, Pooling from faq.encoder import FAQEncoder + class FAQModel(TrainableModel): ... def configure_encoders(self) -> Union[Encoder, Dict[str, Encoder]]: @@ -293,20 +271,25 @@ class FAQModel(TrainableModel): return encoder ``` -Ok, we have a model, we have raw data, we know how to prepare input for encoders, we don't know yet -how to pass this data to our model and how it has to look like. +### Data preparation -Currently, Quaterion assumes two kind of data representation - pairs and groups. In pairs, you -should label if a couple of objects are similar or dissimilar, for example, it is a suitable input -for _ContrastiveLoss_. And the groups format assumes that all objects split into groups of similar -objects. All objects inside one group are similar, and all other objects outside this group -considered dissimilar to them. +Okay, we have raw data and a trainable model. But we don't know yet how to feed this data to our model. -In our case we have a collection of questions and answers. We can use here both approaches, but -pairs seems more intuitive for this case. -To represent pairs we have `SimilarityPairSample` in `quaterion.dataset.similarity_samples`. Let's take -a look at it: +Currently, Quaterion takes two types of similarity representation - pairs and groups. + +The groups format assumes that all objects split into groups of similar objects. All objects inside +one group are similar, and all other objects outside this group considered dissimilar to them. + +But in the case of pairs, we can only assume similarity between explicitly specified pairs of objects. + +We can apply any of the approaches with our data, but pairs one seems more intuitive. + +The format in which Similarity is represented determines which loss can be used. +For example, _ContrastiveLoss_ and _MultipleNegativesRankingLoss_ works with pairs format. + +[SimilarityPairSample](https://quaterion.qdrant.tech/quaterion.dataset.similarity_samples.html#quaterion.dataset.similarity_samples.SimilarityPairSample) could be used to represent pairs. +Let's take a look at it: ```python @dataclass @@ -317,19 +300,21 @@ class SimilarityPairSample: subgroup: int = 0 ``` -Here might occur some questions: why `score` is float and what is a `subgroup`? +Here might be some questions: what `score` and `subgroup` are? -Well, `score` is a measure of samples similarity, generally it is just being converted into _bool_ - -_0.0_ into _False_ and all other values into _True_. But you also can make your own implementations -to tune impact of particular pairs on a loss function. +Well, `score` is a measure of expected samples similarity. +If you only need to specify if two samples are similar or not, you can use `1.0` and `0.0` respectively. -Up to this point pairs might look like an absolutely separated entities which can't have any -relation between them. That's where subgroup came into a play: we can union some pairs into a -subgroup, so other objects from the same subgroup can't be considered as negative examples, but -objects from the others subgroups can. By default, all objects are in the same subgroup, so we -consider only explicitly set links between points. +`subgroups` parameter is required for more granular description of what negative examples could be. +By default, all pairs belong the subgroup zero. +That means that we would need to specify all negative examples manually. +But in most cases, we can avoid this by enabling different subgroups. +All objects from different subgroups will be considered as negative examples in loss, and thus it +provides a way to set negative examples implicitly. + + +With this knowledge, we now can create our `Dataset` class in `dataset.py` to feed our model: -After this introduction we can create our `Dataset` class in `dataset.py` to feed our model: ```python import json from typing import List, Dict @@ -347,6 +332,8 @@ class FAQDataset(Dataset): def __getitem__(self, index) -> SimilarityPairSample: line = self.dataset[index] question = line["question"] + # All questions have a unique subgroup + # Meaning that all other answers are considered negative pairs subgroup = hash(question) return SimilarityPairSample( obj_a=question, @@ -365,102 +352,70 @@ class FAQDataset(Dataset): return [json.loads(json_line) for json_line in fd] ``` +We assigned a unique subgroup for each question, so all other objects which have different question will be considered as negative examples. -We assign a unique subgroup for each question-answer pair, so all other objects which do not belong -to a particular pair will be considered as negative examples. +### Evaluation Metric -We are already able to train our model, but at the moment the only obvious way to measure its -performance is loss observations. To make any additional evaluations on embeddings, e.g. perform -metrics calculation, one should override `process_results` method of `TrainableModel`. +We still haven't added any metrics to the model. For this purpose Quaterion provides `configure_metrics`. +We just need to override it and attach interested metrics. -Quaterion has some popular retrieval metrics such as _retrieval precision @ k_ or _retrieval -reciprocal rank_ implemented, they can be found in `quaterion.eval` module. But there is quite a -few of metrics, it is assumed that desirable ones will be made by user or taken from another -libraries. You will probably need to inherit from `PairMetric` or `GroupMetric` to implement a new -one. +Quaterion has some popular retrieval metrics implemented - such as _precision @ k_ or _mean reciprocal rank_. +They can be found in [quaterion.eval](https://quaterion.qdrant.tech/quaterion.eval.html) package. +But there are just a few metrics, it is assumed that desirable ones will be made by user or taken from another libraries. +You will probably need to inherit from `PairMetric` or `GroupMetric` to implement a new one. -Let's bring it all together and add some visibility and control to our training process. +In `configure_metrics` we need to return a list of `AttachedMetric`. +They are just wrappers around metric instances and helps to log metrics more easily. +Under the hood `logging` is handled by `pytorch-lightning`. +You can configure it as you want - pass required parameters as keyword arguments to `AttachedMetric`. +For additional info visit [logging documentation page](https://pytorch-lightning.readthedocs.io/en/stable/extensions/logging.html) + +Let's add mentioned metrics for our `FAQModel`. Add this code to `model.py`: ```python ... -from quaterion.utils.enums import TrainStage from quaterion.eval.pair import RetrievalPrecision, RetrievalReciprocalRank +from quaterion.eval.attached_metric import AttachedMetric class FAQModel(TrainableModel): def __init__(self, lr=10e-5, *args, **kwargs): self.lr = lr super().__init__(*args, **kwargs) - self.retrieval_precision = RetrievalPrecision(k=1) - self.retrieval_reciprocal_rank = RetrievalReciprocalRank() ... - def process_results( - self, - embeddings: Tensor, - targets: Dict[str, Any], - batch_idx: int, - stage: TrainStage, - **kwargs, - ): - device = embeddings.device - - self.retrieval_reciprocal_rank.update( - embeddings, - **targets, - device=device - ) - - self.log( - f"{stage}.rrk", - self.retrieval_reciprocal_rank.compute().mean(), - prog_bar=True, - on_step=False, - on_epoch=True - ) - self.retrieval_reciprocal_rank.reset() - - self.retrieval_precision.update( - embeddings, - **targets, - device=device - ) - self.log( - f"{stage}.rp@1", - self.retrieval_precision.compute().mean(), - on_step=False, - on_epoch=True, - prog_bar=True, - ) - - self.retrieval_precision.reset() + def configure_metrics(self): + return [ + AttachedMetric( + "RetrievalPrecision", + RetrievalPrecision(k=1), + prog_bar=True, + on_epoch=True, + ), + AttachedMetric( + "RetrievalReciprocalRank", + RetrievalReciprocalRank(), + prog_bar=True, + on_epoch=True + ), + ] ``` -When we calculate metrics like this, they can fluctuate a lot depending on a batch size. It might -be useful to calculate metrics not among batch but among whole available dataset, similar to a real -application. Unfortunately, it is not always possible to send a whole dataset inside one batch:) -Raw data may consume a huge amount of memory, but embeddings most probably will consume less. -Quaterion metrics are designed to accumulate embeddings across batches (via `update` call) and then -`compute` them, when you want, e.g. before validation starts or right after its end (you can -override callback). +### Fast training with Cache -However, in this particular `process_result` implementation, we calculate new values per-batch. +Quaterion has one more cherry on top of the cake when it comes to non-trainable encoders. +If encoders are frozen, they are deterministic and emit the exact embeddings for the same input data on each epoch. +It provides a way to avoid repeated calculations and reduce training time. +For this purpose Quaterion has a cache functionality. -Quaterion has one more cherry on top of the cake when it comes to non-trainable encoders. If -encoders are frozen, they are deterministic and emits exactly the same embeddings for the same -input data each epoch. So why not to avoid repeated embeddings calculations and reduce training -time? Actually, Quaterion has a cache exactly for this purpose. -Before training starts, cache runs one epoch to calculate all embeddings from frozen encoders and -then store them on a device you chose (currently CPU or GPU). Everything you need to do is to -define which encoders are trainable and which are not and set cache settings. And that's it: -everything else Quaterion will handle for you. +Before training starts, the cache runs one epoch to pre-calculate all embeddings with frozen encoders and then store them on a device you chose (currently CPU or GPU). +Everything you need is to define which encoders are trainable or not and set cache settings. +And that's it: everything else Quaterion will handle for you. -To configure cache you need override `configure_cache` method in `TrainableModel`. This method -should return instance of `CacheConfig` object, in most of the situations it will just point to -device on which you want to store calculated embeddings and set `batch_size` to be used during -caching. +To configure cache you need to override `configure_cache` method in `TrainableModel`. +This method should return an instance of [CacheConfig](https://quaterion.qdrant.tech/quaterion.train.cache.cache_config.html#quaterion.train.cache.cache_config.CacheConfig). Let's add cache to our model: ```python @@ -470,11 +425,15 @@ from quaterion.train.cache import CacheConfig, CacheType class FAQModel(TrainableModel): ... def configure_caches(self) -> Optional[CacheConfig]: - return CacheConfig(CacheType.AUTO, batch_size=1024) + return CacheConfig(CacheType.AUTO) ... ``` -`CacheType` determines device to store embeddings, `AUTO` chooses GPU if it is available, else CPU. +[CacheType](https://quaterion.qdrant.tech/quaterion.train.cache.cache_config.html#quaterion.train.cache.cache_config.CacheType) determines how the cache will be stored in memory. + + +### Training + Now we need to combine all our code together in `train.py` and launch a training process. ```python @@ -507,7 +466,7 @@ def train(model, train_dataset_path, val_dataset_path, params): ) Quaterion.fit(model, trainer, train_dataloader, val_dataloader) - + if __name__ == "__main__": import os from pytorch_lightning import seed_everything @@ -528,22 +487,51 @@ if __name__ == "__main__": ``` Here are a couple of unseen classes, `PairsSimilarityDataLoader`, which is a native dataloader for -`SimilarityPairSample` objects, and `Quaterion` as an entry point to training process. +`SimilarityPairSample` objects, and `Quaterion` is an entry point to the training process. -As you could already notice, Quaterion framework is split into two separate libraries: Quaterion -and Quaterion-models. The former one contains training related stuff like losses, cache, -`pytorch-lightning` dependency, etc. While the latter one contains only necessary for serving -modules: encoders, heads and `MetricModel` itself. The most important benefits here are: -- less amount of entities you need to operate in a production environment -- reduced memory footprint. (Memory footprint is important, because there are plenty of libraries in -the wild which make you download all auxiliary modules and dependencies due to their architecture -decisions and cause your project or your docker images to blow up and lead to e.g. significance -growth of deployment time.) +### Dataset-wise evaluation -The very last row of `train.py` - `faq_model.save_servable(...)` saves encoders and model in a -fashion eliminating all Quaterion dependencies and storing only the most necessary data to run a -model in a production. -At this point we should train our model, I do it via `python3 -m faq.train`. +Up to this moment we've calculated only batch-wise metrics. +Such metrics can fluctuate a lot depending on a batch size and can be misleading. +It might be helpful if we can calculate a metric on a whole dataset or some large part of it. +Raw data may consume a huge amount of memory, and usually we can't fit it into one batch. +Embeddings, on the contrary, most probably will consume less. + +That's where `Evaluator` enters the scene. +At first, having dataset of `SimilaritySample`, `Evaluator` encodes it via `SimilarityModel` and compute corresponding labels. +After that, it calculates a metric value, which could be more representative than batch-wise ones. + +However, you still can find yourself in a situation where evaluation becomes too slow, or there is no enough space left in the memory. +A bottleneck might be a squared distance matrix, which one needs to calculate to compute a retrieval metric. +You can mitigate this bottleneck by calculating a rectangle matrix with reduced size. +`Evaluator` accepts `sampler` with a sample size to select only specified amount of embeddings. +If sample size is not specified, evaluation is performed on all embeddings. + +Fewer words! Let's add evaluator to our code and finish `train.py`. + +```python +... +from quaterion.eval.evaluator import Evaluator +from quaterion.eval.pair import RetrievalReciprocalRank, RetrievalPrecision +from quaterion.eval.samplers.pair_sampler import PairSampler +... + +def train(model, train_dataset_path, val_dataset_path, params): + ... + + metrics = { + "rrk": RetrievalReciprocalRank(), + "rp@1": RetrievalPrecision(k=1) + } + sampler = PairSampler() + evaluator = Evaluator(metrics, sampler) + results = Quaterion.evaluate(evaluator, val_dataset, model.model) + print(f"results: {results}") +``` + +### Train Results + +At this point we can train our model, I do it via `python3 -m faq.train`. |epoch|train_precision@1|train_reciprocal_rank|val_precision@1|val_reciprocal_rank| |-----|-----------------|---------------------|---------------|-------------------| @@ -554,71 +542,59 @@ At this point we should train our model, I do it via `python3 -m faq.train`. |400 |0.695 |0.772 |0.694 |0.773 | |500 |0.701 |0.778 |0.700 |0.777 | +Results obtained with `Evaluator`: -After training all the metrics have been increased. Of course one can say, that growth is -insignificant, but all this training was done in 3 minutes on a single gpu! There is no overfitting -and the results are steadily growing, although I think there is still room for improvement and -experimentation. +| precision@1 | reciprocal_rank | +|-------------|-----------------| +| 0.577 | 0.675 | -The only remaining part is serving. It's time to sort it out. +After training all the metrics have been increased. +And this training was done in just 3 minutes on a single gpu! +There is no overfitting and the results are steadily growing, although I think there is still room for improvement and experimentation. -As we got rid of Quaterion dependency, we need a new means to supply our model with data. We can -just create a new dataset and dataloader dependent only on torch for it: +## Model serving + +As you could already notice, Quaterion framework is split into two separate libraries: `quaterion` +and [quaterion-models](https://quaterion-models.qdrant.tech/). +The former one contains training related stuff like losses, cache, `pytorch-lightning` dependency, etc. +While the latter one contains only modules necessary for serving: encoders, heads and `SimilarityModel` itself. + +The reasons for this separation are: + +- less amount of entities you need to operate in a production environment +- reduced memory footprint + +It is essential to isolate training dependencies from the serving environment cause the training step is usually more complicated. +Training dependencies are quickly going out of control, significantly slowing down the deployment and serving timings and increasing unnecessary resource usage. + + +The very last row of `train.py` - `faq_model.save_servable(...)` saves encoders and the model in a fashion that eliminates all Quaterion dependencies and stores only the most necessary data to run a model in production. + +In `serve.py` we load and encode all the answers and then look for the closest vectors to the questions we are interested in: ```python -import os +import os import json -from typing import List import torch -from torch.utils.data import DataLoader, Dataset - - -class ServeFAQDataset(Dataset): - "Dataset class to process .jsonl files with FAQ from popular cloud providers""" - def __init__(self, dataset_path): - self.dataset: List[str] = self.read_dataset(dataset_path) - - def __getitem__(self, index) -> str: - return self.dataset[index] - - def __len__(self): - return len(self.dataset) - - @staticmethod - def read_dataset(dataset_path) -> List[str]: - with open(dataset_path) as fd: - return [json.loads(json_line)["answer"] for json_line in fd] -``` - -It is our new dataset, it looks pretty concise and brief for me, doesn't it for you? - -Nevertheless, we are ready for serving, let's extend our `serve.py` and finish this extensive -tutorial:) - -```python -... -from quaterion_models.model import MetricModel +from quaterion_models.model import SimilarityModel from quaterion.distances import Distance + from faq.config import DATA_DIR, ROOT_DIR -... + if __name__ == "__main__": device = "cuda:0" if torch.cuda.is_available() else "cpu" - model = MetricModel.load(os.path.join(ROOT_DIR, "servable")) + model = SimilarityModel.load(os.path.join(ROOT_DIR, "servable")) model.to(device) - path = os.path.join(DATA_DIR, "val_cloud_faq_dataset.jsonl") - dataset = ServeFAQDataset(path) - dataloader = DataLoader(dataset) + dataset_path = os.path.join(DATA_DIR, "val_cloud_faq_dataset.jsonl") + + with open(dataset_path) as fd: + answers = [json.loads(json_line)["answer"] for json_line in fd] # everything is ready, let's encode our answers - answer_embeddings = torch.Tensor().to(device) - for batch in dataloader: - answer_embeddings = torch.cat( - [answer_embeddings, model.encode(batch, to_numpy=False)] - ) + answer_embeddings = model.encode(answers, to_numpy=False) - # probably it's worthwhile to index your vectors and search among them with some kind of vector search engine like Qdrant :) # Some prepared questions and answers to ensure that our model works as intended questions = [ "what is the pricing of aws lambda functions powered by aws graviton2 processors?", @@ -642,22 +618,22 @@ if __name__ == "__main__": answers_indices = question_answers_distances.min(dim=1)[1] for q_ind, a_ind in enumerate(answers_indices): print("Q:", questions[q_ind]) - print("A:", dataset[a_ind], end='\n\n') + print("A:", answers[a_ind], end="\n\n") assert ( - dataset[a_ind] == ground_truth_answers[q_ind] - ), f"<{dataset[a_ind]}> != <{ground_truth_answers[q_ind]}>" + answers[a_ind] == ground_truth_answers[q_ind] + ), f"<{answers[a_ind]}> != <{ground_truth_answers[q_ind]}>" ``` We stored our collection of answer embeddings in memory and perform search directly in Python. -For production purposes, it's probably better to use some sort of vector search engine like Qdrant -to get durability, speed boost, and a bunch of other features. +For production purposes, it's better to use some sort of vector search engine like [Qdrant](https://qdrant.tech/). +It provides durability, speed boost, and a bunch of other features. So far, we've implemented a whole training process, prepared model for serving and even applied a -trained model today with Quaterion. +trained model today with `Quaterion`. -Thank you for being with us. I hope you enjoyed this huge tutorial and will use Quaterion for your -metric learning projects. +Thank you for your time and attention! +I hope you enjoyed this huge tutorial and will use `Quaterion` for your similarity learning projects. -All ready to use code can be found here: some awesome repo +All ready to use code can be found [here](https://github.com/qdrant/demo-cloud-faq/tree/tutorial). Stay tuned!:) \ No newline at end of file diff --git a/qdrant-landing/static/articles_data/faq-question-answering/icon.svg b/qdrant-landing/static/articles_data/faq-question-answering/icon.svg new file mode 100644 index 000000000..05d664f10 --- /dev/null +++ b/qdrant-landing/static/articles_data/faq-question-answering/icon.svg @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/qdrant-landing/static/articles_data/faq-question-answering/preview.png b/qdrant-landing/static/articles_data/faq-question-answering/preview.png new file mode 100644 index 0000000000000000000000000000000000000000..7131c1f1cb0fa610e796a56d85375cf40103ae7d GIT binary patch literal 25226 zcmeFZbx@Vj+CEH!gfvKpv^1M;>29Ta)7{-7jS7l%skDGJ2uL>wC`f~pbhBx`wLRxO z=e+a%X1;l6zTbbZGj8{OVm<4**B#e&-D`~1R9C>kB*#QRK)_K_l+{K+K=c6Ln=sJ8 zCx1}wAOZrRRDiC&m$rp3t(%9djlC0;*2~`wN(=R~w?RPgoBfz>@Q#7?joiI~99OuO z)H`I1tVz*7>oDSBaXxxphkmx)VPiF7RIuD6i>+z7kf$qkL3dQg{%T8o?VJ}OYAL>ZFT;E9uFP+5!lx>a z`nZsT7m6ZAk*T&Gd9K$XRo)J;q_A!>9_}@0)5u$}e-p_0%QcX@Tgxt3K z6xb$4GH&V!4KHaQ+amRZda|Slb}s1#guL`KQx5EFPm`4jAuW<@i}7rf_<1##5;8bQ za}Y8xF4-+X@O|raZ?K~c zaSdkvyGprWuA|n~uRR<$Q;)g%W3!if@os~kEJcikLn>+h@h^myTf48T+^)ho(=bgrzn>M`(o`~y?t%Thz%8rmrR0e5;g8#q;ah&!_Mr@6oV~Pa z4blKjtNd#r+l>xfHK07G&>TVvHJJqhAEDiOU zGPq;g7WXlqz0epubNw3Ko_eGqx9c7XRlWO2$XWV;xSr$l8_Lg#I5q^J85ypGV(|S~@u1v|4deDLrU|meRAO zhj*v;e!$y5)yRdKZm(pGN9+-9P`KgqyYoi?XVH3rf;5%9G(@gUhL44z7K!%l z!{XGiR&sUJmXFAzr;En~>hXTM9B&D>TdkFKha_Sx3sw~Z0z~q?=yL`9tilIa{S>nH zz8IrDc2;XC(mNMQsUloyy-mF?&5Dal8u{j}-58oY{Fdxd{W7b1gBmL3R^reESy#$P z9Ib~`!T!uiYRxNTl=u3*Dx(OmzHa+6zGycJkV!fnO6^D*_IMr4$H(o}ZHcH5fE7@^ z@RMLR#q?vbc_QMdHG%mV<*Qg~l^0n{Gs|VzGSYEgwhxhaX}Jmg@cQ(x@Hn-$GV;D5 zCu={!j9d4HJw+_x&425>D1(bsqY3FnF@0h#i;sA#im^l^&8P!?YAz);GTp_d*E^Ah}ACbLiU`_&J62BE3e}T&)xx z^JyE-hoK)k`5BJ7+k3Lyo8|tJMVg|QIXDMZs{OL3(Q~-$a&mXa9WeLk9)u6dQLO$| z32&U1b9iuaZDqL-LcBurtM|3-pCut~fA)RFA0M$V@I0kyH-u*!g}6i|TBbx^!#wk{ zcs{*zaXyA+&8|$5%Y89|C2y0m;Rv4)1(8aGU5j<#>4-GSj#Q=27+ra}L=7aePV9=; z*C^?i=gEr2>;3sH_IJb$YrI5NN*{h2FrTMv-%8!0JG3v|oJB>IL+msbQ) zQ^i1)>S)Mw9!9O$f38t==m2$>3lZbBaxBivF-?tOYDZ5Jp@=3c>y+>2(HlY3>b^Gk z{@64I9U(A162Gz&$yZQ0WziFRJjNsM)gCS5R<`Gp)6Ya_FXR;0QIhK4J$}U~QYZcL zl)n>qSS1dngp(%5=S^S0#+8gzC>|cp>_@vi(dn&G*7q%u5m^+JyckV%L`n}?O3Dh} zeKQEdvR@@=)#T~#lzfgCVvNKibH(YN6l3MJvE-IW#CSp3q?Q9oOt_+#=MkDFbj`Sk zI^r+Zf#rth-z+xbk*;fxrwV+d_A0C1#ojm99+&<;^1;cY4k7Vluo!f;5zjM$Ft(7+ zU1`wYLI1E%FQ(wp?>|0<)E;8k(#wwLRVWgL`~-)A5g93%s0|J5xL{F_q%r7B-kS;%D z`AEh7Az=URN%#PD41(ZT*`v_*Ru|1GP{e|8lcS$dKYx!+T7%)uTe_j|FoCI(pC&%B z9%;^r6+RTIgl5?jJ|A>fDTQZv-xk@rPHt z;>Ps!cej(2RIB@^qdtC!JAoG?&RSMTsYejwyQaF6PT7*6$O9-mmH$GPkt%~%)~ zSY@X$75&ApR>qcND*8QwR(=ikSmcRUF2RuZ`h|g)`e&AIeFT;b2A4Sdg zqvsfPd+HR#q!pvjWu{(4G@18ylp}>c2#@*`{t(rkgb6iDGs*7RHYt&}g>#I2R^4Wn z10@w26VqJrNkr&RtA63Ip7*<|$Y_;x`-u{izZc$((?jIYZ=!cXy*%VomAoF6=`&%= z89@p#9>}{{=()W|wwH$~Oe5P#b7K{y(Y-|;JD#Bxe(@>)XX@GaImOr7Q`>EBT!E*R zsszpUv3Zyh_$IU{KH4aI2oerGozN+)?-QvnMy+w43z$9_=a|7RHZ`yP!pLOZBc_X|i9^dCPfd~knr98h?ayk(Rp`R13w`!{s z4=Pi$XX%o;cyd=W^dSlgcJr#YH#3$xJLAis-fj#Q0r_*O6Q-0ToNqiclK-`x2R zl!PkZ@6T-H_UhMjwewk+@}u04WAVhHVY?4HL)&|k&Lom{`wmBb#Fak`3euR%3dF{O zXf1j2T4=uomB>fa7Hcqt){-+K9ls%($96)A)ux-~ujxRFj*jnFn|ApaRn){&!r77c zWK&7OkuEigDI+Y(i{@cYE|1%@#y@pmiQN#1e(sS;yUAoCle_Z%@TtaqDq9tsk~$@6 zqdMSbqS!3~U5b*)hkQ@QkwBo@13pJbTh7sq7W>$BL7QTM9AOphZ z{$g<^UTbU=QJ8l!c12W3$+8EDxom?a&XXglOeuBE*Uwmt=>3uP>yclG{CxFx^atDX zW=`ig6IZ6?O(zu-L#>I3e5I*s4c~95daQxHPrM$!H+rZXo2MnB8)HGL)w;YGC0E6h zp3i!Z^M0ip777kte;WNuDGq-y@9pEE^P!J5=CQ4(Tx7VcA)%6*Cvks6I4x-a80 z#gB^b`F#M=KRGj*o1gc~9N&H#=3?1O`vXA{uka!=_Kov!?d-1es#<!`CNxJ}4diD4Cd1-TgNNTe?tNOPp|Zjthw``#CD zAU#geaw%j%dPwf8ayP&eLc;s$(Xdg%J3he#^knSL$2=mptq)4BWdw^VuO6VvaktWL z9!S`AroajM5*7)|3W3p5cSK<3xB;L>tICXW+?IM$omZjk=kz-NZ$p- z+ovc7Dl_fvY52*BsAFmc$~ICcYL?LL`&csS#R8YbZ{D&MH5QAW<1;xgcgqd#F4-d_ z*L%cSwI14I7Wavg5ufe%iwsiIT&#zok%;39j#6(ynO?u}e}B?YwYq!|NN{a_fRbP zU*x$nRXSF+@0ag6hV_xgwQ@$ca!R+aWgHyZy7X|eHwBdF9shWmvxF*;T7Sn)rh;FEdh;CM>u93%7?Q z40e%POmQ>`FFI^)WvcFp@|f4bw)AT=;28M#x?OH?B`&Z=D2*=8JmyAp^| zY={xZs@KX)TAilhSXOW6x;vUZTr$2{ty^ht5+Z#k!64u#+icK&_VZhAxu)M3`TI)cJS&DQvwK3lG@dOq%?o6*~ z@Zso48JKd~*oRN6 zxFv#yFMeGnDS_vmqdx2rkJJ=@L`C_X!Bx&vvx}ii!>8LX?M4Na(p(aV8ycWC&)6Xe`Sj_6vN_Nv6rqYcNb`BguhsD?o&cUnDH=zO z0gmj(h@#c4C8+ox*&;ks?=4i~2ii75unWT*gNO6l`IJirbF5+A?Fy5dWnwhEO|Cd<{m6TNbRKvyMnkqo<9Hb*nMfe)mWt*YoL$hG6;RpBWLi|j{Tca&V&sghG?62;kvuMd*aLf<=aHh)^VM+|7=?)MFzHv-~X zl)a3Mrjm@ze`#qzLCXq$Ew0!lMcHGfQ%6tCYJ-PApqb4c6DQ?|Ua58d~SFGzT(DPgM%t$hokVP$E=BioPA zAqxUJv5l_=K4a!T_o73h?ut>2gAT`9611)EP$Ska_UOO6Z<17~%%~xvrNtX#i?%0p z#nW(|MOzKFG-Xw0I}$#6uSCiaV%@xZM6nt%L^-rJ;^mO@o|y#GnZ2P}8^Zsa^>=aU zR_Yj3>=~IW&(xsaM}Lj;PB(Qk0~?VMI)!Z?D#D*Qq{=ANf<=c^KT($idT zRG6MWz!WCY3`ZL{i@fbPaH+Uii%KI(LeAz&I^jWz@(Gt8ac@heDso3c$qj$fx?cGs zJY8eLk%J=u<qu~x1cLoG zn?EuoB~l^kpefAX*Ei7hKF9tXhO8;=ksdu4-v~5MKYO5U>Z_@WSh+fLT3EYULOK1M z-GKIqfFLI6=VoE$2=$`1gxcD>h|?dmwA0htTZ_{_=U3xabCZGE*((NkKy?Dtb*%y% zt%R-VB_%M${6xS2&QLE4T0dte7f%sCar(dGih%F%!(8;Ve=qTJ6sOl$)1;Mg^?=es zI3b+e9CCj4KD_i2n6zRZ);1#Avhx2N0{jxExAXFH6XD|W_4VcS<>Pequ;t z=H=q$23|S@?0dcs_;?@sBZNp`KPA_HJJGt}e9jF)b`zy}iWg>A`v0|C*n( zo0{6chIjG&rxk!bxcn^KxOh0ZxtyK3{__q`FF7AD$Ui6af4sv}7f8fh+E7ndZx1V| zoDbB+>+ydMVQuxV``x@foc_L!wG|iC3F-_k^#o7l`EQR@P*T(U*Bx*Z*xEb0{k;nq z`@dc3WpDFuWBs?e!O#4Co&TH&xcy(_{)roe46agB6Onbb@`g`ONmiU5{(KQ@ zS1WsKk-uLBEucJnLV^$uC{##@17gi@!C?up{`)_=P#RA^bK1d=P#s2-N03hS2b^2VQC6 z^q;eW4`mI8vfvgJ60)&^azHI1JRA@kUI>St z&K6)f?VT-bpX5-t4Bf1OqX;bd*$Wg%VhwP{36_Z zPq@L6|G3`O+TOXCt^dvwjP;LGR(2LHwos5g{^_p&b=>~{;0hMPP)kdwH4g``r3LUNONbDM zu#g2mhXprOkO#tJWdRZXhgttMx~Hp+m#>8fRN5BU5!ed2=U-dVviw~r*8e^iUppv# z6@bAwxCJ==Z7|3`4aW7)7v_TJjQ@ILF|PlYO~n3Q@E=JA-1m=TAbEjY$n~#e_)nXG z`TpPh^Ut;TzxfDS+W&pX|H!`oYp(w_*Z;@@|D%il*Sr4LT>m2r{EsgFU+?<=&0LuO zrc+QCfP#DhS?Xm%x&=fF%~C}{_Ali^xRhxL1>YXHDH?bpAUy1X|3&O`Ew%;+(Y=(^ z&kb(y^>cA!I~r>BRqg1K5ZZQ3<9mVC6@XGV})^yyP{;C|D~flqYsS2aM=Vrk@8zY(i%i-vXii)qoN$E&YR-V4j%pCev zZS?>bw~~3#n*>(LaZyk}4q;`*WW$J)GbjF-%V`rRc@e3`!gruZ>R>|V>`udMXG6ij?$_-9;XHpcX{bi$! z;X_XzolTpaG_>C)hJ_&{tH^2*4cqXQmVR-X;T@}2FG~6Hg?HWsWR3K71MJ+IAUjGm?RihevDn2BV1J4bas#a@#AraI&2Do-K?bCjEuM$CznIS@fKg#MgL8zPimRHujpi0o;+EN#Fa2V zUK@Zo{7`J8g-?o;NJ~r0Y4L0N#DsR`I(cIBOpUD-Z~A*~i>Iws(xIWD84!Dq+0~Hq z8QZVtvyL~bH1~OWWzSS|C9-*}dh^EZte55bQKcJSb$55;5fbXS`uq6_+0>+e`0ydX zea%)zQ4vF@NF6$sBYv}`5o&q*XPoyJ{aOU=4Rvwp;5!Z@ec&`NUc4~&UGVq=zD|Mu!kHJ$fG%YJjb{p6?o zal7L3a_isSv0tzETfFw?4T9z)fLV>P2L}fm@9(bNTNb`vZI_{-pi}{O0Tz^>{G_(^ zt9q3zx!Qh`$)#}xtv8uwOa@n67Q7+?c*h<)$V7#a;KbzQ)?(}ROuL68#P$piU$!50 zrF8RfrORod`E8TuCa>S2If97q{_yeokU_O2YDr1SAFs1emyaJm1|9W~rlzIwh2912 zH}7y#`>)|A#~`DiqVhloiL%t037OT`&gWbuciwwuj`Vg!qOW5Sz+*^NC|iBrMR0t4 zJO|2@65H4)A02^pD;+Z^Gb6-IrsIR;C& zJa7~G)#A%_cXtPIU7W8G6B4>8e_@74M;cZ34)}XSWaP|Of4+f%fj?d%=esjqAejB{ zE;oY>S2ZaDF2)sTRy>;C3pkV4*o<^8E#-bTX$f#{@I5fK8O>5!Y70*1cYHTyod+B& zC+Jw|_v)(uVF!9ElY(47+;fDGm6VhcCfKJX`y z7vHL73RK&UQ8TG!$ll-G`tLVwk`uX0v%^CL0WC7zv9k4@^IZMP=4KyAlA}$ZmF(!|i!BW;WvCkAPU$aA8P(!R>l(ug1f+04~DBka!B#k;oO zzAIEnMp^~~#dEbOZjNSeeb0}KX{3G1iIGMQ_h#>DlX8<5lCYm;&po?v1gk>tLv;0e z);;hE2%Z+oCyGIBO*qr0mzsWXk$w*9uA|3j=L50)6}A{CtDs<2Gt)mf*a>RE+0Jxe zDJ@%EXIt|8-Hzg3{8lE0&d&6>KtIl(6ApRf)?r z!Z&Z;=-1fbls&IJcPe7RBg9>|)T(;@&bxfR|^ z%g6}m4atMwiP4UXkB@I{ZEapN^L22<0Yd&sBR8qXa;5Eh|HR?!Qz=_OE48GgB$<>n zNa~!Zz)?uzKQ4OI%p`!4#lY~v^aLsmcY770hB-AbeY22D#-TJWbXs=y1S8khx36B| zZ9QbTkln|WZj6!*TJ23d-I{!v&S%SO{tW@`A!WY`S8F#&Px`F^b+Y9cAkVq6rlh6y zHM*|?U&fkiaNR5^%CQ9HWp}-;g?dLM_gggx!%8+o`|PYEn6EZ(a! zsWK5tOG^)D@W;C>G#`P;7re0|X&BOrzW_Of;M<;0jrYzo;GopG7B9esr+ag0{wEtp zgHI~`ewvmPmX^kW(w6e-6)iH`U#lf9xb&^K;Zah~n@(WgJrLE zn$bqng%`Ueij)CaPzfEU%2#F`D=V0P2xKOL8Vb+7z*YSW1qai&Ti|a;Ys}D_NC6Lo zM*>cds6>5EAx=I`#=A2$FF}qy>OiLfCE;--qF5xW7yF8n+ZhtGec_aC!_b~8gxueIU|oKgqPytfByWrw!RKLiZ0Tx8i$k& zX+`;i?N|;xz}U^&vOqWyfNK11#orB7?;OA14+3EOBODF$9?x(%b@e_L6hypz@*Iw+ z!Vr0SF-;cn*r| zdmbw^fU;7;i9n4TO$M2LF<^(|d~dEVM?A1n8-o`0);fgduQqh)CL~=i4S}(M$ex0){IH0 zfGSNU5#$&7mh6Qji90!Fr(SevsjYZ884-~zaQ4{bWKw)MYWr(MX}rIv<{uw$ga3zt z|JwyCPigC(eKt4dBvmh(JlS*;+U6*`_LZq?S#S{qM>a+D^xQ?VbJ(R@O%M3451i5{ z66FE@LyJVLcouJ3sZcbTKe_dN=S;g*6D%H=2`F`(X+tR@Cd>)z>+3|s#9F$#vPw!1 z7#SH;-@er{F;P=f!|`dMYFS7cT0bVRl+)5ALT#Aa^Bxu%2jG__F5+3)8HSBaPhzbh z2LP!pIAo-Cwmt_`s&t)(oIh*+IyY<;W>0R8BTV#&^yhCCZ-$h}+bVMRqJ-R)Xi4a~oH8 z>cW^TYG&M8769V5YM85@aRQSZyBDQ=ICf7X>XQHxHZU*%5m{pX1Z6+tN=w1wkAG65 z0738o8=HZSZl&D81uFvMQZJ#j6rhZgQPG1}DJfmw^JTU`ZW0eTl`5L_FIRDAgt&Fx z?K<$Jt0F{2MP=2r46L~M%`1Dm@};MzgX$3m3VHNHD&z;qk`ogXvl}ID* zbROW~wC45s2vLI9eFM-pKy6iq6}ozQS_TG6YHEpT#D;=R{-yU{U0O$MPBL!1nv8KU z4YHT6bYJW0oh7Ix?~LB=perpLsh=55H^nB%7f#C*?2q)AbDKlRtM%Cdu(qhyKZL=U zSy*BKf9mY&8r!17!^8XL+^`@I+h6d%J)0&IbfE-U%;N-;ehmqp(d!(ik3l*EAl+^1 z+MwbY+mk22vw^WwUcZ)p_Kf86@)Arjp9K@FZhX&gchMiQnKMqkph$oJt_c{0eBt=x zfTF9ZvGtRiaa~>AWiOGE;^MsW@`SiJ%&|wrq@*dCnL9^sMYGj+eMEut1K{`c=~G2@ z^-lo3prD}s5;{AWuHG#QX*1(GAr>@!2{Hg{LQ&i6 z8xi@h`|aML-dbYvh4?80YwpAb>?9zmj*6Ck*w3=Jw+E&2O~X=1=FLqI8ZLE$dQll{ zd27Cj58&T|;Olc!TzDpISJtm{AOem6VnO=!Tg&A`buy2%Q}C@e;x+mj;T8XDR?C0Yq49Hy+{mxheYF zHz^I?$t~(UfEDHg4_b%Q`CfvGqOGH|<4j0St_+sw@~DpnRDj0$A8=oZn%uGmu0HR- znT_KD3gaj!7$Z>+2<@gnFW4_|2A=O)0Ep}~SD*Ojj~xL40VvB1jEv(=p00Gz81L~Jx<-=$zd z4e>J~KAYh~vP_Af3pbz`^Z`i4V4(eOs;Vlq7j}0ymkDHipQQ0CRi60+ z{2dz;v%Hb%x^cBr3HwlJHUiulkZKG}%u?IE>9<^Pj#K&d7?VoGYkA7Z)!A(rOtb0k zYUdKxem?>jW02uKVA?=^u(Y&X+aTd&W$-?v*S5tMICZ!Oua5XCD{I#?7!(%(p@;~D*wm_vR99NW*LNICeTKSk}O3%g?2V{gOK;0mXX24nl!@~@M zg43Hk=np9s?d?mjt7xBw@V^H6VP~o$7}FmtYi{s4S?JwKZtzuiwmLKXmrFzDOCX2T zenwn51q25U#K3g@00(1bWwm#kGjeHqQC`lT`*g8b>#=~#{I6w{N|2b%EC9y__T$MM zk(j0t@rqenTSHsmEPGbbx#z>@_2)ZUGF=u05IiaYIZ&6La86TjPc3 zTU%Qa9FFll6XxmRd3m(%_5jp9rw9S?j8E8vlN3~u9QGa^kQmENTQgX7J}EM-$XWjR zp3iRZB?>r1zz)cSAgzMQUJzYYRdFAsV1WU!0Qo>cN595KW@llcWo3OJDcK&dQ(9Ig z9B|44$4Js&6#-@rINw#WvH3K12Amvlk*1E|;MlKlTZ9u45kdZt%v4z*17sw@EDch~ z2$0L3R34I^e44hiTl^}FMap^FB*Dx31_)Z$#u?JT7Z(}|skdBTHA`+581W*>9BLi^*=c-Xf zS(%*Z`1si40sjqE@e`$2+1VzpCbF`!kvK&%GBQ01oM4I^`V~`XFXp$r2&kw`T>U|A z00p7wz_{@cpo&Y?J5ZMZ>2eY#;?JPH?)w3- zT>y+OjW>N-O3KU28|JNf`RywHmVf#+0BmvgJ&)IvKvjbthA7a%7^gHd2kaZ>f-x^k zf>ltZqi!&JSKKa$JTgb7%y9=YGBSor_h#4C&CSgt3(Cjsf-swAoFuF?wBd>Fsaun= zL>KO!Phg~X+n!Gg#w8>Dnn?D;mt8BKO+O3)>8m8zZtlD$xQL$37S2{LDLWSRXugbz zW_vc}&D)&SG%wk{+;!Nf&AFzX5vEPlGbul^?%{9X2Ed`t(CT#MAMfeIonX!Z3I*S- zC4d(7a17@`rV~5+p%VnSJUr)=>OC221|bs8)qX<|R-#iB@Wr#NwA62}Zu$a{i$7Bp z6d>#ycV(+V_7Gh3i);;qO|%C38rD1c_Xak7j?|Hp>mf_7oZjJz|HSm1JT4%;oq1z@ zy&5`10yHR2GM?u|FQ7y3#>)-(2nY#VWiPv93Cj)Z;y~#2sPq%dS%9Dbpy72^R*({w zzlqXxYAO+sI=BYL8eJAd+HP(<&-Z8db%r&v`fWGcWO71rcXj6UXSUo9CpQvPXB_+P zXZKCSJgJD+*PSlxU)ts(wg%?eaaN|y9>-w{xGX%BQjK{bz*-dgh!vA^zB)XnQ8q%R zYxKhH;8&d}&>bFuS@j%~(n*T4v9b99npJIB0U~?_caJx;t~wCr0t(Q@_0qj*rOX%& zoPNV@4y?8N7B%9N+9(2vD8)OUhvq7o_J%}j1f(X(xS%+6Jq*1%m}bv_A-z^3cyvD) z_-Le#^Zl2U%b?2%?4LR>E|C-?Ke)V`m~4N3gOn%$`BvE$P+ z)4JWh#vYKXA#tnw)LyQC><(Qt;M8ua(u`}}G}IOVR!|GqPsSwM*OW2gB4i+SXa?&E z>4aujY=M+g2=&yPrS1u*54Z+-;>MT5X;b@ua7nHKO)&r$ zy#my23yRrL6!__dO0`e~8p)f`4r99@A@&Cm&l%&S8g=|;%g^WOntxX$XEOB$;AU#k#) z?rrQm&#^u!7;_idZ zGx^60_nTb1rgfCD;_X#9~cN<)e!x5U!Zyn8Bxpq8_MZUlgwi}u(7s89}dcgI$5 ztm9N7KI9ii6f`=oh-zjK-;Q<*0ay%nD>1AG(#QYk`lLQUBDptlH3tB zkst+uMf|nO%m5*z^^^G28_PJ)nsJK6U6X5M6NSa-N&C(I`oz>Vsv>)31xq6)s|(>D zTK0L=V`QlFdmDdpoY{jLVgTXFBfA*%|G-yd@L5;?cG0kq4rIL5^WSDK}eYYxZh&vJ$1 z-z&`|qj72WYtU^C49Ec4uVlVr;^BD>823bx2F)eXozzVi{hyH4Jtu=J;>5WReib#( zpCA1!tBFx_lDXl}{#6%1mM+@BdpO$e?lx&yzWx!S?37r*V!byz*yjDj{@rX&L?PKQ znP`@U4t0@13#2wc{AGQ4&l-(v&>TxWhgo<2$QE|L#wk zr6C2NcF!?!1%NO@LpUnxOp~a^2Z4Hq^9|t6*!XzlW$^Ftd!9%j2=ReN3n19JLB|iq zB(B-Hz8*epC8z_i2(AbMxv!w0;8(Nv6Sze|Aq1icTEko-pja5K06zysH}|3gJo?jt zhddGR?7vJJM7c=z=MEpwPW#2)EOi1=Z`;*Qjs7vAi};_V-xg1a8%a`X1+VyLc~6UHI={C|^bhA#J@SGfC*(*_6y{0_U;7?amaHmm@u5TxJUzcN zNhKxG%u16{^xB1{RReJh;WM;lVv49Z2d2zG~N|}2fm9O7VeEiXe zkv*Mt4eI9O6kHPrTrj_PX=|Gvev=Lop&J&ubyF{008!PuEOp#>p6#tymxg&Zx0jYW zU7br({yKbIU7_RFVyv%%SyKNlx8aqjlfa<*2I1V7#OabcG~}acue?cbL5l~N9eY02 zd+k_FF2B1_q7@2C+3dSqI?c~lQw(6M~)-#1k z4apo7JkU@<4NEf%Ne2xt&{cqAE1)ABQk!Qd0mZU9_}Zr#v;itfF7stx`T@#UrO!?Y zU}(MbT)-j%yb*A@o;Gd=fvbQwCu2bUq6EqZ+!VL4r9LNtRgWe>DJUgFvWjv7;?hh5 z;H?cHo_5VQd0v8A#@Pn*JlAG{9DIL!$|Yfb#(Yq~QdeMej+^J5e3#JZd&T;+$ddc| zwWm304|CG0kMOvi_Yd7?0Y8Vww@;~e7T3M1?sh`?&YJ`=?1>ci^MiMnsKZ3WCgRgT5I3v;a~6;x=&EjPO!*Hw=HnyK)6?_e)29GqN&ubd`1y$eC#u<3 z)YQa#+kT}r4J3z{#6(60hA3cXz{BMX4AP5SH$i(t*zb@QWH7k(1_zaa)4<`gK}|Hc zXuQ$g3L7o5Nb@=14sbD$oBI_A)-N?RN~)@p`*`!QsA844w?A}=-X!aZa-jS zIQ!Gr>w4eSb|yK&l1~iSnWGXu$7H~FA=bCVCrh-|GJ;-ckI@Ayjxrd;&#s>np$2|@ zn=)e{aVOxWYYw7bvVY)_R%R}hJQTw>xnV1vveBIJ5{GeGdj6m+%ee?>Q(Et>Z0mp(}I zNd+uHpdAG;0dQb=XUO1-^#kCE3?d>Kebj+KD`OiR8X6cJJf;ipN-kqmTy2SUwJ5|GaWNZQV_!$FxFoGlI>)##(cRRRb~GX6k8-yf5D;j} z;Clg{T9NYw9T(nQbWq4boTcX$BdgvK6_hU1P|6!yzS*SyzI{4>{=@O+@Jptf{)`reXzJF;x?~?=I=1`e>fMK+drHyDH3>7owl&gL1W(2jl-6iw zRjwkEA4zO!{hq5=7`L;VJ`j-zu;fiNWWcAKiIJ4#Ps63jmF=y^AzI(i)Y1*>aH9Ko z;9W}tG3J#5CN~4^a+=T^p31hXQrq7ifSbW}l$L|GOc2)Yz({;PyZYpUF2FjJLT+KV z*RYf2NL&GZ*2+;=mpVg0N~Mb?SA1GPcL87J9XlO8eFUJV$|@@2?@WM93Z!JK-Cx!7 zpe8vjwPgdO3|34oS0We#tBga~(9|@ky}doi-}d_OAx_Z&tAu-ny28O6(+_`51Po`Q zzOxkSTSf_^+x0-FnU8`TE$8PRuZvw`>?RUom=<5F968T=70zpQ=yy+C6XY)gOQM=3QsTy4+7B{l0bP^(ZJ(e}jH4ZQ#jy<^GR@w(F$V zj}_n@;N-Cxmj=)4y@nbfNV&9K+U&PoyTJR2?IE{7z-2e8PuQ9)y2h47fh!WAvCo8v zhzK?UpuFsozCPar5-{kDQNdjlp6)8mG|bJNYvC{k-WfVNIufFM2$&?$^XULn%?-JV zEy@ka0EuA5ON5n^b7SQt&AU+<(seZ9zV&wRxs!{aQp`PcoXyE1YRmQZrH#eVV7ZW9 zX^0_JoJT24UfLB>eY_f~cTwK-31?_yeRQ4?#-Vq-M+zsE|opfBNNt`(uNc%W<#FWik_`MZqps)HjqugP_)Svj5Z?%mX~EqA)=>A}*H%Eb;? ziW1mmo2tjaCS$C4e4kw_1kYHf8Ivr3faBN%H#bQ*8O(h-!11n9Mgsxo#be8Acx^Qp= zEJ-*K4`Sv_C_Z|`wHu8)grrr88tE|xq&!HJ50p+e%1Y~syR6hn&5WE`%AM~*Z@e7e zjJ137t*>Uk$iRWw(04~h--+;9=rNZvmIoQ|_unaw%o19T;u7}L$-i?$TV9BoU6Lje z7;5R64t6$a@R{#2Kt!^w!+@#86|b9AjJk21jMYM>ERt*V_4H=F4pek&y5mFklXJhO zy~Z7-XAn6@N?&=E>mDee^?r=<@E~w9iyWO-f(L#HI1Y-V)dE**PY> zpS8j!^7Btkdl7%n-A%OUXgP2k+_#Coe2qKud-+pc(0!#aib5aH#pi3gH}j^l@Etjv z_;ka(X8 zJ&e95R$RwN<=3&UbK%dj1HpeEvb9D%ui{iDkmD$G`!Vd6+oZngZz>(cb+;? zT|v*Em26I+I&=Z|kD84LHnsHX6zi{2wwZ=~h4V@?CD|aP0daJ^lwag}FW%g&I+>md z19l9m&~X4uDrwI0@+Yr{&ZaWfL0GdmlY%p3bI z8K}m{T*J4;fAz3({oOZnwA?!`aNgACHpZ8B(E}L8jJ-yKBlVSptn;Ht~u5lB|>YYC5?v*=P z%ky9je=Rzfv=nj2XK-rT_-PEUdGW$lBFCR*T@F=<%10pRgtaBac=+293?N*>aR=|l z17cT~*?6YvtrzMYU0v++@Riq%S?cfFPuM0&$~z>5Qh$WV7{Gea zQ;BoF{n<>eyI^n*IPJ4LD!hN+UcG;EiSVnmy!?LGg!(FoA5dxg9(>!+~f^P(5~s+D#srBXti|dO$));7{C|qe&~_j?6(|ToaElCg03ml=)x0BZ>6}{44|V$AK>v-VfC7aa&S5}nV|sD{?8!s})eY@S#c2L&>X`Xeup*xXizQNF`Dgbm;-#4+i>q zdrB9=&*7$QyT1#Bn{$lE!NCFS>`5T!k3&R8*omy8h0liYf>~U_cZMda(BNAuV|8Z31ujYV!k*+>3YT8< zdN;2zANov2O~Eny2iQz|S6aJ9u*X^tVhP^TKMnD(0g7t9(@ZDdm{=re?(69RI{Yn( zN)&)hWU$EaxB#0xU)kpPM+2Ui6_ni(@t_0uvubVz0^Qs2esJ`6UA~nv;I*!Ba^xJOI$2briBM1)27qC+&)#4h@srU@pknIjS3Cns_Ljjp zXyQyXxSGNDBEgnIlZV!y7LL2kIX4`2;Y#{p+GJxHHMp#Q%5*tSof;V#fv$AEfNzRS zNWca=O%Z?CBs?VJjY7pDF*c$9tOZg5X`&J1YH>-gmQLd2vz@dl2XEs^Ta3GoeCnaE zx8GJCagaa^ITr3uNdpq86-Mu}vxGb_+VeCaB2;_m4~tChcls9_koCOzHfE-eaB&oD zxw-#*zUFzPX8r$kY~F!XzHb=UkdaM7Wo2b#oRiHlqL7u9L*;8k$)=DfBO@}S;@GnH zI>t#xMkh&DR{gAGWoI4y?$MAPFI(7%=Ji)#OPYEnAxdjivXNJmZGVk;$i z?+aV1@BAh4RTLWy-m;_*yZzOONjY7*uM(WKRPlPPcmYvtRTz**$Ov7!X+QtB?^`u6 zB~ScFWPt5%Z+pX}(suW7*Q_+q%9n6ZHRL|?A*45*eAjVxW5;FleDs|1R*$=+0GAEZ zsKmkRtxn>P=&biI!VgLGr{c{=8p31la>6uC$EAD&uqHV6a&OsF1BWIkEu9z`cpPvP z!eA7eWQ)!=85k>yQAtnh^nsZjuJaIt`My$7Q85J>9iZL;@mg<0dXtfn0bFxYYR^7c zE_VX`=7V?@6p{h8aNqc2Sn351dltZEK>d9K+NIPBz^QBilaHwBwfM z@5cheH0E1@Bg1j-!^L8uw*sCz4ml|4+16Z;(Z(HvcH`T*dz;;seG@l6;A+NuEJ z0={7l?9e^GTaCcXz;FKIr|87XOq8`laK zxJ$e6on7;GmKhkNX1~R__L2xVOdEhN=R~y>tlKCN+zeKYz;Ov!5PfN=k+wT)Gaw)O z$gie+F1P9U0ixn^)1iORgdema);}A45X(PxnipUz+4HGR?j9a*Wn4I;F$ege*7R3U zJtbkL*FXGrw-2as1lMl~Ijyub>Hky=+-IQ05U@r~t>w(>g$^LS7jo@eYY%y(CFiQF zbI|`IUv;i`eY6+$n*MD=jpTm@`e`mPU`~<)j!#xO`~?d6tvYu*rHp}vBM2KX=n9Wn zJ?Hv40pR4X>*{W9Hx9ui;j(WfZC>bS)C@li4h=QBeH)k}zt*mJ!3C$jd?P^wn+D(g zGhlIlMyd;QU&gdf00z08TCY z834WWdw?O+`Sy()G$5#^gYk%%v#j*E&FJZt)KuHH3O5^z!yOcQ*@dxVS(!S2AtE}@%N35DL^j21CGDn zLdIzX4M4lTb>?Ha(YQFKpWainoB^a&k-xPFcMlqzB6~YLQvT@CC2*ztnZX)vhYLL$ zp=v$n(ijWX7Y;XHLMo6-fO&VFdg8jvguR>Se%t7Gy{VHIBg%_D5?&)of0Fkd|DBmi z`RG!Q{!N-DMSr6`r>#)ttwYd zK>-kQ*jndq;D|tJny6nWxrjz5Ai0HVY>zi*ZXW;u0$|BItel753aRxHY;WLQr^k>a z;09HLP5}u1xl+4ZTtvhM2y)tc{?NcfW&}ivgv7)Oh^+i=T-PH*ax1U6FqfN-a2}Fq zd0X365->jzY0XyY;1L*`4c*%TChLi=nx>|vuSPODxXU(EKTSDxinNPDWDJ~MsA*vi zeE8UyWsh&M>1etOX{aMSD$1@cj8S>Wu^I~^1LP1k@XQaTlkJ}x7{my!8v$3jM*PhC z3-5OE?$gcJ`Y(~1$fN!u=Jo4Ox-#RQKW707``=VF=3n;a;mDGt!#8fYEr7t3ZNB%+ zmDqtXkt#3N)NWnwIu#{T;qps4?SmCUWQr>QSpf?S3?eiMzxtd8#X5-?%{dtCd zz@k7CviPWBR^P;=G+=0dW_r2|DxvRiZ0<$q=|Ir-C#_y9wE+OY^0FBLIb4BMxegTJ zS1sF6+H>t-w`ju7B{=3MD=KZ4=H-RK8UEGC&5+?_d8fEeTztHo+vLf3LH&PWscC6k zNEiY_Ii+S4{BW`$J#(YadZlvcC+q*9Q$+3uDc35+;tlGdbX5-qA_5Ngex?4V^Fokq zczOdC{_iQ;p^|tnmUqA;reb0O<^#k~&H`3`Wq5daQ=1HtTT*vt0}*~C5i}xbtFG5N zk1J|qA~6CmFMzB8Jeyc}F}m4Gw)8z6sBS-(Jj>*H@=igF_R;J zM&SLyqG5CX9n$;e!8tq#F={Q zg@iO`MFc6{Zp|FKu1XJAYT_O9iO7tBWEtjgM<4nK@ekY9g!XCsAk8efW#QBo7ZRPx{UBb; zW&~b4JlJYVU`ev7Je>3=_Vn1##pM_e^?$2zk|OwSuH6~~Y~OxiFfFg7cm`o(+CE&9g-8=E5Q7HCpv_PxTy1B$qC`~<=X93Wr4gDd%~ zN6%N_7;V{Oe#K5sT5~=n{RjX38Jf|a{LEbB$*B}SnFioK`}PCDgW2|ETa^PccaLb) z@2?i-^|qIwiIUpNsFC5eRn89`YB?f0b?7!j!DA@QqeKmKmbuym50u z8cGgNIn@!)pK0n1={};F=07I^>*ihtOR!2i(QSQw3-&X&IEcJnBs5e^)3 z%`&_SXNc!E1PcqaAL-GpW4zNGM{4Y>nvNk6x0AP#zr-*l5xahiS!msDdNtNKR=LJ6 z646pJ-TX)}*(N}1G=08jVsx1`78f5>P%w+;$rGX;{H#5gU24}+ zf*W()m%s|tv{{#Tf^K{|(6wC7&_Km^7c1>F9@sh|Tl{QD_kOD0B#}tm`QIzW&{Y*TSvpM^~himm*E^7>>At@spQ)r5Bcr9j@S4FBE6? zI>;u(s{ZcHs&eXk!4vd?fr%r~^x~#2;2x^my5OvL-?Q2JsZ@j|L@Z9pIVAMy>pfrz znSLOntIsmg#LryY+Lb;iIsNI{nh`!kJE>Cj^!S|TTHmV+=Vd*^lnGDz^3|^e2C4H| z8X_pmdh?8XYyfXs$jc8if68rSabNo)@TVv>bfO!zBS-1ejpbZV{JG*FwHDreiLXHG z#d;pm5`L%$tz=i0rh z%PcYsS%0)-%{8YYnO&OHxOy8~h8c^#C~v?~R%2q4gU=2=6y1*7 zr}&4O4{&bb^=dA4;A1dhxXewLd&1F+e7xR^M-*M9$}@E0{v$D^+I!#l_Q|GMc34;C zS;&)G^{jT4<#U#2cGVOPB5G940NvlV_(EEb8Y(lz&b(xVEJd(~+A)HvPN?2eXLlF> ztoI!!IWrB_LLqdm`Y7`mwxQ)~10Usl%1zF6cWEE@A8XzWmpp$xj2rdsMm}}{GqkWN zyC@x@$6EGq{ZvIor6D7x4#^+Q+T#=!uWDXnVCB>_Hzy}=p#L_}Nk#I|%bZTiI!N65 zk6hL>Wl$~z?Zo)$d7_)`JM{GIOw3!iut4UWxSdN#NGAl&Av88*MI~S)QNh)zds`#y zTp9ihhS?-M%JmX;VjvS*F6RR-Jt7cm(}7n=PS~$2awsU<^A@0v2*V>caaQ%a;rSAj zQ{cP)V8UvKE%}?gkGkvIu5nW$QNwtTQ*e-m+-XkNP=nG){^G+>#jjdb5$yL>@FQD;ZGmW zQAy;69Ev4Yh`M0%y-kO)c7#sX7e#GDLjXrkMQN&ox~maFzBpWsLmLUb$@VNpN?cOr zyD3k3DjMqar7KzqN#?_~kGZbVadD%jB;YW0$Yw}9`YT8Q!3LtFSrc6ERQ>n&=cSzc zAt+vDP~ut0$BQ%TO&Hyd@6 zZp@9#wCu-Kw=agAwIaCT^U1354CqPJS%lo8M-fbd=vc`8t^^w- zj9`UD9@}isNk}T)WOEGXsYqw~WC%QyM%z(sQjc;jeo^7mQcK)Hkn}Ics zgphM#H_31}Y>tvsTIs@8w6y63_QD5FOe=RAS+`QN43<_1L3d7C%08KPW(+=EGBST( zY<(?^o7%aFZs@{LhL-bj=`C`(8f5KS$W==FnAsRS+pluwnr4*g8tJ_v&~47 z4isJ;m@DDWT!tpca3&4B{n2U1jR!|h55MsUEP27rZi!snUmUs$-N|Q~3olX%W>s91^wy2eVr{gPd!o1+t2 zf{ut#RcZ~+vB-)E&_(^EswtmoGh|nph$LZRh~fUHrqNbN=wC}i?`nyfRp9>t7b0^1 literal 0 HcmV?d00001