From 8965db839f6f98a3800b7bc978ecb852c1bf6a3a Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 9 Jan 2020 14:59:33 -0800 Subject: [PATCH 01/19] Create LICENSE --- LICENSE | 201 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..261eeb9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. From 3131cf882662c0fd5dcf92341e51261d6af18c25 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Mon, 8 Jun 2020 14:15:58 -0700 Subject: [PATCH 02/19] Update model_lstm.py --- neural_ner/model_lstm.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/neural_ner/model_lstm.py b/neural_ner/model_lstm.py index f42d207..cfab61c 100644 --- a/neural_ner/model_lstm.py +++ b/neural_ner/model_lstm.py @@ -108,11 +108,10 @@ def forward(self, batch): h = self.tanh_layer(h) logits = self.hidden2tag(h) logits = logits.view(b, t_k, -1) - + logits = F.log_softmax(logits, dim=2) return logits def neg_log_likelihood(self, logits, y, s_lens): - log_smx = F.log_softmax(logits, dim=2) loss = F.nll_loss(log_smx.transpose(1, 2), y, ignore_index=Constants.TAG_PAD_ID, reduction='none') loss = loss.sum(dim=1) / s_lens.float() loss = loss.mean() From cf945ab9ba98cb10d60f258132c3ab26b5a83811 Mon Sep 17 00:00:00 2001 From: atulkum Date: Wed, 17 Jun 2020 12:00:22 -0700 Subject: [PATCH 03/19] added bert version --- NER_BERT/LICENSE | 21 +++ NER_BERT/README.md | 1 + NER_BERT/bert_data_util.py | 92 ++++++++++ NER_BERT/const.py | 44 +++++ NER_BERT/crf.py | 182 ++++++++++++++++++++ NER_BERT/dataset_utils.py | 320 +++++++++++++++++++++++++++++++++++ NER_BERT/eval_util.py | 74 ++++++++ NER_BERT/lstm_data_util.py | 80 +++++++++ NER_BERT/model_bert_train.py | 280 ++++++++++++++++++++++++++++++ NER_BERT/model_lstm.py | 155 +++++++++++++++++ NER_BERT/model_lstm_train.py | 258 ++++++++++++++++++++++++++++ NER_BERT/model_utils.py | 99 +++++++++++ NER_BERT/transformer.py | 110 ++++++++++++ 13 files changed, 1716 insertions(+) create mode 100644 NER_BERT/LICENSE create mode 100644 NER_BERT/README.md create mode 100644 NER_BERT/bert_data_util.py create mode 100644 NER_BERT/const.py create mode 100644 NER_BERT/crf.py create mode 100644 NER_BERT/dataset_utils.py create mode 100644 NER_BERT/eval_util.py create mode 100644 NER_BERT/lstm_data_util.py create mode 100644 NER_BERT/model_bert_train.py create mode 100644 NER_BERT/model_lstm.py create mode 100644 NER_BERT/model_lstm_train.py create mode 100644 NER_BERT/model_utils.py create mode 100644 NER_BERT/transformer.py diff --git a/NER_BERT/LICENSE b/NER_BERT/LICENSE new file mode 100644 index 0000000..642dede --- /dev/null +++ b/NER_BERT/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2019 Atul Kumar + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/NER_BERT/README.md b/NER_BERT/README.md new file mode 100644 index 0000000..0f0b7d0 --- /dev/null +++ b/NER_BERT/README.md @@ -0,0 +1 @@ +# WIP: BERT vs BiLSTM NER diff --git a/NER_BERT/bert_data_util.py b/NER_BERT/bert_data_util.py new file mode 100644 index 0000000..3acc676 --- /dev/null +++ b/NER_BERT/bert_data_util.py @@ -0,0 +1,92 @@ +from __future__ import absolute_import, division, print_function + +import os + +from transformers import BertTokenizer + +import torch +import torch.nn as nn + +import const + +bert_tokenizer = BertTokenizer.from_pretrained(const.MODEL_TYPE, do_lower_case=True) + +def get_bert_data(examples, tag2id, config): + bert_data = [] + for orig_tokens, orig_tag in examples: + input_ids, label_ids, segment_ids, tokens = prepare_bert_input(orig_tokens, orig_tag, tag2id, config) + bert_data.append((input_ids, label_ids, segment_ids)) + return bert_data + +def prepare_bert_input(orig_tokens, orig_tag, tag2id, config): + tokens = [] + label_ids = [] + + assert len(orig_tag) == len(orig_tokens), orig_tag + orig_tokens + + for i, t in enumerate(orig_tokens): + label_t = tag2id[orig_tag[i]] + bert_tokens = bert_tokenizer.tokenize(t) + bert_tokens_len = len(bert_tokens) + if bert_tokens_len > 0: + tokens.extend(bert_tokens) + label_ids.append(label_t) + + # pad label if multiple tokens for a single word + if bert_tokens_len > 1: + label_ids.extend([const.label_pad_id] * (bert_tokens_len - 1)) + + assert len(tokens) == len(label_ids) + ###truncate large sequence### + tokens = tokens[:config.max_seq_length] + label_ids = label_ids[:config.max_seq_length] + ############################## + segment_ids = [const.sequence_a_segment_id] * len(tokens) + + tokens = [const.cls_token] + tokens + [const.sep_token] + + label_ids = [const.label_pad_id] + label_ids + [const.label_pad_id] + segment_ids = [const.cls_token_segment_id] + segment_ids + [const.sequence_a_segment_id] + input_ids = bert_tokenizer.convert_tokens_to_ids(tokens) + + assert len(input_ids) == len(label_ids) + assert len(input_ids) == len(segment_ids) + + input_ids = torch.tensor(input_ids).long() + label_ids = torch.tensor(label_ids).long() + segment_ids = torch.tensor(segment_ids).long() + return input_ids, label_ids, segment_ids, tokens + +def create_batch(train_data, batch_ids, is_cuda): + max_len = max([len(train_data[bi][0]) for bi in batch_ids]) + batch_input_ids = [] + batch_label_ids = [] + batch_segment_ids = [] + for bi in batch_ids: + input_ids, label_ids, segment_ids = train_data[bi] + pad_len = max_len - len(input_ids) + padding_op = nn.ConstantPad1d((0, pad_len), const.pad_token_id) + batch_input_ids.append(padding_op(input_ids).unsqueeze(0)) + padding_op = nn.ConstantPad1d((0, pad_len), const.label_pad_id) + batch_label_ids.append(padding_op(label_ids).unsqueeze(0)) + padding_op = nn.ConstantPad1d((0, pad_len), const.pad_token_segment_id) + batch_segment_ids.append(padding_op(segment_ids).unsqueeze(0)) + + batch_input_ids = torch.cat(batch_input_ids) + batch_label_ids = torch.cat(batch_label_ids) + batch_segment_ids = torch.cat(batch_segment_ids) + + att_mask = batch_input_ids.ne(const.pad_token_id) + + if is_cuda: + batch_input_ids = batch_input_ids.cuda() + batch_label_ids = batch_label_ids.cuda() + batch_segment_ids = batch_segment_ids.cuda() + att_mask = att_mask.cuda() + + inputs = {'input_ids': batch_input_ids, + 'attention_mask': att_mask, + 'token_type_ids': batch_segment_ids, + 'labels': batch_label_ids} + return inputs + diff --git a/NER_BERT/const.py b/NER_BERT/const.py new file mode 100644 index 0000000..7624956 --- /dev/null +++ b/NER_BERT/const.py @@ -0,0 +1,44 @@ +from torch.nn import CrossEntropyLoss + +pad_token_id = 0 + +ENTITY_OTHER = 'O' +ENTITY_BEGIN = 'B-' +ENTITY_CONT = 'I-' +ENTITY_SINGLE = 'S-' +ENTITY_END = 'E-' + +UNK_INTENT = 'unknown' + +label_pad_id = CrossEntropyLoss().ignore_index +sep_token = "[SEP]" +cls_token = "[CLS]" +cls_token_segment_id = 0 +sequence_a_segment_id = 0 +pad_token_segment_id = 0 + +MODEL_TYPE = 'bert-base-uncased' + +ENTITY_NAMES=[ +"Appeal_to_Authority", +"Appeal_to_fear-prejudice", +"Black-and-White_Fallacy", +"Causal_Oversimplification", +"Doubt", +"Exaggeration,Minimisation", +"Flag-Waving", +"Loaded_Language", +"Name_Calling,Labeling", +"Obfuscation,Intentional_Vagueness,Confusion", +"Repetition", +"Slogans", +"Thought-terminating_Cliches", +"Whataboutism,Straw_Men,Red_Herring", +"Bandwagon,Reductio_ad_hitlerum" +] + +_UNK = "" +_PAD = "" +_START_VOCAB = [_UNK, _PAD] +UNK_ID = 0 +PAD_ID = 1 \ No newline at end of file diff --git a/NER_BERT/crf.py b/NER_BERT/crf.py new file mode 100644 index 0000000..d5ebe09 --- /dev/null +++ b/NER_BERT/crf.py @@ -0,0 +1,182 @@ +from __future__ import unicode_literals, print_function, division + +import torch +import torch.nn as nn +from torch.nn.utils.rnn import pad_sequence +import numpy as np + +is_cuda = torch.cuda.is_available() + +class CRF_Loss(nn.Module): + def __init__(self, tagset_size, pad_token_id, tag_pad_id): + super(CRF_Loss, self).__init__() + self.start_tag = tagset_size + self.end_tag = tagset_size + 1 + self.num_tags = tagset_size + 2 + self.tag_pad_id = tag_pad_id + self.pad_token_id = pad_token_id + + self.transitions = nn.Parameter(torch.Tensor(self.num_tags, self.num_tags)) + nn.init.constant_(self.transitions, -np.log(self.num_tags)) + + self.transitions.data[self.end_tag, :] = -10000 + self.transitions.data[:, self.start_tag] = -10000 + + def get_log_p_z(self, emissions, mask): + seq_len = emissions.shape[1] + log_alpha = emissions[:, 0].clone() + log_alpha += self.transitions[self.start_tag, : self.start_tag].unsqueeze(0) + + for idx in range(1, seq_len): + broadcast_emissions = emissions[:, idx].unsqueeze(1) + broadcast_transitions = self.transitions[ : self.start_tag, : self.start_tag].unsqueeze(0) + broadcast_logprob = log_alpha.unsqueeze(2) + score = broadcast_logprob + broadcast_emissions + broadcast_transitions + + score = torch.logsumexp(score, 1) + log_alpha = score * mask[:, idx].unsqueeze(1) + log_alpha.squeeze(1) * (1.0 - mask[:, idx].unsqueeze(1)) + + log_alpha += self.transitions[: self.start_tag, self.end_tag].unsqueeze(0) + return torch.logsumexp(log_alpha.squeeze(1), 1) + + def get_log_p_Y_X(self, emissions, mask, orig_tags): + seq_len = emissions.shape[1] + tags = orig_tags.clone() + tags[tags < 0] = 0 + + llh = self.transitions[self.start_tag, tags[:, 0]].unsqueeze(1) + llh += emissions[:, 0, :].gather(1, tags[:, 0].view(-1, 1)) * mask[:, 0].unsqueeze(1) + + for idx in range(1, seq_len): + old_state, new_state = ( + tags[:, idx - 1].view(-1, 1), + tags[:, idx].view(-1, 1), + ) + emission_scores = emissions[:, idx, :].gather(1, new_state) + transition_scores = self.transitions[old_state, new_state] + llh += (emission_scores + transition_scores) * mask[:, idx].unsqueeze(1) + + last_tag_indices = mask.sum(1, dtype=torch.long) - 1 + last_tags = tags.gather(1, last_tag_indices.view(-1, 1)) + + llh += self.transitions[last_tags.squeeze(1), self.end_tag].unsqueeze(1) + + return llh.squeeze(1) + + def log_likelihood(self, emissions, tags, mask): + log_z = self.get_log_p_z(emissions, mask) + log_p_y_x = self.get_log_p_Y_X(emissions, mask, tags) + return log_p_y_x - log_z + + def get_crf_loss(self, logits, y): + mask = y.ne(self.tag_pad_id) + s_lens = mask.sum(1) + loss = -1 * self.log_likelihood(logits, y, mask.float()) + loss = loss / s_lens.float() + loss = loss.mean() + return loss + + def viterbi_decode(self, emissions, mask): + mask = mask.float() + b, seq_len, d = emissions.shape + log_prob = emissions[:, 0].clone() + log_prob += self.transitions[self.start_tag, : self.start_tag].unsqueeze(0) + + end_scores = log_prob + self.transitions[: self.start_tag, self.end_tag].unsqueeze(0) + + best_scores_list = [] + best_scores_list.append(end_scores.unsqueeze(1)) + + best_paths_0 = torch.Tensor().long() + if is_cuda: + best_paths_0 = best_paths_0.cuda() + best_paths_list = [best_paths_0] + + for idx in range(1, seq_len): + broadcast_emissions = emissions[:, idx].unsqueeze(1) + broadcast_transmissions = self.transitions[: self.start_tag, : self.start_tag].unsqueeze(0) + broadcast_log_prob = log_prob.unsqueeze(2) + score = broadcast_emissions + broadcast_transmissions + broadcast_log_prob + max_scores, max_score_indices = torch.max(score, 1) + best_paths_list.append(max_score_indices.unsqueeze(1)) + end_scores = max_scores + self.transitions[: self.start_tag, self.end_tag].unsqueeze(0) + + best_scores_list.append(end_scores.unsqueeze(1)) + log_prob = max_scores + + best_scores = torch.cat(best_scores_list, 1).float() + best_paths = torch.cat(best_paths_list, 1) + + max_scores, max_indices_from_scores = torch.max(best_scores, 2) + + valid_index_tensor = torch.tensor(0).long() + padding_tensor = torch.tensor(self.tag_pad_id).long() + + if is_cuda: + valid_index_tensor = valid_index_tensor.cuda() + padding_tensor = padding_tensor.cuda() + #alternative to where + #curr_mask = mask[:, seq_len - 1].float() + #labels = max_indices_from_scores[:, seq_len - 1] * curr_mask + torch.logical_not(curr_mask) * padding_tensor + + labels = max_indices_from_scores[:, seq_len - 1] + labels = torch.where(mask[:, seq_len - 1] != 1.0, padding_tensor, labels) + all_labels = labels.unsqueeze(1).long() + ##### + labels_score = max_scores[:, seq_len - 1] + all_labels_score = labels_score.unsqueeze(1) + #### + for idx in range(seq_len - 2, -1, -1): + indices_for_lookup = all_labels[:, -1].clone() + indices_for_lookup = torch.where(indices_for_lookup == self.tag_pad_id, + valid_index_tensor, + indices_for_lookup) + + indices_from_prev_pos = best_paths[:, idx, :].gather(1, indices_for_lookup.view(-1, 1).long()).squeeze(1) + indices_from_prev_pos = torch.where(mask[:, idx + 1] != 1.0, padding_tensor, indices_from_prev_pos) + + indices_from_max_scores = max_indices_from_scores[:, idx] + indices_from_max_scores = torch.where(mask[:, idx + 1] == 1.0, padding_tensor, indices_from_max_scores) + + labels = torch.where(indices_from_max_scores == self.tag_pad_id, + indices_from_prev_pos, + indices_from_max_scores) + # Set to ignore_index if present state is not valid. + labels = torch.where(mask[:, idx] != 1.0, padding_tensor, labels) + all_labels = torch.cat((all_labels, labels.view(-1, 1).long()), 1) + ###### + labels_score = max_scores[:, idx] + all_labels_score = torch.cat((all_labels_score, labels_score.view(-1, 1)), 1) + #### + #think about squeezing this score between 0 and 1 + last_tag_indices = mask.sum(1, dtype=torch.long) - 1 + sentence_score = max_scores.gather(1, last_tag_indices.view(-1, 1)).squeeze(1) + all_labels = torch.flip(all_labels, [1]) + all_labels_score = torch.flip(all_labels_score, [1]) + + return sentence_score, all_labels, all_labels_score + + def structural_perceptron_loss(self, emissions, tags): + mask = tags.ne(self.tag_pad_id).float() + + best_scores, pred = self.viterbi_decode(emissions, mask, is_cuda) + log_p_y_x = self.get_log_p_Y_X(emissions, mask, tags) + + delta = torch.sum(tags.ne(pred).float()*mask, 1) + + margin_loss = torch.clamp(best_scores + delta - log_p_y_x, min=0.0) + return margin_loss + + def bert_output2crf_input(self, logits_ner, labels): + mask = labels.ne(self.tag_pad_id) + lens = mask.sum(1).view(-1).tolist() + + logits_selected = torch.masked_select(logits_ner, mask.unsqueeze(2)).view(-1, logits_ner.size()[-1]) + logits_split = torch.split(logits_selected, lens) + logits_padded = pad_sequence(logits_split, batch_first=True, padding_value=self.pad_token_id) + + labels_selected = torch.masked_select(labels, mask) + labels_split = torch.split(labels_selected, lens) + labels_padded = pad_sequence(labels_split, batch_first=True, padding_value=self.tag_pad_id) + + return logits_padded, labels_padded diff --git a/NER_BERT/dataset_utils.py b/NER_BERT/dataset_utils.py new file mode 100644 index 0000000..fc45444 --- /dev/null +++ b/NER_BERT/dataset_utils.py @@ -0,0 +1,320 @@ +from __future__ import absolute_import, division, print_function +import glob +import os.path +import codecs +from pathlib import Path +import os +from collections import defaultdict +from sklearn.model_selection import train_test_split +import json +import re + +import const + +def tokenize(w): + q = w + contr_dict = {"’": "'", + "i\'m": "i am", + "won\'t": " will not", + "\'s": " s", + "\'ll": " will", + "\'ve": " have", + "n\'t": " not", + "\'re": " are", + "\'d": " would", + "y'all": " all of you"} + + for contr in contr_dict: + q = q.replace(contr, contr_dict[contr]) + + q_arr = re.findall(r"[\w]+[']*[\w]+|[\w]+|[.,!?;:]", q, re.UNICODE) + + q = ' '.join(q_arr) + q = re.sub('[0-9]{5,}', '#####', q) + q = re.sub('[0-9]{4}', '####', q) + q = re.sub('[0-9]{3}', '###', q) + q = re.sub('[0-9]{2}', '##', q) + + q = q.strip().lower().split() + if len(q) == 0: + return [w] + return q + +def tagid2tag_seq(tag_vocab, tagid_seq): + return [[tag_vocab[t] for t in tag_seq] for tag_seq in tagid_seq] + +def get_tokenize_tag(tagging_type, t, n): + if n == 1: + return [t] + if t == const.ENTITY_OTHER or t.startswith(const.ENTITY_CONT) or tagging_type == 'B': + return [t] * n + + name = t[2:] + start = t[:2] + tags = [] + if start == const.ENTITY_BEGIN: + tags.append(t) + for i in range(1, n): + tags.append(const.ENTITY_CONT + name) + return tags + + if tagging_type == "BIOES": + if start == const.ENTITY_SINGLE: + tags.append(t) + for i in range(1, n-1): + tags.append(const.ENTITY_CONT + name) + tags.append(const.ENTITY_END + name) + elif start == const.ENTITY_END: + for i in range(n-1): + tags.append(const.ENTITY_CONT + name) + tags.append(t) + + return tags + +def get_tag_vocab(config): + tags = [const.ENTITY_OTHER] + if config.tagging_type == 'B': + tags.append(const.ENTITY_BEGIN) + else: + tags.extend([const.ENTITY_BEGIN + t for t in const.ENTITY_NAMES]) + tags.extend([const.ENTITY_CONT + t for t in const.ENTITY_NAMES]) + + if config.tagging_type == "BIOES": + tags.extend([const.ENTITY_END + t for t in const.ENTITY_NAMES]) + tags.extend([const.ENTITY_SINGLE + t for t in const.ENTITY_NAMES]) + + return tags + +def read_articles_from_file_list(folder_name, file_pattern="*.txt"): + file_list = glob.glob(os.path.join(folder_name, file_pattern)) + articles = [] + for filename in sorted(file_list): + article_id = os.path.basename(filename).split(".")[0][7:] + with codecs.open(filename, "r", encoding="utf8") as f: + articles.append((article_id, f.read())) + return articles + + +def parse_label(label_path): + labels = [] + f = Path(label_path) + + if not f.exists(): + return labels + + for line in open(label_path): + parts = line.strip().split('\t') + labels.append({'start': int(parts[2]), 'end': int(parts[3]), 'type': parts[1]}) + + labels.sort(key=lambda s: (s['start'], -s['end'])) + return labels + +def clean_text(article): + sentences = article.split('\n') + end = -1 + res = [] + for sentence in sentences: + start = end + 1 + end = start + len(sentence) # length of sequence + if sentence != "": # if not empty line + res.append({'start': start, 'end': end, 'sentence': sentence}) + return res + +def get_overlapping_entities(entities): + etree = defaultdict(set) + # inefficeint but clean + for a in range(len(entities)): + ea = entities[a] + for b in range(a + 1, len(entities)): + eb = entities[b] + overlap_start = max(ea['start'], eb['start']) + overlap_end = min(ea['end'], eb['end']) + if overlap_start <= overlap_end: + # if eb['end'] > ea['end']: + # print('partial', ea, eb) + etree[a].add(b) + etree[b].add(a) + assert all([(a in etree) for k in etree for a in etree[k]]) + # assert all([len(etree[k]) == 1 for k in etree]) + return etree + +def get_per_sentence_entity(entities, ds, de): + d_entities = [] + for a in range(len(entities)): + ea = entities[a] + overlap_start = max(ea['start'], ds) + overlap_end = min(ea['end'], de) + if overlap_start <= overlap_end: + d_entities.append({ + 'type': ea['type'], + 'start': overlap_start - ds, + 'end': overlap_end - ds + }) + d_entities.sort(key=lambda s: (s['start'], -s['end'])) + return d_entities + + +def get_non_overlapping_seq(etree, n): + # get non overlapping entity sequence + ex = set() + if len(etree) > 0: + for k in etree: + ex.add(tuple([i for i in range(n) if i not in etree[k]])) + else: + ex.add(tuple(range(n))) + return ex + + +def get_tag_seq(n, name, tagging_type): + tags = [] + for i in range(n): + tag = None + if tagging_type == 'IOBES': + if i == 0: + if n == 1: + tag = const.ENTITY_SINGLE + else: + tag = const.ENTITY_BEGIN + elif i == n - 1: + tag = const.ENTITY_END + else: + tag = const.ENTITY_CONT + elif tagging_type == 'IOB': + if i == 0: + tag = const.ENTITY_BEGIN + else: + tag = const.ENTITY_CONT + elif tagging_type == 'B': + tag = const.ENTITY_BEGIN + else: + raise Exception('tagging_type no recognized') + assert tag is not None + if tagging_type == 'B': + tags.append(tag) + else: + tags.append(tag + name) + + return tags + +def encode_tokens_json(e, sentence, d_entities): + tokens = [] + labels = [] + curr = 0 + for a in sorted(e): + ea = d_entities[a] + pre = sentence[curr:ea['start']].split() + span = sentence[ea['start']:ea['end']].split() + + tokens.extend(pre) + start = len(tokens) + tokens.extend(span) + end = len(tokens) + labels.append({ 'type': ea['type'], + 'start': start, + 'end' : end}) + curr = ea['end'] + + pre = sentence[curr:].split() + tokens.extend(pre) + + return {'tokens': tokens, + 'labels': labels} + +def get_data_train_dev(root_dir, filename, config): + examples = [] + for line in codecs.open(os.path.join(root_dir, filename), "r", encoding="utf8"): + datum = json.loads(line) + orig_tokens = datum['tokens'] + assert len(orig_tokens) > 0, line + orig_tags = [const.ENTITY_OTHER]* len(datum['tokens']) + for e in datum['labels']: + orig_tags[e['start']:e['end']] = get_tag_seq(e['end'] - e['start'], e['type'], config.tagging_type) + + tokens = [] + tags = [] + for i, w in enumerate(orig_tokens): + w_i = tokenize(w) + t_i = orig_tags[i] + tokens.extend(w_i) + tokenized_tags = get_tokenize_tag(config.tagging_type, t_i, len(w_i)) + tags.extend(tokenized_tags) + + assert len(w_i) == len(tokenized_tags), f'{w_i} => {tokenized_tags}' + assert len(tokens) == len(tags) and len(tokens) > 0, f'{tokens} => {tags}' + examples.append((tokens, tags)) + return examples + + +def get_data_test(root_dir, filename): + examples = [] + metadata = [] + + for line in codecs.open(os.path.join(root_dir, filename), "r", encoding="utf8"): + datum = json.loads(line) + orig_tokens = datum['tokens'] + tokens = [] + ignore_mapping = [] + data_tokens = [] + for i, w in enumerate(orig_tokens): + w_i = tokenize(w) + tokens.extend(w_i) + ignore = [0]*len(w_i) + ignore[0] = 1 + ignore_mapping.extend(ignore) + data_tokens.extend([w]*len(w_i)) + + tags = [const.ENTITY_OTHER] * len(tokens) + examples.append((tokens, tags)) + metadata.append({'article_id': datum['article_id'], + 'start_sentence': datum['start_sentence'], + 'end_sentence': datum['end_sentence'], + 'ignore_mapping': ignore_mapping, + 'data_tokens': data_tokens + }) + return examples, metadata + + +def dump_data(root_dir): + train_data = read_articles_from_file_list(os.path.join(root_dir, 'train-articles')) + label_dir = os.path.join(root_dir, 'train-labels-task2-technique-classification') + + examples = [] + for article_id, line in train_data: + entities = parse_label(os.path.join(label_dir, f'article{article_id}.task2-TC.labels')) + + for d in clean_text(line): + d_entities = get_per_sentence_entity(entities, d['start'], d['end']) + etree = get_overlapping_entities(d_entities) + ex = get_non_overlapping_seq(etree, len(d_entities)) + + sentence = d['sentence'] + for e in ex: + datum = encode_tokens_json(e, sentence, d_entities) + datum['article_id'] = article_id + examples.append(json.dumps(datum)) + + train_data, dev_data = train_test_split(examples, test_size=0.2) + with codecs.open(os.path.join(root_dir, 'train.jsonl'), "w", encoding="utf8") as f: + f.write('\n'.join(train_data) + '\n') + with codecs.open(os.path.join(root_dir, 'dev.jsonl'), "w", encoding="utf8") as f: + f.write('\n'.join(dev_data) + '\n') + +def dump_data_test(root_dir): + examples = [] + dev_data = read_articles_from_file_list(os.path.join(root_dir, 'dev-articles')) + + for article_id, line in dev_data: + for d in clean_text(line): + datum = {'tokens':d['sentence'].split(), + 'start_sentence': d['start'], + 'end_sentence': d['end'], + 'article_id':article_id} + examples.append(json.dumps(datum)) + + with codecs.open(os.path.join(root_dir, 'test_phase0.jsonl'), "w", encoding="utf8") as f: + f.write('\n'.join(examples) + '\n') + +if __name__ == "__main__": + data_dir = os.path.join(os.path.expanduser("~"), 'prop/datasets') + dump_data(data_dir) + dump_data_test(data_dir) diff --git a/NER_BERT/eval_util.py b/NER_BERT/eval_util.py new file mode 100644 index 0000000..d3e8a0f --- /dev/null +++ b/NER_BERT/eval_util.py @@ -0,0 +1,74 @@ +from __future__ import absolute_import, division, print_function + +from seqeval.metrics import precision_score, recall_score, f1_score + +import const +import os +import dataset_utils + +# eval +def get_chunks(seq, ignore_I_mismatch=False): + chunks = [] + chunk_type, chunk_start = None, None + for i, tok in enumerate(seq): + if tok == const.ENTITY_OTHER: + if chunk_type is not None: + chunks.append((chunk_type, chunk_start, i)) + chunk_type, chunk_start = None, None + else: + curr_chunk_type = tok[2:] + chunk_prefix = tok[:2] + if chunk_prefix == const.ENTITY_BEGIN: + if chunk_type is not None: + chunks.append((chunk_type, chunk_start, i)) + + chunk_type, chunk_start = curr_chunk_type, i + elif chunk_prefix == const.ENTITY_CONT and not ignore_I_mismatch: + if chunk_type is not None and chunk_type != curr_chunk_type: + chunks.append((chunk_type, chunk_start, i)) + chunk_type, chunk_start = None, None + # end condition + if chunk_type is not None: + chunks.append((chunk_type, chunk_start, len(seq))) + + chunks = list(set(chunks)) + chunks.sort(key=lambda s: s[0]) + return chunks + +def evaluate(gold_label_list, preds_list, config): + tag_vocab = dataset_utils.get_tag_vocab(config) + gold_label_list = dataset_utils.tagid2tag_seq(tag_vocab, gold_label_list) + preds_list = dataset_utils.tagid2tag_seq(tag_vocab, preds_list) + + results = { + "precision": precision_score(gold_label_list, preds_list), + "recall": recall_score(gold_label_list, preds_list), + "f1": f1_score(gold_label_list, preds_list) + } + return results + + +def dump_result(preds_list, metadata, test_data, root_dir, filename, config): + tag_vocab = dataset_utils.get_tag_vocab(config) + preds_list = dataset_utils.tagid2tag_seq(tag_vocab, preds_list) + + tc = os.path.join(root_dir, 'tc_' + filename) + si = os.path.join(root_dir, 'si_' + filename) + + with open(tc, "w") as tc_writer, open(si, "w") as si_writer: + for i, t in enumerate(preds_list): + article = metadata[i] + sen_start = article['start_sentence'] + article_id = article['article_id'] + ignore_mapping = article['ignore_mapping'] + data_tokens = article['data_tokens'] + + for type, start, end in get_chunks(t): + #adjust start end + orig_tokens = [data_tokens[i] for i in range(start) if ignore_mapping[i] == 1] + start_boundary = len(' '.join(orig_tokens)) + sen_start + orig_tokens = [data_tokens[i] for i in range(start, end) if ignore_mapping[i] == 1] + end_boundary = start_boundary + len(' '.join(orig_tokens)) + si_writer.write(f'{article_id}\t{start_boundary}\t{end_boundary}\n') + tc_writer.write(f'{article_id}\t{type}\t{start_boundary}\t{end_boundary}\n') + diff --git a/NER_BERT/lstm_data_util.py b/NER_BERT/lstm_data_util.py new file mode 100644 index 0000000..c460b8b --- /dev/null +++ b/NER_BERT/lstm_data_util.py @@ -0,0 +1,80 @@ +import torch +import torch.nn as nn +import const +import dataset_utils + +def get_data(examples, vocab, config): + tag_vocab = dataset_utils.get_tag_vocab(config) + tag2id = {t: i for i, t in enumerate(tag_vocab)} + + all_data = [] + for orig_tokens, orig_tag in examples: + input_ids, char_ids, label_ids = prepare_input(orig_tokens, orig_tag, tag2id, vocab, config) + all_data.append((input_ids, char_ids, label_ids)) + return all_data + +def create_char_batch(char_id_seq): + batch_char_ids = [] + max_len = max([len(char_ids) for char_ids in char_id_seq]) + for char_ids in char_id_seq: + pad_len = max_len - len(char_ids) + padding_op = nn.ConstantPad1d((0, pad_len), const.pad_token_id) + batch_char_ids.append(padding_op(char_ids).unsqueeze(0)) + + batch_char_ids = torch.cat(batch_char_ids) + mask = batch_char_ids.ne(const.pad_token_id) + + return batch_char_ids, mask + +def prepare_input(orig_tokens, orig_tag, tag2id, vocab, config): + input_ids = [] + label_ids = [] + char_id_seq = [] + + assert len(orig_tag) == len(orig_tokens), orig_tag + orig_tokens + + for i, w in enumerate(orig_tokens): + label_id = tag2id[orig_tag[i]] + w_id = vocab['word2id'][w] if w in vocab['word2id'] else const.UNK_ID + input_ids.append(w_id) + label_ids.append(label_id) + w_char_ids = [vocab['char2id'][c] if c in vocab['char2id'] else const.UNK_ID for c in w] + w_char_ids = torch.tensor(w_char_ids).long() + + char_id_seq.append(w_char_ids) + + char_ids = create_char_batch(char_id_seq) + input_ids = torch.tensor(input_ids).long() + label_ids = torch.tensor(label_ids).long() + return input_ids, char_ids, label_ids + +def create_batch(train_data, batch_ids, is_cuda): + max_len = max([len(train_data[bi][0]) for bi in batch_ids]) + batch_input_ids = [] + batch_char_ids = [] + batch_label_ids = [] + + for bi in batch_ids: + input_ids, char_ids, label_ids = train_data[bi] + batch_char_ids.append(char_ids) + pad_len = max_len - len(input_ids) + padding_op = nn.ConstantPad1d((0, pad_len), const.pad_token_id) + batch_input_ids.append(padding_op(input_ids).unsqueeze(0)) + padding_op = nn.ConstantPad1d((0, pad_len), const.label_pad_id) + batch_label_ids.append(padding_op(label_ids).unsqueeze(0)) + + batch_input_ids = torch.cat(batch_input_ids) + batch_label_ids = torch.cat(batch_label_ids) + + att_mask = batch_input_ids.ne(const.pad_token_id) + + if is_cuda: + batch_input_ids = batch_input_ids.cuda() + batch_label_ids = batch_label_ids.cuda() + att_mask = att_mask.cuda() + + inputs = {'word_ids': batch_input_ids, + 'mask': att_mask, + 'char_ids':batch_char_ids, + 'labels': batch_label_ids} + return inputs diff --git a/NER_BERT/model_bert_train.py b/NER_BERT/model_bert_train.py new file mode 100644 index 0000000..12006d3 --- /dev/null +++ b/NER_BERT/model_bert_train.py @@ -0,0 +1,280 @@ +from __future__ import absolute_import, division, print_function + +import os.path + +import os +import torch +import torch.nn as nn +import numpy as np +import random +import time +import json + +import const +import eval_util +import bert_data_util +import dataset_utils + +from torch.nn import CrossEntropyLoss +from transformers import BertModel, BertConfig +from transformers import AdamW, get_linear_schedule_with_warmup + +is_cuda = torch.cuda.is_available() + +class Config(object): + def __init__(self): + self.num_epoch = 10 + self.weight_decay = 0.0 # 1e-8 + self.batch_size = 8 + self.eval_batch_size = 8 + self.max_grad_norm = 1.0 + self.learning_rate = 5e-5 + self.adam_epsilon = 1e-8 + self.print_interval = 1000 * self.batch_size + self.warmup_steps = 0.0 + self.max_seq_length = 100 # =32 - 2 + self.seed = 42 + self.tagging_type = 'B' + +config = Config() + +#####set seed +random.seed(config.seed) +np.random.seed(config.seed) +torch.manual_seed(config.seed) +if is_cuda > 0: + torch.cuda.manual_seed_all(config.seed) +#####set seed end + +class MyBertForTokenClassification(nn.Module): + def __init__(self, num_tags): + super(MyBertForTokenClassification, self).__init__() + self.bert = BertModel.from_pretrained(const.MODEL_TYPE) + + self.config = BertConfig.from_pretrained(const.MODEL_TYPE) + self.config.num_tags = num_tags + + self.dropout_ner = nn.Dropout(self.config.hidden_dropout_prob) + self.classifier_ner = nn.Linear(self.config.hidden_size, num_tags) + + self.classifier_ner.weight.data.normal_(mean=0.0, std=self.config.initializer_range) + self.classifier_ner.bias.data.zero_() + + def forward( + self, + input_ids=None, + attention_mask=None, + token_type_ids=None, + position_ids=None, + head_mask=None, + inputs_embeds=None, + labels=None + ): + outputs = self.bert( + input_ids, + attention_mask=attention_mask, + token_type_ids=token_type_ids, + position_ids=position_ids, + head_mask=head_mask, + inputs_embeds=inputs_embeds, + ) + # NER + sequence_output = outputs[0] + sequence_output = self.dropout_ner(sequence_output) + logits_ner = self.classifier_ner(sequence_output) + + outputs = (logits_ner,) + outputs[2:] # add hidden states and attention if they are here + return outputs # (loss), scores_ner, (hidden_states), (attentions) + + def save_pretrained(self, save_directory): + self.config.save_pretrained(save_directory) + output_model_file = os.path.join(save_directory, "pytorch_model.bin") + torch.save(self.state_dict(), output_model_file) + + def get_loss_greedy(self, logits_ner, labels, attention_mask): + loss_fct = CrossEntropyLoss() + # Only keep active parts of the loss + active_loss = attention_mask.view(-1) == 1 + active_logits = logits_ner.view(-1, self.model_config.num_tags)[active_loss] + active_labels = labels.view(-1)[active_loss] + loss = loss_fct(active_logits, active_labels) + return loss + + def get_loss_crf(self, logits_ner, labels, attention_mask): + logits_padded, labels_padded = self.crf.bert_output2crf_input(logits_ner, labels) + loss = self.crf.get_crf_loss(logits_padded, labels_padded) + return loss + + def predict_ner_greedy(self, logits_ner, labels): + logits_ner = logits_ner.softmax(dim=2) + score_tags, pred_tags = logits_ner.max(dim=2) + return -1.0, pred_tags, score_tags, labels + + def predict_ner_viterbi(self, logits_ner, labels): + logits_padded, labels_padded = self.crf.bert_output2crf_input(logits_ner, labels) + mask = labels_padded.ne(self.crf.tag_pad_id) + score_sentence, pred_tags, score_tags = self.crf.viterbi_decode(logits_padded, mask) + return score_sentence, pred_tags, score_tags, labels_padded + +def get_optimizer(model, config, t_total): + # Prepare optimizer and schedule (linear warmup and decay) + no_decay = ["bias", "LayerNorm.weight"] + optimizer_grouped_parameters = [ + {"params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], + "weight_decay": config.weight_decay}, + {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], "weight_decay": 0.0} + ] + optimizer = AdamW(optimizer_grouped_parameters, lr=config.learning_rate, eps=config.adam_epsilon) + scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=config.warmup_steps, + num_training_steps=t_total) + + return optimizer, scheduler + + +def predictions(dev_data, model, config): + data_size = len(dev_data) + ids = np.arange(data_size) + eval_loss = 0 + model.eval() + + gold_label_list = [] + preds_list = [] + + for i in range(0, data_size, config.eval_batch_size): + batch_ids = ids[i:i + config.eval_batch_size] + + inputs = bert_data_util.create_batch(dev_data, batch_ids, is_cuda) + outputs_t = model(**inputs) + loss_t, scores_ner_t = outputs_t[:2] + + eval_loss += loss_t.item() + max_value_tag_t, pred_tag_t = torch.max(scores_ner_t, dim=2) + + pred_tag = pred_tag_t.cpu().data.numpy() + gold_tag = inputs['labels'].cpu().data.numpy() + + for k, bi in enumerate(batch_ids): + s_len = len(dev_data[bi][0]) + predict_list = [] + gold_list = [] + for j in range(s_len): + if gold_tag[k][j] == const.label_pad_id: + continue + predict_list.append(pred_tag[k][j]) + gold_list.append(gold_tag[k][j]) + + gold_label_list.append(gold_list) + preds_list.append(predict_list) + eval_loss /= data_size + return preds_list, gold_label_list, eval_loss + +def train(output_root_dir, train_data_seq, dev_data_seq): + tag_vocab = dataset_utils.get_tag_vocab(config) + tag2id = {t: i for i, t in enumerate(tag_vocab)} + + train_data = bert_data_util.get_bert_data(train_data_seq, tag2id, config) + dev_data = bert_data_util.get_bert_data(dev_data_seq, tag2id, config) + + model = MyBertForTokenClassification(num_tags=len(tag2id)) + if is_cuda: + model = model.cuda() + + data_size = len(train_data) + num_batch = np.ceil(data_size / config.batch_size) + t_total = config.num_epoch * num_batch + + optimizer, scheduler = get_optimizer(model, config, t_total) + + exp_loss = None + global_step = 0 + best_dev_f1 = 0 + model.zero_grad() + ids = np.arange(data_size) + for epoch in range(config.num_epoch): + np.random.shuffle(ids) + for i in range(0, data_size, config.batch_size): + batch_ids = ids[i:i + config.batch_size] + + model.train() + inputs = bert_data_util.create_batch(train_data, batch_ids, is_cuda) + outputs = model(**inputs) + loss = outputs[0] + + loss.backward() + nn.utils.clip_grad_norm_(model.parameters(), config.max_grad_norm) + optimizer.step() + scheduler.step() + model.zero_grad() + global_step += 1 + + exp_loss = 0.99 * exp_loss + 0.01 * loss.item() if exp_loss else loss.item() + + if global_step > 0 and global_step % config.print_interval == 0: + print(f'{global_step} / {t_total} train loss: {exp_loss} lr: {scheduler.get_lr()[0]}', flush=True) + + preds_list, gold_label_list, eval_loss = predictions(dev_data, model, config) + results = eval_util.evaluate(preds_list, gold_label_list, tag_vocab) + print(f'{global_step}/{t_total} NER: p/r/f1 {results["precision"]:.5f}/{results["recall"]:.5f}/{results["f1"]:.5f}', flush=True) + + f1 = results['f1'] + if f1 > best_dev_f1: + # output_dir = os.path.join(output_root_dir, 'checkpoint-{}'.format(epoch)) + output_dir = os.path.join(output_root_dir, 'checkpoint') + if not os.path.exists(output_dir): + os.makedirs(output_dir) + + print(f"Saving model checkpoint to {output_dir}", flush=True) + + model.save_pretrained(output_dir) + bert_data_util.bert_tokenizer.save_pretrained(output_dir) + + with open(os.path.join(output_dir, "training_config.json"), 'w') as fout: + json.dump(vars(config), fout) + + print(f'{global_step} / {t_total} train loss: {exp_loss} lr: {scheduler.get_lr()[0]}', flush=True) + +def get_model(model_dir, num_tag): + model = MyBertForTokenClassification(num_tags=num_tag) + + #load model + save_directory = os.path.join(root_dir, model_dir + '/checkpoint') + model_file_path = os.path.join(save_directory, "pytorch_model.bin") + print(f'reading model from {model_file_path}') + state_dict = torch.load(model_file_path, map_location=lambda storage, location: storage) + model.eval() + model.load_state_dict(state_dict, strict=False) + + if is_cuda: + model = model.cuda() + return model + +def process_train(root_dir, data_dir): + output_root_dir = os.path.join(root_dir, f'dl_model_{int(time.time())}') + if not os.path.exists(output_root_dir): + os.makedirs(output_root_dir) + print(f'model out dir {output_root_dir}', flush=True) + + train_data_seq = dataset_utils.get_data_train_dev(data_dir, 'train.jsonl', config) + dev_data_seq = dataset_utils.get_data_train_dev(data_dir, 'dev.jsonl', config) + + train(output_root_dir, train_data_seq, dev_data_seq) + +def process_eval(root_dir, model_dir, data_dir, filename): + tag_vocab = dataset_utils.get_tag_vocab(config) + tag2id = {t: i for i, t in enumerate(tag_vocab)} + + model = get_model(model_dir, len(tag2id)) + + test_data, metadata = dataset_utils.get_data_test(data_dir, filename) + test_data_bert = bert_data_util.get_bert_data(test_data, tag2id, config) + + preds_list, _, _ = predictions(test_data_bert, model, config) + eval_util.dump_result(preds_list, metadata, tag_vocab, test_data, root_dir, 'boundary_model.txt') + +if __name__ == "__main__": + root_dir = os.path.join(os.path.expanduser("~"), 'private-projects/propganda/exp') + data_dir = os.path.join(os.path.expanduser("~"), 'private-projects/propganda/datasets') + #process_train(root_dir, data_dir) + model_dir = os.path.join(os.path.expanduser("~"), 'private-projects/propganda/exp/dl_model_1579925959') + process_eval(root_dir, model_dir, data_dir, 'test_phase0.jsonl') + diff --git a/NER_BERT/model_lstm.py b/NER_BERT/model_lstm.py new file mode 100644 index 0000000..179dbc2 --- /dev/null +++ b/NER_BERT/model_lstm.py @@ -0,0 +1,155 @@ +from __future__ import unicode_literals, print_function, division + +import torch +import torch.nn as nn +import torch.nn.functional as F +from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence +from torch.nn import Conv1d, ReLU + +from crf import CRF_Loss +import model_utils + +class CHAR_CONV(torch.nn.Module): + def __init__(self, + embedding_dim, + num_filters, + ngram_filter_sizes=(2, 3, 4, 5)): + super(CHAR_CONV, self).__init__() + + self.convolution_layers = torch.nn.ModuleList() + for ngram_size in ngram_filter_sizes: + conv_maxpool = torch.nn.ModuleList() + conv_maxpool.extend([Conv1d( + in_channels=embedding_dim, + out_channels=num_filters, + kernel_size=ngram_size), + ReLU(), + torch.nn.MaxPool1d(kernel_size=num_filters)]) + + self.convolution_layers.append(conv_maxpool) + + def forward(self, tokens, mask): + tokens = tokens * mask.unsqueeze(-1).float() + tokens = torch.transpose(tokens, 1, 2) + + filter_outputs = [] + for conv_maxpool in self.convolution_layers: + filter_outputs.append(conv_maxpool(tokens)) + + maxpool_output = torch.cat(filter_outputs, dim=1) if len(filter_outputs) > 1 else filter_outputs[0] + + return maxpool_output + +class NER_SOFTMAX_CHAR(nn.Module): + def __init__(self, emb_matrix, config, num_tags): + super(NER_SOFTMAX_CHAR, self).__init__() + + embd_vector = torch.from_numpy(emb_matrix['word']).float() + self.word_embeds = nn.Embedding.from_pretrained(embd_vector, freeze=False) + + embd_vector = torch.from_numpy(emb_matrix['char']).float() + self.char_embeds = nn.Embedding.from_pretrained(embd_vector, freeze=False) + + self.lstm_char = nn.LSTM(self.char_embeds.embedding_dim, + config.char_lstm_dim, + num_layers=1, bidirectional=True, batch_first=True) + + input_size = self.word_embeds.embedding_dim + config.char_lstm_dim * 2 + + self.lstm = nn.LSTM(input_size, + config.word_lstm_dim, + num_layers=1, bidirectional=True, batch_first=True) + + self.dropout = nn.Dropout(config.dropout_rate) + self.hidden_layer = nn.Linear(config.word_lstm_dim * 2, config.word_lstm_dim) + self.tanh_layer = torch.nn.Tanh() + + self.hidden2tag = nn.Linear(config.word_lstm_dim, num_tags) + + self.config = config + + model_utils.init_lstm_wt(self.lstm_char) + model_utils.init_lstm_wt(self.lstm) + model_utils.init_linear_wt(self.hidden_layer) + model_utils.init_linear_wt(self.hidden2tag) + + def forward(self, word_ids, mask, char_ids): + lengths = mask.sum(1, dtype=torch.long) + + max_length = torch.max(lengths) + char_emb = [] + word_embed = self.word_embeds(word_ids) + for chars, char_mask in char_ids: + char_len = char_mask.sum(1, dtype=torch.long) + seq_embed = self.char_embeds(chars) + seq_lengths, sort_idx = torch.sort(char_len, descending=True) + _, unsort_idx = torch.sort(sort_idx) + seq_embed = seq_embed[sort_idx] + packed = pack_padded_sequence(seq_embed, seq_lengths, batch_first=True) + output, hidden = self.lstm_char(packed) + lstm_feats, _ = pad_packed_sequence(output, batch_first=True) + lstm_feats = lstm_feats.contiguous() + b, t_k, d = list(lstm_feats.size()) + + seq_rep = lstm_feats.view(b, t_k, 2, -1) #0 is fwd and 1 is bwd + + last_idx = char_len - 1 + seq_rep_fwd = seq_rep[unsort_idx, 0, 0] + seq_rep_bwd = seq_rep[unsort_idx, last_idx, 1] + + seq_out = torch.cat([seq_rep_fwd, seq_rep_bwd], 1) + # fill up the dummy char embedding for padding + seq_out = F.pad(seq_out, (0, 0, 0, max_length - seq_out.size(0))) + char_emb.append(seq_out.unsqueeze(0)) + + char_emb = torch.cat(char_emb, 0) #b x n x c_dim + + word_embed = torch.cat([char_emb, word_embed], 2) + word_embed = self.dropout(word_embed) + + lengths = lengths.view(-1).tolist() + packed = pack_padded_sequence(word_embed, lengths, batch_first=True, enforce_sorted=False) + output, hidden = self.lstm(packed) + + lstm_feats, _ = pad_packed_sequence(output, batch_first=True) # h dim = B x t_k x n + lstm_feats = lstm_feats.contiguous() + + b, t_k, d = list(lstm_feats.size()) + + h = self.hidden_layer(lstm_feats.view(-1, d)) + h = self.tanh_layer(h) + logits = self.hidden2tag(h) + logits = logits.view(b, t_k, -1) + + return logits + + +class NER_SOFTMAX_CHAR_CRF(nn.Module): + def __init__(self, emb_matrix, config, tag_pad_id, num_tags): + super(NER_SOFTMAX_CHAR_CRF, self).__init__() + self.featurizer = NER_SOFTMAX_CHAR(emb_matrix, config, num_tags) + self.crf = CRF_Loss(num_tags, config, tag_pad_id) + self.config = config + + def forward(self, word_ids, mask, char_ids, labels=None): + output = self.featurizer(word_ids, mask, char_ids) + if labels is not None: + loss = self.get_loss(output, labels, mask) + output = (loss, output) + return output + + def get_loss(self, logits, y, mask): + if self.config.is_structural_perceptron_loss: + loss = self.crf.structural_perceptron_loss(logits, y) + else: + loss = -1 * self.crf.log_likelihood(logits, y) + + s_lens = mask.sum(1, dtype=torch.long) + + loss = loss / s_lens.float() + loss = loss.mean() + return loss + + def predict(self, emissions, mask): + best_scores, pred = self.crf.viterbi_decode_batch(emissions, mask) + return best_scores, pred diff --git a/NER_BERT/model_lstm_train.py b/NER_BERT/model_lstm_train.py new file mode 100644 index 0000000..6d36b1c --- /dev/null +++ b/NER_BERT/model_lstm_train.py @@ -0,0 +1,258 @@ +from __future__ import absolute_import, division, print_function + +import os.path + +import os +import torch +import torch.nn as nn +import numpy as np +import random +import time +import json +import codecs + +import const +import eval_util +import bert_data_util +import dataset_utils +import model_lstm +import model_utils +import lstm_data_util + +from torch.optim import Adam +from torch.optim.lr_scheduler import LambdaLR + +is_cuda = torch.cuda.is_available() + +class Config(object): + def __init__(self): + self.num_epoch = 10 + self.weight_decay = 0.0 # 1e-8 + self.batch_size = 8 + self.eval_batch_size = 8 + self.max_grad_norm = 1.0 + self.learning_rate = 5e-5 + self.adam_epsilon = 1e-8 + self.print_interval = 1000 * self.batch_size + self.warmup_steps = 0.0 + self.max_seq_length = 100 # =32 - 2 + self.seed = 42 + self.tagging_type = 'B' + self.word_emdb_dim = 100 + self.word_lstm_dim = 100 + self.char_embd_dim = 30 + self.char_lstm_dim = 64 + self.dropout_rate = 0.15 + self.is_structural_perceptron_loss = False + +config = Config() + +#####set seed +random.seed(config.seed) +np.random.seed(config.seed) +torch.manual_seed(config.seed) +if is_cuda > 0: + torch.cuda.manual_seed_all(config.seed) +#####set seed end +def get_linear_schedule_with_warmup(optimizer, num_warmup_steps, num_training_steps, last_epoch=-1): + """ Create a schedule with a learning rate that decreases linearly after + linearly increasing during a warmup period. + """ + + def lr_lambda(current_step): + if current_step < num_warmup_steps: + return float(current_step) / float(max(1, num_warmup_steps)) + return max( + 0.0, float(num_training_steps - current_step) / float(max(1, num_training_steps - num_warmup_steps)) + ) + + return LambdaLR(optimizer, lr_lambda, last_epoch) + +def get_optimizer(model, config, t_total): + optimizer = Adam(model.parameters(), amsgrad=True) + scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=config.warmup_steps, + num_training_steps=t_total) + return optimizer, scheduler + + +def predictions(dev_data, model, config): + data_size = len(dev_data) + ids = np.arange(data_size) + eval_loss = 0 + model.eval() + + gold_label_list = [] + preds_list = [] + + for i in range(0, data_size, config.eval_batch_size): + batch_ids = ids[i:i + config.eval_batch_size] + + inputs = lstm_data_util.create_batch(dev_data, batch_ids, is_cuda) + outputs_t = model(**inputs) + loss_t, scores_ner_t = outputs_t[:2] + best_scores, pred_tag_t = model.predict(scores_ner_t, inputs['mask']) + + eval_loss += loss_t.item() + + pred_tag = pred_tag_t.cpu().data.numpy() + gold_tag = inputs['labels'].cpu().data.numpy() + + for k, bi in enumerate(batch_ids): + s_len = len(dev_data[bi][0]) + predict_list = [] + gold_list = [] + for j in range(s_len): + if gold_tag[k][j] == const.label_pad_id: + continue + predict_list.append(pred_tag[k][j]) + gold_list.append(gold_tag[k][j]) + + gold_label_list.append(gold_list) + preds_list.append(predict_list) + eval_loss /= data_size + return preds_list, gold_label_list, eval_loss + +def train(output_root_dir, embd, train_data, dev_data): + num_tags = len(dataset_utils.get_tag_vocab(config)) + model = model_lstm.NER_SOFTMAX_CHAR_CRF(embd, config, const.label_pad_id, num_tags) + + if is_cuda: + model = model.cuda() + + data_size = len(train_data) + num_batch = np.ceil(data_size / config.batch_size) + t_total = config.num_epoch * num_batch + + optimizer, scheduler = get_optimizer(model, config, t_total) + + exp_loss = None + global_step = 0 + best_dev_f1 = 0 + model.zero_grad() + ids = np.arange(data_size) + for epoch in range(config.num_epoch): + np.random.shuffle(ids) + for i in range(0, data_size, config.batch_size): + batch_ids = ids[i:i + config.batch_size] + + model.train() + inputs = lstm_data_util.create_batch(train_data, batch_ids, is_cuda) + outputs = model(**inputs) + loss = outputs[0] + + loss.backward() + nn.utils.clip_grad_norm_(model.parameters(), config.max_grad_norm) + optimizer.step() + scheduler.step() + model.zero_grad() + global_step += 1 + + exp_loss = 0.99 * exp_loss + 0.01 * loss.item() if exp_loss else loss.item() + + if global_step > 0 and global_step % config.print_interval == 0: + print(f'{global_step} / {t_total} train loss: {exp_loss} lr: {scheduler.get_lr()[0]}', flush=True) + + preds_list, gold_label_list, eval_loss = predictions(dev_data, model, config) + results = eval_util.evaluate(preds_list, gold_label_list, config) + print(f'{global_step}/{t_total} NER: p/r/f1 {results["precision"]:.5f}/{results["recall"]:.5f}/{results["f1"]:.5f}', flush=True) + + f1 = results['f1'] + if f1 > best_dev_f1: + # output_dir = os.path.join(output_root_dir, 'checkpoint-{}'.format(epoch)) + output_dir = os.path.join(output_root_dir, 'checkpoint') + if not os.path.exists(output_dir): + os.makedirs(output_dir) + + print(f"Saving model checkpoint to {output_dir}", flush=True) + + model.save_pretrained(output_dir) + bert_data_util.bert_tokenizer.save_pretrained(output_dir) + + with open(os.path.join(output_dir, "training_config.json"), 'w') as fout: + json.dump(vars(config), fout) + + print(f'{global_step} / {t_total} train loss: {exp_loss} lr: {scheduler.get_lr()[0]}', flush=True) + + +def process_train(root_dir, data_dir, glove_path): + output_root_dir = os.path.join(root_dir, f'dl_model_{int(time.time())}') + if not os.path.exists(output_root_dir): + os.makedirs(output_root_dir) + print(f'model out dir {output_root_dir}', flush=True) + + train_data_seq = dataset_utils.get_data_train_dev(data_dir, 'train.jsonl', config) + dev_data_seq = dataset_utils.get_data_train_dev(data_dir, 'dev.jsonl', config) + + word_emb_matrix, word2id, id2word = model_utils.get_word_embd(config, glove_path, train_data_seq) + char_emb_matrix, char2id, id2char = model_utils.get_char_embd(config, id2word) + + vocab = { + 'word2id': word2id, + 'id2word': id2word, + 'char2id': char2id, + 'id2char': id2char + } + embd = { + 'word': word_emb_matrix, + 'char': char_emb_matrix + } + with codecs.open(os.path.join(output_root_dir, 'word.vocab'), "w", encoding="utf8") as f: + f.write('\n'.join(id2word) + '\n') + with codecs.open(os.path.join(output_root_dir, 'char.vocab'), "w", encoding="utf8") as f: + f.write('\n'.join(id2char) + '\n') + + train_data = lstm_data_util.get_data(train_data_seq, vocab, config) + dev_data = lstm_data_util.get_data(dev_data_seq, vocab, config) + + train(output_root_dir, embd, train_data, dev_data) + + +def get_model(model_dir, vocab, num_tag): + embd = model_utils.get_random_embedding(vocab, config) + model = model_lstm.NER_SOFTMAX_CHAR_CRF(embd, config, const.label_pad_id, num_tag) + + #load model + save_directory = os.path.join(root_dir, model_dir + '/checkpoint') + model_file_path = os.path.join(save_directory, "pytorch_model.bin") + print(f'reading model from {model_file_path}') + state_dict = torch.load(model_file_path, map_location=lambda storage, location: storage) + model.eval() + model.load_state_dict(state_dict, strict=False) + + if is_cuda: + model = model.cuda() + return model + +def process_eval(root_dir, model_dir, data_dir, filename): + tag_vocab = dataset_utils.get_tag_vocab(config) + tag2id = {t: i for i, t in enumerate(tag_vocab)} + + with codecs.open(os.path.join(model_dir, 'word.vocab'), "r", encoding="utf8") as f: + id2word = f.readlines().split('\n') + word2id = {v: k for k, v in enumerate(id2word)} + with codecs.open(os.path.join(model_dir, 'char.vocab'), "r", encoding="utf8") as f: + id2char = f.readlines().split('\n') + char2id = {v: k for k, v in enumerate(id2char)} + vocab = { + 'word2id': word2id, + 'id2word': id2word, + 'char2id': char2id, + 'id2char': id2char + } + model = get_model(model_dir, vocab, len(tag2id)) + + test_data_seq, metadata = dataset_utils.get_data_test(data_dir, filename) + test_data = lstm_data_util.get_data(test_data_seq, vocab, config) + + preds_list, _, _ = predictions(test_data, model, config) + eval_util.dump_result(preds_list, metadata, test_data, root_dir, 'boundary_model.txt', config) + +if __name__ == "__main__": + prefix = 'prop' #'private-projects/propganda' + root_dir = os.path.join(os.path.expanduser("~"), prefix + '/exp') + data_dir = os.path.join(os.path.expanduser("~"), prefix + '/datasets') + glove_dir = os.path.join(os.path.expanduser("~"), 'dl_entity/glove.6B/glove.6B.100d.txt') + process_train(root_dir, data_dir, glove_dir) + #model_dir = os.path.join(os.path.expanduser("~"), 'private-projects/propganda/exp/dl_model_1579925959') + #process_eval(root_dir, model_dir, data_dir, 'test_phase0.jsonl') + diff --git a/NER_BERT/model_utils.py b/NER_BERT/model_utils.py new file mode 100644 index 0000000..631c54d --- /dev/null +++ b/NER_BERT/model_utils.py @@ -0,0 +1,99 @@ +import numpy as np +import codecs +import re +from collections import Counter + +import const + +def init_lstm_wt(lstm): + for names in lstm._all_weights: + for name in names: + if name.startswith('weight_'): + wt = getattr(lstm, name) + drange = np.sqrt(6. / (np.sum(wt.size()))) + wt.data.uniform_(-drange, drange) + + elif name.startswith('bias_'): + # set forget bias to 1 + bias = getattr(lstm, name) + n = bias.size(0) + start, end = n // 4, n // 2 + bias.data.fill_(0.) + bias.data[start:end].fill_(1.) + + +def init_linear_wt(linear): + drange = np.sqrt(6. / (np.sum(linear.weight.size()))) + linear.weight.data.uniform_(-drange, drange) + + if linear.bias is not None: + linear.bias.data.fill_(0.) + + +def get_glove(glove_path): + print("Loading GLoVE vectors from file: {}".format(glove_path)) + word_to_vector = {} + + # go through glove vecs + with codecs.open(glove_path, 'r', 'utf-8') as fh: + for line in fh: + line = re.split('\s+', line.strip()) + word = line[0] + vector = list(map(float, line[1:])) + word_to_vector[word] = vector + + return word_to_vector + +def get_word_embd(config, glove_path, examples): + word_to_vector = get_glove(glove_path) + + word_freq_map = Counter() + for tokens, tags in examples: + word_freq_map.update(tokens) + + word_freq_map.update(word_to_vector.keys()) + orig_tokens = set([w for w, ct in word_freq_map.most_common()]) + orig_tokens = sorted(list(orig_tokens.union(word_to_vector.keys()))) + + id_to_word = const._START_VOCAB.copy() + id_to_word.extend(orig_tokens) + + word_to_id = {v: k for k, v in enumerate(id_to_word)} + + word_emb_matrix = np.random.uniform(low=-1.0, high=1.0, + size=(len(id_to_word), config.word_emdb_dim)) + pretrained_init = 0 + for wid, w in enumerate(id_to_word): + if w in word_to_vector: + word_emb_matrix[wid, :] = word_to_vector[w] + pretrained_init += 1 + + return word_emb_matrix, word_to_id, id_to_word + +def get_char_embd(config, id_to_word): + char_freq_map = Counter() + for w in id_to_word: + char_freq_map.update([c for c in w]) + + id_to_char = const._START_VOCAB.copy() + id_to_char.extend([c for c, ct in char_freq_map.most_common()]) + + char_to_id = {v: k for k, v in enumerate(id_to_char)} + + char_emb_matrix = np.random.uniform(low=-1.0, high=1.0, + size=(len(id_to_char), config.char_embd_dim)) + + return char_emb_matrix, char_to_id, id_to_char + +def get_random_embedding(vocab, config): + word_emb_matrix = np.random.uniform(low=-1.0, high=1.0, + size=(len(vocab['id2word']), config.word_emdb_dim)) + + char_emb_matrix = np.random.uniform(low=-1.0, high=1.0, + size=(len(vocab['id2char']), config.char_embd_dim)) + + embd = { + 'word': word_emb_matrix, + 'char': char_emb_matrix + } + return embd diff --git a/NER_BERT/transformer.py b/NER_BERT/transformer.py new file mode 100644 index 0000000..a2ce657 --- /dev/null +++ b/NER_BERT/transformer.py @@ -0,0 +1,110 @@ +#Code is based on http://nlp.seas.harvard.edu/2018/04/03/attention.html + +from __future__ import unicode_literals, print_function, division + +import torch +import torch.nn as nn +import torch.nn.functional as F +import logging +import math + +logging.basicConfig(level=logging.INFO) + +class PositionalEncoding(nn.Module): + def __init__(self, d_model, dropout, max_len=5000): + super(PositionalEncoding, self).__init__() + self.dropout = nn.Dropout(p=dropout) + + pe = torch.zeros(max_len, d_model) + position = torch.arange(0, max_len).float().unsqueeze(1) + div_term = torch.exp(torch.arange(0, d_model, 2).float() * + -(math.log(10000.0) / d_model)) + pe[:, 0::2] = torch.sin(position * div_term) + pe[:, 1::2] = torch.cos(position * div_term) + pe = pe.unsqueeze(0) + self.register_buffer('pe', pe) + + def forward(self, x): + x = x + self.pe[:, :x.size(1)] + return self.dropout(x) + +class MultiHeadedAttention(nn.Module): + def __init__(self, num_head, d_model, dropout=0.1): + super(MultiHeadedAttention, self).__init__() + assert d_model % num_head == 0 + self.d_k = d_model // num_head #d_k == d_v + self.h = num_head + + self.linear_key = nn.Linear(d_model, d_model) + self.linear_value = nn.Linear(d_model, d_model) + self.linear_query = nn.Linear(d_model, d_model) + self.linear_out = nn.Linear(d_model, d_model) + + self.dropout = nn.Dropout(p=dropout) + + def attention(self, query, key, value, mask, dropout=None): + d_k = query.size(-1) + scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(d_k) + scores = scores.masked_fill(mask == 0, -1e9) + + p_attn = F.softmax(scores, dim=-1) + if dropout is not None: + p_attn = dropout(p_attn) + return torch.matmul(p_attn, value), p_attn + + def forward(self, query, key, value, mask): + nbatches = query.size(0) + query = self.linear_query(query).view(nbatches, -1, self.h, self.d_k).transpose(1, 2) + key = self.linear_key(key).view(nbatches, -1, self.h, self.d_k).transpose(1, 2) + value = self.linear_value(value).view(nbatches, -1, self.h, self.d_k).transpose(1, 2) + + mask = mask.unsqueeze(1) + x, attn = self.attention(query, key, value, mask, dropout=self.dropout) + x = x.transpose(1, 2).contiguous().view(nbatches, -1, self.h * self.d_k) + return self.linear_out(x) + +class AffineLayer(nn.Module): + def __init__(self, dropout, d_model, d_ff): + super(AffineLayer, self).__init__() + self.w_1 = nn.Linear(d_model, d_ff) + self.w_2 = nn.Linear(d_ff, d_model) + self.dropout = nn.Dropout(dropout) + + def forward(self, x): + return self.w_2(self.dropout(F.relu(self.w_1(x)))) + +class EncoderLayer(nn.Module): + def __init__(self, num_head, dropout, d_model, d_ff): + super(EncoderLayer, self).__init__() + + self.att_layer = MultiHeadedAttention(num_head, d_model, dropout) + self.norm_att = nn.LayerNorm(d_model) + self.dropout_att = nn.Dropout(dropout) + + self.affine_layer = AffineLayer(dropout, d_model, d_ff) + self.norm_affine = nn.LayerNorm(d_model) + self.dropout_affine = nn.Dropout(dropout) + + def forward(self, x, mask): + x_att = self.norm_att(x*mask) + x_att = self.att_layer(x_att, x_att, x_att, mask) + x = x + self.dropout_att(x_att) + + x_affine = self.norm_affine(x*mask) + x_affine = self.affine_layer(x_affine) + return x + self.dropout_affine(x_affine) + +class Encoder(nn.Module): + def __init__(self, N, num_head, dropout, d_model, d_ff): + super(Encoder, self).__init__() + self.position = PositionalEncoding(d_model, dropout) + self.layers = nn.ModuleList() + for _ in range(N): + self.layers.append(EncoderLayer(num_head, dropout, d_model, d_ff)) + self.norm = nn.LayerNorm(d_model) + + def forward(self, word_embed, mask): + x = self.position(word_embed) + for layer in self.layers: + x = layer(x, mask) + return self.norm(x*mask) From 8eada512ed1e0524c55fa200a0c09da5b70100d7 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Wed, 17 Jun 2020 12:47:28 -0700 Subject: [PATCH 04/19] Add files via upload --- NER_BERT/decoder.py | 110 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 NER_BERT/decoder.py diff --git a/NER_BERT/decoder.py b/NER_BERT/decoder.py new file mode 100644 index 0000000..4806b6b --- /dev/null +++ b/NER_BERT/decoder.py @@ -0,0 +1,110 @@ +import torch +import numpy as np +import torch.nn.functional as F +from torch.nn.utils.rnn import pad_sequence + + +@torch.jit.script +def viterbi_decode_single_jit(tag_sequence, transition_matrix): + top_k = 1 + sequence_length, num_tags = tag_sequence.size() + num_tags = num_tags + 2 + + zero_sentinel = torch.zeros(1, num_tags) + extra_tags_sentinel = torch.ones(sequence_length, 2) * -10000 + tag_sequence = torch.cat([tag_sequence, extra_tags_sentinel], -1) + tag_sequence = torch.cat([zero_sentinel, tag_sequence, zero_sentinel], 0) + + path_indices = torch.zeros(num_tags, dtype=torch.long, device=tag_sequence.device).unsqueeze(0) + path_scores = tag_sequence[0, :].unsqueeze(0) + + for t in range(1, tag_sequence.size(0)): + summed_potentials = path_scores.unsqueeze(2) + transition_matrix + summed_potentials = summed_potentials.view(-1, num_tags) + + scores, paths = torch.topk(summed_potentials, k=top_k, dim=0) + + path_scores = tag_sequence[t, :].unsqueeze(0) + scores + path_indices = torch.cat([path_indices, paths], 0) + + path_indices = path_indices[1:] + path_scores_v = path_scores.view(-1) + viterbi_scores, best_paths = torch.topk(path_scores_v, k=top_k, dim=0) + + n_paths_indices = path_indices.size(0) + + viterbi_paths = torch.zeros(sequence_length, dtype=torch.long, device=tag_sequence.device).unsqueeze(0) + tag_scores = torch.zeros(sequence_length, device=tag_sequence.device).unsqueeze(0) + for i in range(top_k): + viterbi_path = best_paths[0].unsqueeze(0) + + for k in range(n_paths_indices): + t_rev = n_paths_indices - k - 1 + backward_timestep = path_indices[t_rev, :] + tag_id = torch.index_select(backward_timestep.view(-1), 0, viterbi_path[-1]) + viterbi_path = torch.cat([viterbi_path, tag_id], -1) + + viterbi_path = viterbi_path.flip(0) + viterbi_path = viterbi_path % num_tags + viterbi_path = viterbi_path[1:-1] + viterbi_paths = torch.cat([viterbi_paths, viterbi_path.unsqueeze(0)], 0) + + tag_score = torch.gather(tag_sequence[1:-1], 1, viterbi_path.unsqueeze(-1)).view(-1) + tag_scores = torch.cat([tag_scores, tag_score.unsqueeze(0)], 0) + viterbi_paths = viterbi_paths[1:] + tag_scores = tag_scores[1:] + return viterbi_paths, tag_scores.exp(), viterbi_scores.exp() + +def predict_ner_single_jit(logits_ner, labels, transition_matrix, tag_pad_id): + mask = labels.ne(tag_pad_id) + logits_padded = torch.masked_select(logits_ner, mask.unsqueeze(2)).view(-1, logits_ner.size()[-1]) + return viterbi_decode_single_jit(logits_padded, transition_matrix) + + +####### +def viterbi_decode_single_python(e, t): + num_tags = len(e[0]) + seq_len = len(e) + start_tag_id = num_tags + end_tag_id = num_tags + 1 + + dp_links = [] + dp = [0.] * num_tags + curr_dp_links = [] + for j in range(num_tags): + dp[j] = t[start_tag_id, j] + e[0][j] + curr_dp_links.append(-1) + dp_links.append(curr_dp_links) + + for i in range(1, seq_len): + new_dp = [] + curr_dp_links = [] + for j in range(num_tags): + all_candidates = [np.logaddexp(t[k, j] + e[i][j], dp[k]) for k in range(num_tags)] + max_k = max(range(num_tags), key=lambda i: all_candidates[i]) + new_dp.append(all_candidates[max_k]) + curr_dp_links.append(max_k) + dp = new_dp + dp_links.append(curr_dp_links) + + all_candidates = [np.logaddexp(t[k, end_tag_id], dp[k]) for k in range(num_tags)] + max_k = max(range(num_tags), key=lambda i: all_candidates[i]) + sentence_score = all_candidates[max_k] + + all_labels = [max_k] + all_labels_score = [t[max_k, end_tag_id]] + + for i in range(seq_len - 1, 0, -1): + curr_k = dp_links[i][max_k] + all_labels.append(curr_k) + all_labels_score.append(t[curr_k, max_k] + e[i][max_k]) + max_k = curr_k + + return sentence_score, all_labels[::-1], all_labels_score[::-1] + +def predict_ner_single_python(logits_ner, labels, transitions, tag_pad_id): + mask = labels.ne(tag_pad_id) + logits_ner_padded = torch.masked_select(logits_ner, mask.unsqueeze(2)).view(-1, logits_ner.size()[-1]) + logits_ner_padded_lsf = F.log_softmax(logits_ner_padded, dim=-1) + score_sentence, pred_tags, score_tags = viterbi_decode_single_python(logits_ner_padded_lsf.cpu().data.numpy(), transitions.cpu().data.numpy()) + return [pred_tags], [np.exp(score_tags)], [np.exp(score_sentence)] From 0d2c902d4741196c2c00cd66ca99912006682922 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Wed, 17 Jun 2020 12:49:24 -0700 Subject: [PATCH 05/19] Update decoder.py --- NER_BERT/decoder.py | 1 + 1 file changed, 1 insertion(+) diff --git a/NER_BERT/decoder.py b/NER_BERT/decoder.py index 4806b6b..80c1545 100644 --- a/NER_BERT/decoder.py +++ b/NER_BERT/decoder.py @@ -3,6 +3,7 @@ import torch.nn.functional as F from torch.nn.utils.rnn import pad_sequence +#reference https://github.com/allenai/allennlp/blob/a7265c04078964ea2b80a78fc3967bde8d16072d/allennlp/nn/util.py#L403 @torch.jit.script def viterbi_decode_single_jit(tag_sequence, transition_matrix): From c35f3b701c54f3fece11f385c5caad4f89256a51 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Wed, 17 Jun 2020 12:54:24 -0700 Subject: [PATCH 06/19] Update README.md --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index dafe84b..767a2f5 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ ### Datset - - [x] conll2003 - - [ ] atis + ### Neural NER - - [x] CharLSTM+WordLSTM+CRF: [Lample .etc, NAACL16](http://www.aclweb.org/anthology/N/N16/N16-1030.pdf) - - [x] Make a CoNLL-2003 batcher @@ -12,6 +13,8 @@ - - [x] Implement CharLSTM + WordLSTM + softmax - - [x] Implement CharLSTM + WordLSTM + CRF - - [x] Tranformer encoder + CRF +- - [x] BERT encoder + CRF +- - [x] pytorch JIT compilable Viterbi Decoder https://github.com/atulkum/sequence_prediction/blob/master/NER_BERT/decoder.py#L9 ### Slot Filling + intent prediciton - - [ ] [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://arxiv.org/abs/1609.01454) From 8dfd318bb0928e8e0baeacc0a538f0a88322477b Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Fri, 26 Jun 2020 16:30:56 -0700 Subject: [PATCH 07/19] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 767a2f5..6c2cc79 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Sequence Prediction +# Various BERT and BiLSTM based model for NER ## TO-DO ### Datset From 490f44bae365095bb2f5ca23ac21516dc5329f75 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Fri, 26 Jun 2020 16:31:12 -0700 Subject: [PATCH 08/19] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6c2cc79..9cc25a9 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Various BERT and BiLSTM based model for NER +# Various BERT-CRF and BiLSTM-CRF based model for NER ## TO-DO ### Datset From e04c043b6a4e23484e4c0511824f81b4f9fc1674 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 4 Feb 2021 16:15:38 -0800 Subject: [PATCH 09/19] Add files via upload --- dataset_roberta.py | 154 ++++++++++++++++++++++++++++++++++++ model.py | 168 ++++++++++++++++++++++++++++++++++++++++ multichoice_dataset.py | 120 ++++++++++++++++++++++++++++ multichoice_train.py | 103 ++++++++++++++++++++++++ pooled-vs-unpooled.pptx | Bin 0 -> 53429 bytes train_roberta.py | 62 +++++++++++++++ 6 files changed, 607 insertions(+) create mode 100644 dataset_roberta.py create mode 100644 model.py create mode 100644 multichoice_dataset.py create mode 100644 multichoice_train.py create mode 100644 pooled-vs-unpooled.pptx create mode 100644 train_roberta.py diff --git a/dataset_roberta.py b/dataset_roberta.py new file mode 100644 index 0000000..1a61c01 --- /dev/null +++ b/dataset_roberta.py @@ -0,0 +1,154 @@ + +import torch +from torch.nn.utils.rnn import pad_sequence +from torch.utils.data import Dataset +from transformers import RobertaTokenizer +import pandas as pd +from ast import literal_eval +from torch.nn import CrossEntropyLoss + +tokenizer = RobertaTokenizer.from_pretrained('roberta-base') +id2tag = ['O', 'B-toxic', 'I-toxic'] +tag2id = {v:k for k, v in enumerate(id2tag)} +tag_pad_id = CrossEntropyLoss().ignore_index + +def encode_roberta(sentence): + sentence_tokens = [tokenizer.tokenize(sentence[0])] + \ + [tokenizer.tokenize(f' {t}') for t in sentence[1:]] + sentence_ids = [tokenizer.convert_tokens_to_ids(t) for t in sentence_tokens] + start_idx_mask = [] + all_ids = [] + for subwords in sentence_ids: + curr_mask = [1] + if len(subwords) > 1: + curr_mask += [0] * (len(subwords) - 1) + start_idx_mask.extend(curr_mask) + all_ids.extend(subwords) + special_token_mask = tokenizer.get_special_tokens_mask(all_ids) + + prefix_offset = 0 + while prefix_offset < len(special_token_mask) and special_token_mask[prefix_offset] == 1: + prefix_offset += 1 + suffix_offset = len(special_token_mask) - len(start_idx_mask) - prefix_offset + start_idx_mask = [0] * prefix_offset + start_idx_mask + [0] * suffix_offset + + sentence_inputs = tokenizer.prepare_for_model(all_ids, add_special_tokens=True) + input_ids = sentence_inputs["input_ids"] + attention_mask = sentence_inputs["attention_mask"] + ####### + inputs = tokenizer( + text=' '.join(sentence), + add_special_tokens=True + ) + assert inputs["input_ids"] == input_ids + assert inputs["attention_mask"] == attention_mask + ####### + return input_ids, attention_mask, start_idx_mask + +def get_labels_tokens(orig_sentence, chunks): + curr = 0 + labels = [] + tokens = [] + for s, e in chunks: + other_txt = orig_sentence[curr:s].split() + label_txt = orig_sentence[s:e + 1].split() + curr = e + 1 + tokens.extend(other_txt) + labels.extend(['O'] * len(other_txt)) + + tokens.append(label_txt[0]) + labels.append('B-toxic') + for k in range(1, len(label_txt)): + tokens.append(label_txt[k]) + labels.append('I-toxic') + if curr < len(orig_sentence): + other_txt = orig_sentence[curr:].split() + tokens.extend(other_txt) + labels.extend(['O'] * len(other_txt)) + return tokens, labels + +def get_chunks(span): + chunks = [] + curr_start = None + for span_i, t in enumerate(span): + if span_i == 0 or curr_start is None: + curr_start = t + elif t > span[span_i - 1] + 1: + chunks.append((curr_start, span[span_i - 1])) + curr_start = t + if curr_start is not None: + chunks.append((curr_start, span[-1])) + return chunks + +def get_text_from_ids(input_ids): + return tokenizer.convert_tokens_to_string( + [tokenizer._convert_id_to_token(input_id) for input_id in input_ids]) + +class SpanDataset(Dataset): + def __getitem__(self, n): + return self._features[n] + + def __len__(self): + return len(self._features) + + def __init__(self, phase): + self._phase = phase + self.init_dataset() + + def init_dataset(self): + train = pd.read_csv("tsd_train.csv") + sentences = train['text'] + if self._phase in {'train', 'dev'}: + spans = train.spans.apply(literal_eval) + max_seq_len = -1 + max_token_len = -1 + features = [] + for i, orig_sentence in enumerate(sentences): + chunks = [] + if self._phase in {'train', 'dev'}: + chunks = get_chunks(spans[i]) + + tokens, labels = get_labels_tokens(orig_sentence, chunks) + # roberta tokenization + input_ids, attention_mask, start_idx_mask = encode_roberta(tokens) + max_seq_len = max(max_seq_len, len(input_ids)) + max_token_len = max(max_token_len, len(labels)) + labels_ids = [tag2id[k] for k in labels] + padded_labels_ids = labels_ids + [tag_pad_id]*(200 - len(labels_ids)) + datum = { + 'input_ids': torch.LongTensor(input_ids), + 'attention_mask': torch.LongTensor(attention_mask), + 'start_idx_mask': torch.BoolTensor(start_idx_mask), + 'labels': torch.LongTensor(labels_ids), + 'padded_labels': torch.LongTensor(padded_labels_ids) + } + features.append(datum) + print(f'max_seq_len {max_seq_len} max_token_len {max_token_len}') + self._features = features + +def variable_collate_fn(batch): + batch_features = {} + + batch_features['input_ids'] = pad_sequence([x['input_ids'] for x in batch], + batch_first=True, + padding_value=tokenizer.pad_token_id) + batch_features['attention_mask'] = pad_sequence([x['attention_mask'] for x in batch], + batch_first=True, + padding_value=0) + batch_features['start_idx_mask'] = pad_sequence([x['start_idx_mask'] for x in batch], + batch_first=True, + padding_value=0) + if 'labels' in batch[0]: + batch_features['labels'] = pad_sequence([x['labels'] for x in batch], + batch_first=True, + padding_value=tag_pad_id) + batch_features['padded_labels'] = pad_sequence([x['padded_labels'] for x in batch], + batch_first=True, + padding_value=tag_pad_id) + return batch_features + +if __name__ == '__main__': + data_iter = SpanDataset('dev') + for d in data_iter: + print(d) + break diff --git a/model.py b/model.py new file mode 100644 index 0000000..1bdc153 --- /dev/null +++ b/model.py @@ -0,0 +1,168 @@ +import torch.nn as nn +import numpy as np +from torch.nn.utils.rnn import pad_sequence +import torch.nn.functional as F +from transformers import BertPreTrainedModel, RobertaModel, RobertaConfig +import torch +from torch.nn import CrossEntropyLoss + +tag_pad_id = CrossEntropyLoss().ignore_index + +class CRFBert(BertPreTrainedModel): + config_class = RobertaConfig + base_model_prefix = "roberta" + def __init__(self, config): + super(CRFBert, self).__init__(config) + num_tags = 3 + self.roberta = RobertaModel(config) + self.dropout = nn.Dropout(config.hidden_dropout_prob) + self.classifier = nn.Linear(config.hidden_size, num_tags) + self.init_weights() + + #crf + self.start_tag, self.end_tag = num_tags, num_tags + 1 + self.transitions = nn.Parameter(torch.Tensor(num_tags + 2, num_tags + 2)) + nn.init.constant_(self.transitions, -np.log(num_tags)) + self.transitions.data[self.end_tag, :] = -10000 + self.transitions.data[:, self.start_tag] = -10000 + + def forward(self, input_ids, attention_mask, start_idx_mask, labels, **kwargs): + outputs = self.roberta( + input_ids, + attention_mask=attention_mask, + output_attentions=True, + output_hidden_states=True, + return_dict=True + ) + token_embedding = self.dropout(outputs.last_hidden_state) + #get start idx embedding of each tokens + start_idx_lens = start_idx_mask.sum(1).view(-1) + embd_selected = torch.masked_select(token_embedding, + start_idx_mask.unsqueeze(2)).view(-1,token_embedding.size()[-1]) + embd_split = torch.split(embd_selected, start_idx_lens.tolist()) + embd_padded = pad_sequence(embd_split, batch_first=True, padding_value=0) + logits = self.classifier(embd_padded) + logits= F.log_softmax(logits, dim=-1) + mask = labels.ne(tag_pad_id) + loss = self.get_crf_loss(logits, labels, mask) + outputs = (loss,) + if not self.training: + sentence_score, pred_tag = self.viterbi_decode_batch(logits, mask) + pred_tag = nn.ConstantPad1d((0, 200 - pred_tag.size(1)), tag_pad_id)(pred_tag) + outputs += (pred_tag,) + return outputs + + def viterbi_decode_batch(self, emissions, mask): + seq_len = emissions.shape[1] + options = dict(dtype=emissions.dtype, device=emissions.device) + + log_prob = emissions[:, 0].clone() + log_prob += self.transitions[self.start_tag, : self.start_tag].unsqueeze(0) + + end_scores = log_prob + self.transitions[: self.start_tag, self.end_tag].unsqueeze(0) + + best_scores_list = [] + best_scores_list.append(end_scores.unsqueeze(1)) + + best_paths_0 = torch.Tensor().long().to(emissions.device) + best_paths_list = [best_paths_0] + + for idx in range(1, seq_len): + broadcast_emissions = emissions[:, idx].unsqueeze(1) + broadcast_transmissions = self.transitions[: self.start_tag, : self.start_tag].unsqueeze(0) + broadcast_log_prob = log_prob.unsqueeze(2) + score = broadcast_emissions + broadcast_transmissions + broadcast_log_prob + max_scores, max_score_indices = torch.max(score, 1) + best_paths_list.append(max_score_indices.unsqueeze(1)) + end_scores = max_scores + self.transitions[: self.start_tag, self.end_tag].unsqueeze(0) + + best_scores_list.append(end_scores.unsqueeze(1)) + log_prob = max_scores + + best_scores = torch.cat(best_scores_list, 1).float() + best_paths = torch.cat(best_paths_list, 1) + + max_scores, max_indices_from_scores = torch.max(best_scores, 2) + + valid_index_tensor = torch.tensor(0, **options).long() + padding_tensor = torch.tensor(tag_pad_id, **options).long() + + labels = max_indices_from_scores[:, seq_len - 1] + labels = torch.where(mask[:, seq_len - 1] != 1.0, padding_tensor, labels) + all_labels = labels.unsqueeze(1).long() + + for idx in range(seq_len - 2, -1, -1): + indices_for_lookup = all_labels[:, -1].clone() + indices_for_lookup = torch.where(indices_for_lookup == tag_pad_id, valid_index_tensor, + indices_for_lookup) + + indices_from_prev_pos = best_paths[:, idx, :].gather(1, indices_for_lookup.view(-1, 1).long()).squeeze(1) + indices_from_prev_pos = torch.where(mask[:, idx + 1] != 1.0, padding_tensor, indices_from_prev_pos) + + indices_from_max_scores = max_indices_from_scores[:, idx] + indices_from_max_scores = torch.where(mask[:, idx + 1] == 1.0, padding_tensor, indices_from_max_scores) + + labels = torch.where(indices_from_max_scores == tag_pad_id, indices_from_prev_pos, + indices_from_max_scores) + + # Set to ignore_index if present state is not valid. + labels = torch.where(mask[:, idx] != 1.0, padding_tensor, labels) + all_labels = torch.cat((all_labels, labels.view(-1, 1).long()), 1) + + last_tag_indices = mask.sum(1, dtype=torch.long) - 1 + sentence_score = max_scores.gather(1, last_tag_indices.view(-1, 1)).squeeze(1) + + return sentence_score, torch.flip(all_labels, [1]) + + def get_log_p_z(self, emissions, mask): + seq_len = emissions.shape[1] + log_alpha = emissions[:, 0].clone() + log_alpha += self.transitions[self.start_tag, : self.start_tag].unsqueeze(0) + + for idx in range(1, seq_len): + broadcast_emissions = emissions[:, idx].unsqueeze(1) + broadcast_transitions = self.transitions[: self.start_tag, : self.start_tag].unsqueeze(0) + broadcast_logprob = log_alpha.unsqueeze(2) + score = broadcast_logprob + broadcast_emissions + broadcast_transitions + + score = torch.logsumexp(score, 1) + log_alpha = score * mask[:, idx].unsqueeze(1) + log_alpha.squeeze(1) * (1.0 - mask[:, idx].unsqueeze(1)) + + log_alpha += self.transitions[: self.start_tag, self.end_tag].unsqueeze(0) + return torch.logsumexp(log_alpha.squeeze(1), 1) + + def get_log_p_Y_X(self, emissions, mask, orig_tags): + seq_len = emissions.shape[1] + tags = orig_tags.clone() + tags[tags < 0] = 0 + + llh = self.transitions[self.start_tag, tags[:, 0]].unsqueeze(1) + llh += emissions[:, 0, :].gather(1, tags[:, 0].view(-1, 1)) * mask[:, 0].unsqueeze(1) + + for idx in range(1, seq_len): + old_state, new_state = ( + tags[:, idx - 1].view(-1, 1), + tags[:, idx].view(-1, 1), + ) + emission_scores = emissions[:, idx, :].gather(1, new_state) + transition_scores = self.transitions[old_state, new_state] + llh += (emission_scores + transition_scores) * mask[:, idx].unsqueeze(1) + + last_tag_indices = mask.sum(1, dtype=torch.long) - 1 + last_tags = tags.gather(1, last_tag_indices.view(-1, 1)) + + llh += self.transitions[last_tags.squeeze(1), self.end_tag].unsqueeze(1) + + return llh.squeeze(1) + + def log_likelihood(self, emissions, tags, mask): + log_z = self.get_log_p_z(emissions, mask) + log_p_y_x = self.get_log_p_Y_X(emissions, mask, tags) + return log_p_y_x - log_z + + def get_crf_loss(self, logits, y, mask): + s_lens = mask.sum(1) + loss = -1 * self.log_likelihood(logits, y, mask.float()) + loss = loss / s_lens.float() + loss = loss.mean() + return loss \ No newline at end of file diff --git a/multichoice_dataset.py b/multichoice_dataset.py new file mode 100644 index 0000000..7047316 --- /dev/null +++ b/multichoice_dataset.py @@ -0,0 +1,120 @@ +from torch.nn.utils.rnn import pad_sequence +from torch.utils.data import Dataset, DataLoader +import glob +import torch +from transformers import BertTokenizer + +tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') +pad_token_id = tokenizer.pad_token_id + +class MultiChoiceDataset(Dataset): + _MAX_LEN = 30 + def __getitem__(self, n): + return self._features[n] + + def __len__(self): + return len(self._features) + + def __init__(self, phase, data_dir): + self._phase = phase + self.init_dataset(data_dir) + + def init_dataset(self, data_dir): + if self._phase == 'train': + prefix = 'wsj.2-21.txt.dep.pp' + else: + prefix = 'wsj.23.txt.dep.pp' + preps = [] + for filename in glob.glob(data_dir + f'/{prefix}.preps.words'): + with open(filename) as f: + preps.extend(f.readlines()) + preps = [t.strip() for t in preps] + + children = [] + for filename in glob.glob(data_dir + f'/{prefix}.children.words'): + with open(filename) as f: + children.extend(f.readlines()) + children = [t.strip() for t in children] + heads = [] + for filename in glob.glob(data_dir + f'/{prefix}.heads.words'): + with open(filename) as f: + heads.extend(f.readlines()) + heads = [t.strip() for t in heads] + + n_heads = [] + for filename in glob.glob(data_dir + f'/{prefix}.nheads'): + with open(filename) as f: + n_heads.extend(f.readlines()) + n_heads = [int(l.strip()) for l in n_heads] + + n_pres = len(preps) + assert n_pres == len(children) + assert n_pres == len(heads) + assert n_pres == len(n_heads) + + if self._phase in {'train', 'dev'}: + labels = [] + for filename in glob.glob(data_dir + f'/{prefix}.labels'): + with open(filename) as f: + labels.extend(f.readlines()) + labels = [int(l.strip()) for l in labels] + + assert n_pres == len(labels) + + features = [] + for i, h in enumerate(heads): + assert len(h.split()) == n_heads[i] + single_feature = [] + for hi, hh in enumerate(h.split()): + inputs = tokenizer( + h, + f'{hh} {preps[i]} {children[i]}', + add_special_tokens=False, + max_length=self._MAX_LEN, + padding="max_length", + truncation=True, + return_overflowing_tokens=False, + + ) + input_ids = inputs["input_ids"] + attention_mask = inputs["attention_mask"] + token_type_ids = inputs["token_type_ids"] + single_feature.append({ + 'input_ids': input_ids, + 'attention_mask':attention_mask, + 'token_type_ids': token_type_ids + }) + datum = { + 'input_ids': torch.LongTensor([x['input_ids'] for x in single_feature]), + 'attention_mask': torch.LongTensor([x['attention_mask'] for x in single_feature]), + 'token_type_ids': torch.LongTensor([x['token_type_ids'] for x in single_feature]), + } + datum['n_heads'] = n_heads[i] + if self._phase in {'train', 'dev'}: + datum['labels'] = labels[i] - 1 + + features.append(datum) + self._features = features + +def variable_collate_fn(batch): + batch_features = {} + + batch_features['input_ids'] = pad_sequence([x['input_ids'] for x in batch], + batch_first=True, + padding_value=pad_token_id) + batch_features['attention_mask'] = pad_sequence([x['attention_mask'] for x in batch], + batch_first=True, + padding_value=pad_token_id) + batch_features['token_type_ids'] = pad_sequence([x['token_type_ids'] for x in batch], + batch_first=True, + padding_value=pad_token_id) + batch_features['n_heads'] = torch.LongTensor([x['n_heads'] for x in batch]) + if 'labels' in batch[0]: + batch_features['labels'] = torch.LongTensor([x['labels'] for x in batch]) + return batch_features +''' +def get_dataloader(dataset, batch_size, is_shuffle, num_worker = 0): + dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=is_shuffle, + num_workers=num_worker, collate_fn=variable_collate_fn) + return dataloader +''' diff --git a/multichoice_train.py b/multichoice_train.py new file mode 100644 index 0000000..2dd5a14 --- /dev/null +++ b/multichoice_train.py @@ -0,0 +1,103 @@ +import os +import time + +import torch +import torch.nn as nn + +from transformers import Trainer, TrainingArguments +from transformers import BertModel, BertPreTrainedModel +from torch.nn import CrossEntropyLoss +import torch.nn.functional as F +from multichoice_dataset import MultiChoiceDataset, variable_collate_fn + +class PPAttachmentBert(BertPreTrainedModel): + def __init__(self, config): + super(PPAttachmentBert, self).__init__(config) + self.bert = BertModel(config) + self.dropout = nn.Dropout(config.hidden_dropout_prob) + self.classifier = nn.Linear(config.hidden_size, 1) + self.init_weights() + + def forward(self, input_ids, attention_mask, token_type_ids, n_heads, labels): + num_choices = input_ids.shape[1] + input_ids = input_ids.view(-1, input_ids.size(-1)) + attention_mask = attention_mask.view(-1, attention_mask.size(-1)) + token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)) + + outputs = self.bert( + input_ids, + attention_mask=attention_mask, + token_type_ids=token_type_ids) + pooled_output = outputs[1] + pooled_output = self.dropout(pooled_output) + logits = self.classifier(pooled_output) + reshaped_logits = logits.view(-1, num_choices) + + ones = n_heads.new_ones(n_heads.size(0), torch.max(n_heads)) + range_tensor = ones.cumsum(dim=1) + label_mask = (n_heads.unsqueeze(1) >= range_tensor).long() + + reshaped_logits = reshaped_logits + (label_mask + 1e-45).log() + #reshaped_logits = F.log_softmax(reshaped_logits, dim=1) + _, pred_intent = reshaped_logits.max(dim=1) + + loss_fct = CrossEntropyLoss() + loss = loss_fct(reshaped_logits, labels) + + return (loss, pred_intent) + +def train(data_dir): + train_dataset = ( + MultiChoiceDataset('train', data_dir) + ) + eval_dataset = ( + MultiChoiceDataset('dev', data_dir) + ) + + model = PPAttachmentBert.from_pretrained('bert-base-uncased') + + training_args = TrainingArguments( + output_dir=f'./results_{int(time.time())}', # output directory + num_train_epochs=3, # total # of training epochs + per_device_train_batch_size=16, # batch size per device during training + per_device_eval_batch_size=16, # batch size for evaluation + warmup_steps=500, # number of warmup steps for learning rate scheduler + weight_decay=0.01, # strength of weight decay + logging_dir='./logs', # directory for storing logs + learning_rate=5e-5, + save_total_limit=1 + ) + # Initialize our Trainer + trainer = Trainer( + model=model, + args=training_args, + train_dataset=train_dataset, + eval_dataset=eval_dataset, + compute_metrics=compute_metrics, + data_collator=variable_collate_fn, + ) + trainer.train() + trainer.save_model() + result = trainer.evaluate() + print(result) +def simple_accuracy(preds, labels): + return (preds == labels).mean() + +def compute_metrics(p): + preds = p.predictions + return {"acc": simple_accuracy(preds, p.label_ids)} + +if __name__ == '__main__': + data_dir = os.path.join(os.path.expanduser('~'), + 'pp-attachment/dataset/Belinkov2014/pp-data-english') + train(data_dir) + ''' + dataset = MultiChoiceDataset('train', data_dir) + + dataloader = get_dataloader(dataset, 4, True) + for i_batch, sample_batched in enumerate(dataloader): + print(sample_batched) + break + ''' + + diff --git a/pooled-vs-unpooled.pptx b/pooled-vs-unpooled.pptx new file mode 100644 index 0000000000000000000000000000000000000000..df0cf1cffbefe010a035c281183ab533a3d1f2f7 GIT binary patch literal 53429 zcmeFZQ;;WL*XLX8VwcrrySmIS+qP|^%eHOXwr$(CjbCx<`M)z0?>X~K%*DAo@8m^h zWbB=hu`<`%JJ)A_SISC&f}sHY0D=Sp0wMsS6wUT-0R{pR{|N+y3H2#mW*t4-AAn2MFZ5|Nq_o3qOI0#8C02$b4tU`0cn-e<6p!D znbEk6U=N+D3QHI!W{im#r5OZkuER*GDiDNL(8io^-8f<5`Ysy{b=G0jmn=n@gQ7W^ zRduhLftQ^o4wNS+p~l?(JAX5!_>p^eCX%@wX6fb&P=gS(`S1k zD9O&)xg!P(EwV^x5}KG@aKY+L>6ak@bJ({6gZnaw;eyW&DV1*e4@5NFbvIGH-8TW@5Y&L9-sS~Qv`#Y?^zCeLw z{~rNw`r?=i`rWek9nUb|0k30kWa&Wr`(OM28|eRqqx(OZUJgXv=u*T>raP*L`7m+P&f5(^(|NR*L*O>Vwl_k>&2Lyzf0|bQn-NV+_ zk=DV|%+Sc;KMwT&nkFx_G;G#~ki2+$0PucU_CA&!&l0l5_|BtEytKj>PlYF-M6*q@ zVe%A1PO=Z(*=lidR%%fRf{LYSUNtVOSSH8&to7~n?`P-hW-KjY;e&=r*>Iy&oY(~b zQi;r7H+2H&7WCj_ILKXZmjPxdpsK)@_vgdw`nU z=I8BE1%~7he)TTCOc8$4u9!otgSy`cwI#-reg{UCOJDOH=RVDMG+k?-$SQ z^4z*TT^ViERaI1#_gc(fM;t3r&$^181MR}k(z78hFYHE-jOJ6w*LO+@2>X?W!#AN; z%f&Z+dyaRiC^PcBlzfaRgkb^^y@8)$6!VBmwS|(Omk!7PH^qX)f^H2n*w|<{Gmc-V z^@A669?WS{=~vq3KOGTvBd1=HK8+xY2QUL6XORX|EKtZV5Xhne2cY-_-kgcROz-cS z-NNFJQqzOCS9F~F)>ae}Y$Hp{UVZF>Nao?)f2I`{n7aw^pF_FMkc^(Y){=LNnaun0 zu;HNL--$1c{$#3?Zj+R6EVYJuPr|EPAe*>rJ|z+DXDuF6gJNh@qi+$bo7ZE&^bLxC zU2})i;e}*$C_lg|W3_)%PB&;C;44mG1L{Kn)g+mXlo{hyWoQ@ciaa0OifD&q{LO#Ns zPBB0~_qsRciS9kQ|HhJKeC`%MvkNO8rBI8jEidjh zFTh>N6Z6fDDzlu=gJ)a7cDvmJZ`*&|lHtp!^j4w|oCSQUI%cYtPQE zD1tVPThmEfYhHN3olV(1QL>I44}J{ZVaBg6Me+Ku2shhsyj&0B#d_!iZib8LYJ5F< zXY2U|NlHU+u&KYytN1Uxvb;TMYB+*kEWfZr5I+sr9GK`3yJcptdqAQ&loIxYiF$f% zf?hvBSP{{npw#-Y@bPv zq4|!$%B`HN1tSY1YPFg^`;fN=oOi-Mb9%D1k>N$1Rt}LWXyIHAz$)Ch#q|0Lrr85o z2vYX}J4Frt=)ipq96H*v6t7B&DHvkR_rqvqA)Q}O7&cU}e@z*4p~=766@=>7$I<|s zf)$Xri;Rf;l}F7KAs_%kqfx-HUmWYWFKx$H5Ys8Ox}NY>O93#iIlx6|=ioqsh}~g{ zK!~*U(Ht-+An6y?r7VQiFE(5uL}+AyX2ljHAxf)ecb?IvF|twMR!EP#9xXc1FP9p1 z6(PWs8?JD|gvy~9poT|GF-Jp;N_8ZTpxJghvrYzbkxh_4P#IM09MX6?2QX>4Rgxu%gmJ-=r7 zR>z_;?=GW}o~cD%R0$(Qj_?wQi3o}s92+Y-7>6ubo4CteJl=^j>(^KcW+2HLACy3D zE@)$g`$maU>uHyKa<=yyX)w3OkI<@-kNPRRT^|cyLK0&zO^9&ADZ>a|6``fAgdyHZ%ELT%obmgUgbY8iCG(*l> zO9;L(qyXEEg)|{q3@Td_wJ|Z=-^Z1|y=c9Lv^%d&(10~C#?aPvTju z)I2uFas`3f@a+cj2(^ZclOaW1=}w*#I#ref&GpX!HXQh?!0f+EY!Rji;Qai9NJPTy zExK#;#J?KZ`XWMVCWYet=qnm3`t>4QP<{%O0Jm4?Vinc&tp|N}o^)=!=?(8=|HUp4 z+67;P<;n8Lm5E=6Ad0@p0K8CB#5Ln#b-cgdinV4^7^$SnLs6mDgRz&x)}{R^0G&jP z5-KT`^=hc5qM{C@xBTTHP0s3OmAJ})VhyqK{O}w-pADa)nQf+^6^V*Z4RRQdBBxIg z#zr<0fDJ27Mx8V_IMjr7ZN*z^F?pP!K)LKHV@IgOC{Nx)w!aAE$us{%n`QRqbh|#@ zTfKa6sQYP8_i7~*Z{_CR;&Wx*!iz1wFNi`2`-zlS$6tEayX|XF+QbjpH63%EB>cND zcc<%#VEYRxYrUaTOr^MdAHhYC?G&9Fg8a$g%eTPmLI;ZoLp9KF*(ZHEEOb7>4}SGt zbJ`)=CF&~qcw+3`Bkj1&JTfh1ujdEaFI4&i3>6>wT?EDQN+OOp$LxAYW1e_OJPkz< zPBIkiXHMq^&f`YI%wtKFVIMC~pU?9}|0? z)QZK&t^3EVKW|Hl%5O`6iD0SD5EjZ2L)mpl%_0RyXqGuqpqob>)_UdvzFS$xZdn@p z?4M?w&xpL%kgN1sKUWd^prm1im>jt+qZCjE6yLZWP>f*j%buQ|Y=*G0l(Dh|#eufh zs@6R(HtsHd0eY`f*>#W$JaKx;SI5E3~AlxCWp)SxkCty;t=$w z3lXd!s=vo|tE^AiyU8;}GX_bgV7pekNKv!w-{}%j5IG)J#!_>05V~} zqTv1t)W%VoK}-_LWI_tU-fh-sghEA9?cz<}uBC;nnT!EWZtW%;l~=Zrzk^E@6J8-L z+SfX-Znomq>S{Ou9bLbgY_&!7zBhapJ^9fpEdt#i{X4ptJzXcYweO{byw)22w7!LH zel$y}SQ$|{^`t6`suFqTl=Jz@V*ava30&ZSwoU(2Pt}`X34+BkSZN_e$MSO$Q&R9N z%oX>K(N#eInF&DZsZd#21V$ ztyFL4CWnF}M z43B?M!-?s1)-CM;zpBV}si|cf;|76ptE~AHeE_#OtlqyR-D`1;amlWmc@Eeyqw&n9 zF8`5rmG51uyXcVp{6!C2nrDAIFY0=H_dx+6c=6VsEn|8e(6Xm8f0M9N4c*L5Aq&m~@M{BeYxFr6KewdZT&@2sj~3Z}*@>HaQD%-? zZZz8VoRP{^)|_OJ=WlGkdDN>`lO+OnW@QoRwHmD$cWRKz0gg6$_dNnfQ!)pGH1{Iw{kVD9d|aYZUE1}&39Y3shsz}Q{ny8 zf1mF%{71ffZoNN*`l*w1jenv7N^dcIvn8`i6&JxMlyXDhn3yNgdNGJjjZ z`3L*cHja=Bd*ZwT+hnj|0zNqV4sc_G_xTdsJ+(C*ehXxteE+<+f!*W%0xuT<$ZHe zjyWa5SG%7tRSQTh2;$PeNwtN#x-Z#3@ysN=diCvm{;bctul;8gd@%33%;&Xlce2j1 zznp`Eql2^K=SNyy{=K2pslN~eL^H>u%_8j8=XuHCxyOVE-LZ)fcTE3Z>g|d1!IIzQ zqq{fTX5y8i@=@|HL2OOfs{vRE@ zAJ$Brrp4~P%yfJK&7AupuQmc~Bva&qAxY$en)#x;7KQ{TkQ} z*Bf2$Yk)R_rL|NfO*(4R6j0y9fZmNvW{jk`vdo`Q`DD30_TqKV_2$~%i88c5KF4=! zK;j_OO;qScica!GF%%e)UMhG{TWM7j@a`3Iemn>ZO!AIwpf2uEQo#kUnu1|Vr->Y| zve{U}zd;yYOh*I8R%7SL_UrhysKlr@*unjT+CNDB z?u)5rr@TdpqMGCPCUkVDubqb%73_aoP-a7?>izQ4DaD;7=%x9ypV=@oaF*r=sxgD@ zmXqrS;Vhu~*>GIG1$$YEh5b)8gq)I;H%^P08pvSLq7T!-B`$~19$kDC5zQRfAlyu8 zZ9inOv}zxF{e)%c6p2+`kp8x$2+uTV?=dqc%!k><+s})oo#(%u?JZqSZ!8TE#e;@V z(ctyX_{k{}l}e!YrKl$Mm8M0QSWYT?TmH`-nf-?c%v_b=?IQX+C6@BUhc2?-X~5_U zi^L%ZO?Wzn{)#Igt>*;Y;O07*caM)I_}3;LzC}_s`ffxODQ9JK{jdtuuncT*q6)v| z5)guI1@}`5gUrO8A*p)*f|Ped{4vAnX~|pL5e}CZr(>jQf+x^q#vp<<&1F5A!8ODI zPF<;J7gTXV7n7&T$PxWxg&@&+C9V~KP0u36ED_N72P{?2eO616p$Y|RW=`JL{E83; z3@SN+bw^&9YRTX{Q8(OX2<@D_2}t?(U@UI}Wf=A`MIC?PNNb8cK|~|x8cEZHb`7{& z2*Tn@V44Bt^_!Pq_!67qjt%0b!MT&Jgyj(0HF=XE*pOld!T_wEE)xH+91_d{n2Q#* z5piTwWQe6djo)ZiW&irsG6$+KhQUz!J+3h(Bc&O9{W36P>o4QHe7byTiAy>WNx6Mb zI^PQWok$Aff)M2IDFP?x0nq0 zz*vf+y^JE!R-Rr&_9B5#`X1v{)FD7Yj~O*gE-$N;!ue-oL85=mjo}9U@g>lJ=r{IjhY%;qpO(L2Ej+C_I4^Q9{pQ4IpuF^r~T@)GN?Zp;NCldF0S-FRWnjsfcw|E z#+Y{BHKJ{!>>*QP1t<4O&gduw%FK>V+-a*C9js-7XCp>^o=pZwu1` ztM<~`EUadOPG``sYjyyAaP-&5arg6=e=$l{6Z*~9P6kc&^0KM;ZMypBPk7JGb9S>L$0+izSxb zR!d8WwYKI0o&K{RqCzOi1QZH;96q{hy3Pt@KwKjIKE2Fp6U*0@LXn~=C3-q5hQiJC zS_VaLknGrNf!T0;kyrC{*%X?VQ`FDD(#U88lCXS;M-+1d90P?%&?rqniwz=~lMQ<_ z4*B(g&jBh1ik(wl{v7E~9B;K*eyY~P9VAX|ZC`Vt#CN`Q1HH_=84kgOc?JV{kT6Ih zu}SM!-5&CWntr0U?oEEhIlR?Di5Q&=uIuu_kMhT%S$z!bpEboZi<++D^ojCxhAknn zbz4ATiToHFfhodFg?p{-iq1zA{f!6Ha+Fs>$Fo&ccsvydUNo2xy;u^*r77>AjWM_3^Apa1p zQ*KOb1*e9XqHn~X^0u_42Aq-ubC1@8uhi|jmuUKY`JnEmhT?DF65WA)=$-cE8ky#< z=q@7SFUCGq4dLxp+F zkBLwdtXb4G=OnGTjc5!VvVk0NJ}Hod^voQsKut#_UNejEZ%3n834xG2!AygTxlQK?|4c*{5sw zh7HO&&T7=3I_@(!m6ZCbzTol{dPL-eu@U@3AnqW&?)%spQQ30jU)>>0CeRiWs#KGa zV6z`X1?-!tOve()U%T-|?dG?C4v|BC^T%QOW84NqQS!wo*NPG+*%dTzhP6j>KFe+n z^JQW%&i}4FlC+&tnPNEyf3-k5@%Z(o1L?T6Do~(+Q>;uchDLH2#)&Obc;y(Rw8zs%dv^66rO z`e0}c0&@Vy<=Q$?=t4rFhC|2fVF*aN4oGd)TZ^R$+ewmhq3QD>rjkk5c>3B_? zv|DX0j%HU;Iw?Q~QhGKr18t1nj_ND;HpJSS!6W=7QXsm8pB?3;&22mm?m5bV;EcQy zRq!K=_TLK-uh8*|6>LsUgHH>*C)S_`6-#%$^Xr2AI%Hy8SI=Ov-*x>oYAGu8ij_hKwj?Ehp&1Ndc zrJVY=cF^FVwwh+AI@&UA@7^aX#TG+!I{L0%_qdHW=hbsZr!?-fNoqG(G_B@5>*)T2 z5vPgDUt?w`$Cd{4T) z{#clJSGN=0QyI=p73P?P1U37l8uzssY@V}4$X%+lGL?r(#i8KC-4g4b~iJ0z$ z{?-dY+-Tk4^j(rJ@(HH3h2XP=a`(h)uIr`;XrjD588__gxaf5AsNikw<7+3ZfLH6$ zP$3*h)l4~KUNz5~L-;Yeg$vvi=n)19)N3xNVA3TMU1=H0p-f}GX6$QIuy93=FbLim z%>^;(|28^P6uj?=S|}=OqV&X?nUJ(=!kk;J;@c<^{aVSj%FH#On~wGPM9g)uP5R9i zX!4u7D5~xUMfY2MVI(zTPMZ4UmKFE^2h;z-BL6i_|3?Aw|2NY?geHW4eLMRa7!VL1 z5G3%w9R5E%-PYd7;aeoB=V)eQ{XhNvTRsK+t&{w|;J^0Pu`ny%Plxhj$Gz7X@_4K0 z(lLyBGZL~LLGT>Z@X`!RGCpR&Yd^l*l@Tyj;Lku|{zPp02G7F?&GA{ns(<(`hqY1$VF9UdgkdA3 zBAj-|v$%49cYCqqmOR9lgm;eX7mXcNp5pNyH zhpQ>!MvtnvaE^_OF=r+o5AR>gC=?9m3&khSSCovsyS!8AM0~w`c7L72-#drvbNH@rOQwCCp!9uRgxcLZ3ND0!wb-_&#Sq%Ez|&YT(93DPHtB(GLrJ3pRl zovE0Z)QjP&#FvkESgr}5b&OzQ0QU4*bf`7bujke}rqE|3OFV5vuXm?N$p#m^X^)DX zvflFVUS%HN4L@_?lDj)UWuuK1a+X~bjgxtkZsnYFsnEQVmUnkdD_;S3bnea$$~vz$ zEzPr&w-_N`qu#gx2Y?h~S_2tbvk4jcwiWrGR#BD&ZuP&mj>QVzDwjLBTT$v;Q^Ou0 z)9)Q4BK8}Rhv`D~*o#VBe{jEuq)Ji6#^j(yLJY=ZrCRtXj5u;Bof1a9U)Kb?(idEF z9+s)12=00n+A^fh%;hqfb)?G1uk2`Bf8ZkQ;!MB9e)c0shcZGV=8=fgFO$o&31Z}R*h>`6)9-u)fb=6+@EwSV76vNS|S(xR(6a{~*ejP%3dtF*T!L2+|FhRzKM zx&l}1$y%E>jpuOja0CxK+5+D-$La4+f6MUS;Q{oGM84NY3bAi~LI8RmP;*`zrUlr5 z2`#zks`@>b3@Or{-GWZXSgHG`hMt?{NFybpcj%w5b0+n9s*Rh>HLaITsZ|~=U-gVF zu}VfAop)uzC#{q=R60{*Wm^lpcIyrrT_my+DS-B)A?5jR&>DU7vqOtDusjSXsv<*E z5#NSXYLmN0e>USKYK$)0-B5cnBNHQ45C!$U?XGtw6o}`!oW{8koraS5cKnL6z+*Qa_ron?q5t4JGafkeUP=gbwE{p6K9F_t%&;m_wxFqs zUdtZn`cYu~VWWuW5+qPa1_-cc*^Eo|(&kM1&PI0t4mCk6y&+U(dI!M52}lzYZB+xR zH)!Hvc0tXiW5D8EzHuv6xq>>(DE|qw2K^_@>b@W=j3X!@H`*dk%qt<$TiO~;jexE> zN759bY}vC!)%%YqMB!T$q8UcyORj{%KBZt7C?4xTf5?wv;o2Z$nd+|_ZgI1!Xz^hd zRqAoFi%7$PEbFTx^x#vu>`PONqtQ{k5{HPPV8@T3DhCV8bHXOB@0$U%9ab1oFk598 ztyLq`gRu_L!fk`IS>y&FVc<19Bb}Ra>?J3~Q?}e;qIC*ul z6Q6s-g#9CCadnth3Ye2?fwtK~d{srNG$1~V-A}(5);?`mR3wck1G>CaHUjCeLlZ9_Q1R~f~3nWhDZ{X7vSI*ai}Xh)Tc+W zztHj!V;)T&+Ua^+ToRu}86$CI2N%NUVLJRY zMD8jO|E3CpxxwDTn->%sq++Z9=^PYc2BiR4RHXp~7y*QP!YzwKI3(!Nj$4l%&@NCP znGapS*s`3ctqxg_6m`DfFp4dD;RxkaEX!}cQiUl@Ac~}RVYX}#fzk%1B4$xO^uA~U zUwfZjZQT);-Wt!I5C2>x#9cp&1G)#e{u>G{d3;z(h9rW@aBPI2A+oM>X)H=xq{B8H zwjNub-FV#=!Ci`)xJqkb&&UZu;w4J)N_Y)jlp=mgHV)} z%h@CfWS|;l5(Yl@!E*;k=la$ZrUy60A-y`sV6TesZBOW%7ioiWQ^r`Hx7@jIGm221U?CasR z11f-a$CBB}R%HFrsj5e`ilPy7&@L`d)Gj3P!Q@AA~A_d?f`6l?(>>Kv9E&W^tzz|f1bO+ zm)*_hDGlNIsY=UVfhM#S%i*a7ekzCr;6>frVZonj6+oP&*Ovx%UUgpD+IJ_<>fwg(DIW|l__9O(zy9S{WB30(Ywxq zs6>uxD})fF;5xGD*02=u_A9hk4ny&&a_q=|%xljr@Dht>r0b|t*8oElD;&j+eh)?Q zZ$cXi2`rqQA(4Jow+{<#F?{k(>q0-(BT z>Lacm;@-vcv8Rw^0I4{}m`QlbR`vqL0_E2)65_qJnPJIP9&YCe(KTx~Lb}1R3icgI zAZ*j&uK-HI#hR4e#FG0>6g4*`lzilt=#GqoZVl|OW1X)q{4c=r*9Yyp&dedO2JX$3 z?v|&_&X+gKNuNmMZh!D>MaB_p{x;WqKnj;wGLQTFmAV=_Fn2Y!f7r=c>Eb~5-%xRsu|{$U8@#*gp+E8ocqOt9{HU6Eo+d>Gq%Psf;94A+q230 z0aM#7v5DS9sSuBd=66S5IBR-?vOu7ed!ps8p%1uJSKfu z`b+c?-NK$7-87+vG^Wktrg+@4U>BPos6lBv`H+$*!#6K|sGP?N z-V~3u*=EgtcCt{kCQ&5o zmvQ)=<39KB=X4OqYPP4@M7=TG{|vs#z9Lsirr_chcxM-WnY&LXK3tz4WhsubNiz%7 z>I73#qmzpx^CHo&=~Ed*>KIok*$x$KXS_^KHP(HAJ2Z7y#y&GkAie)7>xJecZGwZs zB>f^0Ru`T##}xW;t(1dzgzCWWdv==i2X4NyT06(RbspsG%@u-VZ5f~(vnEN2MkLkv z4OY44|0h^|h*^gTsWB~q8G{j$w5Z;w-D6J1Z(paG4;``$XLVs5=DZ`>(GY>0x5mQ-(+kVCu>N=kucwkd1%inbs!)o*v z+yL?NaA)rdH?1uhg+_y_uQXH|OaZFQ$2sk1gD1nA$14OA_LuytQtsQhV*zj@xx)!%a>M9+- z!H+*A|AVadUE6YAEw_Kq?Zpy0XFZ&_AYnF%!>8jeX&rL z_M8>iAe%RPZrLmjbMwrPHrt+tW^Pz8%ibPqozy^oe5z7IfQjHK%QgL3l2D%deF-oZ z|B$BtLYoyyBKnh*uS0F}_nft3-KyE|q8>cj+U($T34EHXF#PoG3*z6Q09Xdq)}rrv zyzuWLT$FEk^e@o#|3*v~$(z>cbSOU8WS2NGHe}*2WOAa54F+ioVsi7`J}}b$=9Y%( zOs5OZ+qIy+1!)TqJJ9RL%XmW>jNMx*3u}l35>xUA+EOt>s#D={odqYS^!3PwVu5PD z6eh$%5QMXjdvDFIT;<@g0TdNI3m~kLQ?_AE;){&Kv4A8+Rqfj}x)`gLk{Z8K7dvR* z?H*4d921UU|gZ_Lk%`{Xxob6jy5mh)_ImAf<^1 zXi?1r>Z2*|iRobxGCj_kr-sK?my?Yz>MXbcK0J4fZ>7_r`%3k(v@Zm9=zB~aITSJ% zJ=`x9!6x&}9#g4Jd^R&wLw~F?+)?eyMc=qV?DNJ&@rL~1#2WG?kx05XGpo8^WivQV z7Bi(aEz2o1k~%vn@`k=sL|)q6&WL&nVVJ0X*Q@d!Y0HK1;1 z(VGL(wP`BER|%g;tk#1D>bf|2#@XYL4SBswIzffE{qgo0200q161_Ke4@MqJV}(NI z+lU<);9e2F-?U(@7j~Sc*i)e*;X~)MV0snExBv`t8och~c|Ht)DLGAc&hh?hEmout z2PI98EfD!FQv3&`sDc@=5EuTm1mQh82C&(8iwh7r-1}`5x11R;2^YSUNk@TiiOe+v56tFa zAKQ*ys;;$<;3PFmuplY4~4xvf)~n{+l!i-Bp*<1j+>m1*3c|qw3RO}H@uuPf%P>yC)OFNUB{1}vL2t# z@3Xz#7AIf7MS#y9L^Or@mnMJ>ZHd)M>|)xbiqSX5ty*PI4ZGO``OYwnViW*GVS6dg zN!yCqU0LJs$6m^ZM`Iau(vALnjWV1#Sh+*#`}MNYvdmhWL4%Xuqj==%N27%OcBv^~ z60gZ9U_iOk?Wr}aU3IB&@hDQgrOI%UIuVEJs8TU4f_GaBX6T_}^OLg8$eCoKt9hMD zPpNQ;@v1rVwx8Tsd%ZH14ZdkxdfuZ+BUCt=^4F5JHPyt^>*)z|Q@k=gHCphZz9%5S zvQ|1f<}WGe6uPjJqQ66QzpnzD0Ec5z1Ri>42IpFcxwZEMgQ!1~0KNjFk7gK$>)N`wrSy(uN!6h)O0z;C2)2OIPCDpJe?1ew_io>3vK^5p4HYNt8s=&V zBgigUOlm(cl37AvdQudGFwGjsd4v{nI5glpdZ@nR0zRcHI6^XA-s0ZvrDvg!{fLW=QN>o0KYI3yq~K_D%Jl5q1~0Nc+GARccUn!>h<_d>$a^TfZB z=xeqGnFVvoXB(`ud*nBS2|_*rGvH9d+#w7|K#6AJq1k~11kUp~d5RcgWqN8(?T8ol z`PY!qYKD)I5f(|!&ra0Sx&GKEI4cDgSQG1al<0u#XxA6-7BorCakk~n9f_t~^f4Am zyR-o{XwAHex2!Q90w^&00u+K15vYi)Y%3~?A8jA4qk-a^11-o;PxJhdWF^T?N(|?h zMXjztE;szZ(U7p@a0)deC}(4IKTrlCU5JC!{3voMT)lc0;iJb^<5SWZF9x)N__u!& zIq4Lx^y9-IfQRFU_*7#``jHoM!(O#ZRufnyIuElXPiKFA9^MywOV4Oc(0$xWs{JYSmiDU{&}nd{JgQi?5Aj7)S8?6o5NU#Ow8uPX-%oM zNWkMnN{g~IH|){dtjEV4?z%^3%kWw{+6^2Uv)s=eZg>c(-404666|=w@cld$u!1&-5IkfAT2V;(002lW6Cc})Zik+2l7yZspflwG+<%&_a$K{?vt3^1 zt_gRL-b`{HYE-nC50Ewe>w&<)hfK2?Y7|=#tVp*uhd$pC7kqTDKLse#)!YhY3*#E& zMNn{Cs{ydSWu`8bu zkNH9r8KN*i*Di5H=j>cmwcpf2e`VuXE;fVz8z58>&!S}&uXt1G+kCy*pgbZk;9Y(9OojVP^?>xayDpS_LX5(6U-)$;9w?bO#;LHUT<9m$D6htgiR29|%1h9L z*oz+p(A@WdE%ZV-!@Am~O5|@kjIE) z0@Ku4*Tp?C?N3n1VXVv_qAj^$mmiQBdIek=5>nV z+#_gFiPzq?-$F@FotXI1Ml|)6&&**)W2KS5FVGl^tfYZK7{M+BvMUjD?{yK@%cN&U>J79(q z7z2tY1O-Vh#tXKOK>TVf*@8Ee0uC?ZqJzp|@@yZ4yOR)#_@tHsh7YF;BrmT2@hBqT z^9amHW`}Jc=v*vmLvF>?W&ocogZK4$I6v&|)pM$!eu`H`ao)sgP8QVa@+u?_)VW_vHyX?=W$nQ8s> zwM^k8Qm;+rJJ+So4fWWCCu^AtS(5TH_8 z3-Lmm!&H^=JI(bgb$Ec+iIdTM`@CE0xLd6z3+l8`9DjhqIdzmIT@3Gf^TT}C< zx9@Kb$~zY=a{kghYsadQbhw+q=>9I#h+9!5TD~}o>rk#_H*9I+yzRBKVA%Ni!kNQV z=CDBSo6E%e3LCsAvuH!oWYCtt!7S$5p?k7q&dcgQk;g= zdx?p4?-g2>mV!8aSfRUes5Q~h1np;1k}nBlV=ij3b2H@h(Bi&;A0*%jJ8(5z6PW&( zMs)7HR=!bg)>den36}|va06^OVJDMEw*p{Bwi`Rn%m%PUX2FXSZ4bBS87+xa@Xw1(uK#}uN5Vevsf;-%=z zwHP4Qhv82!37TP`X-N5+#e}hS@;L5n*4NsS^$J1rlkfp!xJ^P0w;uz7@Dq=UH^2ZR z4Mt=Z913ywv8bxK$i;Y77rM!;`qS{Pnu&jX0rsWthBU3S?8QUfcH)sm@jmwEg)S1e z=qCc@TM{G^4yZJpWW>C z$gBx#QbkWF#r&9H)vLgLLar7c zE##V^sd2NOlip{$%OKuR~=(`i#8?f5TK1M#`lFY?C0 zkb)EE3#XK14^QN)y*@w>_TQtN*I|yzXGj3x9q&I!Ip)7mPVHYumK_#LB*JovGX*pF z!60T|CXT@<;&3b$w@6mS|1q*$B_th`*(s2jg9Ayz4v!m&KD>IF`jz{7zabopUEW|; zU-mOQhSgQSz(|u+&8>;|$(1NJ1;YwGQ9MD#sExN(YJIf*2hh>gxNLN@$Iwz9p8`bG zZG_*GhgX$96wR|}7YHwOm7X*NxZIRc=D}O1*PG1=9r|kRDTL8yqsb}D1pS_**)4zK z3ZKsjOBPml<`#=Tkb}B!4zI8lZE#0#XcO1)oz1UKU(j0GwsO(V)Xd0G5iRzSg4<|S z>os6t2Yl?RWnma8_JDO$os^8Ol`Yt;Jr{^8EK;d-v)oniSl{|0lWk?01 z3$l7?R5=>B8$7PAMBLLHNat47U=0nvgNwB*b}QUbcOFHw_+V6L4~6G{^V$_|DSU9u z&~2kHy)I+DXySBqJR~{2Uidzqf6ivvKAUG0e=f@-DbD>h0Z!bMFo-*-BZgf}ULjy1 ziHG?KojL8QvqhYlwwpA+3%YyIEOtEa+u#6ytD78H{tK4r5%PcxE zm zQ?mHBHok7Xq;*i*Rk|3>k`EGgBS!k8K?vXEtzyG3+vGQSet87bD95hXg0q={=`?`` zAS|s}hi^X{GNX!osp%&z<&A{=*x)W#*t~EeIUTH5?g@jMSj|_QfLy?B4SO%IZ4w>hM5#A*mm^PoFc)aDlEo425rTd9a)bNIqSC0 zF&H?;uoCu#REn=(M#joYL6Y!wPiOPv9N#$|EkFD-`V-b_ zJ0>1*5}YV?;r}*_Wo`$9U*Gofty5>&&^Hlk0@G*dhPu+Lwb)RTu0_Xc89>nDXXlr= z9oe4{%=!i$r%XYziwukOuCkc$X@@si4r~o?r-il>A9*-qMe)+iWm&n8glbO5qI`Mn zq_OGQXHogJYi-OumvXJwSS)0h7y$~rl_h>8e@>vuKqDns&ou42)uQ~=OzpBPT4vm3 zK?U3FsjWs(QpW((1#@9iOKs+ z-lLa{S~@Y6B<xJyVW*O$9}p6>6m z@nf`Zjb4N8HJvuF8g|{_i@SV@S^Wr%{0xXy>|r$-FmJejJsX!aD133&;JmVBJzrRK zTU$^n`WPa0jhUvBUuPxzu4X@ug|OBsvb03CV>0H`axnQ)!lP~$i_#`=eV4BeQ-9fZ zk>`p3(aiIR{?Fbe!Ea|bQ@}{-4KRlBpG7HH{=7>jlu!S}OW%2QF9}S$F~w1w7|qd( z4B^4%$V%p+bV)hT%GN2?8a2TG7Ns}_zDv~NKzd`~o`~gx58u7G=F*#k+v@HG?v2(b ztS%=Szs>Gh$+0XuyzHV1)>_?#u|ia@iA-f0MPd=Bi9S%DeQ(w*AGV(Mx zOE@tOAKz4#xcMwO=lRu3f3>LER?_AS+|%iC(PCe55-dJ2GKjvvro2BlRyZKCxY7EC z#mp9)xAW=xve{@^cC)S66&RoBjjVk)kvLi5Z!A+xY%wj%x589vg~~Od&3X3JLRId! zK-4t*HePZZ-_;%`aHLiLq2+vfA&>Rr;YHFcG0>R~Yi-Jvt}J!g*oI|_`{vYa+$1=% z;_g#t4ZH(30q03^zR^xjg(u%^zHF$r!G$R+Gjhig` zMq3*5+>pJwn?A^_MjxDhg>XmS|0s+kM2}bdS4iOsf=xps`jkB-9F-5sm-db;kUXc< z@g^JA6Kb2&SgbGo0IBE*-4p8Wr&R1Vp1TRNQg{Q^&2*CaH@So{_hApx0;8S&NJ!eYo?NNS zK)a(1P|h!VAMhQ#AlY`L72{!t`s>ONLcP@qU`U@Cu;u_p{70ZjVxMLS17`+f(~ZOa=z0kbPJGntQ`$(gSz)GdZ5Wz>e8i}_mf;9_xTY|(Ak7Z(mVwR@4eD$ z5`S4%T{p+uGD`oDSVg0g0^Ks9P$l~sz^J<(XN=(%K%!k|zl&H*JKYo+2e$G4(+;LG z9fNh12W&SB{Y2N_T`MuLX7lny_5na%jbaPx+V27}6y9z~K^#Cd=81qOClGR2$n=B{ z2>xI=TFp<{MNeTsd9h9?Ak51}Cqm8F=z@%ODF5ysoGa`85du8_X zc#T%T*~-jYo7<&eI!gtB07LgX-W1d=XM|EM@)*g&-TEgBq8IIq58{G}IyFQ$d(CUpqsfjGq0Z**zW9~|v^%Cuj2VoFAwW22w%Wxh#5(QL zXzxwE%-QNJYwx&~1&mmx>oKz|2D05jo%2OnYlb$q zFT1O{ZK_@veI2Uzj=|+xzbiwFh*#)5XEZB%sj(NrJ{meXl1^f2pGMCSx9x}61fZNIK4L#}RH zL)phR{Hrr=PIcqA3bSSKSLVSkPHoEKDyfUOUH9=Hq?T>xQSraH#ky;YR5vaZEtE0* z9NL}zO5heKyOor6 z_-{kzn`o4H?>W7c@6UV+MzJy|Qs=A!J~NnS8n=GS;5tI763K!rCn9285bA7twDMreN0e z#@osE5FvVvKc5(~bOt%uhbb|hRPr#}w_#2gLCs~%7U{*vXXTL$@r)5md>@NP>@C2Q zgpS??pIIyN71U~uTV{nKrA2Vh`wp_0W5Whx7|dgtd#P*^iL#NIbjyBU5)8&z=B8Wo zG2o`61w;Do`dtt2f=;i&m#e()V;QSfeb=j+UvI1Y^GA+3F+HvQ5Mx3BhzLwEcEWdj z>b|;(=v6YZr_*w);zjCaAoH>JsA9+R0+!;!sI(~Q#7aSB&dtTN8Xr-{`+MIF1-Tr* zd_$%$?B_e-0Jjm$DedysTq6+t8S?pPU^jZC8eZvs`#H;Yw{YDq_F0j(K8XL z%2c|Y&)qzhj&p64O){1C!psX&q=tD#Qg|v=I8v-XcQfd@H)Odaa`Z*1p6NkOLGk=j{B~w!S9NwM{&vLAZQiqbXM$s8HuMOm3jd0Drsy3=*JBN(`<1<58jNq0In(&`jTOt`v9nECc!}AIF`2fm z2n#~r5t7vDNL9XZEIJAsf~^)a%quw`vk>M}Ur6puE>j)-L*K|7lQp1X6~_r}o_>qo zP9da=+p6xNQ;GFEh0=1aHET$2%E=0ih?aPxD6#dU0#tK*NBscaZ>{P1#{3D0+Rbk) zzhQ<3aKnyl*YS|j3vp7`iCf{*U)3YK0*DjnD(i*8T(@MyL&$D~mCI(d!Y0R^Ak!eo z!fXk;1U63a#^)TF!NTGZ6q8o&%0;QFOh)iuewk3o={sQg;vhYk@bpRK@xZX_v!I|F z(B2*)B$~!n+Yknq$Ft+viMTyyNz%7}mAz|9dkH;cgPYkhtx`^OE`BbpDRD5(qFTsLve{PRi(b2eyzBX`bctI! zRGifRc;R}H#2f9&<2d@`>Ebr%8v_0l&dv!FJhA7<(dXP??nRRd%VF*n9BuCr#IsG1eqio#4EK}IBTY1~+ZV}_EjTiu( z^&RG#-e`oFT1eay%KJt}?)-o# z-U@mcdtkzsXu_3ax?8awe`X(sY&iRpTxzOV6qRwA4lN0-J}WadFVcMZg1bo2eVD8{ zN@wdu@SD@LR!!*qKH)ehBJ?VmKSe=?>GQmC z82np5-+Z_>P8zX}oKG9A^)6u!p1UcyN9=UjlxiuQ+|-Q(-WjWDS)*p~&ijkfMCt}X zyV(H?A6~XokObs}aQ0F=8=SR$_;ubI!Fx2-A9&zxXYDZ%k~N)*;spNuMWM$e1Y*EL zq)SKtxj#nNd6%J+%ItGF=YnRJ=le@CZT80#{6!wZ!FtiE6K>=hj9~We2<#$jBlzr! zi(HH?BqfqW{tPS;gTk5_uDcGW?QgpChDoT|M(JE$P|kabY^AbID62F3V0!B^F4H~d z#|W^r*1WdNijfsrRIO3Z zSDN2>tZTBMSBiVpx0zOQEq(_ZFywfp4@g}%H^Q3NI9@hj*yqBQ44qwk{yPUa=|MrKY?;-#$iZuNGvC+zZFWg%WyHrnXQ|Y zCCASX5knI}5uOoPNJ1Hn`GkTL96S*2xzm#E$l@#;$HMsYH8u*X{{mNgoXhOADytk_ zTnio)(+M&`pW}O(=+msTAQ8mp*h5Ybe>BmIYec&vi0<~r^>tccth?i$tL+T`7%f?D2X=U1k#J5d}&T{EyRC~o*FUJN<= z2nT+K*?yLtJM~H=`(CGLWn@-Wv20gdx5VH8mIX$D zRa2Y;Wh3=@cSV|A2fdIn%wDqyT{y!qyiM#E|6Z#^g1hE3o6rCPj*ul5K~$)a9?B;& zP#rpfEbTm9FL$m}tJ_}!Mk$=G-^;e~NG)|rY3@UkqUEUevv%*1f+_9@-PDbKW?>_Y zm!oiLu}Uvpf40;lJHmaYrLT3X?E4Y=GfivgKl-GPxh?ses`|53~j$<;R4A62aK zmXesZ<11XV% ztVr@O1ru~BxycI%zsO#Otoe`yH4AG_Ksk``)`fpbck37M(ZzOQ8-mO8DPzY{Vb5dV z;bwfXa~SYA@B()j7-kvv>LE_q<=IA>u*1KCK4?L31$X!f<+*~x%mr>a#klp+SE`cb zwsq$H7#JV9i8WyUVHMA#U^5u?&HsB_$fPCPdAnl}cq7C$?lM9MdFbToyoW{E1>4t) zRi7F4N&2U9NK@+wkpo+8*2Ok|J$_<8wiWbGnkPLneVG2KNpYO9P)o07TyMoaxntwh zW7WUTuAEEZ0-FnsoC>eQsjFB%DVi8c6ocC=f=~0l^ISx!ho7jQx~f>6rItxgrEn70 z1N^wB5#wzUlM^IJxOxjCQX+_6ju%-}tZ19NlkH#?Sg9_s$ulqIe%h7Zi)SXqr)Bat3dG%euHwV6Ml!h~3oacFx$Rhu_z38?R4JeXCa2#c(OX1YBS@mo zVYY)OF1|ZFT|aGY0Rj9H`a+*(*JR9YN!ns2 zDxr^*Ct1vV$CQAJDO?khls(}&ZrYQ4Va<%LvmEz`ONvwajnE>UgLTWHmV3kmwLdE= zu)n9i6+`WUxem9I>|K5tVL8tWm(@l^grsnF-D`f)RPR1_9Z~D$5#;6UpQmk*e|!fw z116^ffYd}HAh`caF@t|XO{Ay$3%ga;f!K}crK9`7Kf_bsCdZmDqc=9E@7DmcbkS80 zWCSG&eG_hX4hOtdZcHC1pT)%OkjZEii=nyEQ<}T7F&<8&xIdYzs0<~AR_#qLIBKsH z^`>Q@8#ypDZ_zNKq>7N6Da?eiTAE*|nRIcy+$ATU7nFxNv>jrb0Yv7dj}|l~S7oTn z$CoRLHefgFAU7*pN=gGn?Tk?Uy#Z%!b(-t&6=_M4>}Tl3XNToVsrBsBtjrdM2t=yF zq>o#b<4xGJYYH>!y&S%??KbTs58s4fLu1D?%}hCK15t;$e$3nmD`9-D)YDTN3&V^d z*rlbTaZtyUY>Hp2ut@f&XSQhS4tJi_{?z$0c`uJ)lsf~7GLNMVXZ&2ddJ^hn^0LTQ z7{4L0U>;C`(OwF(cfMF%xYPqfe2IS3Z{F3_GR&e={Gh6_3$mmVTvp9_q&^m>*3SF~&Lxye?V*d42E`r0s zpV<180lj1Q>l4x89YLI&1z-6ODD}kT>^U{2aia7bwg!6O>=goJeQXH$(GC)L0a~%@ zU~u2*-IDy8AlKj?QKGi{jKfRL{9k+tJB^3uLwV#1k>~C+Ns8=PXV1y*_9NFhpsmCm zE9HWly(Q?!1t)s~7hj9;)$j5*5c!0Jm5%EmJGE~2#_9{GZXSa6AiqkObrfY9_RSm$ zT7iXqszFe2%LS?I?_CmmY7Tu6*|AylwNsnTsB}~6vAbYI{SiqIkGm{b8V-}jp7Vnu zEncHudw8n`eGTPI-=lX7yH=XPW8cka5XLSfdoKne9*|m1FJMYS zDP))`Lt)e!HTyv-d^-G`&apPy*gGND&7T{^%`S+6cYx9klr`c#c;{=*N?SA{Or^5H z>Ya~o^jxZnt;)2?7qH^>p{&*&MWxrx@eb9*-BB;_{N$x3H-Qobq?Pi{^a+NNhom3Qm-FU+AxMdMD@!2OQqgY33;e?Y7B~_y>A1{nJ%LjMcUU#qfq;LXAU{sm>eVd6wX*vx+j4e01~n3*S>5 z&-i&X%4oo|3|^Wpg?{rXRLA~k4top?!yl69C^{ch4lh_@EfAt@lt=+`JcI^$yt%XA z{^{I+2KdvCEFe%BBK{YE>aQ{!6RK7=>nw=xJbweKII*xIe~!RA7C5nWU6}{1#3R*` zO_ABE`aHgmrF(~GNoOF5+HP(DGmTJ3Dy)rjbK~Y`?De=xds|)8-}3^278-e@>v@(m zhxhHknS1LeN=qnei!A!qI-#51PS3ztLVJ7{)l^-%b3S#=>@E4~Pi-J|kxk|=>LM+W zx_D*1IL4uHMM0Y>V2t1E{3kPzs$c@l3_M?Byg4^gIUqGZ6`@69MTp56eFD;`fti7& zHiHT$`CHu~(=A7FEhofpBqCY~Wys-0Ei0=@vB?^)yQ{@9{+?>6%JGXeS5P5IZpvyR zoBk1M==#MGs#{uOgtrx|?^NwZE;Nz%>F2EtV*(o3l`B(^`Y4RilS@(=V0YtYX_;rr zbxXvC!)udPfZq}y9PXbpYOeL@x2g|Ln_Wf@PWRx=OxAjyj1_Pfw|}17UZG6!%RA0R z2%2=JX|9jhTl+{T;5>^<$bDiI+K9=c%LS33{{@p(h(vZ)zjVJMP7?@n)~rvIHEivt zmE3Ci;#{b`XT@D-OoYFEHj89(eo?%#PZ;VM5pq}@B9cIeR}b+??W|31;ti!ZdsxbM zdfqh8cBp3<^=QHDM~&s<94S0{$m4=x?)#6yiwJa~$p^`?cb*M(cgm0$?e{6^?npp{)=<5oZPjCu4y(+E@9kJU02 zNhc+ko0(($-30JoRP(q(Oh+5`F-pa9N(cDVi|Tn4iFX{Qi7`o`Sn6=9v)XW@i?dii zrZzwCuD*2;H@g*s@mAMqR;-u#o?kH*5*)yy34;Jx7VOtB(%-iWLDG=WW)gl2Ppr%< zd%g6_c9>Ikkrp+h0a72R* zm8uqX5ee;gKN{^)A=f5>+lua*b%8p*RV%WomEFE{@pTjsM(xMEcJy!$A4TII^C}~U~S=;MVDQar^YPy+8u|mKU$C1@V&yF21#S#4{#le&GC&j@s&kEPawRKpt4n5x! z+u4luSQrHax2~zSW(~o(wVyFW^$TYD@Od6cxG%StJ;(on2dzBU_y$Bb0p$NYQu=pr zJGV|3LwW0->jKmFk=5(c4LMPSoG>qJh+H}bL&O*^@%$FCQJ_@tH;&-_6rlEI)^e_( z{FQ!J?_i?y0SA5=xN1@dRb}Pbb6%+li}(IoQbj!(?MZ#=+bPf;hla*cE#{~=d?k^t8H0!%lXRgiM z7ZpUy(7Q70m^KmP6U*)vkW-Q8Ci8UMY&uSfT>^v^H>saJwT@&(r$-%(jQS0m(kGWL zFICek(OoK>i@-8E1)Y=yZJXx_6*vS59lfM*L_5=n)*J^X-s4C}_QC^l!9_w^2?c-o zLr@SZQio&<1-K`>v);qk$zVymkC2<`U7}?w#X3oERVonXCS0UEiSlfs)6NWR7C$2vy5YLE ztJBAnR-d~kC=_uh6xkiyZoxSRcEuY_z&LF9@kYQTglU^_^;oLrqy~gNXL*EAgS2yh zKh%rYIoc?;g}|HV&|tl~&4f=es!lq|OEB*EW>=n^VA2@6b(KE8m`%6pleL<~?^5VBJ*P;KbHW9B#anEU*tOs$Tzm~Ti_)ds+SG|p<6uMq6qK5KwQt@PqG4M&#q0^2FZSa%Ps2eK%>Btz{PyOK!k2XQ{e{r zlXcM!>@Tn#_T*AgrJ|Um%XDo~2>z0GQu8XzlP|hcQYnKE?Z@$VG55dueCXtz;arM# z_u1<`Wa+V>vB0b$;bCz|+Z=X0Bz=HLWZ{l@fNLn18PGT=H&TnOE8%v1v)BY$O>h5V z5%2uHP2BEdmY7S#Zd$U@W%1>B%XFz-+R6Qg?YsQ>U^rVim|vbQQVxMGD8#@Q)7)&f zG99iJnJaU64J@etu&MuW|Yt$0SOtUleSKs$>*SnR@cfF_{{oSCV;`-UM5$c1o{5Xb{JZ zb!-wdE}HAXOS(upUd;RO2LE$@_2O*Y?+&2r`llY>w}0ls{vTEO{^r%v|A%GTDu0A9 zIU)7-3g3c6?X5wslkDXkt*6iW|Je0mh$y zwpMa$#mC7wIBG6NuJNO#i}=kTxXK|2zL$&a`bjikXI71a*hC_5a6w^-VBf{3NpTz* zq|9pJY~%i)PBQ#|FvDN|{6DK!_dmo8)c>Ws;jf7F|9UI@m4E#oW(Bi69YRzfDhvZX z%Kz(Lkk)f>G_rU2dlkdKrsDo>*S`Y8z@oHyKOG{(wpXqTU41kuXJ~xF$ZCKXy7V)$Y1FqMP?ZCg_Y`J-G;Gl z+UvVN&b_!T!_Gw4poc!k$gv1=FrW)8^!Mh7qg#rnyC_Xcq@=5OibMRoKII=et|stM zgTqtnCrjBh443Ttxfn-fJN)8zZi<3sxqOR@bZwu~%{&jYpv9Bq^fX^U>+xi}W=uZbkwJM+%oW6!+|nCR=2aZePmpo^XCDxC6?U z_9j6^ni>6a&V1D^%9*bRD|YIwV5nR8HwpYJT|HFU)I_WA0BL*Tk7u)L-RV zz;Jv|sc>o!iH0{I@K$N;xkm|*gjeje5H7bRf|+dw9xBwa?CBWTB?Cdjs`LWCl^e@( zG^Q;G+F>iLX6|M5qHYIkf=<&?y^k`5As8_-2ns=WXp~^@Jj4IIdY`Z~5kaw2qWH)+ z`eZL+Ow}@17%+lcEyO&{8M&r1R+@*Dyi=uXzR~z9ZwUVhRk4UcS-o4c$r#DRF1DC8 zs01!)zwan6g@JpIBZkPR8CnvM(9`Q&YJL~|hHvw#qd()~b11ffPwMP%hoh#jj;{ej zSc^Jju-q3v02}YRGi30xzo7$4g&HUG(k%q?31^uT2NJv;dUyM=e|t4cN3vL_+ywRa zQ!K%^BG6nn66A|+VGyUsZ^_h^V-plBX{fR>N1j?ci$sUA=p}I*dVb=KJNmy3L-$~i zZ=OK>rpL*J4&z}^+xJoM&3ICyhGll1{7{FlWLzY0zTOUZ1&%C`=W-JAZ+Cj~jDkB> zXX}zSq`0^VvoGsRe#HwKJKj7k$H~?mB$yv-dhg2HDND-#5{gv~OX8J@wp{a-BB2g5 z{R57~Ook-6oS-y}OY%C;wE!Ef*{YoUTD}2gtI=vh);eXqy)whkJ08n9tYJ6Hp9GEf{a_%%gD&iBg}LXOKk4+tU2HL2wFDXIx?r8>n0RP8La zuqK_l^;`*?1Ek@0juqYX{yX3`XM%>Jc#iF1u15#PNhR1_UWozo;YTKpS&c06FkU-S z!=%EQsj;D@z<03Zq*>Oe1 zAv#x1eGu-Y5rL(1qdArFyKAT3#dy)3Y%;6@YH(E(yG5^gq8z7w33BYJK|W*F+=u2MM=Tp^ezYBmjv^jqAQ z8(H8Cjx^9`L=kjt`gDwKzN4(1w+2VX%PeIz-B;U`u7m8J1jiH@UC7&}3zG`sj&Po1VxOk>yDJ178)G)pbsIQa zY%Zh=q{&}qsk<%S)E7IUzEe~3LWN{DVNOYk9qKAdeu;vjoNcHmQF(CJB}G*ns&pFS zD$a3|9xwCLSg$2E<$~6{gCc625^fVaVxTpw2MKWGck3T0(w7DXowViv zC3=pi-G@8ob@BbeA^5t6vn01Z@F%Qh;pd+gys-ASSl4X>tv`OAC)%&yp+=C8Y~k%8AHv&LL$!vF zSeR`AJAJVab%1K!`oVRpaV{Iv%{@Xog}c{_W-oadd_acPJ zY|TFS8(HHE{Swp|TN!gbU(hQJ9Lqgi*}9BYx#!%F$la`1ykCz5-+UeFrZ<0pS_Q;= z-Jg}!(y}g*gPpJc$m@YgxCgdYYfBlt8D`$!BH!_QM)^FP!E!Dr#qm!vxtDoKk2hrM zw=~Oq6p!%D*SGA}yMwadfTZsKKL(cnI&S|r)%O2ad9wdu;r;(rR`?HBUie>Sg}*8Z z{10<7{`;=hUo(yWM_Az#FqZX42mv7RcmsqKz!q==5-tITHU{7AZEPKA9ZiADPg(1k zS<;x>8ku~od~5>{B}BwU03bm3T^INR`1l3z6?Qc<1^}d`0aU>6LIc1-umC{n=3mP- z@PJ_dOIrkl0sxFN0`~)pS_GQ`fXS)9`p5u3|A;Pd1D^KppI|v4fA0<4mjn8b-XMS+ zuz$1xF~Ht{k8=RqcT*cj8wXPxTLMNJIslu1xHR~mvjgR?HtAnY#0A`l)o4H+_^~fc z%GslU_eT!^2^tUsybcHv02B!X3<>080Jt6n00aW$FZt_hz&{|MVBiptP|z@+V1XSP z5dokeU|^u&U=R?0E(78PJPrUyf5Z<_rZy^w%R777#IX31W+KMA^!;UC+NTA)87T|FM&7|Og&jAh& z4h8&0goS}c{BO(02Jp5-{8$6PfdOxDBrqfZFW~JpFHFD|CI=#~o4}gaYAdoN7)vRu9JTL<>ob=PQV>EQv{@ zoj1{2^#{Nd0rrGgVliG+GMkEBO?frl1~^?WmWF|Ix6??{*RGLt9EQJ}5=%DJ_P0u0 zoObXjH|i6F#f$Z;Wzq>ohM9(8tm4k>Gcw2rAgS^BMVrT?-!-GPSO43k_L9rm)`5%B zY9o#cYh9P0Fp7$=yPh!0cMxHeQEYw2yET~C5|F#pQcaznCH+AnyKj9YU3z*UA*5lt zI8A&#Tuukw)VDh8P`k}x%*RZTq>iqJ@l&p`z-gjL{(W?w^GR$y`tADYJUhU@|2d!C$cH{p~E8jX<}Z5_i=noE|uT$eQQ&NJ4fe9 zr_!B>#2UmR6_HroR1-h_WeJuc4f@zBSa5fYcXl!xTsUl7_puh0FpAPIqDzkQ!k!#f zK2hZP{?vz z`~f#=vnccDTK&SP2M6p!Wp30Agw<W$y97>mrP%klk9zhbF*+yr>5KlQ?cy>!JjxFH+vi;8I3CNZ zO-ptf@nlkQ-J7vb2IpD*E4m&^Z4X;$2VaG?b=>wV;$fH4co3T2LV563hzF}r)FZDi z-hSrU!RgtuS(2p7oMpsD@lB)Nn)F2ncA-Bq=5HKrqSv=y(-yx7)s54&8e8Rx(4lOD z3?X;ML|j?Wt%_T^99JN^gmy1x7d!IGa7XGz>`-fy!DqTh7SBCn{km>LF`ATtVNhH+ z!)(F~uz6c$hznCGL+2ViNPI)8~A^8?VOWAh#;8D9^Z z#-!|WpIPTK+~eW%d=vnD$U*mg)LT)mrrNh-4Rw?VqUYj&Xhbqe3Zp1SP~?RH{;_?` z{sA`%z@M!;l-G=Q&0>fblFQ2AS2Ti|GkVj5GK}-IBPk7TVJtK@t_afa1{8Gd676{ICgPi4M1Xlj4!9mS|M@DAcL=`Zk&mK);8l95)wl z{>}U3@CN|54C?;yvFHQv`yu~0&ZGt%srsFrg8VmGm(f-9K&EJK)0g9WwEJdQt|F`X zQWH_r*P<0cI3rlfdxP@`ALPfb(}79gBF$l&>#`zI&<5<#I2Sf4uLJQX-ia^NY=R)? zINoKIo0r!-Fo!8w8ZRc^OS~$)M$=L61droBJJ?zd;x+Rk@u>AfOBMtsj7)ybskrr# zKY-OAybXPNc~&*lepk&QJab3OJ{(lfyT2+#(mB>VB}y)7{0)H&jdBt=S9GK#@*=b**SOT z<&nlJqad~>oN9alhm6YSQK;KY5}NufTVN?Kj5}ssl(dSE@+NNxQ+9&o0%OmBxX=OF z^Yi2C*M_=Bay1uwNk;V%aq?1UcGLUZbS=h z%17p3K8UVi=a+Zb>n?7^@z|LGRXG^|4giNr-E!FbR2xh6uS@?f)LEx>cE^_&4HGrp zR0(lxOwC^j*K7pKjVd=y*7$eb7Ahe6|sS-U)tZ zCBvzpy9<}g;y74euO|2ft?>_Yhg1ty!@m`Z;e)(d%7VRc$R1GHx5;IZC6HBR#QC^? z#bm`|jN2f8!iSiCaq*OBlclnOH=SaPz5Z#4H-OJd@Dirsv);c`K!3)87f*rXKW;0g zV^4aBSNyv=!ttB44T1%W`gu5EYfYkYw9nU0c1+B>svE_}=CTu(>wbvL%y3D|d;T17 z@J1h6dLnuPJ%djOVDdy@hdA@8#n{Q$ri2Z2RDL94rb0E$L+W`(^?c37l? zdZ)tr`9+Smu6UXQ?e5Bl@+-ftZIvhitxCD>8AIuF2@(WEgkWa-u-6TiCZ5{-o87_g zrq1F3PnC*CeSFxBF~{)HbGU39fjSHl%65ys0kDW&7re%Bs8AfEfibXxssV7@PD{M#6VO=4y3k08Ow*Wkl% zv01`CeDd2Btuw{Ex_fIX57{;fH<-WSDHRjE3gc|@J^;k$Hjl0}*&hG_rmaUhop*X2 zqgTSmX!OBo5LMR$Y{+@Ery?D~0_4PNBh&Xsr77pbLT|WJEZ@aO$x3Ofyn1vRO+nDJ zNRYGq=Evo;;xnfY00~(C2jFh$;R7I1P@N1cdeEy|^Sv^-c$}Q1*(TphC-&^>z%FD! zpH!xzD$d5RE&uq0BCnf_XXOw0ea!s}@PU0*UuLrTd>n6~P0$%EQ`5YDAZxqBHgETd zl@SqA9N>|5{n8!o1HudgzWh!(Lt`k{z|_og!FM}K9=7jeVe`bgh3eA|goeWJ%X>l5 z+gqP4u#^giUrCnk(XYR9UHt@l=U9Bx3<2L+7Lo|$7jaXm)C-aozyL{QDNYgx|z3o z486L23_Vz-EsGs)4tqJZFSG)K6BQq~o~mh?Yob6!&{D#Htt6!aMbXi&Pn8oD;~`(j zvFQ`1KCKErW%vJ1IAN63G9;GQkf-vZ&f5X~-Of7)EIGs|LFVcL0>>xQFGD*`i;*eg zW5w{i{nN6@DwdviH?k6RP@sBG%j-i^MsV zpRo<9#YfnFs)?EbUmblm9MNKGp1nXY%X?FvNa#`>ly{lzSoV}+bi`tc_F#?X?%ak; zxP>b->8QpYsS*|i7b2Cm$$6iK^}wc9WY#k=5u0qCxxLutqf}nMBy-x$H@YFW^@lNpvRQy|S^46vpn~ zqQfKzvC7%UPLYbIO|@HEu$4#Z@QoQSp~jQ<`(NFnRIKlg8k)+gyGAjasOy!@dp3tX zM4Q+AW|xGm+&W_tm??NW&6@0l@gz~hHF2O(BZNs(D06KOVh)+HLw3XVInBZEATwGK z2=%l0VXd-S6v#L1{@FZHs}DU?zpam!gzCQML_|9U_6yeCbcjEaaFi*bWk?SGG>M6B z0V7YYo(l55P z$EuJ^Na;v;)YXj5IiN$Thv<>df!D`^X@sReurjyZOvbo6VXuRI7yzfnQVQ9xB0O=| zM!XEGgJSJQ&Quvv9Y_I z)Z7G-Em;z^;O`W?k30qH2VK-NGMtg->H@h}F~h>ayq)Y({=OYisW0+hi{r1)key;=fEDpJx-ZA*r1){e%6P&QoRe^=B^(g zTzrcV=B;4xef>8m(DoOnQ5*SDc#2|doe+C`_d=UGl_iaT=;a$UBM^20L-ia{GPa_U zlI&T0w#CYFB363erl6Y^TBG@iUr0d&@_4jT(8VMiDZY|>&)p$=qR-vOJ!a;PQVukc zMBq^{(h~u!OJLcx@QVFh0=Ic)j(yp?WDMk)=3VM;Bnq_kvd@8WERf{!W2_stQr)#cW?2R-$U1((dCmhxfmW_>`p@FJR(^G+@{+qe zZX|2Xr4TfNZDCB0?oGg|QA2s=4Fcm9f5n2#+A|`f8ucAD7u>(==Dn$SW4zU|;kdqw zP|KF$)h_<3B1yLJ-XGO}K9+UY+(kfagY*fVZ7*dX9a;=D(QRNs_|0NiA_5-Vpn0}dU z84TY<^kNQ#6_(KaqZtCS3CzeQv%?5bAU_QK6EHBm*AXI@vo?edi4u2g@Owj^mh75E z(xX84HNOYgfnSI6Kc|)StTxrm0O7~B3jXkg1CIlzlNOuz z&J@-`g4(UgHNJ0ezSG}u50-r!sCsJhP}vW&2bnOl#0JW-~W&n%UbN2X@~4o?^So9sugVos|k3^IuNg5 zf?k2UDsN|OP)&E!BAwnN2O_ZpU(;c~YNtHV=-^oIkc5s5FCm-*TI%~7HJdzSic+fw zY1fkX5rXoB3@k+}4m=x`UT=3YTJ%S6wa%7?=2m$)&SvyUc5$TKKa`A77JJEE)$!gw z@ttNktnrGQ88V0qFgiU95utD8X@>$u3eMS{qEEq=V{@DJ2|T%ynzS*r_wU6N-IBe| zjiceNhDRiXx%{@>RG7uvy`F5%E|d`**id^|YevCKT28W(-1z=mqiR;Ieb<7^*fH$(*xGT6>AGG=eGZwLu`RR?)*fmaOrQ7Opo7Z%gZ0;T921911G$8xJ@+gq^0SYvC zaKb6+y6{duw7%adZ&2{*g`%$kQVaTZ_B*~pP7-~fo;&u>9Meu z-Ofwi1sW?kHQ++03~Z{cl$f&H>k-vgzga=8 zxdwkcmnQdK?#-zn{l@yRx>xV3tnCg5Uyk9zXNcb9gE$07h97J>g&lbbSOT<@J%fy3 z%T~Qg9B<8Kq0X7lm!dogT$>ZonW3++PS0?}%C~SCds<1RQY4>GERoMi&u!&mMYzea z7L1I9Hr^ogc9=tY55rFIoEOdDRjt^sP6mZ76F=)jvO4D8Nq&{9Gtx`gnF1p*8#)n9 zP*=TeXFAX#za^=sIEh;QLaa4Icd?$kN-|efE=%S-w2`v1@ec%8Uqz#htf&m%#qi8WIR0_-^@fH|Z{ehjD5yfYt6ii^*MA+lk%v$aLE*svA+oR9|d9n+OTIr=Wb#QgrpNmh-^zD6UNyQYOG-&+H`8o@5@mr2r!G7XU&gW zz+%9dKe|<`U^xqe+Yb+q8s8$dladu*2|e{uXP8y+$N?VQpzyLCL9FLRrcK z=4v%iYN%r@P3F8?6k)TScQfD;ox9ERPsq<_+_g!c;5)pwPW0Ax z$-A9q=gbk^v8KLAI?Z^lpvNyBq6tr5M*x9(KRkHhl;*eEtqmZcMVXtP0|bV`d}CeJ z#^lr2FgNDiMgnfMD8~q#0evG&ErLI+0c}`e2)5M(MnJ3aB164i_@?-M++cGK>9J~Y zx4A7l^eLyRH*TZWa4N`E(=}FQwL-Nz>Q=dF`J<=H)TBmO6d*g9Ww&g3*vS3aCe5pI zV>KbZ&PIVY+%FtaAYsd8CWuY0ulo71F0pq5M%k=ZNsq^VQ&+zTdJP?!o#42l9q zY?Z}U^k^7l)++}KL|VL8+5j6fs)JEf$@0HKaGQP7nXI=uQW#c|3e2Y=dZ8*U*ctl0 zcwZ5!7E-6ts5GkiI=GU1g99G#Y}BhbiAk#Y3Sn^e!IN@ZbxWaeM?L$XbG7|W1eMI3 z9R|aFo2gqyTX;si0>U)9l9F=4?x`m?hjXu}IDT4>*v)N$jG#cX{7vkE6(~^fzL&hb zU#I$8`as4=!&|Qci8gUb@D`LLEZc2SAiz4$9VGUi#j6D_hLY`$EwouYW@0NLAaymW zqp@5igxi9B!9V=rr=h=0A_2F{kQwBeG-AEQ*+t}}fq?>bK0lMkezU{CBoS?1^EkDJ ztl%qr6|S4^5Iu45eA9Ca=n$xO%#hwE{AXunz^FoY%XGHM(H-JyDEFo~hA1`q2^=h}Jo4TOz<_!mep_Q?6s@zet z!-$f^F`T?!FV>bc=WEUr%5BYUGvYV{2Ld$q`oO}r(G76}1^QrGhyp=@FXj$3VB@(n zqy^(L6*l<+sdr%Z*b>22!0dKlcy*uK=BDo>(X6bP8rsf>8O;OrCJr`F#$gaRc1PGL zE%F-|V+{SZt~YcR1L70{7zQWPn#)J)Ytyzw2m!c!DqOmmvy~vt{D|#V^Nk2ltgCY) z0bxf&l|GWHafa|mw}Qa-L4m^hQ_d=(L>9yGhk<2g5RfAwaiS7LBQ`t2S0`r)p2ae>%Rr-lB!68V5q_WcBhvFVS|v>x|u{T3TzkE6LoH$&w)J7_m&zey} zh9I`5mQ=%Z6Sw2uN2DuGH%BEbdG_@J)z2P(VXnvq#Dr)8k$-}D?G z#Hsc>5s0c@ z#vetUCb>M$8vaS8WEuq`zmaIUiUN&r zjV*!$6K_}~yVdty?qpU;Dw14fJO~^2G9RcbBi;t~EdVN>AGH zbM2|mh@;PC{lb=a)9lTJBXp(|)~|8NUXogg2tKp?Ov}`Nb8vs-X=h{qcmmVyfN@`Z z!XCY7yc|2N8moP3kOy*aTIHo);SQVJ#0WGNK7|6^a%}aDAInVwVs{vbt8q=*ybFsl z_U<$8sx%@C?rd}FoN^3!GW7vuXeBXf<pPR^PrXUl4y*?`L9DZGVZ!cy470j|PIALFQ}kX0>TzBgVFin2nu}%|r?&Q@mD7zF zN;j?zmyf08?=>&8?)LV8c&=hRmb9F|Rl|l3WR{I+4L#s(jLk(rT z*8U3Rk}JijDo?PBtgql$f*2zmu)e@f*kM)5d`2>*{kWG@kyM5(<;J3b1{YK%R4%F+ z_TUYvfyMwrc_EIgaSh2oIFm6g*Cp*cP^yCvytRi4vdqzFJ;$gdF>UGU5&dbz4f3alj1pHDl z$D+{9x=5=?D5iqX()KV~i~@K#rWOtrYts>dno?|O$35Zh-(p+HFWb&|6Qz9?KI!gZ z6G7<{nZ&HuAw~Kwyvh!T?6mQRDH3*W5qk0`wV*(4+ps}dSnm`f2KE*K1ie!}V)SXV z&bJS%v>>cRXmXmP1a-MsAvw`*Ms zH0F!|a8T{WCZwjbKQQf{I;QOO*-FU-)<-}ThZ1~Jd~(H$RC57)>=mhs7#KD5v!}Sk zyg<}Q-(=RQI&x6g?P;F^9ahZMd91hcogiYiN^cW7No)PJZHHAxxavhWl@aB}#PA4E zZ98-JD3I>d@pS~*_A&hFm~-m&uA}zPa%5(l?FE=@g6N__5ih#D;7bulo>UQrJ9$}a zlBMKS+Gp;$g?bTt?uRCtgiI_p!?iZCyJjoV;9|Xvjfep!6bQ5EaOFASo-_T?p+)@b3xP&*OsscNVeyrQ-7`6G>O zxaDIn*{XI`Rq74g5>%%M{j zS?r~5vBy+H_2;4W(xiJ&7wj@B zO&h}yHnm*lIZ1H;a!tf33{G0x)p=$P_zz?{tRHwB@kh#452f*~2NjjxU){J?6n5#| zPmRr}M()A}54)j;8lgZ~qY%3WRW*fDQ8_rD}1jAfMshosF@k>^OX%4L;f5)Y}n| zHHPtEN_5P;)bN_TDvh4I*%VNaA~nY{?JW}G27M3QoXOP3hHOu_80Ooy9$m;j7@ACe zJ%=SBx<_)rd#KCW#KXby`T958w=F}vweg#7f<2mQ#M9FomPo!`udq(m4ckXCY-zP{ z=S!`^1v)-FZrx4UrGlK^sg+?>4ARz!x`419ws;WGrtsFb z6D*|qj16SyoCnq)g^aMsFUWi!sh_iN)jlcPLE^J|@WX))TsjDDT8-pA^9*a+n;|5Q zu^@lwd56ueqm1E}tY^j@w2pOHj1Ne&>{1r+R04AjUASZ|O|BQCul}Z9nyQQuaYZ3| z(NpfqDDQiwwa-k>5=BXSzA*GHn5vJ~Zm4ZBN4i{_iY%v}erFt|%D;ZLUDwf{7Sg3E z>RO~Z%o*2 zglswA}^)CS7N^ ziYU;_QhkI=s{f%B9MhJ^^{Sh-PgBP#bkw<#8Ca9V))ZyT`k^)F$ewiq{nuqX7Y3f= z6u_Pl`#)UMCn@zuABOqAyuqkZWqEWAQVKNY6j0Q25HF43F>YHmxCPTYxj&|Lbh`B1NIOTJCBz|EX7(OVkoZPOyKiuw z=s7X2S9r`Crvu|zX!YhG=gtoGJfDv#}#Y&RN)k8)tkB_OYT-zxU- zQme&E>u$MLqqTBao&Iw8LqMc)#j6~6$n!kyjq6;7POVpLPrv7B=`?UZ5+v9A?0M_; zQ#L7AO7JByA#&MB0SC@T5}r%CFPH>pn?@N7F<$cD$06a+{jBa6bI&sqqEuQW{mnht zp>O4@j1@k!Xefm}R}{?=i@CF7IC*cP2n@b|+&Gm4dA^UB$?YsJU2st*UR-!`w%-kb zc&;GM#=~-HVV140g^>&F`qhZJUX&_3zV?)e`|*DJ>#bVk+EVK^x44uPVMKV-V}aqF zK*y-&Jh4s;ctGv@CPQDDCm$1hFEPlpKmLkNNx8g+`yc{yxG|l^)=`F>Z+hWa&u3e_ zX({sYi{XzWpiNFGk6@i)C&liV(gtCiji^0$T33=}qw$G)zc}nY$|jNg&v>`2zD{IJ zJCM#|$V-bDsRgq=b-m2FULh^@4u>zIhh}!4C_jEaoSO1mG|omi1)?DDDh7Ov!~Vu=>=ps_)x?Od7Bxv1GKr-_&w)rFc`i?K0~i`JtPhT_eo4pjB*fQ>bma zH1=f^vZX}u!N8jvxNq)bsQa-^dI!uP6&_eKvhkI6D0V3(lqXlI>^IT7TVocfoWj8w zhsOr$y(_~)lFTNB9n1~ZLwsm+M!PF=dfyFl zgR>Lc!ovZ?WHSehPKE(hHTT#ukc*(E256z|}LY>BE z6rke9PDII+3TlJgzE3~jBkc*sVM^N1Q!l++ySPJ3m60;Iy6QrL5#H633}`~%&%fRO zO^Um+MJIE(T3Np|=Q7i1>CK3QE_#*HaPu52{reKR>yI86vE9*@^SX(9R=9tK`XPZT z`CLsLm$#iyI4vtAZI^W`u_EJju*Ka`aD_98S&RP+urrjc#03^Dv$7q^*DE8dq6 ze8^D9S4y2KbRv~3g|gj#PHiqP&cXLyy{@x%{fy(-IG-okrS))&225xTIGS+ zRqaHkW8=gIXAecjQi5(LM|D$J`I{7F!If&I$zfb;`fhLZ2k+w#5j!&k$JCj3wy3iD z2CCn#jb-X;I>}X`UWTW{Dw-+-4QL>!CsgguWrzOk`-RKF0vr7X+|LHl^1rY;+pZn( z302@mO!+@^ewZf1-nJ=zOq2CQA^ojbSWZEw$V!*sO4$)BbQt#;RQXF0LcBfGBSy7z zYe-&iDpwO5THLoaNc%#?m{jx3Y4awcP>H;=^ysh zd*t(Bo|h#EnyL5fd9n7erHc-F?x}%=<>TX3SCc2*YqNi?yYzkGjaVSCWt|(?>V6d% z2Y<+4x>%UF{@hUiL+vs+vhyJz10Z=MgYa1pQS_F3@rjuVBSG%bGXvXq7SHBKVq3j- z4uYeKit4=IZwZHP1&g?k6vtcFr4*#kk1{Rb1s4@_DYolpW@HNWa9mcywZ@P3By0nx z$B~n0T9H~UC0?yFxG7h2rJhXI+W|AXF2dP>skEqu>}9HL+zXDdgK)Kz!yJTX7Y+nvio$p zCP~#_dU zk_}}_QI@bCZ@6`hCI;E;C7YM=#PV7?KpHvdLl-?pj?G)J>aMs<#`;?*9d0+;+pc$Q zxGvjfSS;;vbVN+DoedA-Z#d8RrslRAQ*&q3*1eqohl-@3w)+42W`PuAI?E8C-nf3N zH#-Lt$N$z_R-cOP_zfb^^pWf-;k7Lc4!+QAffyQUi$2=)FK1|NEjaOUx$#}kkPOgq74snp+qoo? zOlss;q|9A5Ul}%^VE?FkJ{_`hS%BQ$L7N`=DGQcu3Q^st2y;!fwo(;+l`E>N(cb>~ zy09zexU`@TrxFnv=VgiQ|t}hu0-b;LSlHHd7twYL4_`yRSk|+Hix_brp>z`sRw5 zc?$8|q{;DaBzreR?ql5Jbz>fM)HE7#1@B}kHddI2kCE7r{iMA`kwM-ST=$ZxG+}!s zcgIXp&9$%T<_UIKDUOa30oKyVWg>61mNyPeRVJ*<@lti*9{n-&FPCTnXQQwN0u$@` zzj)vHSX7tMxe#C>gRc9Wr^Wu!&|vxJGnPA0u>#0n7A_h$OT`Ru%pBm<%vI23wErH? zF|u_q`BAhRA2HCFKKwj70|5TtefdV}$v1M4`fXvN?zelz-F^@D&6&lnR53J=+_{@- zrjsWt=TK4sv(+wLRo>?EO4_sAO>Z9bw}@nee|Z1?Dqp+IC5>|J@xfqW&At6ddbG4` zles%xZ3LIfS(KEn3|?6fzEfpcpW+w(APQfin7csA3NuYSyS|Mh&^KhJ&z+8w_AIDG zwYrl`IniWk0Mw!VF}0e*TAFT@)smNGJj51DJe!Hfpzi4htA=I%*VBq{M*|sXxYS(E zE6BI|0)1fdFoR*|sJq~=WssEIFhZq9bv5hL9MbTDPcdR4_|dq-FZBhiIkns_+ZzSu zuX~3+=w%@!e$Sq3-%o!;MRx|G7Cmz2=bAS1e4FNXV(R(;`S28tP-n8%vV9kgvbVV= zja_6ww(326bvC?ezSMQ_U9|Aro$kjS?#<-_LL_R3L$)qFC^nu`d?fEKPlARp`MRzG zJCp)=B0K_z6S6`8AN-$9Z+#D3MhbYQBXHu2=y&flw6pvF&UxM#s~)M?j&Tq{lZsFV z3)K{3fi7)jUp^5Lnn>O%>b3E_IYd2PAfj=&=4vl@UE|T`grchL$;r5|&ZWKlwI%H+ z8u*8-x27re9CQX(SYt*X7AlHDDm!EX8f22QUs{y-hwJV2CXICHV2iR`#$Zr*W2zo( zBVobB@=21yVALS3sOH+$`?YB$chu2JOW-O#f%~2U)M{>_2J~&*!#Jkdv&Q5ojYnf2 zSJ;K;J^Y`)Tj0C$KJz+!84>htI9x=V9lf>;(P54|8sex;cIzXZ=jSW7K5TSaK@xAx z?@MYNGQwk;1HOrpv%P%5mTI1OZ$3;;f=T#=A#i0twlBF3yDk}=;&kBYMb zU6dcJ$tu@wPqb3H6Qf;+WQ;y^V^HYQ)~2G*(hErJ%?^BNbxB65(%9R}2As*>?OHOe z_cSwRj-w8{NE-b0;bDZXB>v3QxRUtp)t&st4gHBYx5XXbnK-XTHNxykYa#wmxu+Y> z=C_NU@2*x~FIO99Ni?3w=zQ>n$S~7Og;?h*80dtVDiSI#| z{oNrC3!E8*WhQ7Qy_SisUiNrQZkxq$=!9eUj7t=9(T>q{NYB+1B^6yKC|}$J$5Ns2 z|I?DtF0p~m112moNw6}YmB|6W|1pgH`O>*>YvW{MZOmj4rA4tGih zPM?8hfd&a&bokqA2bk6T(oXt$boF9FO2tH_Y61}G`Uen*?6+hf5Cb4|^dNA{nZ zJ3k{g7ZK%Ctgdr_C@D4bFWN86j|nJH z<%Uicwl+U1{4Zg1qe$uz!196r!}8Br|H#_U%KK@si>1TkdR(3kIL&Lo^GN^5&nux{ zh>JP7NRXVCvy=x)qX1;&<{tz-(cd!hA04lsS^4$NU%EoN0_6rTFrT z?$<@!#p?~vaq*h}jQc63b`f{+%C&Qxy3Rl2ehL6y#9h4E=NuHlq)f8lq(?w+{FyC@2O&P%rZUwD7nE#h5}VLs=lLt_vw*KE zo&N&&TW9pk0>9~$UM%_FqZ?fQ75ZHJ^PkW^Wc|)x?XLd@eXghZ@5#Mx|Au_7QF)Pk zQR(qK&x7v2k^dt`_nW0J>ONj1|8D3%t-<5*8~Hyj`y2V!kIp3z|2_Gh=f5GJ%Lra1 zUlijzw}zP4Z{+`2<7fB&n;kCV{@w&3uYSY*tqJ~qWc_<`l;6K0|NRW$@5yid{|))? zXDWYBP73%p7oQhi zBwUQPo)eDBe-ZwOy!@;^d=7n5I%JDn%D1m>|n2A>O2sEdF{=` wK_Ho$->mqbNYBOOe=fd%Prg|9Tk`+A6jzqV#QO0!%)o~&9w4mQ^y9n#14n(zH2?qr literal 0 HcmV?d00001 diff --git a/train_roberta.py b/train_roberta.py new file mode 100644 index 0000000..1be5ddc --- /dev/null +++ b/train_roberta.py @@ -0,0 +1,62 @@ +import time +import random +import numpy as np + +import torch +from transformers import Trainer, TrainingArguments +from torch.utils.data import random_split + +from .dataset_roberta import SpanDataset, variable_collate_fn +from .eval_utils import compute_metrics +from .model import CRFBert + +def set_seed(seed): + random.seed(seed) + np.random.seed(seed) + torch.manual_seed(seed) + if torch.cuda.is_available() > 0: + torch.cuda.manual_seed_all(seed) + +def train(): + all_dataset = SpanDataset('train') + train_size = int(0.99 * len(all_dataset)) + test_size = len(all_dataset) - train_size + train_dataset, eval_dataset = random_split(all_dataset, [train_size, test_size]) + + model = CRFBert.from_pretrained('roberta-base') + n_params = sum(p.numel() for p in model.parameters() if p.requires_grad) + print(f'Number of trainable parameter: {n_params}') + + training_args = TrainingArguments( + output_dir=f'./results_{int(time.time())}', # output directory + num_train_epochs=3, # total # of training epochs + per_device_train_batch_size=8, # batch size per device during training + per_device_eval_batch_size=8, # batch size for evaluation + warmup_steps=500, # number of warmup steps for learning rate scheduler + weight_decay=0.01, # strength of weight decay + logging_dir='./logs', # directory for storing logs + save_total_limit=1, + seed=42, + label_names=["padded_labels"] + ) + # Initialize our Trainer + trainer = Trainer( + model=model, + args=training_args, + train_dataset=train_dataset, + eval_dataset=eval_dataset, + compute_metrics=compute_metrics, + data_collator=variable_collate_fn, + ) + #trainer.train() + #trainer.save_model() + result = trainer.evaluate() + for d in data_iter: + print(d) + +if __name__ == '__main__': + set_seed(42) + train() + + + From c5cbbc1e75a6fb6613ff890a9953bfb670d9306f Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 4 Feb 2021 16:16:52 -0800 Subject: [PATCH 10/19] Rename dataset_roberta.py to transformer_models/dataset_roberta.py --- dataset_roberta.py => transformer_models/dataset_roberta.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename dataset_roberta.py => transformer_models/dataset_roberta.py (100%) diff --git a/dataset_roberta.py b/transformer_models/dataset_roberta.py similarity index 100% rename from dataset_roberta.py rename to transformer_models/dataset_roberta.py From 166a6525c980bb8871e08c57f65592114634b1e5 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 4 Feb 2021 16:17:10 -0800 Subject: [PATCH 11/19] Rename model.py to transformer_models/model.py --- model.py => transformer_models/model.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename model.py => transformer_models/model.py (99%) diff --git a/model.py b/transformer_models/model.py similarity index 99% rename from model.py rename to transformer_models/model.py index 1bdc153..5fccdb4 100644 --- a/model.py +++ b/transformer_models/model.py @@ -165,4 +165,4 @@ def get_crf_loss(self, logits, y, mask): loss = -1 * self.log_likelihood(logits, y, mask.float()) loss = loss / s_lens.float() loss = loss.mean() - return loss \ No newline at end of file + return loss From 97368e84ef39556f22c7b343e36c15eecf186037 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 4 Feb 2021 16:17:33 -0800 Subject: [PATCH 12/19] Rename train_roberta.py to transformer_models/train_roberta.py --- train_roberta.py => transformer_models/train_roberta.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename train_roberta.py => transformer_models/train_roberta.py (100%) diff --git a/train_roberta.py b/transformer_models/train_roberta.py similarity index 100% rename from train_roberta.py rename to transformer_models/train_roberta.py From 6152da4faec8778c983d25a546515f00f7eeed45 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 4 Feb 2021 16:17:51 -0800 Subject: [PATCH 13/19] Rename multichoice_dataset.py to transformer_models/multichoice_dataset.py --- .../multichoice_dataset.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename multichoice_dataset.py => transformer_models/multichoice_dataset.py (100%) diff --git a/multichoice_dataset.py b/transformer_models/multichoice_dataset.py similarity index 100% rename from multichoice_dataset.py rename to transformer_models/multichoice_dataset.py From 3a652259e875a9c1f8418d43f7a55ba1ef4795ab Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 4 Feb 2021 16:18:09 -0800 Subject: [PATCH 14/19] Rename multichoice_train.py to transformer_models/multichoice_train.py --- multichoice_train.py => transformer_models/multichoice_train.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename multichoice_train.py => transformer_models/multichoice_train.py (100%) diff --git a/multichoice_train.py b/transformer_models/multichoice_train.py similarity index 100% rename from multichoice_train.py rename to transformer_models/multichoice_train.py From dc18f8b2324deb7828ec08c6cb36e74f0427e447 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 4 Feb 2021 16:18:56 -0800 Subject: [PATCH 15/19] Delete pooled-vs-unpooled.pptx --- pooled-vs-unpooled.pptx | Bin 53429 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 pooled-vs-unpooled.pptx diff --git a/pooled-vs-unpooled.pptx b/pooled-vs-unpooled.pptx deleted file mode 100644 index df0cf1cffbefe010a035c281183ab533a3d1f2f7..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 53429 zcmeFZQ;;WL*XLX8VwcrrySmIS+qP|^%eHOXwr$(CjbCx<`M)z0?>X~K%*DAo@8m^h zWbB=hu`<`%JJ)A_SISC&f}sHY0D=Sp0wMsS6wUT-0R{pR{|N+y3H2#mW*t4-AAn2MFZ5|Nq_o3qOI0#8C02$b4tU`0cn-e<6p!D znbEk6U=N+D3QHI!W{im#r5OZkuER*GDiDNL(8io^-8f<5`Ysy{b=G0jmn=n@gQ7W^ zRduhLftQ^o4wNS+p~l?(JAX5!_>p^eCX%@wX6fb&P=gS(`S1k zD9O&)xg!P(EwV^x5}KG@aKY+L>6ak@bJ({6gZnaw;eyW&DV1*e4@5NFbvIGH-8TW@5Y&L9-sS~Qv`#Y?^zCeLw z{~rNw`r?=i`rWek9nUb|0k30kWa&Wr`(OM28|eRqqx(OZUJgXv=u*T>raP*L`7m+P&f5(^(|NR*L*O>Vwl_k>&2Lyzf0|bQn-NV+_ zk=DV|%+Sc;KMwT&nkFx_G;G#~ki2+$0PucU_CA&!&l0l5_|BtEytKj>PlYF-M6*q@ zVe%A1PO=Z(*=lidR%%fRf{LYSUNtVOSSH8&to7~n?`P-hW-KjY;e&=r*>Iy&oY(~b zQi;r7H+2H&7WCj_ILKXZmjPxdpsK)@_vgdw`nU z=I8BE1%~7he)TTCOc8$4u9!otgSy`cwI#-reg{UCOJDOH=RVDMG+k?-$SQ z^4z*TT^ViERaI1#_gc(fM;t3r&$^181MR}k(z78hFYHE-jOJ6w*LO+@2>X?W!#AN; z%f&Z+dyaRiC^PcBlzfaRgkb^^y@8)$6!VBmwS|(Omk!7PH^qX)f^H2n*w|<{Gmc-V z^@A669?WS{=~vq3KOGTvBd1=HK8+xY2QUL6XORX|EKtZV5Xhne2cY-_-kgcROz-cS z-NNFJQqzOCS9F~F)>ae}Y$Hp{UVZF>Nao?)f2I`{n7aw^pF_FMkc^(Y){=LNnaun0 zu;HNL--$1c{$#3?Zj+R6EVYJuPr|EPAe*>rJ|z+DXDuF6gJNh@qi+$bo7ZE&^bLxC zU2})i;e}*$C_lg|W3_)%PB&;C;44mG1L{Kn)g+mXlo{hyWoQ@ciaa0OifD&q{LO#Ns zPBB0~_qsRciS9kQ|HhJKeC`%MvkNO8rBI8jEidjh zFTh>N6Z6fDDzlu=gJ)a7cDvmJZ`*&|lHtp!^j4w|oCSQUI%cYtPQE zD1tVPThmEfYhHN3olV(1QL>I44}J{ZVaBg6Me+Ku2shhsyj&0B#d_!iZib8LYJ5F< zXY2U|NlHU+u&KYytN1Uxvb;TMYB+*kEWfZr5I+sr9GK`3yJcptdqAQ&loIxYiF$f% zf?hvBSP{{npw#-Y@bPv zq4|!$%B`HN1tSY1YPFg^`;fN=oOi-Mb9%D1k>N$1Rt}LWXyIHAz$)Ch#q|0Lrr85o z2vYX}J4Frt=)ipq96H*v6t7B&DHvkR_rqvqA)Q}O7&cU}e@z*4p~=766@=>7$I<|s zf)$Xri;Rf;l}F7KAs_%kqfx-HUmWYWFKx$H5Ys8Ox}NY>O93#iIlx6|=ioqsh}~g{ zK!~*U(Ht-+An6y?r7VQiFE(5uL}+AyX2ljHAxf)ecb?IvF|twMR!EP#9xXc1FP9p1 z6(PWs8?JD|gvy~9poT|GF-Jp;N_8ZTpxJghvrYzbkxh_4P#IM09MX6?2QX>4Rgxu%gmJ-=r7 zR>z_;?=GW}o~cD%R0$(Qj_?wQi3o}s92+Y-7>6ubo4CteJl=^j>(^KcW+2HLACy3D zE@)$g`$maU>uHyKa<=yyX)w3OkI<@-kNPRRT^|cyLK0&zO^9&ADZ>a|6``fAgdyHZ%ELT%obmgUgbY8iCG(*l> zO9;L(qyXEEg)|{q3@Td_wJ|Z=-^Z1|y=c9Lv^%d&(10~C#?aPvTju z)I2uFas`3f@a+cj2(^ZclOaW1=}w*#I#ref&GpX!HXQh?!0f+EY!Rji;Qai9NJPTy zExK#;#J?KZ`XWMVCWYet=qnm3`t>4QP<{%O0Jm4?Vinc&tp|N}o^)=!=?(8=|HUp4 z+67;P<;n8Lm5E=6Ad0@p0K8CB#5Ln#b-cgdinV4^7^$SnLs6mDgRz&x)}{R^0G&jP z5-KT`^=hc5qM{C@xBTTHP0s3OmAJ})VhyqK{O}w-pADa)nQf+^6^V*Z4RRQdBBxIg z#zr<0fDJ27Mx8V_IMjr7ZN*z^F?pP!K)LKHV@IgOC{Nx)w!aAE$us{%n`QRqbh|#@ zTfKa6sQYP8_i7~*Z{_CR;&Wx*!iz1wFNi`2`-zlS$6tEayX|XF+QbjpH63%EB>cND zcc<%#VEYRxYrUaTOr^MdAHhYC?G&9Fg8a$g%eTPmLI;ZoLp9KF*(ZHEEOb7>4}SGt zbJ`)=CF&~qcw+3`Bkj1&JTfh1ujdEaFI4&i3>6>wT?EDQN+OOp$LxAYW1e_OJPkz< zPBIkiXHMq^&f`YI%wtKFVIMC~pU?9}|0? z)QZK&t^3EVKW|Hl%5O`6iD0SD5EjZ2L)mpl%_0RyXqGuqpqob>)_UdvzFS$xZdn@p z?4M?w&xpL%kgN1sKUWd^prm1im>jt+qZCjE6yLZWP>f*j%buQ|Y=*G0l(Dh|#eufh zs@6R(HtsHd0eY`f*>#W$JaKx;SI5E3~AlxCWp)SxkCty;t=$w z3lXd!s=vo|tE^AiyU8;}GX_bgV7pekNKv!w-{}%j5IG)J#!_>05V~} zqTv1t)W%VoK}-_LWI_tU-fh-sghEA9?cz<}uBC;nnT!EWZtW%;l~=Zrzk^E@6J8-L z+SfX-Znomq>S{Ou9bLbgY_&!7zBhapJ^9fpEdt#i{X4ptJzXcYweO{byw)22w7!LH zel$y}SQ$|{^`t6`suFqTl=Jz@V*ava30&ZSwoU(2Pt}`X34+BkSZN_e$MSO$Q&R9N z%oX>K(N#eInF&DZsZd#21V$ ztyFL4CWnF}M z43B?M!-?s1)-CM;zpBV}si|cf;|76ptE~AHeE_#OtlqyR-D`1;amlWmc@Eeyqw&n9 zF8`5rmG51uyXcVp{6!C2nrDAIFY0=H_dx+6c=6VsEn|8e(6Xm8f0M9N4c*L5Aq&m~@M{BeYxFr6KewdZT&@2sj~3Z}*@>HaQD%-? zZZz8VoRP{^)|_OJ=WlGkdDN>`lO+OnW@QoRwHmD$cWRKz0gg6$_dNnfQ!)pGH1{Iw{kVD9d|aYZUE1}&39Y3shsz}Q{ny8 zf1mF%{71ffZoNN*`l*w1jenv7N^dcIvn8`i6&JxMlyXDhn3yNgdNGJjjZ z`3L*cHja=Bd*ZwT+hnj|0zNqV4sc_G_xTdsJ+(C*ehXxteE+<+f!*W%0xuT<$ZHe zjyWa5SG%7tRSQTh2;$PeNwtN#x-Z#3@ysN=diCvm{;bctul;8gd@%33%;&Xlce2j1 zznp`Eql2^K=SNyy{=K2pslN~eL^H>u%_8j8=XuHCxyOVE-LZ)fcTE3Z>g|d1!IIzQ zqq{fTX5y8i@=@|HL2OOfs{vRE@ zAJ$Brrp4~P%yfJK&7AupuQmc~Bva&qAxY$en)#x;7KQ{TkQ} z*Bf2$Yk)R_rL|NfO*(4R6j0y9fZmNvW{jk`vdo`Q`DD30_TqKV_2$~%i88c5KF4=! zK;j_OO;qScica!GF%%e)UMhG{TWM7j@a`3Iemn>ZO!AIwpf2uEQo#kUnu1|Vr->Y| zve{U}zd;yYOh*I8R%7SL_UrhysKlr@*unjT+CNDB z?u)5rr@TdpqMGCPCUkVDubqb%73_aoP-a7?>izQ4DaD;7=%x9ypV=@oaF*r=sxgD@ zmXqrS;Vhu~*>GIG1$$YEh5b)8gq)I;H%^P08pvSLq7T!-B`$~19$kDC5zQRfAlyu8 zZ9inOv}zxF{e)%c6p2+`kp8x$2+uTV?=dqc%!k><+s})oo#(%u?JZqSZ!8TE#e;@V z(ctyX_{k{}l}e!YrKl$Mm8M0QSWYT?TmH`-nf-?c%v_b=?IQX+C6@BUhc2?-X~5_U zi^L%ZO?Wzn{)#Igt>*;Y;O07*caM)I_}3;LzC}_s`ffxODQ9JK{jdtuuncT*q6)v| z5)guI1@}`5gUrO8A*p)*f|Ped{4vAnX~|pL5e}CZr(>jQf+x^q#vp<<&1F5A!8ODI zPF<;J7gTXV7n7&T$PxWxg&@&+C9V~KP0u36ED_N72P{?2eO616p$Y|RW=`JL{E83; z3@SN+bw^&9YRTX{Q8(OX2<@D_2}t?(U@UI}Wf=A`MIC?PNNb8cK|~|x8cEZHb`7{& z2*Tn@V44Bt^_!Pq_!67qjt%0b!MT&Jgyj(0HF=XE*pOld!T_wEE)xH+91_d{n2Q#* z5piTwWQe6djo)ZiW&irsG6$+KhQUz!J+3h(Bc&O9{W36P>o4QHe7byTiAy>WNx6Mb zI^PQWok$Aff)M2IDFP?x0nq0 zz*vf+y^JE!R-Rr&_9B5#`X1v{)FD7Yj~O*gE-$N;!ue-oL85=mjo}9U@g>lJ=r{IjhY%;qpO(L2Ej+C_I4^Q9{pQ4IpuF^r~T@)GN?Zp;NCldF0S-FRWnjsfcw|E z#+Y{BHKJ{!>>*QP1t<4O&gduw%FK>V+-a*C9js-7XCp>^o=pZwu1` ztM<~`EUadOPG``sYjyyAaP-&5arg6=e=$l{6Z*~9P6kc&^0KM;ZMypBPk7JGb9S>L$0+izSxb zR!d8WwYKI0o&K{RqCzOi1QZH;96q{hy3Pt@KwKjIKE2Fp6U*0@LXn~=C3-q5hQiJC zS_VaLknGrNf!T0;kyrC{*%X?VQ`FDD(#U88lCXS;M-+1d90P?%&?rqniwz=~lMQ<_ z4*B(g&jBh1ik(wl{v7E~9B;K*eyY~P9VAX|ZC`Vt#CN`Q1HH_=84kgOc?JV{kT6Ih zu}SM!-5&CWntr0U?oEEhIlR?Di5Q&=uIuu_kMhT%S$z!bpEboZi<++D^ojCxhAknn zbz4ATiToHFfhodFg?p{-iq1zA{f!6Ha+Fs>$Fo&ccsvydUNo2xy;u^*r77>AjWM_3^Apa1p zQ*KOb1*e9XqHn~X^0u_42Aq-ubC1@8uhi|jmuUKY`JnEmhT?DF65WA)=$-cE8ky#< z=q@7SFUCGq4dLxp+F zkBLwdtXb4G=OnGTjc5!VvVk0NJ}Hod^voQsKut#_UNejEZ%3n834xG2!AygTxlQK?|4c*{5sw zh7HO&&T7=3I_@(!m6ZCbzTol{dPL-eu@U@3AnqW&?)%spQQ30jU)>>0CeRiWs#KGa zV6z`X1?-!tOve()U%T-|?dG?C4v|BC^T%QOW84NqQS!wo*NPG+*%dTzhP6j>KFe+n z^JQW%&i}4FlC+&tnPNEyf3-k5@%Z(o1L?T6Do~(+Q>;uchDLH2#)&Obc;y(Rw8zs%dv^66rO z`e0}c0&@Vy<=Q$?=t4rFhC|2fVF*aN4oGd)TZ^R$+ewmhq3QD>rjkk5c>3B_? zv|DX0j%HU;Iw?Q~QhGKr18t1nj_ND;HpJSS!6W=7QXsm8pB?3;&22mm?m5bV;EcQy zRq!K=_TLK-uh8*|6>LsUgHH>*C)S_`6-#%$^Xr2AI%Hy8SI=Ov-*x>oYAGu8ij_hKwj?Ehp&1Ndc zrJVY=cF^FVwwh+AI@&UA@7^aX#TG+!I{L0%_qdHW=hbsZr!?-fNoqG(G_B@5>*)T2 z5vPgDUt?w`$Cd{4T) z{#clJSGN=0QyI=p73P?P1U37l8uzssY@V}4$X%+lGL?r(#i8KC-4g4b~iJ0z$ z{?-dY+-Tk4^j(rJ@(HH3h2XP=a`(h)uIr`;XrjD588__gxaf5AsNikw<7+3ZfLH6$ zP$3*h)l4~KUNz5~L-;Yeg$vvi=n)19)N3xNVA3TMU1=H0p-f}GX6$QIuy93=FbLim z%>^;(|28^P6uj?=S|}=OqV&X?nUJ(=!kk;J;@c<^{aVSj%FH#On~wGPM9g)uP5R9i zX!4u7D5~xUMfY2MVI(zTPMZ4UmKFE^2h;z-BL6i_|3?Aw|2NY?geHW4eLMRa7!VL1 z5G3%w9R5E%-PYd7;aeoB=V)eQ{XhNvTRsK+t&{w|;J^0Pu`ny%Plxhj$Gz7X@_4K0 z(lLyBGZL~LLGT>Z@X`!RGCpR&Yd^l*l@Tyj;Lku|{zPp02G7F?&GA{ns(<(`hqY1$VF9UdgkdA3 zBAj-|v$%49cYCqqmOR9lgm;eX7mXcNp5pNyH zhpQ>!MvtnvaE^_OF=r+o5AR>gC=?9m3&khSSCovsyS!8AM0~w`c7L72-#drvbNH@rOQwCCp!9uRgxcLZ3ND0!wb-_&#Sq%Ez|&YT(93DPHtB(GLrJ3pRl zovE0Z)QjP&#FvkESgr}5b&OzQ0QU4*bf`7bujke}rqE|3OFV5vuXm?N$p#m^X^)DX zvflFVUS%HN4L@_?lDj)UWuuK1a+X~bjgxtkZsnYFsnEQVmUnkdD_;S3bnea$$~vz$ zEzPr&w-_N`qu#gx2Y?h~S_2tbvk4jcwiWrGR#BD&ZuP&mj>QVzDwjLBTT$v;Q^Ou0 z)9)Q4BK8}Rhv`D~*o#VBe{jEuq)Ji6#^j(yLJY=ZrCRtXj5u;Bof1a9U)Kb?(idEF z9+s)12=00n+A^fh%;hqfb)?G1uk2`Bf8ZkQ;!MB9e)c0shcZGV=8=fgFO$o&31Z}R*h>`6)9-u)fb=6+@EwSV76vNS|S(xR(6a{~*ejP%3dtF*T!L2+|FhRzKM zx&l}1$y%E>jpuOja0CxK+5+D-$La4+f6MUS;Q{oGM84NY3bAi~LI8RmP;*`zrUlr5 z2`#zks`@>b3@Or{-GWZXSgHG`hMt?{NFybpcj%w5b0+n9s*Rh>HLaITsZ|~=U-gVF zu}VfAop)uzC#{q=R60{*Wm^lpcIyrrT_my+DS-B)A?5jR&>DU7vqOtDusjSXsv<*E z5#NSXYLmN0e>USKYK$)0-B5cnBNHQ45C!$U?XGtw6o}`!oW{8koraS5cKnL6z+*Qa_ron?q5t4JGafkeUP=gbwE{p6K9F_t%&;m_wxFqs zUdtZn`cYu~VWWuW5+qPa1_-cc*^Eo|(&kM1&PI0t4mCk6y&+U(dI!M52}lzYZB+xR zH)!Hvc0tXiW5D8EzHuv6xq>>(DE|qw2K^_@>b@W=j3X!@H`*dk%qt<$TiO~;jexE> zN759bY}vC!)%%YqMB!T$q8UcyORj{%KBZt7C?4xTf5?wv;o2Z$nd+|_ZgI1!Xz^hd zRqAoFi%7$PEbFTx^x#vu>`PONqtQ{k5{HPPV8@T3DhCV8bHXOB@0$U%9ab1oFk598 ztyLq`gRu_L!fk`IS>y&FVc<19Bb}Ra>?J3~Q?}e;qIC*ul z6Q6s-g#9CCadnth3Ye2?fwtK~d{srNG$1~V-A}(5);?`mR3wck1G>CaHUjCeLlZ9_Q1R~f~3nWhDZ{X7vSI*ai}Xh)Tc+W zztHj!V;)T&+Ua^+ToRu}86$CI2N%NUVLJRY zMD8jO|E3CpxxwDTn->%sq++Z9=^PYc2BiR4RHXp~7y*QP!YzwKI3(!Nj$4l%&@NCP znGapS*s`3ctqxg_6m`DfFp4dD;RxkaEX!}cQiUl@Ac~}RVYX}#fzk%1B4$xO^uA~U zUwfZjZQT);-Wt!I5C2>x#9cp&1G)#e{u>G{d3;z(h9rW@aBPI2A+oM>X)H=xq{B8H zwjNub-FV#=!Ci`)xJqkb&&UZu;w4J)N_Y)jlp=mgHV)} z%h@CfWS|;l5(Yl@!E*;k=la$ZrUy60A-y`sV6TesZBOW%7ioiWQ^r`Hx7@jIGm221U?CasR z11f-a$CBB}R%HFrsj5e`ilPy7&@L`d)Gj3P!Q@AA~A_d?f`6l?(>>Kv9E&W^tzz|f1bO+ zm)*_hDGlNIsY=UVfhM#S%i*a7ekzCr;6>frVZonj6+oP&*Ovx%UUgpD+IJ_<>fwg(DIW|l__9O(zy9S{WB30(Ywxq zs6>uxD})fF;5xGD*02=u_A9hk4ny&&a_q=|%xljr@Dht>r0b|t*8oElD;&j+eh)?Q zZ$cXi2`rqQA(4Jow+{<#F?{k(>q0-(BT z>Lacm;@-vcv8Rw^0I4{}m`QlbR`vqL0_E2)65_qJnPJIP9&YCe(KTx~Lb}1R3icgI zAZ*j&uK-HI#hR4e#FG0>6g4*`lzilt=#GqoZVl|OW1X)q{4c=r*9Yyp&dedO2JX$3 z?v|&_&X+gKNuNmMZh!D>MaB_p{x;WqKnj;wGLQTFmAV=_Fn2Y!f7r=c>Eb~5-%xRsu|{$U8@#*gp+E8ocqOt9{HU6Eo+d>Gq%Psf;94A+q230 z0aM#7v5DS9sSuBd=66S5IBR-?vOu7ed!ps8p%1uJSKfu z`b+c?-NK$7-87+vG^Wktrg+@4U>BPos6lBv`H+$*!#6K|sGP?N z-V~3u*=EgtcCt{kCQ&5o zmvQ)=<39KB=X4OqYPP4@M7=TG{|vs#z9Lsirr_chcxM-WnY&LXK3tz4WhsubNiz%7 z>I73#qmzpx^CHo&=~Ed*>KIok*$x$KXS_^KHP(HAJ2Z7y#y&GkAie)7>xJecZGwZs zB>f^0Ru`T##}xW;t(1dzgzCWWdv==i2X4NyT06(RbspsG%@u-VZ5f~(vnEN2MkLkv z4OY44|0h^|h*^gTsWB~q8G{j$w5Z;w-D6J1Z(paG4;``$XLVs5=DZ`>(GY>0x5mQ-(+kVCu>N=kucwkd1%inbs!)o*v z+yL?NaA)rdH?1uhg+_y_uQXH|OaZFQ$2sk1gD1nA$14OA_LuytQtsQhV*zj@xx)!%a>M9+- z!H+*A|AVadUE6YAEw_Kq?Zpy0XFZ&_AYnF%!>8jeX&rL z_M8>iAe%RPZrLmjbMwrPHrt+tW^Pz8%ibPqozy^oe5z7IfQjHK%QgL3l2D%deF-oZ z|B$BtLYoyyBKnh*uS0F}_nft3-KyE|q8>cj+U($T34EHXF#PoG3*z6Q09Xdq)}rrv zyzuWLT$FEk^e@o#|3*v~$(z>cbSOU8WS2NGHe}*2WOAa54F+ioVsi7`J}}b$=9Y%( zOs5OZ+qIy+1!)TqJJ9RL%XmW>jNMx*3u}l35>xUA+EOt>s#D={odqYS^!3PwVu5PD z6eh$%5QMXjdvDFIT;<@g0TdNI3m~kLQ?_AE;){&Kv4A8+Rqfj}x)`gLk{Z8K7dvR* z?H*4d921UU|gZ_Lk%`{Xxob6jy5mh)_ImAf<^1 zXi?1r>Z2*|iRobxGCj_kr-sK?my?Yz>MXbcK0J4fZ>7_r`%3k(v@Zm9=zB~aITSJ% zJ=`x9!6x&}9#g4Jd^R&wLw~F?+)?eyMc=qV?DNJ&@rL~1#2WG?kx05XGpo8^WivQV z7Bi(aEz2o1k~%vn@`k=sL|)q6&WL&nVVJ0X*Q@d!Y0HK1;1 z(VGL(wP`BER|%g;tk#1D>bf|2#@XYL4SBswIzffE{qgo0200q161_Ke4@MqJV}(NI z+lU<);9e2F-?U(@7j~Sc*i)e*;X~)MV0snExBv`t8och~c|Ht)DLGAc&hh?hEmout z2PI98EfD!FQv3&`sDc@=5EuTm1mQh82C&(8iwh7r-1}`5x11R;2^YSUNk@TiiOe+v56tFa zAKQ*ys;;$<;3PFmuplY4~4xvf)~n{+l!i-Bp*<1j+>m1*3c|qw3RO}H@uuPf%P>yC)OFNUB{1}vL2t# z@3Xz#7AIf7MS#y9L^Or@mnMJ>ZHd)M>|)xbiqSX5ty*PI4ZGO``OYwnViW*GVS6dg zN!yCqU0LJs$6m^ZM`Iau(vALnjWV1#Sh+*#`}MNYvdmhWL4%Xuqj==%N27%OcBv^~ z60gZ9U_iOk?Wr}aU3IB&@hDQgrOI%UIuVEJs8TU4f_GaBX6T_}^OLg8$eCoKt9hMD zPpNQ;@v1rVwx8Tsd%ZH14ZdkxdfuZ+BUCt=^4F5JHPyt^>*)z|Q@k=gHCphZz9%5S zvQ|1f<}WGe6uPjJqQ66QzpnzD0Ec5z1Ri>42IpFcxwZEMgQ!1~0KNjFk7gK$>)N`wrSy(uN!6h)O0z;C2)2OIPCDpJe?1ew_io>3vK^5p4HYNt8s=&V zBgigUOlm(cl37AvdQudGFwGjsd4v{nI5glpdZ@nR0zRcHI6^XA-s0ZvrDvg!{fLW=QN>o0KYI3yq~K_D%Jl5q1~0Nc+GARccUn!>h<_d>$a^TfZB z=xeqGnFVvoXB(`ud*nBS2|_*rGvH9d+#w7|K#6AJq1k~11kUp~d5RcgWqN8(?T8ol z`PY!qYKD)I5f(|!&ra0Sx&GKEI4cDgSQG1al<0u#XxA6-7BorCakk~n9f_t~^f4Am zyR-o{XwAHex2!Q90w^&00u+K15vYi)Y%3~?A8jA4qk-a^11-o;PxJhdWF^T?N(|?h zMXjztE;szZ(U7p@a0)deC}(4IKTrlCU5JC!{3voMT)lc0;iJb^<5SWZF9x)N__u!& zIq4Lx^y9-IfQRFU_*7#``jHoM!(O#ZRufnyIuElXPiKFA9^MywOV4Oc(0$xWs{JYSmiDU{&}nd{JgQi?5Aj7)S8?6o5NU#Ow8uPX-%oM zNWkMnN{g~IH|){dtjEV4?z%^3%kWw{+6^2Uv)s=eZg>c(-404666|=w@cld$u!1&-5IkfAT2V;(002lW6Cc})Zik+2l7yZspflwG+<%&_a$K{?vt3^1 zt_gRL-b`{HYE-nC50Ewe>w&<)hfK2?Y7|=#tVp*uhd$pC7kqTDKLse#)!YhY3*#E& zMNn{Cs{ydSWu`8bu zkNH9r8KN*i*Di5H=j>cmwcpf2e`VuXE;fVz8z58>&!S}&uXt1G+kCy*pgbZk;9Y(9OojVP^?>xayDpS_LX5(6U-)$;9w?bO#;LHUT<9m$D6htgiR29|%1h9L z*oz+p(A@WdE%ZV-!@Am~O5|@kjIE) z0@Ku4*Tp?C?N3n1VXVv_qAj^$mmiQBdIek=5>nV z+#_gFiPzq?-$F@FotXI1Ml|)6&&**)W2KS5FVGl^tfYZK7{M+BvMUjD?{yK@%cN&U>J79(q z7z2tY1O-Vh#tXKOK>TVf*@8Ee0uC?ZqJzp|@@yZ4yOR)#_@tHsh7YF;BrmT2@hBqT z^9amHW`}Jc=v*vmLvF>?W&ocogZK4$I6v&|)pM$!eu`H`ao)sgP8QVa@+u?_)VW_vHyX?=W$nQ8s> zwM^k8Qm;+rJJ+So4fWWCCu^AtS(5TH_8 z3-Lmm!&H^=JI(bgb$Ec+iIdTM`@CE0xLd6z3+l8`9DjhqIdzmIT@3Gf^TT}C< zx9@Kb$~zY=a{kghYsadQbhw+q=>9I#h+9!5TD~}o>rk#_H*9I+yzRBKVA%Ni!kNQV z=CDBSo6E%e3LCsAvuH!oWYCtt!7S$5p?k7q&dcgQk;g= zdx?p4?-g2>mV!8aSfRUes5Q~h1np;1k}nBlV=ij3b2H@h(Bi&;A0*%jJ8(5z6PW&( zMs)7HR=!bg)>den36}|va06^OVJDMEw*p{Bwi`Rn%m%PUX2FXSZ4bBS87+xa@Xw1(uK#}uN5Vevsf;-%=z zwHP4Qhv82!37TP`X-N5+#e}hS@;L5n*4NsS^$J1rlkfp!xJ^P0w;uz7@Dq=UH^2ZR z4Mt=Z913ywv8bxK$i;Y77rM!;`qS{Pnu&jX0rsWthBU3S?8QUfcH)sm@jmwEg)S1e z=qCc@TM{G^4yZJpWW>C z$gBx#QbkWF#r&9H)vLgLLar7c zE##V^sd2NOlip{$%OKuR~=(`i#8?f5TK1M#`lFY?C0 zkb)EE3#XK14^QN)y*@w>_TQtN*I|yzXGj3x9q&I!Ip)7mPVHYumK_#LB*JovGX*pF z!60T|CXT@<;&3b$w@6mS|1q*$B_th`*(s2jg9Ayz4v!m&KD>IF`jz{7zabopUEW|; zU-mOQhSgQSz(|u+&8>;|$(1NJ1;YwGQ9MD#sExN(YJIf*2hh>gxNLN@$Iwz9p8`bG zZG_*GhgX$96wR|}7YHwOm7X*NxZIRc=D}O1*PG1=9r|kRDTL8yqsb}D1pS_**)4zK z3ZKsjOBPml<`#=Tkb}B!4zI8lZE#0#XcO1)oz1UKU(j0GwsO(V)Xd0G5iRzSg4<|S z>os6t2Yl?RWnma8_JDO$os^8Ol`Yt;Jr{^8EK;d-v)oniSl{|0lWk?01 z3$l7?R5=>B8$7PAMBLLHNat47U=0nvgNwB*b}QUbcOFHw_+V6L4~6G{^V$_|DSU9u z&~2kHy)I+DXySBqJR~{2Uidzqf6ivvKAUG0e=f@-DbD>h0Z!bMFo-*-BZgf}ULjy1 ziHG?KojL8QvqhYlwwpA+3%YyIEOtEa+u#6ytD78H{tK4r5%PcxE zm zQ?mHBHok7Xq;*i*Rk|3>k`EGgBS!k8K?vXEtzyG3+vGQSet87bD95hXg0q={=`?`` zAS|s}hi^X{GNX!osp%&z<&A{=*x)W#*t~EeIUTH5?g@jMSj|_QfLy?B4SO%IZ4w>hM5#A*mm^PoFc)aDlEo425rTd9a)bNIqSC0 zF&H?;uoCu#REn=(M#joYL6Y!wPiOPv9N#$|EkFD-`V-b_ zJ0>1*5}YV?;r}*_Wo`$9U*Gofty5>&&^Hlk0@G*dhPu+Lwb)RTu0_Xc89>nDXXlr= z9oe4{%=!i$r%XYziwukOuCkc$X@@si4r~o?r-il>A9*-qMe)+iWm&n8glbO5qI`Mn zq_OGQXHogJYi-OumvXJwSS)0h7y$~rl_h>8e@>vuKqDns&ou42)uQ~=OzpBPT4vm3 zK?U3FsjWs(QpW((1#@9iOKs+ z-lLa{S~@Y6B<xJyVW*O$9}p6>6m z@nf`Zjb4N8HJvuF8g|{_i@SV@S^Wr%{0xXy>|r$-FmJejJsX!aD133&;JmVBJzrRK zTU$^n`WPa0jhUvBUuPxzu4X@ug|OBsvb03CV>0H`axnQ)!lP~$i_#`=eV4BeQ-9fZ zk>`p3(aiIR{?Fbe!Ea|bQ@}{-4KRlBpG7HH{=7>jlu!S}OW%2QF9}S$F~w1w7|qd( z4B^4%$V%p+bV)hT%GN2?8a2TG7Ns}_zDv~NKzd`~o`~gx58u7G=F*#k+v@HG?v2(b ztS%=Szs>Gh$+0XuyzHV1)>_?#u|ia@iA-f0MPd=Bi9S%DeQ(w*AGV(Mx zOE@tOAKz4#xcMwO=lRu3f3>LER?_AS+|%iC(PCe55-dJ2GKjvvro2BlRyZKCxY7EC z#mp9)xAW=xve{@^cC)S66&RoBjjVk)kvLi5Z!A+xY%wj%x589vg~~Od&3X3JLRId! zK-4t*HePZZ-_;%`aHLiLq2+vfA&>Rr;YHFcG0>R~Yi-Jvt}J!g*oI|_`{vYa+$1=% z;_g#t4ZH(30q03^zR^xjg(u%^zHF$r!G$R+Gjhig` zMq3*5+>pJwn?A^_MjxDhg>XmS|0s+kM2}bdS4iOsf=xps`jkB-9F-5sm-db;kUXc< z@g^JA6Kb2&SgbGo0IBE*-4p8Wr&R1Vp1TRNQg{Q^&2*CaH@So{_hApx0;8S&NJ!eYo?NNS zK)a(1P|h!VAMhQ#AlY`L72{!t`s>ONLcP@qU`U@Cu;u_p{70ZjVxMLS17`+f(~ZOa=z0kbPJGntQ`$(gSz)GdZ5Wz>e8i}_mf;9_xTY|(Ak7Z(mVwR@4eD$ z5`S4%T{p+uGD`oDSVg0g0^Ks9P$l~sz^J<(XN=(%K%!k|zl&H*JKYo+2e$G4(+;LG z9fNh12W&SB{Y2N_T`MuLX7lny_5na%jbaPx+V27}6y9z~K^#Cd=81qOClGR2$n=B{ z2>xI=TFp<{MNeTsd9h9?Ak51}Cqm8F=z@%ODF5ysoGa`85du8_X zc#T%T*~-jYo7<&eI!gtB07LgX-W1d=XM|EM@)*g&-TEgBq8IIq58{G}IyFQ$d(CUpqsfjGq0Z**zW9~|v^%Cuj2VoFAwW22w%Wxh#5(QL zXzxwE%-QNJYwx&~1&mmx>oKz|2D05jo%2OnYlb$q zFT1O{ZK_@veI2Uzj=|+xzbiwFh*#)5XEZB%sj(NrJ{meXl1^f2pGMCSx9x}61fZNIK4L#}RH zL)phR{Hrr=PIcqA3bSSKSLVSkPHoEKDyfUOUH9=Hq?T>xQSraH#ky;YR5vaZEtE0* z9NL}zO5heKyOor6 z_-{kzn`o4H?>W7c@6UV+MzJy|Qs=A!J~NnS8n=GS;5tI763K!rCn9285bA7twDMreN0e z#@osE5FvVvKc5(~bOt%uhbb|hRPr#}w_#2gLCs~%7U{*vXXTL$@r)5md>@NP>@C2Q zgpS??pIIyN71U~uTV{nKrA2Vh`wp_0W5Whx7|dgtd#P*^iL#NIbjyBU5)8&z=B8Wo zG2o`61w;Do`dtt2f=;i&m#e()V;QSfeb=j+UvI1Y^GA+3F+HvQ5Mx3BhzLwEcEWdj z>b|;(=v6YZr_*w);zjCaAoH>JsA9+R0+!;!sI(~Q#7aSB&dtTN8Xr-{`+MIF1-Tr* zd_$%$?B_e-0Jjm$DedysTq6+t8S?pPU^jZC8eZvs`#H;Yw{YDq_F0j(K8XL z%2c|Y&)qzhj&p64O){1C!psX&q=tD#Qg|v=I8v-XcQfd@H)Odaa`Z*1p6NkOLGk=j{B~w!S9NwM{&vLAZQiqbXM$s8HuMOm3jd0Drsy3=*JBN(`<1<58jNq0In(&`jTOt`v9nECc!}AIF`2fm z2n#~r5t7vDNL9XZEIJAsf~^)a%quw`vk>M}Ur6puE>j)-L*K|7lQp1X6~_r}o_>qo zP9da=+p6xNQ;GFEh0=1aHET$2%E=0ih?aPxD6#dU0#tK*NBscaZ>{P1#{3D0+Rbk) zzhQ<3aKnyl*YS|j3vp7`iCf{*U)3YK0*DjnD(i*8T(@MyL&$D~mCI(d!Y0R^Ak!eo z!fXk;1U63a#^)TF!NTGZ6q8o&%0;QFOh)iuewk3o={sQg;vhYk@bpRK@xZX_v!I|F z(B2*)B$~!n+Yknq$Ft+viMTyyNz%7}mAz|9dkH;cgPYkhtx`^OE`BbpDRD5(qFTsLve{PRi(b2eyzBX`bctI! zRGifRc;R}H#2f9&<2d@`>Ebr%8v_0l&dv!FJhA7<(dXP??nRRd%VF*n9BuCr#IsG1eqio#4EK}IBTY1~+ZV}_EjTiu( z^&RG#-e`oFT1eay%KJt}?)-o# z-U@mcdtkzsXu_3ax?8awe`X(sY&iRpTxzOV6qRwA4lN0-J}WadFVcMZg1bo2eVD8{ zN@wdu@SD@LR!!*qKH)ehBJ?VmKSe=?>GQmC z82np5-+Z_>P8zX}oKG9A^)6u!p1UcyN9=UjlxiuQ+|-Q(-WjWDS)*p~&ijkfMCt}X zyV(H?A6~XokObs}aQ0F=8=SR$_;ubI!Fx2-A9&zxXYDZ%k~N)*;spNuMWM$e1Y*EL zq)SKtxj#nNd6%J+%ItGF=YnRJ=le@CZT80#{6!wZ!FtiE6K>=hj9~We2<#$jBlzr! zi(HH?BqfqW{tPS;gTk5_uDcGW?QgpChDoT|M(JE$P|kabY^AbID62F3V0!B^F4H~d z#|W^r*1WdNijfsrRIO3Z zSDN2>tZTBMSBiVpx0zOQEq(_ZFywfp4@g}%H^Q3NI9@hj*yqBQ44qwk{yPUa=|MrKY?;-#$iZuNGvC+zZFWg%WyHrnXQ|Y zCCASX5knI}5uOoPNJ1Hn`GkTL96S*2xzm#E$l@#;$HMsYH8u*X{{mNgoXhOADytk_ zTnio)(+M&`pW}O(=+msTAQ8mp*h5Ybe>BmIYec&vi0<~r^>tccth?i$tL+T`7%f?D2X=U1k#J5d}&T{EyRC~o*FUJN<= z2nT+K*?yLtJM~H=`(CGLWn@-Wv20gdx5VH8mIX$D zRa2Y;Wh3=@cSV|A2fdIn%wDqyT{y!qyiM#E|6Z#^g1hE3o6rCPj*ul5K~$)a9?B;& zP#rpfEbTm9FL$m}tJ_}!Mk$=G-^;e~NG)|rY3@UkqUEUevv%*1f+_9@-PDbKW?>_Y zm!oiLu}Uvpf40;lJHmaYrLT3X?E4Y=GfivgKl-GPxh?ses`|53~j$<;R4A62aK zmXesZ<11XV% ztVr@O1ru~BxycI%zsO#Otoe`yH4AG_Ksk``)`fpbck37M(ZzOQ8-mO8DPzY{Vb5dV z;bwfXa~SYA@B()j7-kvv>LE_q<=IA>u*1KCK4?L31$X!f<+*~x%mr>a#klp+SE`cb zwsq$H7#JV9i8WyUVHMA#U^5u?&HsB_$fPCPdAnl}cq7C$?lM9MdFbToyoW{E1>4t) zRi7F4N&2U9NK@+wkpo+8*2Ok|J$_<8wiWbGnkPLneVG2KNpYO9P)o07TyMoaxntwh zW7WUTuAEEZ0-FnsoC>eQsjFB%DVi8c6ocC=f=~0l^ISx!ho7jQx~f>6rItxgrEn70 z1N^wB5#wzUlM^IJxOxjCQX+_6ju%-}tZ19NlkH#?Sg9_s$ulqIe%h7Zi)SXqr)Bat3dG%euHwV6Ml!h~3oacFx$Rhu_z38?R4JeXCa2#c(OX1YBS@mo zVYY)OF1|ZFT|aGY0Rj9H`a+*(*JR9YN!ns2 zDxr^*Ct1vV$CQAJDO?khls(}&ZrYQ4Va<%LvmEz`ONvwajnE>UgLTWHmV3kmwLdE= zu)n9i6+`WUxem9I>|K5tVL8tWm(@l^grsnF-D`f)RPR1_9Z~D$5#;6UpQmk*e|!fw z116^ffYd}HAh`caF@t|XO{Ay$3%ga;f!K}crK9`7Kf_bsCdZmDqc=9E@7DmcbkS80 zWCSG&eG_hX4hOtdZcHC1pT)%OkjZEii=nyEQ<}T7F&<8&xIdYzs0<~AR_#qLIBKsH z^`>Q@8#ypDZ_zNKq>7N6Da?eiTAE*|nRIcy+$ATU7nFxNv>jrb0Yv7dj}|l~S7oTn z$CoRLHefgFAU7*pN=gGn?Tk?Uy#Z%!b(-t&6=_M4>}Tl3XNToVsrBsBtjrdM2t=yF zq>o#b<4xGJYYH>!y&S%??KbTs58s4fLu1D?%}hCK15t;$e$3nmD`9-D)YDTN3&V^d z*rlbTaZtyUY>Hp2ut@f&XSQhS4tJi_{?z$0c`uJ)lsf~7GLNMVXZ&2ddJ^hn^0LTQ z7{4L0U>;C`(OwF(cfMF%xYPqfe2IS3Z{F3_GR&e={Gh6_3$mmVTvp9_q&^m>*3SF~&Lxye?V*d42E`r0s zpV<180lj1Q>l4x89YLI&1z-6ODD}kT>^U{2aia7bwg!6O>=goJeQXH$(GC)L0a~%@ zU~u2*-IDy8AlKj?QKGi{jKfRL{9k+tJB^3uLwV#1k>~C+Ns8=PXV1y*_9NFhpsmCm zE9HWly(Q?!1t)s~7hj9;)$j5*5c!0Jm5%EmJGE~2#_9{GZXSa6AiqkObrfY9_RSm$ zT7iXqszFe2%LS?I?_CmmY7Tu6*|AylwNsnTsB}~6vAbYI{SiqIkGm{b8V-}jp7Vnu zEncHudw8n`eGTPI-=lX7yH=XPW8cka5XLSfdoKne9*|m1FJMYS zDP))`Lt)e!HTyv-d^-G`&apPy*gGND&7T{^%`S+6cYx9klr`c#c;{=*N?SA{Or^5H z>Ya~o^jxZnt;)2?7qH^>p{&*&MWxrx@eb9*-BB;_{N$x3H-Qobq?Pi{^a+NNhom3Qm-FU+AxMdMD@!2OQqgY33;e?Y7B~_y>A1{nJ%LjMcUU#qfq;LXAU{sm>eVd6wX*vx+j4e01~n3*S>5 z&-i&X%4oo|3|^Wpg?{rXRLA~k4top?!yl69C^{ch4lh_@EfAt@lt=+`JcI^$yt%XA z{^{I+2KdvCEFe%BBK{YE>aQ{!6RK7=>nw=xJbweKII*xIe~!RA7C5nWU6}{1#3R*` zO_ABE`aHgmrF(~GNoOF5+HP(DGmTJ3Dy)rjbK~Y`?De=xds|)8-}3^278-e@>v@(m zhxhHknS1LeN=qnei!A!qI-#51PS3ztLVJ7{)l^-%b3S#=>@E4~Pi-J|kxk|=>LM+W zx_D*1IL4uHMM0Y>V2t1E{3kPzs$c@l3_M?Byg4^gIUqGZ6`@69MTp56eFD;`fti7& zHiHT$`CHu~(=A7FEhofpBqCY~Wys-0Ei0=@vB?^)yQ{@9{+?>6%JGXeS5P5IZpvyR zoBk1M==#MGs#{uOgtrx|?^NwZE;Nz%>F2EtV*(o3l`B(^`Y4RilS@(=V0YtYX_;rr zbxXvC!)udPfZq}y9PXbpYOeL@x2g|Ln_Wf@PWRx=OxAjyj1_Pfw|}17UZG6!%RA0R z2%2=JX|9jhTl+{T;5>^<$bDiI+K9=c%LS33{{@p(h(vZ)zjVJMP7?@n)~rvIHEivt zmE3Ci;#{b`XT@D-OoYFEHj89(eo?%#PZ;VM5pq}@B9cIeR}b+??W|31;ti!ZdsxbM zdfqh8cBp3<^=QHDM~&s<94S0{$m4=x?)#6yiwJa~$p^`?cb*M(cgm0$?e{6^?npp{)=<5oZPjCu4y(+E@9kJU02 zNhc+ko0(($-30JoRP(q(Oh+5`F-pa9N(cDVi|Tn4iFX{Qi7`o`Sn6=9v)XW@i?dii zrZzwCuD*2;H@g*s@mAMqR;-u#o?kH*5*)yy34;Jx7VOtB(%-iWLDG=WW)gl2Ppr%< zd%g6_c9>Ikkrp+h0a72R* zm8uqX5ee;gKN{^)A=f5>+lua*b%8p*RV%WomEFE{@pTjsM(xMEcJy!$A4TII^C}~U~S=;MVDQar^YPy+8u|mKU$C1@V&yF21#S#4{#le&GC&j@s&kEPawRKpt4n5x! z+u4luSQrHax2~zSW(~o(wVyFW^$TYD@Od6cxG%StJ;(on2dzBU_y$Bb0p$NYQu=pr zJGV|3LwW0->jKmFk=5(c4LMPSoG>qJh+H}bL&O*^@%$FCQJ_@tH;&-_6rlEI)^e_( z{FQ!J?_i?y0SA5=xN1@dRb}Pbb6%+li}(IoQbj!(?MZ#=+bPf;hla*cE#{~=d?k^t8H0!%lXRgiM z7ZpUy(7Q70m^KmP6U*)vkW-Q8Ci8UMY&uSfT>^v^H>saJwT@&(r$-%(jQS0m(kGWL zFICek(OoK>i@-8E1)Y=yZJXx_6*vS59lfM*L_5=n)*J^X-s4C}_QC^l!9_w^2?c-o zLr@SZQio&<1-K`>v);qk$zVymkC2<`U7}?w#X3oERVonXCS0UEiSlfs)6NWR7C$2vy5YLE ztJBAnR-d~kC=_uh6xkiyZoxSRcEuY_z&LF9@kYQTglU^_^;oLrqy~gNXL*EAgS2yh zKh%rYIoc?;g}|HV&|tl~&4f=es!lq|OEB*EW>=n^VA2@6b(KE8m`%6pleL<~?^5VBJ*P;KbHW9B#anEU*tOs$Tzm~Ti_)ds+SG|p<6uMq6qK5KwQt@PqG4M&#q0^2FZSa%Ps2eK%>Btz{PyOK!k2XQ{e{r zlXcM!>@Tn#_T*AgrJ|Um%XDo~2>z0GQu8XzlP|hcQYnKE?Z@$VG55dueCXtz;arM# z_u1<`Wa+V>vB0b$;bCz|+Z=X0Bz=HLWZ{l@fNLn18PGT=H&TnOE8%v1v)BY$O>h5V z5%2uHP2BEdmY7S#Zd$U@W%1>B%XFz-+R6Qg?YsQ>U^rVim|vbQQVxMGD8#@Q)7)&f zG99iJnJaU64J@etu&MuW|Yt$0SOtUleSKs$>*SnR@cfF_{{oSCV;`-UM5$c1o{5Xb{JZ zb!-wdE}HAXOS(upUd;RO2LE$@_2O*Y?+&2r`llY>w}0ls{vTEO{^r%v|A%GTDu0A9 zIU)7-3g3c6?X5wslkDXkt*6iW|Je0mh$y zwpMa$#mC7wIBG6NuJNO#i}=kTxXK|2zL$&a`bjikXI71a*hC_5a6w^-VBf{3NpTz* zq|9pJY~%i)PBQ#|FvDN|{6DK!_dmo8)c>Ws;jf7F|9UI@m4E#oW(Bi69YRzfDhvZX z%Kz(Lkk)f>G_rU2dlkdKrsDo>*S`Y8z@oHyKOG{(wpXqTU41kuXJ~xF$ZCKXy7V)$Y1FqMP?ZCg_Y`J-G;Gl z+UvVN&b_!T!_Gw4poc!k$gv1=FrW)8^!Mh7qg#rnyC_Xcq@=5OibMRoKII=et|stM zgTqtnCrjBh443Ttxfn-fJN)8zZi<3sxqOR@bZwu~%{&jYpv9Bq^fX^U>+xi}W=uZbkwJM+%oW6!+|nCR=2aZePmpo^XCDxC6?U z_9j6^ni>6a&V1D^%9*bRD|YIwV5nR8HwpYJT|HFU)I_WA0BL*Tk7u)L-RV zz;Jv|sc>o!iH0{I@K$N;xkm|*gjeje5H7bRf|+dw9xBwa?CBWTB?Cdjs`LWCl^e@( zG^Q;G+F>iLX6|M5qHYIkf=<&?y^k`5As8_-2ns=WXp~^@Jj4IIdY`Z~5kaw2qWH)+ z`eZL+Ow}@17%+lcEyO&{8M&r1R+@*Dyi=uXzR~z9ZwUVhRk4UcS-o4c$r#DRF1DC8 zs01!)zwan6g@JpIBZkPR8CnvM(9`Q&YJL~|hHvw#qd()~b11ffPwMP%hoh#jj;{ej zSc^Jju-q3v02}YRGi30xzo7$4g&HUG(k%q?31^uT2NJv;dUyM=e|t4cN3vL_+ywRa zQ!K%^BG6nn66A|+VGyUsZ^_h^V-plBX{fR>N1j?ci$sUA=p}I*dVb=KJNmy3L-$~i zZ=OK>rpL*J4&z}^+xJoM&3ICyhGll1{7{FlWLzY0zTOUZ1&%C`=W-JAZ+Cj~jDkB> zXX}zSq`0^VvoGsRe#HwKJKj7k$H~?mB$yv-dhg2HDND-#5{gv~OX8J@wp{a-BB2g5 z{R57~Ook-6oS-y}OY%C;wE!Ef*{YoUTD}2gtI=vh);eXqy)whkJ08n9tYJ6Hp9GEf{a_%%gD&iBg}LXOKk4+tU2HL2wFDXIx?r8>n0RP8La zuqK_l^;`*?1Ek@0juqYX{yX3`XM%>Jc#iF1u15#PNhR1_UWozo;YTKpS&c06FkU-S z!=%EQsj;D@z<03Zq*>Oe1 zAv#x1eGu-Y5rL(1qdArFyKAT3#dy)3Y%;6@YH(E(yG5^gq8z7w33BYJK|W*F+=u2MM=Tp^ezYBmjv^jqAQ z8(H8Cjx^9`L=kjt`gDwKzN4(1w+2VX%PeIz-B;U`u7m8J1jiH@UC7&}3zG`sj&Po1VxOk>yDJ178)G)pbsIQa zY%Zh=q{&}qsk<%S)E7IUzEe~3LWN{DVNOYk9qKAdeu;vjoNcHmQF(CJB}G*ns&pFS zD$a3|9xwCLSg$2E<$~6{gCc625^fVaVxTpw2MKWGck3T0(w7DXowViv zC3=pi-G@8ob@BbeA^5t6vn01Z@F%Qh;pd+gys-ASSl4X>tv`OAC)%&yp+=C8Y~k%8AHv&LL$!vF zSeR`AJAJVab%1K!`oVRpaV{Iv%{@Xog}c{_W-oadd_acPJ zY|TFS8(HHE{Swp|TN!gbU(hQJ9Lqgi*}9BYx#!%F$la`1ykCz5-+UeFrZ<0pS_Q;= z-Jg}!(y}g*gPpJc$m@YgxCgdYYfBlt8D`$!BH!_QM)^FP!E!Dr#qm!vxtDoKk2hrM zw=~Oq6p!%D*SGA}yMwadfTZsKKL(cnI&S|r)%O2ad9wdu;r;(rR`?HBUie>Sg}*8Z z{10<7{`;=hUo(yWM_Az#FqZX42mv7RcmsqKz!q==5-tITHU{7AZEPKA9ZiADPg(1k zS<;x>8ku~od~5>{B}BwU03bm3T^INR`1l3z6?Qc<1^}d`0aU>6LIc1-umC{n=3mP- z@PJ_dOIrkl0sxFN0`~)pS_GQ`fXS)9`p5u3|A;Pd1D^KppI|v4fA0<4mjn8b-XMS+ zuz$1xF~Ht{k8=RqcT*cj8wXPxTLMNJIslu1xHR~mvjgR?HtAnY#0A`l)o4H+_^~fc z%GslU_eT!^2^tUsybcHv02B!X3<>080Jt6n00aW$FZt_hz&{|MVBiptP|z@+V1XSP z5dokeU|^u&U=R?0E(78PJPrUyf5Z<_rZy^w%R777#IX31W+KMA^!;UC+NTA)87T|FM&7|Og&jAh& z4h8&0goS}c{BO(02Jp5-{8$6PfdOxDBrqfZFW~JpFHFD|CI=#~o4}gaYAdoN7)vRu9JTL<>ob=PQV>EQv{@ zoj1{2^#{Nd0rrGgVliG+GMkEBO?frl1~^?WmWF|Ix6??{*RGLt9EQJ}5=%DJ_P0u0 zoObXjH|i6F#f$Z;Wzq>ohM9(8tm4k>Gcw2rAgS^BMVrT?-!-GPSO43k_L9rm)`5%B zY9o#cYh9P0Fp7$=yPh!0cMxHeQEYw2yET~C5|F#pQcaznCH+AnyKj9YU3z*UA*5lt zI8A&#Tuukw)VDh8P`k}x%*RZTq>iqJ@l&p`z-gjL{(W?w^GR$y`tADYJUhU@|2d!C$cH{p~E8jX<}Z5_i=noE|uT$eQQ&NJ4fe9 zr_!B>#2UmR6_HroR1-h_WeJuc4f@zBSa5fYcXl!xTsUl7_puh0FpAPIqDzkQ!k!#f zK2hZP{?vz z`~f#=vnccDTK&SP2M6p!Wp30Agw<W$y97>mrP%klk9zhbF*+yr>5KlQ?cy>!JjxFH+vi;8I3CNZ zO-ptf@nlkQ-J7vb2IpD*E4m&^Z4X;$2VaG?b=>wV;$fH4co3T2LV563hzF}r)FZDi z-hSrU!RgtuS(2p7oMpsD@lB)Nn)F2ncA-Bq=5HKrqSv=y(-yx7)s54&8e8Rx(4lOD z3?X;ML|j?Wt%_T^99JN^gmy1x7d!IGa7XGz>`-fy!DqTh7SBCn{km>LF`ATtVNhH+ z!)(F~uz6c$hznCGL+2ViNPI)8~A^8?VOWAh#;8D9^Z z#-!|WpIPTK+~eW%d=vnD$U*mg)LT)mrrNh-4Rw?VqUYj&Xhbqe3Zp1SP~?RH{;_?` z{sA`%z@M!;l-G=Q&0>fblFQ2AS2Ti|GkVj5GK}-IBPk7TVJtK@t_afa1{8Gd676{ICgPi4M1Xlj4!9mS|M@DAcL=`Zk&mK);8l95)wl z{>}U3@CN|54C?;yvFHQv`yu~0&ZGt%srsFrg8VmGm(f-9K&EJK)0g9WwEJdQt|F`X zQWH_r*P<0cI3rlfdxP@`ALPfb(}79gBF$l&>#`zI&<5<#I2Sf4uLJQX-ia^NY=R)? zINoKIo0r!-Fo!8w8ZRc^OS~$)M$=L61droBJJ?zd;x+Rk@u>AfOBMtsj7)ybskrr# zKY-OAybXPNc~&*lepk&QJab3OJ{(lfyT2+#(mB>VB}y)7{0)H&jdBt=S9GK#@*=b**SOT z<&nlJqad~>oN9alhm6YSQK;KY5}NufTVN?Kj5}ssl(dSE@+NNxQ+9&o0%OmBxX=OF z^Yi2C*M_=Bay1uwNk;V%aq?1UcGLUZbS=h z%17p3K8UVi=a+Zb>n?7^@z|LGRXG^|4giNr-E!FbR2xh6uS@?f)LEx>cE^_&4HGrp zR0(lxOwC^j*K7pKjVd=y*7$eb7Ahe6|sS-U)tZ zCBvzpy9<}g;y74euO|2ft?>_Yhg1ty!@m`Z;e)(d%7VRc$R1GHx5;IZC6HBR#QC^? z#bm`|jN2f8!iSiCaq*OBlclnOH=SaPz5Z#4H-OJd@Dirsv);c`K!3)87f*rXKW;0g zV^4aBSNyv=!ttB44T1%W`gu5EYfYkYw9nU0c1+B>svE_}=CTu(>wbvL%y3D|d;T17 z@J1h6dLnuPJ%djOVDdy@hdA@8#n{Q$ri2Z2RDL94rb0E$L+W`(^?c37l? zdZ)tr`9+Smu6UXQ?e5Bl@+-ftZIvhitxCD>8AIuF2@(WEgkWa-u-6TiCZ5{-o87_g zrq1F3PnC*CeSFxBF~{)HbGU39fjSHl%65ys0kDW&7re%Bs8AfEfibXxssV7@PD{M#6VO=4y3k08Ow*Wkl% zv01`CeDd2Btuw{Ex_fIX57{;fH<-WSDHRjE3gc|@J^;k$Hjl0}*&hG_rmaUhop*X2 zqgTSmX!OBo5LMR$Y{+@Ery?D~0_4PNBh&Xsr77pbLT|WJEZ@aO$x3Ofyn1vRO+nDJ zNRYGq=Evo;;xnfY00~(C2jFh$;R7I1P@N1cdeEy|^Sv^-c$}Q1*(TphC-&^>z%FD! zpH!xzD$d5RE&uq0BCnf_XXOw0ea!s}@PU0*UuLrTd>n6~P0$%EQ`5YDAZxqBHgETd zl@SqA9N>|5{n8!o1HudgzWh!(Lt`k{z|_og!FM}K9=7jeVe`bgh3eA|goeWJ%X>l5 z+gqP4u#^giUrCnk(XYR9UHt@l=U9Bx3<2L+7Lo|$7jaXm)C-aozyL{QDNYgx|z3o z486L23_Vz-EsGs)4tqJZFSG)K6BQq~o~mh?Yob6!&{D#Htt6!aMbXi&Pn8oD;~`(j zvFQ`1KCKErW%vJ1IAN63G9;GQkf-vZ&f5X~-Of7)EIGs|LFVcL0>>xQFGD*`i;*eg zW5w{i{nN6@DwdviH?k6RP@sBG%j-i^MsV zpRo<9#YfnFs)?EbUmblm9MNKGp1nXY%X?FvNa#`>ly{lzSoV}+bi`tc_F#?X?%ak; zxP>b->8QpYsS*|i7b2Cm$$6iK^}wc9WY#k=5u0qCxxLutqf}nMBy-x$H@YFW^@lNpvRQy|S^46vpn~ zqQfKzvC7%UPLYbIO|@HEu$4#Z@QoQSp~jQ<`(NFnRIKlg8k)+gyGAjasOy!@dp3tX zM4Q+AW|xGm+&W_tm??NW&6@0l@gz~hHF2O(BZNs(D06KOVh)+HLw3XVInBZEATwGK z2=%l0VXd-S6v#L1{@FZHs}DU?zpam!gzCQML_|9U_6yeCbcjEaaFi*bWk?SGG>M6B z0V7YYo(l55P z$EuJ^Na;v;)YXj5IiN$Thv<>df!D`^X@sReurjyZOvbo6VXuRI7yzfnQVQ9xB0O=| zM!XEGgJSJQ&Quvv9Y_I z)Z7G-Em;z^;O`W?k30qH2VK-NGMtg->H@h}F~h>ayq)Y({=OYisW0+hi{r1)key;=fEDpJx-ZA*r1){e%6P&QoRe^=B^(g zTzrcV=B;4xef>8m(DoOnQ5*SDc#2|doe+C`_d=UGl_iaT=;a$UBM^20L-ia{GPa_U zlI&T0w#CYFB363erl6Y^TBG@iUr0d&@_4jT(8VMiDZY|>&)p$=qR-vOJ!a;PQVukc zMBq^{(h~u!OJLcx@QVFh0=Ic)j(yp?WDMk)=3VM;Bnq_kvd@8WERf{!W2_stQr)#cW?2R-$U1((dCmhxfmW_>`p@FJR(^G+@{+qe zZX|2Xr4TfNZDCB0?oGg|QA2s=4Fcm9f5n2#+A|`f8ucAD7u>(==Dn$SW4zU|;kdqw zP|KF$)h_<3B1yLJ-XGO}K9+UY+(kfagY*fVZ7*dX9a;=D(QRNs_|0NiA_5-Vpn0}dU z84TY<^kNQ#6_(KaqZtCS3CzeQv%?5bAU_QK6EHBm*AXI@vo?edi4u2g@Owj^mh75E z(xX84HNOYgfnSI6Kc|)StTxrm0O7~B3jXkg1CIlzlNOuz z&J@-`g4(UgHNJ0ezSG}u50-r!sCsJhP}vW&2bnOl#0JW-~W&n%UbN2X@~4o?^So9sugVos|k3^IuNg5 zf?k2UDsN|OP)&E!BAwnN2O_ZpU(;c~YNtHV=-^oIkc5s5FCm-*TI%~7HJdzSic+fw zY1fkX5rXoB3@k+}4m=x`UT=3YTJ%S6wa%7?=2m$)&SvyUc5$TKKa`A77JJEE)$!gw z@ttNktnrGQ88V0qFgiU95utD8X@>$u3eMS{qEEq=V{@DJ2|T%ynzS*r_wU6N-IBe| zjiceNhDRiXx%{@>RG7uvy`F5%E|d`**id^|YevCKT28W(-1z=mqiR;Ieb<7^*fH$(*xGT6>AGG=eGZwLu`RR?)*fmaOrQ7Opo7Z%gZ0;T921911G$8xJ@+gq^0SYvC zaKb6+y6{duw7%adZ&2{*g`%$kQVaTZ_B*~pP7-~fo;&u>9Meu z-Ofwi1sW?kHQ++03~Z{cl$f&H>k-vgzga=8 zxdwkcmnQdK?#-zn{l@yRx>xV3tnCg5Uyk9zXNcb9gE$07h97J>g&lbbSOT<@J%fy3 z%T~Qg9B<8Kq0X7lm!dogT$>ZonW3++PS0?}%C~SCds<1RQY4>GERoMi&u!&mMYzea z7L1I9Hr^ogc9=tY55rFIoEOdDRjt^sP6mZ76F=)jvO4D8Nq&{9Gtx`gnF1p*8#)n9 zP*=TeXFAX#za^=sIEh;QLaa4Icd?$kN-|efE=%S-w2`v1@ec%8Uqz#htf&m%#qi8WIR0_-^@fH|Z{ehjD5yfYt6ii^*MA+lk%v$aLE*svA+oR9|d9n+OTIr=Wb#QgrpNmh-^zD6UNyQYOG-&+H`8o@5@mr2r!G7XU&gW zz+%9dKe|<`U^xqe+Yb+q8s8$dladu*2|e{uXP8y+$N?VQpzyLCL9FLRrcK z=4v%iYN%r@P3F8?6k)TScQfD;ox9ERPsq<_+_g!c;5)pwPW0Ax z$-A9q=gbk^v8KLAI?Z^lpvNyBq6tr5M*x9(KRkHhl;*eEtqmZcMVXtP0|bV`d}CeJ z#^lr2FgNDiMgnfMD8~q#0evG&ErLI+0c}`e2)5M(MnJ3aB164i_@?-M++cGK>9J~Y zx4A7l^eLyRH*TZWa4N`E(=}FQwL-Nz>Q=dF`J<=H)TBmO6d*g9Ww&g3*vS3aCe5pI zV>KbZ&PIVY+%FtaAYsd8CWuY0ulo71F0pq5M%k=ZNsq^VQ&+zTdJP?!o#42l9q zY?Z}U^k^7l)++}KL|VL8+5j6fs)JEf$@0HKaGQP7nXI=uQW#c|3e2Y=dZ8*U*ctl0 zcwZ5!7E-6ts5GkiI=GU1g99G#Y}BhbiAk#Y3Sn^e!IN@ZbxWaeM?L$XbG7|W1eMI3 z9R|aFo2gqyTX;si0>U)9l9F=4?x`m?hjXu}IDT4>*v)N$jG#cX{7vkE6(~^fzL&hb zU#I$8`as4=!&|Qci8gUb@D`LLEZc2SAiz4$9VGUi#j6D_hLY`$EwouYW@0NLAaymW zqp@5igxi9B!9V=rr=h=0A_2F{kQwBeG-AEQ*+t}}fq?>bK0lMkezU{CBoS?1^EkDJ ztl%qr6|S4^5Iu45eA9Ca=n$xO%#hwE{AXunz^FoY%XGHM(H-JyDEFo~hA1`q2^=h}Jo4TOz<_!mep_Q?6s@zet z!-$f^F`T?!FV>bc=WEUr%5BYUGvYV{2Ld$q`oO}r(G76}1^QrGhyp=@FXj$3VB@(n zqy^(L6*l<+sdr%Z*b>22!0dKlcy*uK=BDo>(X6bP8rsf>8O;OrCJr`F#$gaRc1PGL zE%F-|V+{SZt~YcR1L70{7zQWPn#)J)Ytyzw2m!c!DqOmmvy~vt{D|#V^Nk2ltgCY) z0bxf&l|GWHafa|mw}Qa-L4m^hQ_d=(L>9yGhk<2g5RfAwaiS7LBQ`t2S0`r)p2ae>%Rr-lB!68V5q_WcBhvFVS|v>x|u{T3TzkE6LoH$&w)J7_m&zey} zh9I`5mQ=%Z6Sw2uN2DuGH%BEbdG_@J)z2P(VXnvq#Dr)8k$-}D?G z#Hsc>5s0c@ z#vetUCb>M$8vaS8WEuq`zmaIUiUN&r zjV*!$6K_}~yVdty?qpU;Dw14fJO~^2G9RcbBi;t~EdVN>AGH zbM2|mh@;PC{lb=a)9lTJBXp(|)~|8NUXogg2tKp?Ov}`Nb8vs-X=h{qcmmVyfN@`Z z!XCY7yc|2N8moP3kOy*aTIHo);SQVJ#0WGNK7|6^a%}aDAInVwVs{vbt8q=*ybFsl z_U<$8sx%@C?rd}FoN^3!GW7vuXeBXf<pPR^PrXUl4y*?`L9DZGVZ!cy470j|PIALFQ}kX0>TzBgVFin2nu}%|r?&Q@mD7zF zN;j?zmyf08?=>&8?)LV8c&=hRmb9F|Rl|l3WR{I+4L#s(jLk(rT z*8U3Rk}JijDo?PBtgql$f*2zmu)e@f*kM)5d`2>*{kWG@kyM5(<;J3b1{YK%R4%F+ z_TUYvfyMwrc_EIgaSh2oIFm6g*Cp*cP^yCvytRi4vdqzFJ;$gdF>UGU5&dbz4f3alj1pHDl z$D+{9x=5=?D5iqX()KV~i~@K#rWOtrYts>dno?|O$35Zh-(p+HFWb&|6Qz9?KI!gZ z6G7<{nZ&HuAw~Kwyvh!T?6mQRDH3*W5qk0`wV*(4+ps}dSnm`f2KE*K1ie!}V)SXV z&bJS%v>>cRXmXmP1a-MsAvw`*Ms zH0F!|a8T{WCZwjbKQQf{I;QOO*-FU-)<-}ThZ1~Jd~(H$RC57)>=mhs7#KD5v!}Sk zyg<}Q-(=RQI&x6g?P;F^9ahZMd91hcogiYiN^cW7No)PJZHHAxxavhWl@aB}#PA4E zZ98-JD3I>d@pS~*_A&hFm~-m&uA}zPa%5(l?FE=@g6N__5ih#D;7bulo>UQrJ9$}a zlBMKS+Gp;$g?bTt?uRCtgiI_p!?iZCyJjoV;9|Xvjfep!6bQ5EaOFASo-_T?p+)@b3xP&*OsscNVeyrQ-7`6G>O zxaDIn*{XI`Rq74g5>%%M{j zS?r~5vBy+H_2;4W(xiJ&7wj@B zO&h}yHnm*lIZ1H;a!tf33{G0x)p=$P_zz?{tRHwB@kh#452f*~2NjjxU){J?6n5#| zPmRr}M()A}54)j;8lgZ~qY%3WRW*fDQ8_rD}1jAfMshosF@k>^OX%4L;f5)Y}n| zHHPtEN_5P;)bN_TDvh4I*%VNaA~nY{?JW}G27M3QoXOP3hHOu_80Ooy9$m;j7@ACe zJ%=SBx<_)rd#KCW#KXby`T958w=F}vweg#7f<2mQ#M9FomPo!`udq(m4ckXCY-zP{ z=S!`^1v)-FZrx4UrGlK^sg+?>4ARz!x`419ws;WGrtsFb z6D*|qj16SyoCnq)g^aMsFUWi!sh_iN)jlcPLE^J|@WX))TsjDDT8-pA^9*a+n;|5Q zu^@lwd56ueqm1E}tY^j@w2pOHj1Ne&>{1r+R04AjUASZ|O|BQCul}Z9nyQQuaYZ3| z(NpfqDDQiwwa-k>5=BXSzA*GHn5vJ~Zm4ZBN4i{_iY%v}erFt|%D;ZLUDwf{7Sg3E z>RO~Z%o*2 zglswA}^)CS7N^ ziYU;_QhkI=s{f%B9MhJ^^{Sh-PgBP#bkw<#8Ca9V))ZyT`k^)F$ewiq{nuqX7Y3f= z6u_Pl`#)UMCn@zuABOqAyuqkZWqEWAQVKNY6j0Q25HF43F>YHmxCPTYxj&|Lbh`B1NIOTJCBz|EX7(OVkoZPOyKiuw z=s7X2S9r`Crvu|zX!YhG=gtoGJfDv#}#Y&RN)k8)tkB_OYT-zxU- zQme&E>u$MLqqTBao&Iw8LqMc)#j6~6$n!kyjq6;7POVpLPrv7B=`?UZ5+v9A?0M_; zQ#L7AO7JByA#&MB0SC@T5}r%CFPH>pn?@N7F<$cD$06a+{jBa6bI&sqqEuQW{mnht zp>O4@j1@k!Xefm}R}{?=i@CF7IC*cP2n@b|+&Gm4dA^UB$?YsJU2st*UR-!`w%-kb zc&;GM#=~-HVV140g^>&F`qhZJUX&_3zV?)e`|*DJ>#bVk+EVK^x44uPVMKV-V}aqF zK*y-&Jh4s;ctGv@CPQDDCm$1hFEPlpKmLkNNx8g+`yc{yxG|l^)=`F>Z+hWa&u3e_ zX({sYi{XzWpiNFGk6@i)C&liV(gtCiji^0$T33=}qw$G)zc}nY$|jNg&v>`2zD{IJ zJCM#|$V-bDsRgq=b-m2FULh^@4u>zIhh}!4C_jEaoSO1mG|omi1)?DDDh7Ov!~Vu=>=ps_)x?Od7Bxv1GKr-_&w)rFc`i?K0~i`JtPhT_eo4pjB*fQ>bma zH1=f^vZX}u!N8jvxNq)bsQa-^dI!uP6&_eKvhkI6D0V3(lqXlI>^IT7TVocfoWj8w zhsOr$y(_~)lFTNB9n1~ZLwsm+M!PF=dfyFl zgR>Lc!ovZ?WHSehPKE(hHTT#ukc*(E256z|}LY>BE z6rke9PDII+3TlJgzE3~jBkc*sVM^N1Q!l++ySPJ3m60;Iy6QrL5#H633}`~%&%fRO zO^Um+MJIE(T3Np|=Q7i1>CK3QE_#*HaPu52{reKR>yI86vE9*@^SX(9R=9tK`XPZT z`CLsLm$#iyI4vtAZI^W`u_EJju*Ka`aD_98S&RP+urrjc#03^Dv$7q^*DE8dq6 ze8^D9S4y2KbRv~3g|gj#PHiqP&cXLyy{@x%{fy(-IG-okrS))&225xTIGS+ zRqaHkW8=gIXAecjQi5(LM|D$J`I{7F!If&I$zfb;`fhLZ2k+w#5j!&k$JCj3wy3iD z2CCn#jb-X;I>}X`UWTW{Dw-+-4QL>!CsgguWrzOk`-RKF0vr7X+|LHl^1rY;+pZn( z302@mO!+@^ewZf1-nJ=zOq2CQA^ojbSWZEw$V!*sO4$)BbQt#;RQXF0LcBfGBSy7z zYe-&iDpwO5THLoaNc%#?m{jx3Y4awcP>H;=^ysh zd*t(Bo|h#EnyL5fd9n7erHc-F?x}%=<>TX3SCc2*YqNi?yYzkGjaVSCWt|(?>V6d% z2Y<+4x>%UF{@hUiL+vs+vhyJz10Z=MgYa1pQS_F3@rjuVBSG%bGXvXq7SHBKVq3j- z4uYeKit4=IZwZHP1&g?k6vtcFr4*#kk1{Rb1s4@_DYolpW@HNWa9mcywZ@P3By0nx z$B~n0T9H~UC0?yFxG7h2rJhXI+W|AXF2dP>skEqu>}9HL+zXDdgK)Kz!yJTX7Y+nvio$p zCP~#_dU zk_}}_QI@bCZ@6`hCI;E;C7YM=#PV7?KpHvdLl-?pj?G)J>aMs<#`;?*9d0+;+pc$Q zxGvjfSS;;vbVN+DoedA-Z#d8RrslRAQ*&q3*1eqohl-@3w)+42W`PuAI?E8C-nf3N zH#-Lt$N$z_R-cOP_zfb^^pWf-;k7Lc4!+QAffyQUi$2=)FK1|NEjaOUx$#}kkPOgq74snp+qoo? zOlss;q|9A5Ul}%^VE?FkJ{_`hS%BQ$L7N`=DGQcu3Q^st2y;!fwo(;+l`E>N(cb>~ zy09zexU`@TrxFnv=VgiQ|t}hu0-b;LSlHHd7twYL4_`yRSk|+Hix_brp>z`sRw5 zc?$8|q{;DaBzreR?ql5Jbz>fM)HE7#1@B}kHddI2kCE7r{iMA`kwM-ST=$ZxG+}!s zcgIXp&9$%T<_UIKDUOa30oKyVWg>61mNyPeRVJ*<@lti*9{n-&FPCTnXQQwN0u$@` zzj)vHSX7tMxe#C>gRc9Wr^Wu!&|vxJGnPA0u>#0n7A_h$OT`Ru%pBm<%vI23wErH? zF|u_q`BAhRA2HCFKKwj70|5TtefdV}$v1M4`fXvN?zelz-F^@D&6&lnR53J=+_{@- zrjsWt=TK4sv(+wLRo>?EO4_sAO>Z9bw}@nee|Z1?Dqp+IC5>|J@xfqW&At6ddbG4` zles%xZ3LIfS(KEn3|?6fzEfpcpW+w(APQfin7csA3NuYSyS|Mh&^KhJ&z+8w_AIDG zwYrl`IniWk0Mw!VF}0e*TAFT@)smNGJj51DJe!Hfpzi4htA=I%*VBq{M*|sXxYS(E zE6BI|0)1fdFoR*|sJq~=WssEIFhZq9bv5hL9MbTDPcdR4_|dq-FZBhiIkns_+ZzSu zuX~3+=w%@!e$Sq3-%o!;MRx|G7Cmz2=bAS1e4FNXV(R(;`S28tP-n8%vV9kgvbVV= zja_6ww(326bvC?ezSMQ_U9|Aro$kjS?#<-_LL_R3L$)qFC^nu`d?fEKPlARp`MRzG zJCp)=B0K_z6S6`8AN-$9Z+#D3MhbYQBXHu2=y&flw6pvF&UxM#s~)M?j&Tq{lZsFV z3)K{3fi7)jUp^5Lnn>O%>b3E_IYd2PAfj=&=4vl@UE|T`grchL$;r5|&ZWKlwI%H+ z8u*8-x27re9CQX(SYt*X7AlHDDm!EX8f22QUs{y-hwJV2CXICHV2iR`#$Zr*W2zo( zBVobB@=21yVALS3sOH+$`?YB$chu2JOW-O#f%~2U)M{>_2J~&*!#Jkdv&Q5ojYnf2 zSJ;K;J^Y`)Tj0C$KJz+!84>htI9x=V9lf>;(P54|8sex;cIzXZ=jSW7K5TSaK@xAx z?@MYNGQwk;1HOrpv%P%5mTI1OZ$3;;f=T#=A#i0twlBF3yDk}=;&kBYMb zU6dcJ$tu@wPqb3H6Qf;+WQ;y^V^HYQ)~2G*(hErJ%?^BNbxB65(%9R}2As*>?OHOe z_cSwRj-w8{NE-b0;bDZXB>v3QxRUtp)t&st4gHBYx5XXbnK-XTHNxykYa#wmxu+Y> z=C_NU@2*x~FIO99Ni?3w=zQ>n$S~7Og;?h*80dtVDiSI#| z{oNrC3!E8*WhQ7Qy_SisUiNrQZkxq$=!9eUj7t=9(T>q{NYB+1B^6yKC|}$J$5Ns2 z|I?DtF0p~m112moNw6}YmB|6W|1pgH`O>*>YvW{MZOmj4rA4tGih zPM?8hfd&a&bokqA2bk6T(oXt$boF9FO2tH_Y61}G`Uen*?6+hf5Cb4|^dNA{nZ zJ3k{g7ZK%Ctgdr_C@D4bFWN86j|nJH z<%Uicwl+U1{4Zg1qe$uz!196r!}8Br|H#_U%KK@si>1TkdR(3kIL&Lo^GN^5&nux{ zh>JP7NRXVCvy=x)qX1;&<{tz-(cd!hA04lsS^4$NU%EoN0_6rTFrT z?$<@!#p?~vaq*h}jQc63b`f{+%C&Qxy3Rl2ehL6y#9h4E=NuHlq)f8lq(?w+{FyC@2O&P%rZUwD7nE#h5}VLs=lLt_vw*KE zo&N&&TW9pk0>9~$UM%_FqZ?fQ75ZHJ^PkW^Wc|)x?XLd@eXghZ@5#Mx|Au_7QF)Pk zQR(qK&x7v2k^dt`_nW0J>ONj1|8D3%t-<5*8~Hyj`y2V!kIp3z|2_Gh=f5GJ%Lra1 zUlijzw}zP4Z{+`2<7fB&n;kCV{@w&3uYSY*tqJ~qWc_<`l;6K0|NRW$@5yid{|))? zXDWYBP73%p7oQhi zBwUQPo)eDBe-ZwOy!@;^d=7n5I%JDn%D1m>|n2A>O2sEdF{=` wK_Ho$->mqbNYBOOe=fd%Prg|9Tk`+A6jzqV#QO0!%)o~&9w4mQ^y9n#14n(zH2?qr From 01d7cdb2ae22c81cfea685ab17c61250118e5cc9 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Thu, 4 Feb 2021 16:19:15 -0800 Subject: [PATCH 16/19] Add files via upload --- transformer_models/pooled-vs-unpooled.pptx | Bin 0 -> 53429 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 transformer_models/pooled-vs-unpooled.pptx diff --git a/transformer_models/pooled-vs-unpooled.pptx b/transformer_models/pooled-vs-unpooled.pptx new file mode 100644 index 0000000000000000000000000000000000000000..df0cf1cffbefe010a035c281183ab533a3d1f2f7 GIT binary patch literal 53429 zcmeFZQ;;WL*XLX8VwcrrySmIS+qP|^%eHOXwr$(CjbCx<`M)z0?>X~K%*DAo@8m^h zWbB=hu`<`%JJ)A_SISC&f}sHY0D=Sp0wMsS6wUT-0R{pR{|N+y3H2#mW*t4-AAn2MFZ5|Nq_o3qOI0#8C02$b4tU`0cn-e<6p!D znbEk6U=N+D3QHI!W{im#r5OZkuER*GDiDNL(8io^-8f<5`Ysy{b=G0jmn=n@gQ7W^ zRduhLftQ^o4wNS+p~l?(JAX5!_>p^eCX%@wX6fb&P=gS(`S1k zD9O&)xg!P(EwV^x5}KG@aKY+L>6ak@bJ({6gZnaw;eyW&DV1*e4@5NFbvIGH-8TW@5Y&L9-sS~Qv`#Y?^zCeLw z{~rNw`r?=i`rWek9nUb|0k30kWa&Wr`(OM28|eRqqx(OZUJgXv=u*T>raP*L`7m+P&f5(^(|NR*L*O>Vwl_k>&2Lyzf0|bQn-NV+_ zk=DV|%+Sc;KMwT&nkFx_G;G#~ki2+$0PucU_CA&!&l0l5_|BtEytKj>PlYF-M6*q@ zVe%A1PO=Z(*=lidR%%fRf{LYSUNtVOSSH8&to7~n?`P-hW-KjY;e&=r*>Iy&oY(~b zQi;r7H+2H&7WCj_ILKXZmjPxdpsK)@_vgdw`nU z=I8BE1%~7he)TTCOc8$4u9!otgSy`cwI#-reg{UCOJDOH=RVDMG+k?-$SQ z^4z*TT^ViERaI1#_gc(fM;t3r&$^181MR}k(z78hFYHE-jOJ6w*LO+@2>X?W!#AN; z%f&Z+dyaRiC^PcBlzfaRgkb^^y@8)$6!VBmwS|(Omk!7PH^qX)f^H2n*w|<{Gmc-V z^@A669?WS{=~vq3KOGTvBd1=HK8+xY2QUL6XORX|EKtZV5Xhne2cY-_-kgcROz-cS z-NNFJQqzOCS9F~F)>ae}Y$Hp{UVZF>Nao?)f2I`{n7aw^pF_FMkc^(Y){=LNnaun0 zu;HNL--$1c{$#3?Zj+R6EVYJuPr|EPAe*>rJ|z+DXDuF6gJNh@qi+$bo7ZE&^bLxC zU2})i;e}*$C_lg|W3_)%PB&;C;44mG1L{Kn)g+mXlo{hyWoQ@ciaa0OifD&q{LO#Ns zPBB0~_qsRciS9kQ|HhJKeC`%MvkNO8rBI8jEidjh zFTh>N6Z6fDDzlu=gJ)a7cDvmJZ`*&|lHtp!^j4w|oCSQUI%cYtPQE zD1tVPThmEfYhHN3olV(1QL>I44}J{ZVaBg6Me+Ku2shhsyj&0B#d_!iZib8LYJ5F< zXY2U|NlHU+u&KYytN1Uxvb;TMYB+*kEWfZr5I+sr9GK`3yJcptdqAQ&loIxYiF$f% zf?hvBSP{{npw#-Y@bPv zq4|!$%B`HN1tSY1YPFg^`;fN=oOi-Mb9%D1k>N$1Rt}LWXyIHAz$)Ch#q|0Lrr85o z2vYX}J4Frt=)ipq96H*v6t7B&DHvkR_rqvqA)Q}O7&cU}e@z*4p~=766@=>7$I<|s zf)$Xri;Rf;l}F7KAs_%kqfx-HUmWYWFKx$H5Ys8Ox}NY>O93#iIlx6|=ioqsh}~g{ zK!~*U(Ht-+An6y?r7VQiFE(5uL}+AyX2ljHAxf)ecb?IvF|twMR!EP#9xXc1FP9p1 z6(PWs8?JD|gvy~9poT|GF-Jp;N_8ZTpxJghvrYzbkxh_4P#IM09MX6?2QX>4Rgxu%gmJ-=r7 zR>z_;?=GW}o~cD%R0$(Qj_?wQi3o}s92+Y-7>6ubo4CteJl=^j>(^KcW+2HLACy3D zE@)$g`$maU>uHyKa<=yyX)w3OkI<@-kNPRRT^|cyLK0&zO^9&ADZ>a|6``fAgdyHZ%ELT%obmgUgbY8iCG(*l> zO9;L(qyXEEg)|{q3@Td_wJ|Z=-^Z1|y=c9Lv^%d&(10~C#?aPvTju z)I2uFas`3f@a+cj2(^ZclOaW1=}w*#I#ref&GpX!HXQh?!0f+EY!Rji;Qai9NJPTy zExK#;#J?KZ`XWMVCWYet=qnm3`t>4QP<{%O0Jm4?Vinc&tp|N}o^)=!=?(8=|HUp4 z+67;P<;n8Lm5E=6Ad0@p0K8CB#5Ln#b-cgdinV4^7^$SnLs6mDgRz&x)}{R^0G&jP z5-KT`^=hc5qM{C@xBTTHP0s3OmAJ})VhyqK{O}w-pADa)nQf+^6^V*Z4RRQdBBxIg z#zr<0fDJ27Mx8V_IMjr7ZN*z^F?pP!K)LKHV@IgOC{Nx)w!aAE$us{%n`QRqbh|#@ zTfKa6sQYP8_i7~*Z{_CR;&Wx*!iz1wFNi`2`-zlS$6tEayX|XF+QbjpH63%EB>cND zcc<%#VEYRxYrUaTOr^MdAHhYC?G&9Fg8a$g%eTPmLI;ZoLp9KF*(ZHEEOb7>4}SGt zbJ`)=CF&~qcw+3`Bkj1&JTfh1ujdEaFI4&i3>6>wT?EDQN+OOp$LxAYW1e_OJPkz< zPBIkiXHMq^&f`YI%wtKFVIMC~pU?9}|0? z)QZK&t^3EVKW|Hl%5O`6iD0SD5EjZ2L)mpl%_0RyXqGuqpqob>)_UdvzFS$xZdn@p z?4M?w&xpL%kgN1sKUWd^prm1im>jt+qZCjE6yLZWP>f*j%buQ|Y=*G0l(Dh|#eufh zs@6R(HtsHd0eY`f*>#W$JaKx;SI5E3~AlxCWp)SxkCty;t=$w z3lXd!s=vo|tE^AiyU8;}GX_bgV7pekNKv!w-{}%j5IG)J#!_>05V~} zqTv1t)W%VoK}-_LWI_tU-fh-sghEA9?cz<}uBC;nnT!EWZtW%;l~=Zrzk^E@6J8-L z+SfX-Znomq>S{Ou9bLbgY_&!7zBhapJ^9fpEdt#i{X4ptJzXcYweO{byw)22w7!LH zel$y}SQ$|{^`t6`suFqTl=Jz@V*ava30&ZSwoU(2Pt}`X34+BkSZN_e$MSO$Q&R9N z%oX>K(N#eInF&DZsZd#21V$ ztyFL4CWnF}M z43B?M!-?s1)-CM;zpBV}si|cf;|76ptE~AHeE_#OtlqyR-D`1;amlWmc@Eeyqw&n9 zF8`5rmG51uyXcVp{6!C2nrDAIFY0=H_dx+6c=6VsEn|8e(6Xm8f0M9N4c*L5Aq&m~@M{BeYxFr6KewdZT&@2sj~3Z}*@>HaQD%-? zZZz8VoRP{^)|_OJ=WlGkdDN>`lO+OnW@QoRwHmD$cWRKz0gg6$_dNnfQ!)pGH1{Iw{kVD9d|aYZUE1}&39Y3shsz}Q{ny8 zf1mF%{71ffZoNN*`l*w1jenv7N^dcIvn8`i6&JxMlyXDhn3yNgdNGJjjZ z`3L*cHja=Bd*ZwT+hnj|0zNqV4sc_G_xTdsJ+(C*ehXxteE+<+f!*W%0xuT<$ZHe zjyWa5SG%7tRSQTh2;$PeNwtN#x-Z#3@ysN=diCvm{;bctul;8gd@%33%;&Xlce2j1 zznp`Eql2^K=SNyy{=K2pslN~eL^H>u%_8j8=XuHCxyOVE-LZ)fcTE3Z>g|d1!IIzQ zqq{fTX5y8i@=@|HL2OOfs{vRE@ zAJ$Brrp4~P%yfJK&7AupuQmc~Bva&qAxY$en)#x;7KQ{TkQ} z*Bf2$Yk)R_rL|NfO*(4R6j0y9fZmNvW{jk`vdo`Q`DD30_TqKV_2$~%i88c5KF4=! zK;j_OO;qScica!GF%%e)UMhG{TWM7j@a`3Iemn>ZO!AIwpf2uEQo#kUnu1|Vr->Y| zve{U}zd;yYOh*I8R%7SL_UrhysKlr@*unjT+CNDB z?u)5rr@TdpqMGCPCUkVDubqb%73_aoP-a7?>izQ4DaD;7=%x9ypV=@oaF*r=sxgD@ zmXqrS;Vhu~*>GIG1$$YEh5b)8gq)I;H%^P08pvSLq7T!-B`$~19$kDC5zQRfAlyu8 zZ9inOv}zxF{e)%c6p2+`kp8x$2+uTV?=dqc%!k><+s})oo#(%u?JZqSZ!8TE#e;@V z(ctyX_{k{}l}e!YrKl$Mm8M0QSWYT?TmH`-nf-?c%v_b=?IQX+C6@BUhc2?-X~5_U zi^L%ZO?Wzn{)#Igt>*;Y;O07*caM)I_}3;LzC}_s`ffxODQ9JK{jdtuuncT*q6)v| z5)guI1@}`5gUrO8A*p)*f|Ped{4vAnX~|pL5e}CZr(>jQf+x^q#vp<<&1F5A!8ODI zPF<;J7gTXV7n7&T$PxWxg&@&+C9V~KP0u36ED_N72P{?2eO616p$Y|RW=`JL{E83; z3@SN+bw^&9YRTX{Q8(OX2<@D_2}t?(U@UI}Wf=A`MIC?PNNb8cK|~|x8cEZHb`7{& z2*Tn@V44Bt^_!Pq_!67qjt%0b!MT&Jgyj(0HF=XE*pOld!T_wEE)xH+91_d{n2Q#* z5piTwWQe6djo)ZiW&irsG6$+KhQUz!J+3h(Bc&O9{W36P>o4QHe7byTiAy>WNx6Mb zI^PQWok$Aff)M2IDFP?x0nq0 zz*vf+y^JE!R-Rr&_9B5#`X1v{)FD7Yj~O*gE-$N;!ue-oL85=mjo}9U@g>lJ=r{IjhY%;qpO(L2Ej+C_I4^Q9{pQ4IpuF^r~T@)GN?Zp;NCldF0S-FRWnjsfcw|E z#+Y{BHKJ{!>>*QP1t<4O&gduw%FK>V+-a*C9js-7XCp>^o=pZwu1` ztM<~`EUadOPG``sYjyyAaP-&5arg6=e=$l{6Z*~9P6kc&^0KM;ZMypBPk7JGb9S>L$0+izSxb zR!d8WwYKI0o&K{RqCzOi1QZH;96q{hy3Pt@KwKjIKE2Fp6U*0@LXn~=C3-q5hQiJC zS_VaLknGrNf!T0;kyrC{*%X?VQ`FDD(#U88lCXS;M-+1d90P?%&?rqniwz=~lMQ<_ z4*B(g&jBh1ik(wl{v7E~9B;K*eyY~P9VAX|ZC`Vt#CN`Q1HH_=84kgOc?JV{kT6Ih zu}SM!-5&CWntr0U?oEEhIlR?Di5Q&=uIuu_kMhT%S$z!bpEboZi<++D^ojCxhAknn zbz4ATiToHFfhodFg?p{-iq1zA{f!6Ha+Fs>$Fo&ccsvydUNo2xy;u^*r77>AjWM_3^Apa1p zQ*KOb1*e9XqHn~X^0u_42Aq-ubC1@8uhi|jmuUKY`JnEmhT?DF65WA)=$-cE8ky#< z=q@7SFUCGq4dLxp+F zkBLwdtXb4G=OnGTjc5!VvVk0NJ}Hod^voQsKut#_UNejEZ%3n834xG2!AygTxlQK?|4c*{5sw zh7HO&&T7=3I_@(!m6ZCbzTol{dPL-eu@U@3AnqW&?)%spQQ30jU)>>0CeRiWs#KGa zV6z`X1?-!tOve()U%T-|?dG?C4v|BC^T%QOW84NqQS!wo*NPG+*%dTzhP6j>KFe+n z^JQW%&i}4FlC+&tnPNEyf3-k5@%Z(o1L?T6Do~(+Q>;uchDLH2#)&Obc;y(Rw8zs%dv^66rO z`e0}c0&@Vy<=Q$?=t4rFhC|2fVF*aN4oGd)TZ^R$+ewmhq3QD>rjkk5c>3B_? zv|DX0j%HU;Iw?Q~QhGKr18t1nj_ND;HpJSS!6W=7QXsm8pB?3;&22mm?m5bV;EcQy zRq!K=_TLK-uh8*|6>LsUgHH>*C)S_`6-#%$^Xr2AI%Hy8SI=Ov-*x>oYAGu8ij_hKwj?Ehp&1Ndc zrJVY=cF^FVwwh+AI@&UA@7^aX#TG+!I{L0%_qdHW=hbsZr!?-fNoqG(G_B@5>*)T2 z5vPgDUt?w`$Cd{4T) z{#clJSGN=0QyI=p73P?P1U37l8uzssY@V}4$X%+lGL?r(#i8KC-4g4b~iJ0z$ z{?-dY+-Tk4^j(rJ@(HH3h2XP=a`(h)uIr`;XrjD588__gxaf5AsNikw<7+3ZfLH6$ zP$3*h)l4~KUNz5~L-;Yeg$vvi=n)19)N3xNVA3TMU1=H0p-f}GX6$QIuy93=FbLim z%>^;(|28^P6uj?=S|}=OqV&X?nUJ(=!kk;J;@c<^{aVSj%FH#On~wGPM9g)uP5R9i zX!4u7D5~xUMfY2MVI(zTPMZ4UmKFE^2h;z-BL6i_|3?Aw|2NY?geHW4eLMRa7!VL1 z5G3%w9R5E%-PYd7;aeoB=V)eQ{XhNvTRsK+t&{w|;J^0Pu`ny%Plxhj$Gz7X@_4K0 z(lLyBGZL~LLGT>Z@X`!RGCpR&Yd^l*l@Tyj;Lku|{zPp02G7F?&GA{ns(<(`hqY1$VF9UdgkdA3 zBAj-|v$%49cYCqqmOR9lgm;eX7mXcNp5pNyH zhpQ>!MvtnvaE^_OF=r+o5AR>gC=?9m3&khSSCovsyS!8AM0~w`c7L72-#drvbNH@rOQwCCp!9uRgxcLZ3ND0!wb-_&#Sq%Ez|&YT(93DPHtB(GLrJ3pRl zovE0Z)QjP&#FvkESgr}5b&OzQ0QU4*bf`7bujke}rqE|3OFV5vuXm?N$p#m^X^)DX zvflFVUS%HN4L@_?lDj)UWuuK1a+X~bjgxtkZsnYFsnEQVmUnkdD_;S3bnea$$~vz$ zEzPr&w-_N`qu#gx2Y?h~S_2tbvk4jcwiWrGR#BD&ZuP&mj>QVzDwjLBTT$v;Q^Ou0 z)9)Q4BK8}Rhv`D~*o#VBe{jEuq)Ji6#^j(yLJY=ZrCRtXj5u;Bof1a9U)Kb?(idEF z9+s)12=00n+A^fh%;hqfb)?G1uk2`Bf8ZkQ;!MB9e)c0shcZGV=8=fgFO$o&31Z}R*h>`6)9-u)fb=6+@EwSV76vNS|S(xR(6a{~*ejP%3dtF*T!L2+|FhRzKM zx&l}1$y%E>jpuOja0CxK+5+D-$La4+f6MUS;Q{oGM84NY3bAi~LI8RmP;*`zrUlr5 z2`#zks`@>b3@Or{-GWZXSgHG`hMt?{NFybpcj%w5b0+n9s*Rh>HLaITsZ|~=U-gVF zu}VfAop)uzC#{q=R60{*Wm^lpcIyrrT_my+DS-B)A?5jR&>DU7vqOtDusjSXsv<*E z5#NSXYLmN0e>USKYK$)0-B5cnBNHQ45C!$U?XGtw6o}`!oW{8koraS5cKnL6z+*Qa_ron?q5t4JGafkeUP=gbwE{p6K9F_t%&;m_wxFqs zUdtZn`cYu~VWWuW5+qPa1_-cc*^Eo|(&kM1&PI0t4mCk6y&+U(dI!M52}lzYZB+xR zH)!Hvc0tXiW5D8EzHuv6xq>>(DE|qw2K^_@>b@W=j3X!@H`*dk%qt<$TiO~;jexE> zN759bY}vC!)%%YqMB!T$q8UcyORj{%KBZt7C?4xTf5?wv;o2Z$nd+|_ZgI1!Xz^hd zRqAoFi%7$PEbFTx^x#vu>`PONqtQ{k5{HPPV8@T3DhCV8bHXOB@0$U%9ab1oFk598 ztyLq`gRu_L!fk`IS>y&FVc<19Bb}Ra>?J3~Q?}e;qIC*ul z6Q6s-g#9CCadnth3Ye2?fwtK~d{srNG$1~V-A}(5);?`mR3wck1G>CaHUjCeLlZ9_Q1R~f~3nWhDZ{X7vSI*ai}Xh)Tc+W zztHj!V;)T&+Ua^+ToRu}86$CI2N%NUVLJRY zMD8jO|E3CpxxwDTn->%sq++Z9=^PYc2BiR4RHXp~7y*QP!YzwKI3(!Nj$4l%&@NCP znGapS*s`3ctqxg_6m`DfFp4dD;RxkaEX!}cQiUl@Ac~}RVYX}#fzk%1B4$xO^uA~U zUwfZjZQT);-Wt!I5C2>x#9cp&1G)#e{u>G{d3;z(h9rW@aBPI2A+oM>X)H=xq{B8H zwjNub-FV#=!Ci`)xJqkb&&UZu;w4J)N_Y)jlp=mgHV)} z%h@CfWS|;l5(Yl@!E*;k=la$ZrUy60A-y`sV6TesZBOW%7ioiWQ^r`Hx7@jIGm221U?CasR z11f-a$CBB}R%HFrsj5e`ilPy7&@L`d)Gj3P!Q@AA~A_d?f`6l?(>>Kv9E&W^tzz|f1bO+ zm)*_hDGlNIsY=UVfhM#S%i*a7ekzCr;6>frVZonj6+oP&*Ovx%UUgpD+IJ_<>fwg(DIW|l__9O(zy9S{WB30(Ywxq zs6>uxD})fF;5xGD*02=u_A9hk4ny&&a_q=|%xljr@Dht>r0b|t*8oElD;&j+eh)?Q zZ$cXi2`rqQA(4Jow+{<#F?{k(>q0-(BT z>Lacm;@-vcv8Rw^0I4{}m`QlbR`vqL0_E2)65_qJnPJIP9&YCe(KTx~Lb}1R3icgI zAZ*j&uK-HI#hR4e#FG0>6g4*`lzilt=#GqoZVl|OW1X)q{4c=r*9Yyp&dedO2JX$3 z?v|&_&X+gKNuNmMZh!D>MaB_p{x;WqKnj;wGLQTFmAV=_Fn2Y!f7r=c>Eb~5-%xRsu|{$U8@#*gp+E8ocqOt9{HU6Eo+d>Gq%Psf;94A+q230 z0aM#7v5DS9sSuBd=66S5IBR-?vOu7ed!ps8p%1uJSKfu z`b+c?-NK$7-87+vG^Wktrg+@4U>BPos6lBv`H+$*!#6K|sGP?N z-V~3u*=EgtcCt{kCQ&5o zmvQ)=<39KB=X4OqYPP4@M7=TG{|vs#z9Lsirr_chcxM-WnY&LXK3tz4WhsubNiz%7 z>I73#qmzpx^CHo&=~Ed*>KIok*$x$KXS_^KHP(HAJ2Z7y#y&GkAie)7>xJecZGwZs zB>f^0Ru`T##}xW;t(1dzgzCWWdv==i2X4NyT06(RbspsG%@u-VZ5f~(vnEN2MkLkv z4OY44|0h^|h*^gTsWB~q8G{j$w5Z;w-D6J1Z(paG4;``$XLVs5=DZ`>(GY>0x5mQ-(+kVCu>N=kucwkd1%inbs!)o*v z+yL?NaA)rdH?1uhg+_y_uQXH|OaZFQ$2sk1gD1nA$14OA_LuytQtsQhV*zj@xx)!%a>M9+- z!H+*A|AVadUE6YAEw_Kq?Zpy0XFZ&_AYnF%!>8jeX&rL z_M8>iAe%RPZrLmjbMwrPHrt+tW^Pz8%ibPqozy^oe5z7IfQjHK%QgL3l2D%deF-oZ z|B$BtLYoyyBKnh*uS0F}_nft3-KyE|q8>cj+U($T34EHXF#PoG3*z6Q09Xdq)}rrv zyzuWLT$FEk^e@o#|3*v~$(z>cbSOU8WS2NGHe}*2WOAa54F+ioVsi7`J}}b$=9Y%( zOs5OZ+qIy+1!)TqJJ9RL%XmW>jNMx*3u}l35>xUA+EOt>s#D={odqYS^!3PwVu5PD z6eh$%5QMXjdvDFIT;<@g0TdNI3m~kLQ?_AE;){&Kv4A8+Rqfj}x)`gLk{Z8K7dvR* z?H*4d921UU|gZ_Lk%`{Xxob6jy5mh)_ImAf<^1 zXi?1r>Z2*|iRobxGCj_kr-sK?my?Yz>MXbcK0J4fZ>7_r`%3k(v@Zm9=zB~aITSJ% zJ=`x9!6x&}9#g4Jd^R&wLw~F?+)?eyMc=qV?DNJ&@rL~1#2WG?kx05XGpo8^WivQV z7Bi(aEz2o1k~%vn@`k=sL|)q6&WL&nVVJ0X*Q@d!Y0HK1;1 z(VGL(wP`BER|%g;tk#1D>bf|2#@XYL4SBswIzffE{qgo0200q161_Ke4@MqJV}(NI z+lU<);9e2F-?U(@7j~Sc*i)e*;X~)MV0snExBv`t8och~c|Ht)DLGAc&hh?hEmout z2PI98EfD!FQv3&`sDc@=5EuTm1mQh82C&(8iwh7r-1}`5x11R;2^YSUNk@TiiOe+v56tFa zAKQ*ys;;$<;3PFmuplY4~4xvf)~n{+l!i-Bp*<1j+>m1*3c|qw3RO}H@uuPf%P>yC)OFNUB{1}vL2t# z@3Xz#7AIf7MS#y9L^Or@mnMJ>ZHd)M>|)xbiqSX5ty*PI4ZGO``OYwnViW*GVS6dg zN!yCqU0LJs$6m^ZM`Iau(vALnjWV1#Sh+*#`}MNYvdmhWL4%Xuqj==%N27%OcBv^~ z60gZ9U_iOk?Wr}aU3IB&@hDQgrOI%UIuVEJs8TU4f_GaBX6T_}^OLg8$eCoKt9hMD zPpNQ;@v1rVwx8Tsd%ZH14ZdkxdfuZ+BUCt=^4F5JHPyt^>*)z|Q@k=gHCphZz9%5S zvQ|1f<}WGe6uPjJqQ66QzpnzD0Ec5z1Ri>42IpFcxwZEMgQ!1~0KNjFk7gK$>)N`wrSy(uN!6h)O0z;C2)2OIPCDpJe?1ew_io>3vK^5p4HYNt8s=&V zBgigUOlm(cl37AvdQudGFwGjsd4v{nI5glpdZ@nR0zRcHI6^XA-s0ZvrDvg!{fLW=QN>o0KYI3yq~K_D%Jl5q1~0Nc+GARccUn!>h<_d>$a^TfZB z=xeqGnFVvoXB(`ud*nBS2|_*rGvH9d+#w7|K#6AJq1k~11kUp~d5RcgWqN8(?T8ol z`PY!qYKD)I5f(|!&ra0Sx&GKEI4cDgSQG1al<0u#XxA6-7BorCakk~n9f_t~^f4Am zyR-o{XwAHex2!Q90w^&00u+K15vYi)Y%3~?A8jA4qk-a^11-o;PxJhdWF^T?N(|?h zMXjztE;szZ(U7p@a0)deC}(4IKTrlCU5JC!{3voMT)lc0;iJb^<5SWZF9x)N__u!& zIq4Lx^y9-IfQRFU_*7#``jHoM!(O#ZRufnyIuElXPiKFA9^MywOV4Oc(0$xWs{JYSmiDU{&}nd{JgQi?5Aj7)S8?6o5NU#Ow8uPX-%oM zNWkMnN{g~IH|){dtjEV4?z%^3%kWw{+6^2Uv)s=eZg>c(-404666|=w@cld$u!1&-5IkfAT2V;(002lW6Cc})Zik+2l7yZspflwG+<%&_a$K{?vt3^1 zt_gRL-b`{HYE-nC50Ewe>w&<)hfK2?Y7|=#tVp*uhd$pC7kqTDKLse#)!YhY3*#E& zMNn{Cs{ydSWu`8bu zkNH9r8KN*i*Di5H=j>cmwcpf2e`VuXE;fVz8z58>&!S}&uXt1G+kCy*pgbZk;9Y(9OojVP^?>xayDpS_LX5(6U-)$;9w?bO#;LHUT<9m$D6htgiR29|%1h9L z*oz+p(A@WdE%ZV-!@Am~O5|@kjIE) z0@Ku4*Tp?C?N3n1VXVv_qAj^$mmiQBdIek=5>nV z+#_gFiPzq?-$F@FotXI1Ml|)6&&**)W2KS5FVGl^tfYZK7{M+BvMUjD?{yK@%cN&U>J79(q z7z2tY1O-Vh#tXKOK>TVf*@8Ee0uC?ZqJzp|@@yZ4yOR)#_@tHsh7YF;BrmT2@hBqT z^9amHW`}Jc=v*vmLvF>?W&ocogZK4$I6v&|)pM$!eu`H`ao)sgP8QVa@+u?_)VW_vHyX?=W$nQ8s> zwM^k8Qm;+rJJ+So4fWWCCu^AtS(5TH_8 z3-Lmm!&H^=JI(bgb$Ec+iIdTM`@CE0xLd6z3+l8`9DjhqIdzmIT@3Gf^TT}C< zx9@Kb$~zY=a{kghYsadQbhw+q=>9I#h+9!5TD~}o>rk#_H*9I+yzRBKVA%Ni!kNQV z=CDBSo6E%e3LCsAvuH!oWYCtt!7S$5p?k7q&dcgQk;g= zdx?p4?-g2>mV!8aSfRUes5Q~h1np;1k}nBlV=ij3b2H@h(Bi&;A0*%jJ8(5z6PW&( zMs)7HR=!bg)>den36}|va06^OVJDMEw*p{Bwi`Rn%m%PUX2FXSZ4bBS87+xa@Xw1(uK#}uN5Vevsf;-%=z zwHP4Qhv82!37TP`X-N5+#e}hS@;L5n*4NsS^$J1rlkfp!xJ^P0w;uz7@Dq=UH^2ZR z4Mt=Z913ywv8bxK$i;Y77rM!;`qS{Pnu&jX0rsWthBU3S?8QUfcH)sm@jmwEg)S1e z=qCc@TM{G^4yZJpWW>C z$gBx#QbkWF#r&9H)vLgLLar7c zE##V^sd2NOlip{$%OKuR~=(`i#8?f5TK1M#`lFY?C0 zkb)EE3#XK14^QN)y*@w>_TQtN*I|yzXGj3x9q&I!Ip)7mPVHYumK_#LB*JovGX*pF z!60T|CXT@<;&3b$w@6mS|1q*$B_th`*(s2jg9Ayz4v!m&KD>IF`jz{7zabopUEW|; zU-mOQhSgQSz(|u+&8>;|$(1NJ1;YwGQ9MD#sExN(YJIf*2hh>gxNLN@$Iwz9p8`bG zZG_*GhgX$96wR|}7YHwOm7X*NxZIRc=D}O1*PG1=9r|kRDTL8yqsb}D1pS_**)4zK z3ZKsjOBPml<`#=Tkb}B!4zI8lZE#0#XcO1)oz1UKU(j0GwsO(V)Xd0G5iRzSg4<|S z>os6t2Yl?RWnma8_JDO$os^8Ol`Yt;Jr{^8EK;d-v)oniSl{|0lWk?01 z3$l7?R5=>B8$7PAMBLLHNat47U=0nvgNwB*b}QUbcOFHw_+V6L4~6G{^V$_|DSU9u z&~2kHy)I+DXySBqJR~{2Uidzqf6ivvKAUG0e=f@-DbD>h0Z!bMFo-*-BZgf}ULjy1 ziHG?KojL8QvqhYlwwpA+3%YyIEOtEa+u#6ytD78H{tK4r5%PcxE zm zQ?mHBHok7Xq;*i*Rk|3>k`EGgBS!k8K?vXEtzyG3+vGQSet87bD95hXg0q={=`?`` zAS|s}hi^X{GNX!osp%&z<&A{=*x)W#*t~EeIUTH5?g@jMSj|_QfLy?B4SO%IZ4w>hM5#A*mm^PoFc)aDlEo425rTd9a)bNIqSC0 zF&H?;uoCu#REn=(M#joYL6Y!wPiOPv9N#$|EkFD-`V-b_ zJ0>1*5}YV?;r}*_Wo`$9U*Gofty5>&&^Hlk0@G*dhPu+Lwb)RTu0_Xc89>nDXXlr= z9oe4{%=!i$r%XYziwukOuCkc$X@@si4r~o?r-il>A9*-qMe)+iWm&n8glbO5qI`Mn zq_OGQXHogJYi-OumvXJwSS)0h7y$~rl_h>8e@>vuKqDns&ou42)uQ~=OzpBPT4vm3 zK?U3FsjWs(QpW((1#@9iOKs+ z-lLa{S~@Y6B<xJyVW*O$9}p6>6m z@nf`Zjb4N8HJvuF8g|{_i@SV@S^Wr%{0xXy>|r$-FmJejJsX!aD133&;JmVBJzrRK zTU$^n`WPa0jhUvBUuPxzu4X@ug|OBsvb03CV>0H`axnQ)!lP~$i_#`=eV4BeQ-9fZ zk>`p3(aiIR{?Fbe!Ea|bQ@}{-4KRlBpG7HH{=7>jlu!S}OW%2QF9}S$F~w1w7|qd( z4B^4%$V%p+bV)hT%GN2?8a2TG7Ns}_zDv~NKzd`~o`~gx58u7G=F*#k+v@HG?v2(b ztS%=Szs>Gh$+0XuyzHV1)>_?#u|ia@iA-f0MPd=Bi9S%DeQ(w*AGV(Mx zOE@tOAKz4#xcMwO=lRu3f3>LER?_AS+|%iC(PCe55-dJ2GKjvvro2BlRyZKCxY7EC z#mp9)xAW=xve{@^cC)S66&RoBjjVk)kvLi5Z!A+xY%wj%x589vg~~Od&3X3JLRId! zK-4t*HePZZ-_;%`aHLiLq2+vfA&>Rr;YHFcG0>R~Yi-Jvt}J!g*oI|_`{vYa+$1=% z;_g#t4ZH(30q03^zR^xjg(u%^zHF$r!G$R+Gjhig` zMq3*5+>pJwn?A^_MjxDhg>XmS|0s+kM2}bdS4iOsf=xps`jkB-9F-5sm-db;kUXc< z@g^JA6Kb2&SgbGo0IBE*-4p8Wr&R1Vp1TRNQg{Q^&2*CaH@So{_hApx0;8S&NJ!eYo?NNS zK)a(1P|h!VAMhQ#AlY`L72{!t`s>ONLcP@qU`U@Cu;u_p{70ZjVxMLS17`+f(~ZOa=z0kbPJGntQ`$(gSz)GdZ5Wz>e8i}_mf;9_xTY|(Ak7Z(mVwR@4eD$ z5`S4%T{p+uGD`oDSVg0g0^Ks9P$l~sz^J<(XN=(%K%!k|zl&H*JKYo+2e$G4(+;LG z9fNh12W&SB{Y2N_T`MuLX7lny_5na%jbaPx+V27}6y9z~K^#Cd=81qOClGR2$n=B{ z2>xI=TFp<{MNeTsd9h9?Ak51}Cqm8F=z@%ODF5ysoGa`85du8_X zc#T%T*~-jYo7<&eI!gtB07LgX-W1d=XM|EM@)*g&-TEgBq8IIq58{G}IyFQ$d(CUpqsfjGq0Z**zW9~|v^%Cuj2VoFAwW22w%Wxh#5(QL zXzxwE%-QNJYwx&~1&mmx>oKz|2D05jo%2OnYlb$q zFT1O{ZK_@veI2Uzj=|+xzbiwFh*#)5XEZB%sj(NrJ{meXl1^f2pGMCSx9x}61fZNIK4L#}RH zL)phR{Hrr=PIcqA3bSSKSLVSkPHoEKDyfUOUH9=Hq?T>xQSraH#ky;YR5vaZEtE0* z9NL}zO5heKyOor6 z_-{kzn`o4H?>W7c@6UV+MzJy|Qs=A!J~NnS8n=GS;5tI763K!rCn9285bA7twDMreN0e z#@osE5FvVvKc5(~bOt%uhbb|hRPr#}w_#2gLCs~%7U{*vXXTL$@r)5md>@NP>@C2Q zgpS??pIIyN71U~uTV{nKrA2Vh`wp_0W5Whx7|dgtd#P*^iL#NIbjyBU5)8&z=B8Wo zG2o`61w;Do`dtt2f=;i&m#e()V;QSfeb=j+UvI1Y^GA+3F+HvQ5Mx3BhzLwEcEWdj z>b|;(=v6YZr_*w);zjCaAoH>JsA9+R0+!;!sI(~Q#7aSB&dtTN8Xr-{`+MIF1-Tr* zd_$%$?B_e-0Jjm$DedysTq6+t8S?pPU^jZC8eZvs`#H;Yw{YDq_F0j(K8XL z%2c|Y&)qzhj&p64O){1C!psX&q=tD#Qg|v=I8v-XcQfd@H)Odaa`Z*1p6NkOLGk=j{B~w!S9NwM{&vLAZQiqbXM$s8HuMOm3jd0Drsy3=*JBN(`<1<58jNq0In(&`jTOt`v9nECc!}AIF`2fm z2n#~r5t7vDNL9XZEIJAsf~^)a%quw`vk>M}Ur6puE>j)-L*K|7lQp1X6~_r}o_>qo zP9da=+p6xNQ;GFEh0=1aHET$2%E=0ih?aPxD6#dU0#tK*NBscaZ>{P1#{3D0+Rbk) zzhQ<3aKnyl*YS|j3vp7`iCf{*U)3YK0*DjnD(i*8T(@MyL&$D~mCI(d!Y0R^Ak!eo z!fXk;1U63a#^)TF!NTGZ6q8o&%0;QFOh)iuewk3o={sQg;vhYk@bpRK@xZX_v!I|F z(B2*)B$~!n+Yknq$Ft+viMTyyNz%7}mAz|9dkH;cgPYkhtx`^OE`BbpDRD5(qFTsLve{PRi(b2eyzBX`bctI! zRGifRc;R}H#2f9&<2d@`>Ebr%8v_0l&dv!FJhA7<(dXP??nRRd%VF*n9BuCr#IsG1eqio#4EK}IBTY1~+ZV}_EjTiu( z^&RG#-e`oFT1eay%KJt}?)-o# z-U@mcdtkzsXu_3ax?8awe`X(sY&iRpTxzOV6qRwA4lN0-J}WadFVcMZg1bo2eVD8{ zN@wdu@SD@LR!!*qKH)ehBJ?VmKSe=?>GQmC z82np5-+Z_>P8zX}oKG9A^)6u!p1UcyN9=UjlxiuQ+|-Q(-WjWDS)*p~&ijkfMCt}X zyV(H?A6~XokObs}aQ0F=8=SR$_;ubI!Fx2-A9&zxXYDZ%k~N)*;spNuMWM$e1Y*EL zq)SKtxj#nNd6%J+%ItGF=YnRJ=le@CZT80#{6!wZ!FtiE6K>=hj9~We2<#$jBlzr! zi(HH?BqfqW{tPS;gTk5_uDcGW?QgpChDoT|M(JE$P|kabY^AbID62F3V0!B^F4H~d z#|W^r*1WdNijfsrRIO3Z zSDN2>tZTBMSBiVpx0zOQEq(_ZFywfp4@g}%H^Q3NI9@hj*yqBQ44qwk{yPUa=|MrKY?;-#$iZuNGvC+zZFWg%WyHrnXQ|Y zCCASX5knI}5uOoPNJ1Hn`GkTL96S*2xzm#E$l@#;$HMsYH8u*X{{mNgoXhOADytk_ zTnio)(+M&`pW}O(=+msTAQ8mp*h5Ybe>BmIYec&vi0<~r^>tccth?i$tL+T`7%f?D2X=U1k#J5d}&T{EyRC~o*FUJN<= z2nT+K*?yLtJM~H=`(CGLWn@-Wv20gdx5VH8mIX$D zRa2Y;Wh3=@cSV|A2fdIn%wDqyT{y!qyiM#E|6Z#^g1hE3o6rCPj*ul5K~$)a9?B;& zP#rpfEbTm9FL$m}tJ_}!Mk$=G-^;e~NG)|rY3@UkqUEUevv%*1f+_9@-PDbKW?>_Y zm!oiLu}Uvpf40;lJHmaYrLT3X?E4Y=GfivgKl-GPxh?ses`|53~j$<;R4A62aK zmXesZ<11XV% ztVr@O1ru~BxycI%zsO#Otoe`yH4AG_Ksk``)`fpbck37M(ZzOQ8-mO8DPzY{Vb5dV z;bwfXa~SYA@B()j7-kvv>LE_q<=IA>u*1KCK4?L31$X!f<+*~x%mr>a#klp+SE`cb zwsq$H7#JV9i8WyUVHMA#U^5u?&HsB_$fPCPdAnl}cq7C$?lM9MdFbToyoW{E1>4t) zRi7F4N&2U9NK@+wkpo+8*2Ok|J$_<8wiWbGnkPLneVG2KNpYO9P)o07TyMoaxntwh zW7WUTuAEEZ0-FnsoC>eQsjFB%DVi8c6ocC=f=~0l^ISx!ho7jQx~f>6rItxgrEn70 z1N^wB5#wzUlM^IJxOxjCQX+_6ju%-}tZ19NlkH#?Sg9_s$ulqIe%h7Zi)SXqr)Bat3dG%euHwV6Ml!h~3oacFx$Rhu_z38?R4JeXCa2#c(OX1YBS@mo zVYY)OF1|ZFT|aGY0Rj9H`a+*(*JR9YN!ns2 zDxr^*Ct1vV$CQAJDO?khls(}&ZrYQ4Va<%LvmEz`ONvwajnE>UgLTWHmV3kmwLdE= zu)n9i6+`WUxem9I>|K5tVL8tWm(@l^grsnF-D`f)RPR1_9Z~D$5#;6UpQmk*e|!fw z116^ffYd}HAh`caF@t|XO{Ay$3%ga;f!K}crK9`7Kf_bsCdZmDqc=9E@7DmcbkS80 zWCSG&eG_hX4hOtdZcHC1pT)%OkjZEii=nyEQ<}T7F&<8&xIdYzs0<~AR_#qLIBKsH z^`>Q@8#ypDZ_zNKq>7N6Da?eiTAE*|nRIcy+$ATU7nFxNv>jrb0Yv7dj}|l~S7oTn z$CoRLHefgFAU7*pN=gGn?Tk?Uy#Z%!b(-t&6=_M4>}Tl3XNToVsrBsBtjrdM2t=yF zq>o#b<4xGJYYH>!y&S%??KbTs58s4fLu1D?%}hCK15t;$e$3nmD`9-D)YDTN3&V^d z*rlbTaZtyUY>Hp2ut@f&XSQhS4tJi_{?z$0c`uJ)lsf~7GLNMVXZ&2ddJ^hn^0LTQ z7{4L0U>;C`(OwF(cfMF%xYPqfe2IS3Z{F3_GR&e={Gh6_3$mmVTvp9_q&^m>*3SF~&Lxye?V*d42E`r0s zpV<180lj1Q>l4x89YLI&1z-6ODD}kT>^U{2aia7bwg!6O>=goJeQXH$(GC)L0a~%@ zU~u2*-IDy8AlKj?QKGi{jKfRL{9k+tJB^3uLwV#1k>~C+Ns8=PXV1y*_9NFhpsmCm zE9HWly(Q?!1t)s~7hj9;)$j5*5c!0Jm5%EmJGE~2#_9{GZXSa6AiqkObrfY9_RSm$ zT7iXqszFe2%LS?I?_CmmY7Tu6*|AylwNsnTsB}~6vAbYI{SiqIkGm{b8V-}jp7Vnu zEncHudw8n`eGTPI-=lX7yH=XPW8cka5XLSfdoKne9*|m1FJMYS zDP))`Lt)e!HTyv-d^-G`&apPy*gGND&7T{^%`S+6cYx9klr`c#c;{=*N?SA{Or^5H z>Ya~o^jxZnt;)2?7qH^>p{&*&MWxrx@eb9*-BB;_{N$x3H-Qobq?Pi{^a+NNhom3Qm-FU+AxMdMD@!2OQqgY33;e?Y7B~_y>A1{nJ%LjMcUU#qfq;LXAU{sm>eVd6wX*vx+j4e01~n3*S>5 z&-i&X%4oo|3|^Wpg?{rXRLA~k4top?!yl69C^{ch4lh_@EfAt@lt=+`JcI^$yt%XA z{^{I+2KdvCEFe%BBK{YE>aQ{!6RK7=>nw=xJbweKII*xIe~!RA7C5nWU6}{1#3R*` zO_ABE`aHgmrF(~GNoOF5+HP(DGmTJ3Dy)rjbK~Y`?De=xds|)8-}3^278-e@>v@(m zhxhHknS1LeN=qnei!A!qI-#51PS3ztLVJ7{)l^-%b3S#=>@E4~Pi-J|kxk|=>LM+W zx_D*1IL4uHMM0Y>V2t1E{3kPzs$c@l3_M?Byg4^gIUqGZ6`@69MTp56eFD;`fti7& zHiHT$`CHu~(=A7FEhofpBqCY~Wys-0Ei0=@vB?^)yQ{@9{+?>6%JGXeS5P5IZpvyR zoBk1M==#MGs#{uOgtrx|?^NwZE;Nz%>F2EtV*(o3l`B(^`Y4RilS@(=V0YtYX_;rr zbxXvC!)udPfZq}y9PXbpYOeL@x2g|Ln_Wf@PWRx=OxAjyj1_Pfw|}17UZG6!%RA0R z2%2=JX|9jhTl+{T;5>^<$bDiI+K9=c%LS33{{@p(h(vZ)zjVJMP7?@n)~rvIHEivt zmE3Ci;#{b`XT@D-OoYFEHj89(eo?%#PZ;VM5pq}@B9cIeR}b+??W|31;ti!ZdsxbM zdfqh8cBp3<^=QHDM~&s<94S0{$m4=x?)#6yiwJa~$p^`?cb*M(cgm0$?e{6^?npp{)=<5oZPjCu4y(+E@9kJU02 zNhc+ko0(($-30JoRP(q(Oh+5`F-pa9N(cDVi|Tn4iFX{Qi7`o`Sn6=9v)XW@i?dii zrZzwCuD*2;H@g*s@mAMqR;-u#o?kH*5*)yy34;Jx7VOtB(%-iWLDG=WW)gl2Ppr%< zd%g6_c9>Ikkrp+h0a72R* zm8uqX5ee;gKN{^)A=f5>+lua*b%8p*RV%WomEFE{@pTjsM(xMEcJy!$A4TII^C}~U~S=;MVDQar^YPy+8u|mKU$C1@V&yF21#S#4{#le&GC&j@s&kEPawRKpt4n5x! z+u4luSQrHax2~zSW(~o(wVyFW^$TYD@Od6cxG%StJ;(on2dzBU_y$Bb0p$NYQu=pr zJGV|3LwW0->jKmFk=5(c4LMPSoG>qJh+H}bL&O*^@%$FCQJ_@tH;&-_6rlEI)^e_( z{FQ!J?_i?y0SA5=xN1@dRb}Pbb6%+li}(IoQbj!(?MZ#=+bPf;hla*cE#{~=d?k^t8H0!%lXRgiM z7ZpUy(7Q70m^KmP6U*)vkW-Q8Ci8UMY&uSfT>^v^H>saJwT@&(r$-%(jQS0m(kGWL zFICek(OoK>i@-8E1)Y=yZJXx_6*vS59lfM*L_5=n)*J^X-s4C}_QC^l!9_w^2?c-o zLr@SZQio&<1-K`>v);qk$zVymkC2<`U7}?w#X3oERVonXCS0UEiSlfs)6NWR7C$2vy5YLE ztJBAnR-d~kC=_uh6xkiyZoxSRcEuY_z&LF9@kYQTglU^_^;oLrqy~gNXL*EAgS2yh zKh%rYIoc?;g}|HV&|tl~&4f=es!lq|OEB*EW>=n^VA2@6b(KE8m`%6pleL<~?^5VBJ*P;KbHW9B#anEU*tOs$Tzm~Ti_)ds+SG|p<6uMq6qK5KwQt@PqG4M&#q0^2FZSa%Ps2eK%>Btz{PyOK!k2XQ{e{r zlXcM!>@Tn#_T*AgrJ|Um%XDo~2>z0GQu8XzlP|hcQYnKE?Z@$VG55dueCXtz;arM# z_u1<`Wa+V>vB0b$;bCz|+Z=X0Bz=HLWZ{l@fNLn18PGT=H&TnOE8%v1v)BY$O>h5V z5%2uHP2BEdmY7S#Zd$U@W%1>B%XFz-+R6Qg?YsQ>U^rVim|vbQQVxMGD8#@Q)7)&f zG99iJnJaU64J@etu&MuW|Yt$0SOtUleSKs$>*SnR@cfF_{{oSCV;`-UM5$c1o{5Xb{JZ zb!-wdE}HAXOS(upUd;RO2LE$@_2O*Y?+&2r`llY>w}0ls{vTEO{^r%v|A%GTDu0A9 zIU)7-3g3c6?X5wslkDXkt*6iW|Je0mh$y zwpMa$#mC7wIBG6NuJNO#i}=kTxXK|2zL$&a`bjikXI71a*hC_5a6w^-VBf{3NpTz* zq|9pJY~%i)PBQ#|FvDN|{6DK!_dmo8)c>Ws;jf7F|9UI@m4E#oW(Bi69YRzfDhvZX z%Kz(Lkk)f>G_rU2dlkdKrsDo>*S`Y8z@oHyKOG{(wpXqTU41kuXJ~xF$ZCKXy7V)$Y1FqMP?ZCg_Y`J-G;Gl z+UvVN&b_!T!_Gw4poc!k$gv1=FrW)8^!Mh7qg#rnyC_Xcq@=5OibMRoKII=et|stM zgTqtnCrjBh443Ttxfn-fJN)8zZi<3sxqOR@bZwu~%{&jYpv9Bq^fX^U>+xi}W=uZbkwJM+%oW6!+|nCR=2aZePmpo^XCDxC6?U z_9j6^ni>6a&V1D^%9*bRD|YIwV5nR8HwpYJT|HFU)I_WA0BL*Tk7u)L-RV zz;Jv|sc>o!iH0{I@K$N;xkm|*gjeje5H7bRf|+dw9xBwa?CBWTB?Cdjs`LWCl^e@( zG^Q;G+F>iLX6|M5qHYIkf=<&?y^k`5As8_-2ns=WXp~^@Jj4IIdY`Z~5kaw2qWH)+ z`eZL+Ow}@17%+lcEyO&{8M&r1R+@*Dyi=uXzR~z9ZwUVhRk4UcS-o4c$r#DRF1DC8 zs01!)zwan6g@JpIBZkPR8CnvM(9`Q&YJL~|hHvw#qd()~b11ffPwMP%hoh#jj;{ej zSc^Jju-q3v02}YRGi30xzo7$4g&HUG(k%q?31^uT2NJv;dUyM=e|t4cN3vL_+ywRa zQ!K%^BG6nn66A|+VGyUsZ^_h^V-plBX{fR>N1j?ci$sUA=p}I*dVb=KJNmy3L-$~i zZ=OK>rpL*J4&z}^+xJoM&3ICyhGll1{7{FlWLzY0zTOUZ1&%C`=W-JAZ+Cj~jDkB> zXX}zSq`0^VvoGsRe#HwKJKj7k$H~?mB$yv-dhg2HDND-#5{gv~OX8J@wp{a-BB2g5 z{R57~Ook-6oS-y}OY%C;wE!Ef*{YoUTD}2gtI=vh);eXqy)whkJ08n9tYJ6Hp9GEf{a_%%gD&iBg}LXOKk4+tU2HL2wFDXIx?r8>n0RP8La zuqK_l^;`*?1Ek@0juqYX{yX3`XM%>Jc#iF1u15#PNhR1_UWozo;YTKpS&c06FkU-S z!=%EQsj;D@z<03Zq*>Oe1 zAv#x1eGu-Y5rL(1qdArFyKAT3#dy)3Y%;6@YH(E(yG5^gq8z7w33BYJK|W*F+=u2MM=Tp^ezYBmjv^jqAQ z8(H8Cjx^9`L=kjt`gDwKzN4(1w+2VX%PeIz-B;U`u7m8J1jiH@UC7&}3zG`sj&Po1VxOk>yDJ178)G)pbsIQa zY%Zh=q{&}qsk<%S)E7IUzEe~3LWN{DVNOYk9qKAdeu;vjoNcHmQF(CJB}G*ns&pFS zD$a3|9xwCLSg$2E<$~6{gCc625^fVaVxTpw2MKWGck3T0(w7DXowViv zC3=pi-G@8ob@BbeA^5t6vn01Z@F%Qh;pd+gys-ASSl4X>tv`OAC)%&yp+=C8Y~k%8AHv&LL$!vF zSeR`AJAJVab%1K!`oVRpaV{Iv%{@Xog}c{_W-oadd_acPJ zY|TFS8(HHE{Swp|TN!gbU(hQJ9Lqgi*}9BYx#!%F$la`1ykCz5-+UeFrZ<0pS_Q;= z-Jg}!(y}g*gPpJc$m@YgxCgdYYfBlt8D`$!BH!_QM)^FP!E!Dr#qm!vxtDoKk2hrM zw=~Oq6p!%D*SGA}yMwadfTZsKKL(cnI&S|r)%O2ad9wdu;r;(rR`?HBUie>Sg}*8Z z{10<7{`;=hUo(yWM_Az#FqZX42mv7RcmsqKz!q==5-tITHU{7AZEPKA9ZiADPg(1k zS<;x>8ku~od~5>{B}BwU03bm3T^INR`1l3z6?Qc<1^}d`0aU>6LIc1-umC{n=3mP- z@PJ_dOIrkl0sxFN0`~)pS_GQ`fXS)9`p5u3|A;Pd1D^KppI|v4fA0<4mjn8b-XMS+ zuz$1xF~Ht{k8=RqcT*cj8wXPxTLMNJIslu1xHR~mvjgR?HtAnY#0A`l)o4H+_^~fc z%GslU_eT!^2^tUsybcHv02B!X3<>080Jt6n00aW$FZt_hz&{|MVBiptP|z@+V1XSP z5dokeU|^u&U=R?0E(78PJPrUyf5Z<_rZy^w%R777#IX31W+KMA^!;UC+NTA)87T|FM&7|Og&jAh& z4h8&0goS}c{BO(02Jp5-{8$6PfdOxDBrqfZFW~JpFHFD|CI=#~o4}gaYAdoN7)vRu9JTL<>ob=PQV>EQv{@ zoj1{2^#{Nd0rrGgVliG+GMkEBO?frl1~^?WmWF|Ix6??{*RGLt9EQJ}5=%DJ_P0u0 zoObXjH|i6F#f$Z;Wzq>ohM9(8tm4k>Gcw2rAgS^BMVrT?-!-GPSO43k_L9rm)`5%B zY9o#cYh9P0Fp7$=yPh!0cMxHeQEYw2yET~C5|F#pQcaznCH+AnyKj9YU3z*UA*5lt zI8A&#Tuukw)VDh8P`k}x%*RZTq>iqJ@l&p`z-gjL{(W?w^GR$y`tADYJUhU@|2d!C$cH{p~E8jX<}Z5_i=noE|uT$eQQ&NJ4fe9 zr_!B>#2UmR6_HroR1-h_WeJuc4f@zBSa5fYcXl!xTsUl7_puh0FpAPIqDzkQ!k!#f zK2hZP{?vz z`~f#=vnccDTK&SP2M6p!Wp30Agw<W$y97>mrP%klk9zhbF*+yr>5KlQ?cy>!JjxFH+vi;8I3CNZ zO-ptf@nlkQ-J7vb2IpD*E4m&^Z4X;$2VaG?b=>wV;$fH4co3T2LV563hzF}r)FZDi z-hSrU!RgtuS(2p7oMpsD@lB)Nn)F2ncA-Bq=5HKrqSv=y(-yx7)s54&8e8Rx(4lOD z3?X;ML|j?Wt%_T^99JN^gmy1x7d!IGa7XGz>`-fy!DqTh7SBCn{km>LF`ATtVNhH+ z!)(F~uz6c$hznCGL+2ViNPI)8~A^8?VOWAh#;8D9^Z z#-!|WpIPTK+~eW%d=vnD$U*mg)LT)mrrNh-4Rw?VqUYj&Xhbqe3Zp1SP~?RH{;_?` z{sA`%z@M!;l-G=Q&0>fblFQ2AS2Ti|GkVj5GK}-IBPk7TVJtK@t_afa1{8Gd676{ICgPi4M1Xlj4!9mS|M@DAcL=`Zk&mK);8l95)wl z{>}U3@CN|54C?;yvFHQv`yu~0&ZGt%srsFrg8VmGm(f-9K&EJK)0g9WwEJdQt|F`X zQWH_r*P<0cI3rlfdxP@`ALPfb(}79gBF$l&>#`zI&<5<#I2Sf4uLJQX-ia^NY=R)? zINoKIo0r!-Fo!8w8ZRc^OS~$)M$=L61droBJJ?zd;x+Rk@u>AfOBMtsj7)ybskrr# zKY-OAybXPNc~&*lepk&QJab3OJ{(lfyT2+#(mB>VB}y)7{0)H&jdBt=S9GK#@*=b**SOT z<&nlJqad~>oN9alhm6YSQK;KY5}NufTVN?Kj5}ssl(dSE@+NNxQ+9&o0%OmBxX=OF z^Yi2C*M_=Bay1uwNk;V%aq?1UcGLUZbS=h z%17p3K8UVi=a+Zb>n?7^@z|LGRXG^|4giNr-E!FbR2xh6uS@?f)LEx>cE^_&4HGrp zR0(lxOwC^j*K7pKjVd=y*7$eb7Ahe6|sS-U)tZ zCBvzpy9<}g;y74euO|2ft?>_Yhg1ty!@m`Z;e)(d%7VRc$R1GHx5;IZC6HBR#QC^? z#bm`|jN2f8!iSiCaq*OBlclnOH=SaPz5Z#4H-OJd@Dirsv);c`K!3)87f*rXKW;0g zV^4aBSNyv=!ttB44T1%W`gu5EYfYkYw9nU0c1+B>svE_}=CTu(>wbvL%y3D|d;T17 z@J1h6dLnuPJ%djOVDdy@hdA@8#n{Q$ri2Z2RDL94rb0E$L+W`(^?c37l? zdZ)tr`9+Smu6UXQ?e5Bl@+-ftZIvhitxCD>8AIuF2@(WEgkWa-u-6TiCZ5{-o87_g zrq1F3PnC*CeSFxBF~{)HbGU39fjSHl%65ys0kDW&7re%Bs8AfEfibXxssV7@PD{M#6VO=4y3k08Ow*Wkl% zv01`CeDd2Btuw{Ex_fIX57{;fH<-WSDHRjE3gc|@J^;k$Hjl0}*&hG_rmaUhop*X2 zqgTSmX!OBo5LMR$Y{+@Ery?D~0_4PNBh&Xsr77pbLT|WJEZ@aO$x3Ofyn1vRO+nDJ zNRYGq=Evo;;xnfY00~(C2jFh$;R7I1P@N1cdeEy|^Sv^-c$}Q1*(TphC-&^>z%FD! zpH!xzD$d5RE&uq0BCnf_XXOw0ea!s}@PU0*UuLrTd>n6~P0$%EQ`5YDAZxqBHgETd zl@SqA9N>|5{n8!o1HudgzWh!(Lt`k{z|_og!FM}K9=7jeVe`bgh3eA|goeWJ%X>l5 z+gqP4u#^giUrCnk(XYR9UHt@l=U9Bx3<2L+7Lo|$7jaXm)C-aozyL{QDNYgx|z3o z486L23_Vz-EsGs)4tqJZFSG)K6BQq~o~mh?Yob6!&{D#Htt6!aMbXi&Pn8oD;~`(j zvFQ`1KCKErW%vJ1IAN63G9;GQkf-vZ&f5X~-Of7)EIGs|LFVcL0>>xQFGD*`i;*eg zW5w{i{nN6@DwdviH?k6RP@sBG%j-i^MsV zpRo<9#YfnFs)?EbUmblm9MNKGp1nXY%X?FvNa#`>ly{lzSoV}+bi`tc_F#?X?%ak; zxP>b->8QpYsS*|i7b2Cm$$6iK^}wc9WY#k=5u0qCxxLutqf}nMBy-x$H@YFW^@lNpvRQy|S^46vpn~ zqQfKzvC7%UPLYbIO|@HEu$4#Z@QoQSp~jQ<`(NFnRIKlg8k)+gyGAjasOy!@dp3tX zM4Q+AW|xGm+&W_tm??NW&6@0l@gz~hHF2O(BZNs(D06KOVh)+HLw3XVInBZEATwGK z2=%l0VXd-S6v#L1{@FZHs}DU?zpam!gzCQML_|9U_6yeCbcjEaaFi*bWk?SGG>M6B z0V7YYo(l55P z$EuJ^Na;v;)YXj5IiN$Thv<>df!D`^X@sReurjyZOvbo6VXuRI7yzfnQVQ9xB0O=| zM!XEGgJSJQ&Quvv9Y_I z)Z7G-Em;z^;O`W?k30qH2VK-NGMtg->H@h}F~h>ayq)Y({=OYisW0+hi{r1)key;=fEDpJx-ZA*r1){e%6P&QoRe^=B^(g zTzrcV=B;4xef>8m(DoOnQ5*SDc#2|doe+C`_d=UGl_iaT=;a$UBM^20L-ia{GPa_U zlI&T0w#CYFB363erl6Y^TBG@iUr0d&@_4jT(8VMiDZY|>&)p$=qR-vOJ!a;PQVukc zMBq^{(h~u!OJLcx@QVFh0=Ic)j(yp?WDMk)=3VM;Bnq_kvd@8WERf{!W2_stQr)#cW?2R-$U1((dCmhxfmW_>`p@FJR(^G+@{+qe zZX|2Xr4TfNZDCB0?oGg|QA2s=4Fcm9f5n2#+A|`f8ucAD7u>(==Dn$SW4zU|;kdqw zP|KF$)h_<3B1yLJ-XGO}K9+UY+(kfagY*fVZ7*dX9a;=D(QRNs_|0NiA_5-Vpn0}dU z84TY<^kNQ#6_(KaqZtCS3CzeQv%?5bAU_QK6EHBm*AXI@vo?edi4u2g@Owj^mh75E z(xX84HNOYgfnSI6Kc|)StTxrm0O7~B3jXkg1CIlzlNOuz z&J@-`g4(UgHNJ0ezSG}u50-r!sCsJhP}vW&2bnOl#0JW-~W&n%UbN2X@~4o?^So9sugVos|k3^IuNg5 zf?k2UDsN|OP)&E!BAwnN2O_ZpU(;c~YNtHV=-^oIkc5s5FCm-*TI%~7HJdzSic+fw zY1fkX5rXoB3@k+}4m=x`UT=3YTJ%S6wa%7?=2m$)&SvyUc5$TKKa`A77JJEE)$!gw z@ttNktnrGQ88V0qFgiU95utD8X@>$u3eMS{qEEq=V{@DJ2|T%ynzS*r_wU6N-IBe| zjiceNhDRiXx%{@>RG7uvy`F5%E|d`**id^|YevCKT28W(-1z=mqiR;Ieb<7^*fH$(*xGT6>AGG=eGZwLu`RR?)*fmaOrQ7Opo7Z%gZ0;T921911G$8xJ@+gq^0SYvC zaKb6+y6{duw7%adZ&2{*g`%$kQVaTZ_B*~pP7-~fo;&u>9Meu z-Ofwi1sW?kHQ++03~Z{cl$f&H>k-vgzga=8 zxdwkcmnQdK?#-zn{l@yRx>xV3tnCg5Uyk9zXNcb9gE$07h97J>g&lbbSOT<@J%fy3 z%T~Qg9B<8Kq0X7lm!dogT$>ZonW3++PS0?}%C~SCds<1RQY4>GERoMi&u!&mMYzea z7L1I9Hr^ogc9=tY55rFIoEOdDRjt^sP6mZ76F=)jvO4D8Nq&{9Gtx`gnF1p*8#)n9 zP*=TeXFAX#za^=sIEh;QLaa4Icd?$kN-|efE=%S-w2`v1@ec%8Uqz#htf&m%#qi8WIR0_-^@fH|Z{ehjD5yfYt6ii^*MA+lk%v$aLE*svA+oR9|d9n+OTIr=Wb#QgrpNmh-^zD6UNyQYOG-&+H`8o@5@mr2r!G7XU&gW zz+%9dKe|<`U^xqe+Yb+q8s8$dladu*2|e{uXP8y+$N?VQpzyLCL9FLRrcK z=4v%iYN%r@P3F8?6k)TScQfD;ox9ERPsq<_+_g!c;5)pwPW0Ax z$-A9q=gbk^v8KLAI?Z^lpvNyBq6tr5M*x9(KRkHhl;*eEtqmZcMVXtP0|bV`d}CeJ z#^lr2FgNDiMgnfMD8~q#0evG&ErLI+0c}`e2)5M(MnJ3aB164i_@?-M++cGK>9J~Y zx4A7l^eLyRH*TZWa4N`E(=}FQwL-Nz>Q=dF`J<=H)TBmO6d*g9Ww&g3*vS3aCe5pI zV>KbZ&PIVY+%FtaAYsd8CWuY0ulo71F0pq5M%k=ZNsq^VQ&+zTdJP?!o#42l9q zY?Z}U^k^7l)++}KL|VL8+5j6fs)JEf$@0HKaGQP7nXI=uQW#c|3e2Y=dZ8*U*ctl0 zcwZ5!7E-6ts5GkiI=GU1g99G#Y}BhbiAk#Y3Sn^e!IN@ZbxWaeM?L$XbG7|W1eMI3 z9R|aFo2gqyTX;si0>U)9l9F=4?x`m?hjXu}IDT4>*v)N$jG#cX{7vkE6(~^fzL&hb zU#I$8`as4=!&|Qci8gUb@D`LLEZc2SAiz4$9VGUi#j6D_hLY`$EwouYW@0NLAaymW zqp@5igxi9B!9V=rr=h=0A_2F{kQwBeG-AEQ*+t}}fq?>bK0lMkezU{CBoS?1^EkDJ ztl%qr6|S4^5Iu45eA9Ca=n$xO%#hwE{AXunz^FoY%XGHM(H-JyDEFo~hA1`q2^=h}Jo4TOz<_!mep_Q?6s@zet z!-$f^F`T?!FV>bc=WEUr%5BYUGvYV{2Ld$q`oO}r(G76}1^QrGhyp=@FXj$3VB@(n zqy^(L6*l<+sdr%Z*b>22!0dKlcy*uK=BDo>(X6bP8rsf>8O;OrCJr`F#$gaRc1PGL zE%F-|V+{SZt~YcR1L70{7zQWPn#)J)Ytyzw2m!c!DqOmmvy~vt{D|#V^Nk2ltgCY) z0bxf&l|GWHafa|mw}Qa-L4m^hQ_d=(L>9yGhk<2g5RfAwaiS7LBQ`t2S0`r)p2ae>%Rr-lB!68V5q_WcBhvFVS|v>x|u{T3TzkE6LoH$&w)J7_m&zey} zh9I`5mQ=%Z6Sw2uN2DuGH%BEbdG_@J)z2P(VXnvq#Dr)8k$-}D?G z#Hsc>5s0c@ z#vetUCb>M$8vaS8WEuq`zmaIUiUN&r zjV*!$6K_}~yVdty?qpU;Dw14fJO~^2G9RcbBi;t~EdVN>AGH zbM2|mh@;PC{lb=a)9lTJBXp(|)~|8NUXogg2tKp?Ov}`Nb8vs-X=h{qcmmVyfN@`Z z!XCY7yc|2N8moP3kOy*aTIHo);SQVJ#0WGNK7|6^a%}aDAInVwVs{vbt8q=*ybFsl z_U<$8sx%@C?rd}FoN^3!GW7vuXeBXf<pPR^PrXUl4y*?`L9DZGVZ!cy470j|PIALFQ}kX0>TzBgVFin2nu}%|r?&Q@mD7zF zN;j?zmyf08?=>&8?)LV8c&=hRmb9F|Rl|l3WR{I+4L#s(jLk(rT z*8U3Rk}JijDo?PBtgql$f*2zmu)e@f*kM)5d`2>*{kWG@kyM5(<;J3b1{YK%R4%F+ z_TUYvfyMwrc_EIgaSh2oIFm6g*Cp*cP^yCvytRi4vdqzFJ;$gdF>UGU5&dbz4f3alj1pHDl z$D+{9x=5=?D5iqX()KV~i~@K#rWOtrYts>dno?|O$35Zh-(p+HFWb&|6Qz9?KI!gZ z6G7<{nZ&HuAw~Kwyvh!T?6mQRDH3*W5qk0`wV*(4+ps}dSnm`f2KE*K1ie!}V)SXV z&bJS%v>>cRXmXmP1a-MsAvw`*Ms zH0F!|a8T{WCZwjbKQQf{I;QOO*-FU-)<-}ThZ1~Jd~(H$RC57)>=mhs7#KD5v!}Sk zyg<}Q-(=RQI&x6g?P;F^9ahZMd91hcogiYiN^cW7No)PJZHHAxxavhWl@aB}#PA4E zZ98-JD3I>d@pS~*_A&hFm~-m&uA}zPa%5(l?FE=@g6N__5ih#D;7bulo>UQrJ9$}a zlBMKS+Gp;$g?bTt?uRCtgiI_p!?iZCyJjoV;9|Xvjfep!6bQ5EaOFASo-_T?p+)@b3xP&*OsscNVeyrQ-7`6G>O zxaDIn*{XI`Rq74g5>%%M{j zS?r~5vBy+H_2;4W(xiJ&7wj@B zO&h}yHnm*lIZ1H;a!tf33{G0x)p=$P_zz?{tRHwB@kh#452f*~2NjjxU){J?6n5#| zPmRr}M()A}54)j;8lgZ~qY%3WRW*fDQ8_rD}1jAfMshosF@k>^OX%4L;f5)Y}n| zHHPtEN_5P;)bN_TDvh4I*%VNaA~nY{?JW}G27M3QoXOP3hHOu_80Ooy9$m;j7@ACe zJ%=SBx<_)rd#KCW#KXby`T958w=F}vweg#7f<2mQ#M9FomPo!`udq(m4ckXCY-zP{ z=S!`^1v)-FZrx4UrGlK^sg+?>4ARz!x`419ws;WGrtsFb z6D*|qj16SyoCnq)g^aMsFUWi!sh_iN)jlcPLE^J|@WX))TsjDDT8-pA^9*a+n;|5Q zu^@lwd56ueqm1E}tY^j@w2pOHj1Ne&>{1r+R04AjUASZ|O|BQCul}Z9nyQQuaYZ3| z(NpfqDDQiwwa-k>5=BXSzA*GHn5vJ~Zm4ZBN4i{_iY%v}erFt|%D;ZLUDwf{7Sg3E z>RO~Z%o*2 zglswA}^)CS7N^ ziYU;_QhkI=s{f%B9MhJ^^{Sh-PgBP#bkw<#8Ca9V))ZyT`k^)F$ewiq{nuqX7Y3f= z6u_Pl`#)UMCn@zuABOqAyuqkZWqEWAQVKNY6j0Q25HF43F>YHmxCPTYxj&|Lbh`B1NIOTJCBz|EX7(OVkoZPOyKiuw z=s7X2S9r`Crvu|zX!YhG=gtoGJfDv#}#Y&RN)k8)tkB_OYT-zxU- zQme&E>u$MLqqTBao&Iw8LqMc)#j6~6$n!kyjq6;7POVpLPrv7B=`?UZ5+v9A?0M_; zQ#L7AO7JByA#&MB0SC@T5}r%CFPH>pn?@N7F<$cD$06a+{jBa6bI&sqqEuQW{mnht zp>O4@j1@k!Xefm}R}{?=i@CF7IC*cP2n@b|+&Gm4dA^UB$?YsJU2st*UR-!`w%-kb zc&;GM#=~-HVV140g^>&F`qhZJUX&_3zV?)e`|*DJ>#bVk+EVK^x44uPVMKV-V}aqF zK*y-&Jh4s;ctGv@CPQDDCm$1hFEPlpKmLkNNx8g+`yc{yxG|l^)=`F>Z+hWa&u3e_ zX({sYi{XzWpiNFGk6@i)C&liV(gtCiji^0$T33=}qw$G)zc}nY$|jNg&v>`2zD{IJ zJCM#|$V-bDsRgq=b-m2FULh^@4u>zIhh}!4C_jEaoSO1mG|omi1)?DDDh7Ov!~Vu=>=ps_)x?Od7Bxv1GKr-_&w)rFc`i?K0~i`JtPhT_eo4pjB*fQ>bma zH1=f^vZX}u!N8jvxNq)bsQa-^dI!uP6&_eKvhkI6D0V3(lqXlI>^IT7TVocfoWj8w zhsOr$y(_~)lFTNB9n1~ZLwsm+M!PF=dfyFl zgR>Lc!ovZ?WHSehPKE(hHTT#ukc*(E256z|}LY>BE z6rke9PDII+3TlJgzE3~jBkc*sVM^N1Q!l++ySPJ3m60;Iy6QrL5#H633}`~%&%fRO zO^Um+MJIE(T3Np|=Q7i1>CK3QE_#*HaPu52{reKR>yI86vE9*@^SX(9R=9tK`XPZT z`CLsLm$#iyI4vtAZI^W`u_EJju*Ka`aD_98S&RP+urrjc#03^Dv$7q^*DE8dq6 ze8^D9S4y2KbRv~3g|gj#PHiqP&cXLyy{@x%{fy(-IG-okrS))&225xTIGS+ zRqaHkW8=gIXAecjQi5(LM|D$J`I{7F!If&I$zfb;`fhLZ2k+w#5j!&k$JCj3wy3iD z2CCn#jb-X;I>}X`UWTW{Dw-+-4QL>!CsgguWrzOk`-RKF0vr7X+|LHl^1rY;+pZn( z302@mO!+@^ewZf1-nJ=zOq2CQA^ojbSWZEw$V!*sO4$)BbQt#;RQXF0LcBfGBSy7z zYe-&iDpwO5THLoaNc%#?m{jx3Y4awcP>H;=^ysh zd*t(Bo|h#EnyL5fd9n7erHc-F?x}%=<>TX3SCc2*YqNi?yYzkGjaVSCWt|(?>V6d% z2Y<+4x>%UF{@hUiL+vs+vhyJz10Z=MgYa1pQS_F3@rjuVBSG%bGXvXq7SHBKVq3j- z4uYeKit4=IZwZHP1&g?k6vtcFr4*#kk1{Rb1s4@_DYolpW@HNWa9mcywZ@P3By0nx z$B~n0T9H~UC0?yFxG7h2rJhXI+W|AXF2dP>skEqu>}9HL+zXDdgK)Kz!yJTX7Y+nvio$p zCP~#_dU zk_}}_QI@bCZ@6`hCI;E;C7YM=#PV7?KpHvdLl-?pj?G)J>aMs<#`;?*9d0+;+pc$Q zxGvjfSS;;vbVN+DoedA-Z#d8RrslRAQ*&q3*1eqohl-@3w)+42W`PuAI?E8C-nf3N zH#-Lt$N$z_R-cOP_zfb^^pWf-;k7Lc4!+QAffyQUi$2=)FK1|NEjaOUx$#}kkPOgq74snp+qoo? zOlss;q|9A5Ul}%^VE?FkJ{_`hS%BQ$L7N`=DGQcu3Q^st2y;!fwo(;+l`E>N(cb>~ zy09zexU`@TrxFnv=VgiQ|t}hu0-b;LSlHHd7twYL4_`yRSk|+Hix_brp>z`sRw5 zc?$8|q{;DaBzreR?ql5Jbz>fM)HE7#1@B}kHddI2kCE7r{iMA`kwM-ST=$ZxG+}!s zcgIXp&9$%T<_UIKDUOa30oKyVWg>61mNyPeRVJ*<@lti*9{n-&FPCTnXQQwN0u$@` zzj)vHSX7tMxe#C>gRc9Wr^Wu!&|vxJGnPA0u>#0n7A_h$OT`Ru%pBm<%vI23wErH? zF|u_q`BAhRA2HCFKKwj70|5TtefdV}$v1M4`fXvN?zelz-F^@D&6&lnR53J=+_{@- zrjsWt=TK4sv(+wLRo>?EO4_sAO>Z9bw}@nee|Z1?Dqp+IC5>|J@xfqW&At6ddbG4` zles%xZ3LIfS(KEn3|?6fzEfpcpW+w(APQfin7csA3NuYSyS|Mh&^KhJ&z+8w_AIDG zwYrl`IniWk0Mw!VF}0e*TAFT@)smNGJj51DJe!Hfpzi4htA=I%*VBq{M*|sXxYS(E zE6BI|0)1fdFoR*|sJq~=WssEIFhZq9bv5hL9MbTDPcdR4_|dq-FZBhiIkns_+ZzSu zuX~3+=w%@!e$Sq3-%o!;MRx|G7Cmz2=bAS1e4FNXV(R(;`S28tP-n8%vV9kgvbVV= zja_6ww(326bvC?ezSMQ_U9|Aro$kjS?#<-_LL_R3L$)qFC^nu`d?fEKPlARp`MRzG zJCp)=B0K_z6S6`8AN-$9Z+#D3MhbYQBXHu2=y&flw6pvF&UxM#s~)M?j&Tq{lZsFV z3)K{3fi7)jUp^5Lnn>O%>b3E_IYd2PAfj=&=4vl@UE|T`grchL$;r5|&ZWKlwI%H+ z8u*8-x27re9CQX(SYt*X7AlHDDm!EX8f22QUs{y-hwJV2CXICHV2iR`#$Zr*W2zo( zBVobB@=21yVALS3sOH+$`?YB$chu2JOW-O#f%~2U)M{>_2J~&*!#Jkdv&Q5ojYnf2 zSJ;K;J^Y`)Tj0C$KJz+!84>htI9x=V9lf>;(P54|8sex;cIzXZ=jSW7K5TSaK@xAx z?@MYNGQwk;1HOrpv%P%5mTI1OZ$3;;f=T#=A#i0twlBF3yDk}=;&kBYMb zU6dcJ$tu@wPqb3H6Qf;+WQ;y^V^HYQ)~2G*(hErJ%?^BNbxB65(%9R}2As*>?OHOe z_cSwRj-w8{NE-b0;bDZXB>v3QxRUtp)t&st4gHBYx5XXbnK-XTHNxykYa#wmxu+Y> z=C_NU@2*x~FIO99Ni?3w=zQ>n$S~7Og;?h*80dtVDiSI#| z{oNrC3!E8*WhQ7Qy_SisUiNrQZkxq$=!9eUj7t=9(T>q{NYB+1B^6yKC|}$J$5Ns2 z|I?DtF0p~m112moNw6}YmB|6W|1pgH`O>*>YvW{MZOmj4rA4tGih zPM?8hfd&a&bokqA2bk6T(oXt$boF9FO2tH_Y61}G`Uen*?6+hf5Cb4|^dNA{nZ zJ3k{g7ZK%Ctgdr_C@D4bFWN86j|nJH z<%Uicwl+U1{4Zg1qe$uz!196r!}8Br|H#_U%KK@si>1TkdR(3kIL&Lo^GN^5&nux{ zh>JP7NRXVCvy=x)qX1;&<{tz-(cd!hA04lsS^4$NU%EoN0_6rTFrT z?$<@!#p?~vaq*h}jQc63b`f{+%C&Qxy3Rl2ehL6y#9h4E=NuHlq)f8lq(?w+{FyC@2O&P%rZUwD7nE#h5}VLs=lLt_vw*KE zo&N&&TW9pk0>9~$UM%_FqZ?fQ75ZHJ^PkW^Wc|)x?XLd@eXghZ@5#Mx|Au_7QF)Pk zQR(qK&x7v2k^dt`_nW0J>ONj1|8D3%t-<5*8~Hyj`y2V!kIp3z|2_Gh=f5GJ%Lra1 zUlijzw}zP4Z{+`2<7fB&n;kCV{@w&3uYSY*tqJ~qWc_<`l;6K0|NRW$@5yid{|))? zXDWYBP73%p7oQhi zBwUQPo)eDBe-ZwOy!@;^d=7n5I%JDn%D1m>|n2A>O2sEdF{=` wK_Ho$->mqbNYBOOe=fd%Prg|9Tk`+A6jzqV#QO0!%)o~&9w4mQ^y9n#14n(zH2?qr literal 0 HcmV?d00001 From 7fdd321853380dcbde1c41973080578fffe28751 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Mon, 8 Feb 2021 14:44:10 -0800 Subject: [PATCH 17/19] Delete multichoice_dataset.py --- transformer_models/multichoice_dataset.py | 120 ---------------------- 1 file changed, 120 deletions(-) delete mode 100644 transformer_models/multichoice_dataset.py diff --git a/transformer_models/multichoice_dataset.py b/transformer_models/multichoice_dataset.py deleted file mode 100644 index 7047316..0000000 --- a/transformer_models/multichoice_dataset.py +++ /dev/null @@ -1,120 +0,0 @@ -from torch.nn.utils.rnn import pad_sequence -from torch.utils.data import Dataset, DataLoader -import glob -import torch -from transformers import BertTokenizer - -tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') -pad_token_id = tokenizer.pad_token_id - -class MultiChoiceDataset(Dataset): - _MAX_LEN = 30 - def __getitem__(self, n): - return self._features[n] - - def __len__(self): - return len(self._features) - - def __init__(self, phase, data_dir): - self._phase = phase - self.init_dataset(data_dir) - - def init_dataset(self, data_dir): - if self._phase == 'train': - prefix = 'wsj.2-21.txt.dep.pp' - else: - prefix = 'wsj.23.txt.dep.pp' - preps = [] - for filename in glob.glob(data_dir + f'/{prefix}.preps.words'): - with open(filename) as f: - preps.extend(f.readlines()) - preps = [t.strip() for t in preps] - - children = [] - for filename in glob.glob(data_dir + f'/{prefix}.children.words'): - with open(filename) as f: - children.extend(f.readlines()) - children = [t.strip() for t in children] - heads = [] - for filename in glob.glob(data_dir + f'/{prefix}.heads.words'): - with open(filename) as f: - heads.extend(f.readlines()) - heads = [t.strip() for t in heads] - - n_heads = [] - for filename in glob.glob(data_dir + f'/{prefix}.nheads'): - with open(filename) as f: - n_heads.extend(f.readlines()) - n_heads = [int(l.strip()) for l in n_heads] - - n_pres = len(preps) - assert n_pres == len(children) - assert n_pres == len(heads) - assert n_pres == len(n_heads) - - if self._phase in {'train', 'dev'}: - labels = [] - for filename in glob.glob(data_dir + f'/{prefix}.labels'): - with open(filename) as f: - labels.extend(f.readlines()) - labels = [int(l.strip()) for l in labels] - - assert n_pres == len(labels) - - features = [] - for i, h in enumerate(heads): - assert len(h.split()) == n_heads[i] - single_feature = [] - for hi, hh in enumerate(h.split()): - inputs = tokenizer( - h, - f'{hh} {preps[i]} {children[i]}', - add_special_tokens=False, - max_length=self._MAX_LEN, - padding="max_length", - truncation=True, - return_overflowing_tokens=False, - - ) - input_ids = inputs["input_ids"] - attention_mask = inputs["attention_mask"] - token_type_ids = inputs["token_type_ids"] - single_feature.append({ - 'input_ids': input_ids, - 'attention_mask':attention_mask, - 'token_type_ids': token_type_ids - }) - datum = { - 'input_ids': torch.LongTensor([x['input_ids'] for x in single_feature]), - 'attention_mask': torch.LongTensor([x['attention_mask'] for x in single_feature]), - 'token_type_ids': torch.LongTensor([x['token_type_ids'] for x in single_feature]), - } - datum['n_heads'] = n_heads[i] - if self._phase in {'train', 'dev'}: - datum['labels'] = labels[i] - 1 - - features.append(datum) - self._features = features - -def variable_collate_fn(batch): - batch_features = {} - - batch_features['input_ids'] = pad_sequence([x['input_ids'] for x in batch], - batch_first=True, - padding_value=pad_token_id) - batch_features['attention_mask'] = pad_sequence([x['attention_mask'] for x in batch], - batch_first=True, - padding_value=pad_token_id) - batch_features['token_type_ids'] = pad_sequence([x['token_type_ids'] for x in batch], - batch_first=True, - padding_value=pad_token_id) - batch_features['n_heads'] = torch.LongTensor([x['n_heads'] for x in batch]) - if 'labels' in batch[0]: - batch_features['labels'] = torch.LongTensor([x['labels'] for x in batch]) - return batch_features -''' -def get_dataloader(dataset, batch_size, is_shuffle, num_worker = 0): - dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=is_shuffle, - num_workers=num_worker, collate_fn=variable_collate_fn) - return dataloader -''' From be318a12ca77fc3244f1b4a45efee1fd168eb64e Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Mon, 8 Feb 2021 14:44:20 -0800 Subject: [PATCH 18/19] Delete multichoice_train.py --- transformer_models/multichoice_train.py | 103 ------------------------ 1 file changed, 103 deletions(-) delete mode 100644 transformer_models/multichoice_train.py diff --git a/transformer_models/multichoice_train.py b/transformer_models/multichoice_train.py deleted file mode 100644 index 2dd5a14..0000000 --- a/transformer_models/multichoice_train.py +++ /dev/null @@ -1,103 +0,0 @@ -import os -import time - -import torch -import torch.nn as nn - -from transformers import Trainer, TrainingArguments -from transformers import BertModel, BertPreTrainedModel -from torch.nn import CrossEntropyLoss -import torch.nn.functional as F -from multichoice_dataset import MultiChoiceDataset, variable_collate_fn - -class PPAttachmentBert(BertPreTrainedModel): - def __init__(self, config): - super(PPAttachmentBert, self).__init__(config) - self.bert = BertModel(config) - self.dropout = nn.Dropout(config.hidden_dropout_prob) - self.classifier = nn.Linear(config.hidden_size, 1) - self.init_weights() - - def forward(self, input_ids, attention_mask, token_type_ids, n_heads, labels): - num_choices = input_ids.shape[1] - input_ids = input_ids.view(-1, input_ids.size(-1)) - attention_mask = attention_mask.view(-1, attention_mask.size(-1)) - token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)) - - outputs = self.bert( - input_ids, - attention_mask=attention_mask, - token_type_ids=token_type_ids) - pooled_output = outputs[1] - pooled_output = self.dropout(pooled_output) - logits = self.classifier(pooled_output) - reshaped_logits = logits.view(-1, num_choices) - - ones = n_heads.new_ones(n_heads.size(0), torch.max(n_heads)) - range_tensor = ones.cumsum(dim=1) - label_mask = (n_heads.unsqueeze(1) >= range_tensor).long() - - reshaped_logits = reshaped_logits + (label_mask + 1e-45).log() - #reshaped_logits = F.log_softmax(reshaped_logits, dim=1) - _, pred_intent = reshaped_logits.max(dim=1) - - loss_fct = CrossEntropyLoss() - loss = loss_fct(reshaped_logits, labels) - - return (loss, pred_intent) - -def train(data_dir): - train_dataset = ( - MultiChoiceDataset('train', data_dir) - ) - eval_dataset = ( - MultiChoiceDataset('dev', data_dir) - ) - - model = PPAttachmentBert.from_pretrained('bert-base-uncased') - - training_args = TrainingArguments( - output_dir=f'./results_{int(time.time())}', # output directory - num_train_epochs=3, # total # of training epochs - per_device_train_batch_size=16, # batch size per device during training - per_device_eval_batch_size=16, # batch size for evaluation - warmup_steps=500, # number of warmup steps for learning rate scheduler - weight_decay=0.01, # strength of weight decay - logging_dir='./logs', # directory for storing logs - learning_rate=5e-5, - save_total_limit=1 - ) - # Initialize our Trainer - trainer = Trainer( - model=model, - args=training_args, - train_dataset=train_dataset, - eval_dataset=eval_dataset, - compute_metrics=compute_metrics, - data_collator=variable_collate_fn, - ) - trainer.train() - trainer.save_model() - result = trainer.evaluate() - print(result) -def simple_accuracy(preds, labels): - return (preds == labels).mean() - -def compute_metrics(p): - preds = p.predictions - return {"acc": simple_accuracy(preds, p.label_ids)} - -if __name__ == '__main__': - data_dir = os.path.join(os.path.expanduser('~'), - 'pp-attachment/dataset/Belinkov2014/pp-data-english') - train(data_dir) - ''' - dataset = MultiChoiceDataset('train', data_dir) - - dataloader = get_dataloader(dataset, 4, True) - for i_batch, sample_batched in enumerate(dataloader): - print(sample_batched) - break - ''' - - From 3fd4abe7dfcc6d3929edbe60058bd05dc67d0fe3 Mon Sep 17 00:00:00 2001 From: Atul Kumar Date: Mon, 8 Feb 2021 14:44:39 -0800 Subject: [PATCH 19/19] Delete pooled-vs-unpooled.pptx --- transformer_models/pooled-vs-unpooled.pptx | Bin 53429 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 transformer_models/pooled-vs-unpooled.pptx diff --git a/transformer_models/pooled-vs-unpooled.pptx b/transformer_models/pooled-vs-unpooled.pptx deleted file mode 100644 index df0cf1cffbefe010a035c281183ab533a3d1f2f7..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 53429 zcmeFZQ;;WL*XLX8VwcrrySmIS+qP|^%eHOXwr$(CjbCx<`M)z0?>X~K%*DAo@8m^h zWbB=hu`<`%JJ)A_SISC&f}sHY0D=Sp0wMsS6wUT-0R{pR{|N+y3H2#mW*t4-AAn2MFZ5|Nq_o3qOI0#8C02$b4tU`0cn-e<6p!D znbEk6U=N+D3QHI!W{im#r5OZkuER*GDiDNL(8io^-8f<5`Ysy{b=G0jmn=n@gQ7W^ zRduhLftQ^o4wNS+p~l?(JAX5!_>p^eCX%@wX6fb&P=gS(`S1k zD9O&)xg!P(EwV^x5}KG@aKY+L>6ak@bJ({6gZnaw;eyW&DV1*e4@5NFbvIGH-8TW@5Y&L9-sS~Qv`#Y?^zCeLw z{~rNw`r?=i`rWek9nUb|0k30kWa&Wr`(OM28|eRqqx(OZUJgXv=u*T>raP*L`7m+P&f5(^(|NR*L*O>Vwl_k>&2Lyzf0|bQn-NV+_ zk=DV|%+Sc;KMwT&nkFx_G;G#~ki2+$0PucU_CA&!&l0l5_|BtEytKj>PlYF-M6*q@ zVe%A1PO=Z(*=lidR%%fRf{LYSUNtVOSSH8&to7~n?`P-hW-KjY;e&=r*>Iy&oY(~b zQi;r7H+2H&7WCj_ILKXZmjPxdpsK)@_vgdw`nU z=I8BE1%~7he)TTCOc8$4u9!otgSy`cwI#-reg{UCOJDOH=RVDMG+k?-$SQ z^4z*TT^ViERaI1#_gc(fM;t3r&$^181MR}k(z78hFYHE-jOJ6w*LO+@2>X?W!#AN; z%f&Z+dyaRiC^PcBlzfaRgkb^^y@8)$6!VBmwS|(Omk!7PH^qX)f^H2n*w|<{Gmc-V z^@A669?WS{=~vq3KOGTvBd1=HK8+xY2QUL6XORX|EKtZV5Xhne2cY-_-kgcROz-cS z-NNFJQqzOCS9F~F)>ae}Y$Hp{UVZF>Nao?)f2I`{n7aw^pF_FMkc^(Y){=LNnaun0 zu;HNL--$1c{$#3?Zj+R6EVYJuPr|EPAe*>rJ|z+DXDuF6gJNh@qi+$bo7ZE&^bLxC zU2})i;e}*$C_lg|W3_)%PB&;C;44mG1L{Kn)g+mXlo{hyWoQ@ciaa0OifD&q{LO#Ns zPBB0~_qsRciS9kQ|HhJKeC`%MvkNO8rBI8jEidjh zFTh>N6Z6fDDzlu=gJ)a7cDvmJZ`*&|lHtp!^j4w|oCSQUI%cYtPQE zD1tVPThmEfYhHN3olV(1QL>I44}J{ZVaBg6Me+Ku2shhsyj&0B#d_!iZib8LYJ5F< zXY2U|NlHU+u&KYytN1Uxvb;TMYB+*kEWfZr5I+sr9GK`3yJcptdqAQ&loIxYiF$f% zf?hvBSP{{npw#-Y@bPv zq4|!$%B`HN1tSY1YPFg^`;fN=oOi-Mb9%D1k>N$1Rt}LWXyIHAz$)Ch#q|0Lrr85o z2vYX}J4Frt=)ipq96H*v6t7B&DHvkR_rqvqA)Q}O7&cU}e@z*4p~=766@=>7$I<|s zf)$Xri;Rf;l}F7KAs_%kqfx-HUmWYWFKx$H5Ys8Ox}NY>O93#iIlx6|=ioqsh}~g{ zK!~*U(Ht-+An6y?r7VQiFE(5uL}+AyX2ljHAxf)ecb?IvF|twMR!EP#9xXc1FP9p1 z6(PWs8?JD|gvy~9poT|GF-Jp;N_8ZTpxJghvrYzbkxh_4P#IM09MX6?2QX>4Rgxu%gmJ-=r7 zR>z_;?=GW}o~cD%R0$(Qj_?wQi3o}s92+Y-7>6ubo4CteJl=^j>(^KcW+2HLACy3D zE@)$g`$maU>uHyKa<=yyX)w3OkI<@-kNPRRT^|cyLK0&zO^9&ADZ>a|6``fAgdyHZ%ELT%obmgUgbY8iCG(*l> zO9;L(qyXEEg)|{q3@Td_wJ|Z=-^Z1|y=c9Lv^%d&(10~C#?aPvTju z)I2uFas`3f@a+cj2(^ZclOaW1=}w*#I#ref&GpX!HXQh?!0f+EY!Rji;Qai9NJPTy zExK#;#J?KZ`XWMVCWYet=qnm3`t>4QP<{%O0Jm4?Vinc&tp|N}o^)=!=?(8=|HUp4 z+67;P<;n8Lm5E=6Ad0@p0K8CB#5Ln#b-cgdinV4^7^$SnLs6mDgRz&x)}{R^0G&jP z5-KT`^=hc5qM{C@xBTTHP0s3OmAJ})VhyqK{O}w-pADa)nQf+^6^V*Z4RRQdBBxIg z#zr<0fDJ27Mx8V_IMjr7ZN*z^F?pP!K)LKHV@IgOC{Nx)w!aAE$us{%n`QRqbh|#@ zTfKa6sQYP8_i7~*Z{_CR;&Wx*!iz1wFNi`2`-zlS$6tEayX|XF+QbjpH63%EB>cND zcc<%#VEYRxYrUaTOr^MdAHhYC?G&9Fg8a$g%eTPmLI;ZoLp9KF*(ZHEEOb7>4}SGt zbJ`)=CF&~qcw+3`Bkj1&JTfh1ujdEaFI4&i3>6>wT?EDQN+OOp$LxAYW1e_OJPkz< zPBIkiXHMq^&f`YI%wtKFVIMC~pU?9}|0? z)QZK&t^3EVKW|Hl%5O`6iD0SD5EjZ2L)mpl%_0RyXqGuqpqob>)_UdvzFS$xZdn@p z?4M?w&xpL%kgN1sKUWd^prm1im>jt+qZCjE6yLZWP>f*j%buQ|Y=*G0l(Dh|#eufh zs@6R(HtsHd0eY`f*>#W$JaKx;SI5E3~AlxCWp)SxkCty;t=$w z3lXd!s=vo|tE^AiyU8;}GX_bgV7pekNKv!w-{}%j5IG)J#!_>05V~} zqTv1t)W%VoK}-_LWI_tU-fh-sghEA9?cz<}uBC;nnT!EWZtW%;l~=Zrzk^E@6J8-L z+SfX-Znomq>S{Ou9bLbgY_&!7zBhapJ^9fpEdt#i{X4ptJzXcYweO{byw)22w7!LH zel$y}SQ$|{^`t6`suFqTl=Jz@V*ava30&ZSwoU(2Pt}`X34+BkSZN_e$MSO$Q&R9N z%oX>K(N#eInF&DZsZd#21V$ ztyFL4CWnF}M z43B?M!-?s1)-CM;zpBV}si|cf;|76ptE~AHeE_#OtlqyR-D`1;amlWmc@Eeyqw&n9 zF8`5rmG51uyXcVp{6!C2nrDAIFY0=H_dx+6c=6VsEn|8e(6Xm8f0M9N4c*L5Aq&m~@M{BeYxFr6KewdZT&@2sj~3Z}*@>HaQD%-? zZZz8VoRP{^)|_OJ=WlGkdDN>`lO+OnW@QoRwHmD$cWRKz0gg6$_dNnfQ!)pGH1{Iw{kVD9d|aYZUE1}&39Y3shsz}Q{ny8 zf1mF%{71ffZoNN*`l*w1jenv7N^dcIvn8`i6&JxMlyXDhn3yNgdNGJjjZ z`3L*cHja=Bd*ZwT+hnj|0zNqV4sc_G_xTdsJ+(C*ehXxteE+<+f!*W%0xuT<$ZHe zjyWa5SG%7tRSQTh2;$PeNwtN#x-Z#3@ysN=diCvm{;bctul;8gd@%33%;&Xlce2j1 zznp`Eql2^K=SNyy{=K2pslN~eL^H>u%_8j8=XuHCxyOVE-LZ)fcTE3Z>g|d1!IIzQ zqq{fTX5y8i@=@|HL2OOfs{vRE@ zAJ$Brrp4~P%yfJK&7AupuQmc~Bva&qAxY$en)#x;7KQ{TkQ} z*Bf2$Yk)R_rL|NfO*(4R6j0y9fZmNvW{jk`vdo`Q`DD30_TqKV_2$~%i88c5KF4=! zK;j_OO;qScica!GF%%e)UMhG{TWM7j@a`3Iemn>ZO!AIwpf2uEQo#kUnu1|Vr->Y| zve{U}zd;yYOh*I8R%7SL_UrhysKlr@*unjT+CNDB z?u)5rr@TdpqMGCPCUkVDubqb%73_aoP-a7?>izQ4DaD;7=%x9ypV=@oaF*r=sxgD@ zmXqrS;Vhu~*>GIG1$$YEh5b)8gq)I;H%^P08pvSLq7T!-B`$~19$kDC5zQRfAlyu8 zZ9inOv}zxF{e)%c6p2+`kp8x$2+uTV?=dqc%!k><+s})oo#(%u?JZqSZ!8TE#e;@V z(ctyX_{k{}l}e!YrKl$Mm8M0QSWYT?TmH`-nf-?c%v_b=?IQX+C6@BUhc2?-X~5_U zi^L%ZO?Wzn{)#Igt>*;Y;O07*caM)I_}3;LzC}_s`ffxODQ9JK{jdtuuncT*q6)v| z5)guI1@}`5gUrO8A*p)*f|Ped{4vAnX~|pL5e}CZr(>jQf+x^q#vp<<&1F5A!8ODI zPF<;J7gTXV7n7&T$PxWxg&@&+C9V~KP0u36ED_N72P{?2eO616p$Y|RW=`JL{E83; z3@SN+bw^&9YRTX{Q8(OX2<@D_2}t?(U@UI}Wf=A`MIC?PNNb8cK|~|x8cEZHb`7{& z2*Tn@V44Bt^_!Pq_!67qjt%0b!MT&Jgyj(0HF=XE*pOld!T_wEE)xH+91_d{n2Q#* z5piTwWQe6djo)ZiW&irsG6$+KhQUz!J+3h(Bc&O9{W36P>o4QHe7byTiAy>WNx6Mb zI^PQWok$Aff)M2IDFP?x0nq0 zz*vf+y^JE!R-Rr&_9B5#`X1v{)FD7Yj~O*gE-$N;!ue-oL85=mjo}9U@g>lJ=r{IjhY%;qpO(L2Ej+C_I4^Q9{pQ4IpuF^r~T@)GN?Zp;NCldF0S-FRWnjsfcw|E z#+Y{BHKJ{!>>*QP1t<4O&gduw%FK>V+-a*C9js-7XCp>^o=pZwu1` ztM<~`EUadOPG``sYjyyAaP-&5arg6=e=$l{6Z*~9P6kc&^0KM;ZMypBPk7JGb9S>L$0+izSxb zR!d8WwYKI0o&K{RqCzOi1QZH;96q{hy3Pt@KwKjIKE2Fp6U*0@LXn~=C3-q5hQiJC zS_VaLknGrNf!T0;kyrC{*%X?VQ`FDD(#U88lCXS;M-+1d90P?%&?rqniwz=~lMQ<_ z4*B(g&jBh1ik(wl{v7E~9B;K*eyY~P9VAX|ZC`Vt#CN`Q1HH_=84kgOc?JV{kT6Ih zu}SM!-5&CWntr0U?oEEhIlR?Di5Q&=uIuu_kMhT%S$z!bpEboZi<++D^ojCxhAknn zbz4ATiToHFfhodFg?p{-iq1zA{f!6Ha+Fs>$Fo&ccsvydUNo2xy;u^*r77>AjWM_3^Apa1p zQ*KOb1*e9XqHn~X^0u_42Aq-ubC1@8uhi|jmuUKY`JnEmhT?DF65WA)=$-cE8ky#< z=q@7SFUCGq4dLxp+F zkBLwdtXb4G=OnGTjc5!VvVk0NJ}Hod^voQsKut#_UNejEZ%3n834xG2!AygTxlQK?|4c*{5sw zh7HO&&T7=3I_@(!m6ZCbzTol{dPL-eu@U@3AnqW&?)%spQQ30jU)>>0CeRiWs#KGa zV6z`X1?-!tOve()U%T-|?dG?C4v|BC^T%QOW84NqQS!wo*NPG+*%dTzhP6j>KFe+n z^JQW%&i}4FlC+&tnPNEyf3-k5@%Z(o1L?T6Do~(+Q>;uchDLH2#)&Obc;y(Rw8zs%dv^66rO z`e0}c0&@Vy<=Q$?=t4rFhC|2fVF*aN4oGd)TZ^R$+ewmhq3QD>rjkk5c>3B_? zv|DX0j%HU;Iw?Q~QhGKr18t1nj_ND;HpJSS!6W=7QXsm8pB?3;&22mm?m5bV;EcQy zRq!K=_TLK-uh8*|6>LsUgHH>*C)S_`6-#%$^Xr2AI%Hy8SI=Ov-*x>oYAGu8ij_hKwj?Ehp&1Ndc zrJVY=cF^FVwwh+AI@&UA@7^aX#TG+!I{L0%_qdHW=hbsZr!?-fNoqG(G_B@5>*)T2 z5vPgDUt?w`$Cd{4T) z{#clJSGN=0QyI=p73P?P1U37l8uzssY@V}4$X%+lGL?r(#i8KC-4g4b~iJ0z$ z{?-dY+-Tk4^j(rJ@(HH3h2XP=a`(h)uIr`;XrjD588__gxaf5AsNikw<7+3ZfLH6$ zP$3*h)l4~KUNz5~L-;Yeg$vvi=n)19)N3xNVA3TMU1=H0p-f}GX6$QIuy93=FbLim z%>^;(|28^P6uj?=S|}=OqV&X?nUJ(=!kk;J;@c<^{aVSj%FH#On~wGPM9g)uP5R9i zX!4u7D5~xUMfY2MVI(zTPMZ4UmKFE^2h;z-BL6i_|3?Aw|2NY?geHW4eLMRa7!VL1 z5G3%w9R5E%-PYd7;aeoB=V)eQ{XhNvTRsK+t&{w|;J^0Pu`ny%Plxhj$Gz7X@_4K0 z(lLyBGZL~LLGT>Z@X`!RGCpR&Yd^l*l@Tyj;Lku|{zPp02G7F?&GA{ns(<(`hqY1$VF9UdgkdA3 zBAj-|v$%49cYCqqmOR9lgm;eX7mXcNp5pNyH zhpQ>!MvtnvaE^_OF=r+o5AR>gC=?9m3&khSSCovsyS!8AM0~w`c7L72-#drvbNH@rOQwCCp!9uRgxcLZ3ND0!wb-_&#Sq%Ez|&YT(93DPHtB(GLrJ3pRl zovE0Z)QjP&#FvkESgr}5b&OzQ0QU4*bf`7bujke}rqE|3OFV5vuXm?N$p#m^X^)DX zvflFVUS%HN4L@_?lDj)UWuuK1a+X~bjgxtkZsnYFsnEQVmUnkdD_;S3bnea$$~vz$ zEzPr&w-_N`qu#gx2Y?h~S_2tbvk4jcwiWrGR#BD&ZuP&mj>QVzDwjLBTT$v;Q^Ou0 z)9)Q4BK8}Rhv`D~*o#VBe{jEuq)Ji6#^j(yLJY=ZrCRtXj5u;Bof1a9U)Kb?(idEF z9+s)12=00n+A^fh%;hqfb)?G1uk2`Bf8ZkQ;!MB9e)c0shcZGV=8=fgFO$o&31Z}R*h>`6)9-u)fb=6+@EwSV76vNS|S(xR(6a{~*ejP%3dtF*T!L2+|FhRzKM zx&l}1$y%E>jpuOja0CxK+5+D-$La4+f6MUS;Q{oGM84NY3bAi~LI8RmP;*`zrUlr5 z2`#zks`@>b3@Or{-GWZXSgHG`hMt?{NFybpcj%w5b0+n9s*Rh>HLaITsZ|~=U-gVF zu}VfAop)uzC#{q=R60{*Wm^lpcIyrrT_my+DS-B)A?5jR&>DU7vqOtDusjSXsv<*E z5#NSXYLmN0e>USKYK$)0-B5cnBNHQ45C!$U?XGtw6o}`!oW{8koraS5cKnL6z+*Qa_ron?q5t4JGafkeUP=gbwE{p6K9F_t%&;m_wxFqs zUdtZn`cYu~VWWuW5+qPa1_-cc*^Eo|(&kM1&PI0t4mCk6y&+U(dI!M52}lzYZB+xR zH)!Hvc0tXiW5D8EzHuv6xq>>(DE|qw2K^_@>b@W=j3X!@H`*dk%qt<$TiO~;jexE> zN759bY}vC!)%%YqMB!T$q8UcyORj{%KBZt7C?4xTf5?wv;o2Z$nd+|_ZgI1!Xz^hd zRqAoFi%7$PEbFTx^x#vu>`PONqtQ{k5{HPPV8@T3DhCV8bHXOB@0$U%9ab1oFk598 ztyLq`gRu_L!fk`IS>y&FVc<19Bb}Ra>?J3~Q?}e;qIC*ul z6Q6s-g#9CCadnth3Ye2?fwtK~d{srNG$1~V-A}(5);?`mR3wck1G>CaHUjCeLlZ9_Q1R~f~3nWhDZ{X7vSI*ai}Xh)Tc+W zztHj!V;)T&+Ua^+ToRu}86$CI2N%NUVLJRY zMD8jO|E3CpxxwDTn->%sq++Z9=^PYc2BiR4RHXp~7y*QP!YzwKI3(!Nj$4l%&@NCP znGapS*s`3ctqxg_6m`DfFp4dD;RxkaEX!}cQiUl@Ac~}RVYX}#fzk%1B4$xO^uA~U zUwfZjZQT);-Wt!I5C2>x#9cp&1G)#e{u>G{d3;z(h9rW@aBPI2A+oM>X)H=xq{B8H zwjNub-FV#=!Ci`)xJqkb&&UZu;w4J)N_Y)jlp=mgHV)} z%h@CfWS|;l5(Yl@!E*;k=la$ZrUy60A-y`sV6TesZBOW%7ioiWQ^r`Hx7@jIGm221U?CasR z11f-a$CBB}R%HFrsj5e`ilPy7&@L`d)Gj3P!Q@AA~A_d?f`6l?(>>Kv9E&W^tzz|f1bO+ zm)*_hDGlNIsY=UVfhM#S%i*a7ekzCr;6>frVZonj6+oP&*Ovx%UUgpD+IJ_<>fwg(DIW|l__9O(zy9S{WB30(Ywxq zs6>uxD})fF;5xGD*02=u_A9hk4ny&&a_q=|%xljr@Dht>r0b|t*8oElD;&j+eh)?Q zZ$cXi2`rqQA(4Jow+{<#F?{k(>q0-(BT z>Lacm;@-vcv8Rw^0I4{}m`QlbR`vqL0_E2)65_qJnPJIP9&YCe(KTx~Lb}1R3icgI zAZ*j&uK-HI#hR4e#FG0>6g4*`lzilt=#GqoZVl|OW1X)q{4c=r*9Yyp&dedO2JX$3 z?v|&_&X+gKNuNmMZh!D>MaB_p{x;WqKnj;wGLQTFmAV=_Fn2Y!f7r=c>Eb~5-%xRsu|{$U8@#*gp+E8ocqOt9{HU6Eo+d>Gq%Psf;94A+q230 z0aM#7v5DS9sSuBd=66S5IBR-?vOu7ed!ps8p%1uJSKfu z`b+c?-NK$7-87+vG^Wktrg+@4U>BPos6lBv`H+$*!#6K|sGP?N z-V~3u*=EgtcCt{kCQ&5o zmvQ)=<39KB=X4OqYPP4@M7=TG{|vs#z9Lsirr_chcxM-WnY&LXK3tz4WhsubNiz%7 z>I73#qmzpx^CHo&=~Ed*>KIok*$x$KXS_^KHP(HAJ2Z7y#y&GkAie)7>xJecZGwZs zB>f^0Ru`T##}xW;t(1dzgzCWWdv==i2X4NyT06(RbspsG%@u-VZ5f~(vnEN2MkLkv z4OY44|0h^|h*^gTsWB~q8G{j$w5Z;w-D6J1Z(paG4;``$XLVs5=DZ`>(GY>0x5mQ-(+kVCu>N=kucwkd1%inbs!)o*v z+yL?NaA)rdH?1uhg+_y_uQXH|OaZFQ$2sk1gD1nA$14OA_LuytQtsQhV*zj@xx)!%a>M9+- z!H+*A|AVadUE6YAEw_Kq?Zpy0XFZ&_AYnF%!>8jeX&rL z_M8>iAe%RPZrLmjbMwrPHrt+tW^Pz8%ibPqozy^oe5z7IfQjHK%QgL3l2D%deF-oZ z|B$BtLYoyyBKnh*uS0F}_nft3-KyE|q8>cj+U($T34EHXF#PoG3*z6Q09Xdq)}rrv zyzuWLT$FEk^e@o#|3*v~$(z>cbSOU8WS2NGHe}*2WOAa54F+ioVsi7`J}}b$=9Y%( zOs5OZ+qIy+1!)TqJJ9RL%XmW>jNMx*3u}l35>xUA+EOt>s#D={odqYS^!3PwVu5PD z6eh$%5QMXjdvDFIT;<@g0TdNI3m~kLQ?_AE;){&Kv4A8+Rqfj}x)`gLk{Z8K7dvR* z?H*4d921UU|gZ_Lk%`{Xxob6jy5mh)_ImAf<^1 zXi?1r>Z2*|iRobxGCj_kr-sK?my?Yz>MXbcK0J4fZ>7_r`%3k(v@Zm9=zB~aITSJ% zJ=`x9!6x&}9#g4Jd^R&wLw~F?+)?eyMc=qV?DNJ&@rL~1#2WG?kx05XGpo8^WivQV z7Bi(aEz2o1k~%vn@`k=sL|)q6&WL&nVVJ0X*Q@d!Y0HK1;1 z(VGL(wP`BER|%g;tk#1D>bf|2#@XYL4SBswIzffE{qgo0200q161_Ke4@MqJV}(NI z+lU<);9e2F-?U(@7j~Sc*i)e*;X~)MV0snExBv`t8och~c|Ht)DLGAc&hh?hEmout z2PI98EfD!FQv3&`sDc@=5EuTm1mQh82C&(8iwh7r-1}`5x11R;2^YSUNk@TiiOe+v56tFa zAKQ*ys;;$<;3PFmuplY4~4xvf)~n{+l!i-Bp*<1j+>m1*3c|qw3RO}H@uuPf%P>yC)OFNUB{1}vL2t# z@3Xz#7AIf7MS#y9L^Or@mnMJ>ZHd)M>|)xbiqSX5ty*PI4ZGO``OYwnViW*GVS6dg zN!yCqU0LJs$6m^ZM`Iau(vALnjWV1#Sh+*#`}MNYvdmhWL4%Xuqj==%N27%OcBv^~ z60gZ9U_iOk?Wr}aU3IB&@hDQgrOI%UIuVEJs8TU4f_GaBX6T_}^OLg8$eCoKt9hMD zPpNQ;@v1rVwx8Tsd%ZH14ZdkxdfuZ+BUCt=^4F5JHPyt^>*)z|Q@k=gHCphZz9%5S zvQ|1f<}WGe6uPjJqQ66QzpnzD0Ec5z1Ri>42IpFcxwZEMgQ!1~0KNjFk7gK$>)N`wrSy(uN!6h)O0z;C2)2OIPCDpJe?1ew_io>3vK^5p4HYNt8s=&V zBgigUOlm(cl37AvdQudGFwGjsd4v{nI5glpdZ@nR0zRcHI6^XA-s0ZvrDvg!{fLW=QN>o0KYI3yq~K_D%Jl5q1~0Nc+GARccUn!>h<_d>$a^TfZB z=xeqGnFVvoXB(`ud*nBS2|_*rGvH9d+#w7|K#6AJq1k~11kUp~d5RcgWqN8(?T8ol z`PY!qYKD)I5f(|!&ra0Sx&GKEI4cDgSQG1al<0u#XxA6-7BorCakk~n9f_t~^f4Am zyR-o{XwAHex2!Q90w^&00u+K15vYi)Y%3~?A8jA4qk-a^11-o;PxJhdWF^T?N(|?h zMXjztE;szZ(U7p@a0)deC}(4IKTrlCU5JC!{3voMT)lc0;iJb^<5SWZF9x)N__u!& zIq4Lx^y9-IfQRFU_*7#``jHoM!(O#ZRufnyIuElXPiKFA9^MywOV4Oc(0$xWs{JYSmiDU{&}nd{JgQi?5Aj7)S8?6o5NU#Ow8uPX-%oM zNWkMnN{g~IH|){dtjEV4?z%^3%kWw{+6^2Uv)s=eZg>c(-404666|=w@cld$u!1&-5IkfAT2V;(002lW6Cc})Zik+2l7yZspflwG+<%&_a$K{?vt3^1 zt_gRL-b`{HYE-nC50Ewe>w&<)hfK2?Y7|=#tVp*uhd$pC7kqTDKLse#)!YhY3*#E& zMNn{Cs{ydSWu`8bu zkNH9r8KN*i*Di5H=j>cmwcpf2e`VuXE;fVz8z58>&!S}&uXt1G+kCy*pgbZk;9Y(9OojVP^?>xayDpS_LX5(6U-)$;9w?bO#;LHUT<9m$D6htgiR29|%1h9L z*oz+p(A@WdE%ZV-!@Am~O5|@kjIE) z0@Ku4*Tp?C?N3n1VXVv_qAj^$mmiQBdIek=5>nV z+#_gFiPzq?-$F@FotXI1Ml|)6&&**)W2KS5FVGl^tfYZK7{M+BvMUjD?{yK@%cN&U>J79(q z7z2tY1O-Vh#tXKOK>TVf*@8Ee0uC?ZqJzp|@@yZ4yOR)#_@tHsh7YF;BrmT2@hBqT z^9amHW`}Jc=v*vmLvF>?W&ocogZK4$I6v&|)pM$!eu`H`ao)sgP8QVa@+u?_)VW_vHyX?=W$nQ8s> zwM^k8Qm;+rJJ+So4fWWCCu^AtS(5TH_8 z3-Lmm!&H^=JI(bgb$Ec+iIdTM`@CE0xLd6z3+l8`9DjhqIdzmIT@3Gf^TT}C< zx9@Kb$~zY=a{kghYsadQbhw+q=>9I#h+9!5TD~}o>rk#_H*9I+yzRBKVA%Ni!kNQV z=CDBSo6E%e3LCsAvuH!oWYCtt!7S$5p?k7q&dcgQk;g= zdx?p4?-g2>mV!8aSfRUes5Q~h1np;1k}nBlV=ij3b2H@h(Bi&;A0*%jJ8(5z6PW&( zMs)7HR=!bg)>den36}|va06^OVJDMEw*p{Bwi`Rn%m%PUX2FXSZ4bBS87+xa@Xw1(uK#}uN5Vevsf;-%=z zwHP4Qhv82!37TP`X-N5+#e}hS@;L5n*4NsS^$J1rlkfp!xJ^P0w;uz7@Dq=UH^2ZR z4Mt=Z913ywv8bxK$i;Y77rM!;`qS{Pnu&jX0rsWthBU3S?8QUfcH)sm@jmwEg)S1e z=qCc@TM{G^4yZJpWW>C z$gBx#QbkWF#r&9H)vLgLLar7c zE##V^sd2NOlip{$%OKuR~=(`i#8?f5TK1M#`lFY?C0 zkb)EE3#XK14^QN)y*@w>_TQtN*I|yzXGj3x9q&I!Ip)7mPVHYumK_#LB*JovGX*pF z!60T|CXT@<;&3b$w@6mS|1q*$B_th`*(s2jg9Ayz4v!m&KD>IF`jz{7zabopUEW|; zU-mOQhSgQSz(|u+&8>;|$(1NJ1;YwGQ9MD#sExN(YJIf*2hh>gxNLN@$Iwz9p8`bG zZG_*GhgX$96wR|}7YHwOm7X*NxZIRc=D}O1*PG1=9r|kRDTL8yqsb}D1pS_**)4zK z3ZKsjOBPml<`#=Tkb}B!4zI8lZE#0#XcO1)oz1UKU(j0GwsO(V)Xd0G5iRzSg4<|S z>os6t2Yl?RWnma8_JDO$os^8Ol`Yt;Jr{^8EK;d-v)oniSl{|0lWk?01 z3$l7?R5=>B8$7PAMBLLHNat47U=0nvgNwB*b}QUbcOFHw_+V6L4~6G{^V$_|DSU9u z&~2kHy)I+DXySBqJR~{2Uidzqf6ivvKAUG0e=f@-DbD>h0Z!bMFo-*-BZgf}ULjy1 ziHG?KojL8QvqhYlwwpA+3%YyIEOtEa+u#6ytD78H{tK4r5%PcxE zm zQ?mHBHok7Xq;*i*Rk|3>k`EGgBS!k8K?vXEtzyG3+vGQSet87bD95hXg0q={=`?`` zAS|s}hi^X{GNX!osp%&z<&A{=*x)W#*t~EeIUTH5?g@jMSj|_QfLy?B4SO%IZ4w>hM5#A*mm^PoFc)aDlEo425rTd9a)bNIqSC0 zF&H?;uoCu#REn=(M#joYL6Y!wPiOPv9N#$|EkFD-`V-b_ zJ0>1*5}YV?;r}*_Wo`$9U*Gofty5>&&^Hlk0@G*dhPu+Lwb)RTu0_Xc89>nDXXlr= z9oe4{%=!i$r%XYziwukOuCkc$X@@si4r~o?r-il>A9*-qMe)+iWm&n8glbO5qI`Mn zq_OGQXHogJYi-OumvXJwSS)0h7y$~rl_h>8e@>vuKqDns&ou42)uQ~=OzpBPT4vm3 zK?U3FsjWs(QpW((1#@9iOKs+ z-lLa{S~@Y6B<xJyVW*O$9}p6>6m z@nf`Zjb4N8HJvuF8g|{_i@SV@S^Wr%{0xXy>|r$-FmJejJsX!aD133&;JmVBJzrRK zTU$^n`WPa0jhUvBUuPxzu4X@ug|OBsvb03CV>0H`axnQ)!lP~$i_#`=eV4BeQ-9fZ zk>`p3(aiIR{?Fbe!Ea|bQ@}{-4KRlBpG7HH{=7>jlu!S}OW%2QF9}S$F~w1w7|qd( z4B^4%$V%p+bV)hT%GN2?8a2TG7Ns}_zDv~NKzd`~o`~gx58u7G=F*#k+v@HG?v2(b ztS%=Szs>Gh$+0XuyzHV1)>_?#u|ia@iA-f0MPd=Bi9S%DeQ(w*AGV(Mx zOE@tOAKz4#xcMwO=lRu3f3>LER?_AS+|%iC(PCe55-dJ2GKjvvro2BlRyZKCxY7EC z#mp9)xAW=xve{@^cC)S66&RoBjjVk)kvLi5Z!A+xY%wj%x589vg~~Od&3X3JLRId! zK-4t*HePZZ-_;%`aHLiLq2+vfA&>Rr;YHFcG0>R~Yi-Jvt}J!g*oI|_`{vYa+$1=% z;_g#t4ZH(30q03^zR^xjg(u%^zHF$r!G$R+Gjhig` zMq3*5+>pJwn?A^_MjxDhg>XmS|0s+kM2}bdS4iOsf=xps`jkB-9F-5sm-db;kUXc< z@g^JA6Kb2&SgbGo0IBE*-4p8Wr&R1Vp1TRNQg{Q^&2*CaH@So{_hApx0;8S&NJ!eYo?NNS zK)a(1P|h!VAMhQ#AlY`L72{!t`s>ONLcP@qU`U@Cu;u_p{70ZjVxMLS17`+f(~ZOa=z0kbPJGntQ`$(gSz)GdZ5Wz>e8i}_mf;9_xTY|(Ak7Z(mVwR@4eD$ z5`S4%T{p+uGD`oDSVg0g0^Ks9P$l~sz^J<(XN=(%K%!k|zl&H*JKYo+2e$G4(+;LG z9fNh12W&SB{Y2N_T`MuLX7lny_5na%jbaPx+V27}6y9z~K^#Cd=81qOClGR2$n=B{ z2>xI=TFp<{MNeTsd9h9?Ak51}Cqm8F=z@%ODF5ysoGa`85du8_X zc#T%T*~-jYo7<&eI!gtB07LgX-W1d=XM|EM@)*g&-TEgBq8IIq58{G}IyFQ$d(CUpqsfjGq0Z**zW9~|v^%Cuj2VoFAwW22w%Wxh#5(QL zXzxwE%-QNJYwx&~1&mmx>oKz|2D05jo%2OnYlb$q zFT1O{ZK_@veI2Uzj=|+xzbiwFh*#)5XEZB%sj(NrJ{meXl1^f2pGMCSx9x}61fZNIK4L#}RH zL)phR{Hrr=PIcqA3bSSKSLVSkPHoEKDyfUOUH9=Hq?T>xQSraH#ky;YR5vaZEtE0* z9NL}zO5heKyOor6 z_-{kzn`o4H?>W7c@6UV+MzJy|Qs=A!J~NnS8n=GS;5tI763K!rCn9285bA7twDMreN0e z#@osE5FvVvKc5(~bOt%uhbb|hRPr#}w_#2gLCs~%7U{*vXXTL$@r)5md>@NP>@C2Q zgpS??pIIyN71U~uTV{nKrA2Vh`wp_0W5Whx7|dgtd#P*^iL#NIbjyBU5)8&z=B8Wo zG2o`61w;Do`dtt2f=;i&m#e()V;QSfeb=j+UvI1Y^GA+3F+HvQ5Mx3BhzLwEcEWdj z>b|;(=v6YZr_*w);zjCaAoH>JsA9+R0+!;!sI(~Q#7aSB&dtTN8Xr-{`+MIF1-Tr* zd_$%$?B_e-0Jjm$DedysTq6+t8S?pPU^jZC8eZvs`#H;Yw{YDq_F0j(K8XL z%2c|Y&)qzhj&p64O){1C!psX&q=tD#Qg|v=I8v-XcQfd@H)Odaa`Z*1p6NkOLGk=j{B~w!S9NwM{&vLAZQiqbXM$s8HuMOm3jd0Drsy3=*JBN(`<1<58jNq0In(&`jTOt`v9nECc!}AIF`2fm z2n#~r5t7vDNL9XZEIJAsf~^)a%quw`vk>M}Ur6puE>j)-L*K|7lQp1X6~_r}o_>qo zP9da=+p6xNQ;GFEh0=1aHET$2%E=0ih?aPxD6#dU0#tK*NBscaZ>{P1#{3D0+Rbk) zzhQ<3aKnyl*YS|j3vp7`iCf{*U)3YK0*DjnD(i*8T(@MyL&$D~mCI(d!Y0R^Ak!eo z!fXk;1U63a#^)TF!NTGZ6q8o&%0;QFOh)iuewk3o={sQg;vhYk@bpRK@xZX_v!I|F z(B2*)B$~!n+Yknq$Ft+viMTyyNz%7}mAz|9dkH;cgPYkhtx`^OE`BbpDRD5(qFTsLve{PRi(b2eyzBX`bctI! zRGifRc;R}H#2f9&<2d@`>Ebr%8v_0l&dv!FJhA7<(dXP??nRRd%VF*n9BuCr#IsG1eqio#4EK}IBTY1~+ZV}_EjTiu( z^&RG#-e`oFT1eay%KJt}?)-o# z-U@mcdtkzsXu_3ax?8awe`X(sY&iRpTxzOV6qRwA4lN0-J}WadFVcMZg1bo2eVD8{ zN@wdu@SD@LR!!*qKH)ehBJ?VmKSe=?>GQmC z82np5-+Z_>P8zX}oKG9A^)6u!p1UcyN9=UjlxiuQ+|-Q(-WjWDS)*p~&ijkfMCt}X zyV(H?A6~XokObs}aQ0F=8=SR$_;ubI!Fx2-A9&zxXYDZ%k~N)*;spNuMWM$e1Y*EL zq)SKtxj#nNd6%J+%ItGF=YnRJ=le@CZT80#{6!wZ!FtiE6K>=hj9~We2<#$jBlzr! zi(HH?BqfqW{tPS;gTk5_uDcGW?QgpChDoT|M(JE$P|kabY^AbID62F3V0!B^F4H~d z#|W^r*1WdNijfsrRIO3Z zSDN2>tZTBMSBiVpx0zOQEq(_ZFywfp4@g}%H^Q3NI9@hj*yqBQ44qwk{yPUa=|MrKY?;-#$iZuNGvC+zZFWg%WyHrnXQ|Y zCCASX5knI}5uOoPNJ1Hn`GkTL96S*2xzm#E$l@#;$HMsYH8u*X{{mNgoXhOADytk_ zTnio)(+M&`pW}O(=+msTAQ8mp*h5Ybe>BmIYec&vi0<~r^>tccth?i$tL+T`7%f?D2X=U1k#J5d}&T{EyRC~o*FUJN<= z2nT+K*?yLtJM~H=`(CGLWn@-Wv20gdx5VH8mIX$D zRa2Y;Wh3=@cSV|A2fdIn%wDqyT{y!qyiM#E|6Z#^g1hE3o6rCPj*ul5K~$)a9?B;& zP#rpfEbTm9FL$m}tJ_}!Mk$=G-^;e~NG)|rY3@UkqUEUevv%*1f+_9@-PDbKW?>_Y zm!oiLu}Uvpf40;lJHmaYrLT3X?E4Y=GfivgKl-GPxh?ses`|53~j$<;R4A62aK zmXesZ<11XV% ztVr@O1ru~BxycI%zsO#Otoe`yH4AG_Ksk``)`fpbck37M(ZzOQ8-mO8DPzY{Vb5dV z;bwfXa~SYA@B()j7-kvv>LE_q<=IA>u*1KCK4?L31$X!f<+*~x%mr>a#klp+SE`cb zwsq$H7#JV9i8WyUVHMA#U^5u?&HsB_$fPCPdAnl}cq7C$?lM9MdFbToyoW{E1>4t) zRi7F4N&2U9NK@+wkpo+8*2Ok|J$_<8wiWbGnkPLneVG2KNpYO9P)o07TyMoaxntwh zW7WUTuAEEZ0-FnsoC>eQsjFB%DVi8c6ocC=f=~0l^ISx!ho7jQx~f>6rItxgrEn70 z1N^wB5#wzUlM^IJxOxjCQX+_6ju%-}tZ19NlkH#?Sg9_s$ulqIe%h7Zi)SXqr)Bat3dG%euHwV6Ml!h~3oacFx$Rhu_z38?R4JeXCa2#c(OX1YBS@mo zVYY)OF1|ZFT|aGY0Rj9H`a+*(*JR9YN!ns2 zDxr^*Ct1vV$CQAJDO?khls(}&ZrYQ4Va<%LvmEz`ONvwajnE>UgLTWHmV3kmwLdE= zu)n9i6+`WUxem9I>|K5tVL8tWm(@l^grsnF-D`f)RPR1_9Z~D$5#;6UpQmk*e|!fw z116^ffYd}HAh`caF@t|XO{Ay$3%ga;f!K}crK9`7Kf_bsCdZmDqc=9E@7DmcbkS80 zWCSG&eG_hX4hOtdZcHC1pT)%OkjZEii=nyEQ<}T7F&<8&xIdYzs0<~AR_#qLIBKsH z^`>Q@8#ypDZ_zNKq>7N6Da?eiTAE*|nRIcy+$ATU7nFxNv>jrb0Yv7dj}|l~S7oTn z$CoRLHefgFAU7*pN=gGn?Tk?Uy#Z%!b(-t&6=_M4>}Tl3XNToVsrBsBtjrdM2t=yF zq>o#b<4xGJYYH>!y&S%??KbTs58s4fLu1D?%}hCK15t;$e$3nmD`9-D)YDTN3&V^d z*rlbTaZtyUY>Hp2ut@f&XSQhS4tJi_{?z$0c`uJ)lsf~7GLNMVXZ&2ddJ^hn^0LTQ z7{4L0U>;C`(OwF(cfMF%xYPqfe2IS3Z{F3_GR&e={Gh6_3$mmVTvp9_q&^m>*3SF~&Lxye?V*d42E`r0s zpV<180lj1Q>l4x89YLI&1z-6ODD}kT>^U{2aia7bwg!6O>=goJeQXH$(GC)L0a~%@ zU~u2*-IDy8AlKj?QKGi{jKfRL{9k+tJB^3uLwV#1k>~C+Ns8=PXV1y*_9NFhpsmCm zE9HWly(Q?!1t)s~7hj9;)$j5*5c!0Jm5%EmJGE~2#_9{GZXSa6AiqkObrfY9_RSm$ zT7iXqszFe2%LS?I?_CmmY7Tu6*|AylwNsnTsB}~6vAbYI{SiqIkGm{b8V-}jp7Vnu zEncHudw8n`eGTPI-=lX7yH=XPW8cka5XLSfdoKne9*|m1FJMYS zDP))`Lt)e!HTyv-d^-G`&apPy*gGND&7T{^%`S+6cYx9klr`c#c;{=*N?SA{Or^5H z>Ya~o^jxZnt;)2?7qH^>p{&*&MWxrx@eb9*-BB;_{N$x3H-Qobq?Pi{^a+NNhom3Qm-FU+AxMdMD@!2OQqgY33;e?Y7B~_y>A1{nJ%LjMcUU#qfq;LXAU{sm>eVd6wX*vx+j4e01~n3*S>5 z&-i&X%4oo|3|^Wpg?{rXRLA~k4top?!yl69C^{ch4lh_@EfAt@lt=+`JcI^$yt%XA z{^{I+2KdvCEFe%BBK{YE>aQ{!6RK7=>nw=xJbweKII*xIe~!RA7C5nWU6}{1#3R*` zO_ABE`aHgmrF(~GNoOF5+HP(DGmTJ3Dy)rjbK~Y`?De=xds|)8-}3^278-e@>v@(m zhxhHknS1LeN=qnei!A!qI-#51PS3ztLVJ7{)l^-%b3S#=>@E4~Pi-J|kxk|=>LM+W zx_D*1IL4uHMM0Y>V2t1E{3kPzs$c@l3_M?Byg4^gIUqGZ6`@69MTp56eFD;`fti7& zHiHT$`CHu~(=A7FEhofpBqCY~Wys-0Ei0=@vB?^)yQ{@9{+?>6%JGXeS5P5IZpvyR zoBk1M==#MGs#{uOgtrx|?^NwZE;Nz%>F2EtV*(o3l`B(^`Y4RilS@(=V0YtYX_;rr zbxXvC!)udPfZq}y9PXbpYOeL@x2g|Ln_Wf@PWRx=OxAjyj1_Pfw|}17UZG6!%RA0R z2%2=JX|9jhTl+{T;5>^<$bDiI+K9=c%LS33{{@p(h(vZ)zjVJMP7?@n)~rvIHEivt zmE3Ci;#{b`XT@D-OoYFEHj89(eo?%#PZ;VM5pq}@B9cIeR}b+??W|31;ti!ZdsxbM zdfqh8cBp3<^=QHDM~&s<94S0{$m4=x?)#6yiwJa~$p^`?cb*M(cgm0$?e{6^?npp{)=<5oZPjCu4y(+E@9kJU02 zNhc+ko0(($-30JoRP(q(Oh+5`F-pa9N(cDVi|Tn4iFX{Qi7`o`Sn6=9v)XW@i?dii zrZzwCuD*2;H@g*s@mAMqR;-u#o?kH*5*)yy34;Jx7VOtB(%-iWLDG=WW)gl2Ppr%< zd%g6_c9>Ikkrp+h0a72R* zm8uqX5ee;gKN{^)A=f5>+lua*b%8p*RV%WomEFE{@pTjsM(xMEcJy!$A4TII^C}~U~S=;MVDQar^YPy+8u|mKU$C1@V&yF21#S#4{#le&GC&j@s&kEPawRKpt4n5x! z+u4luSQrHax2~zSW(~o(wVyFW^$TYD@Od6cxG%StJ;(on2dzBU_y$Bb0p$NYQu=pr zJGV|3LwW0->jKmFk=5(c4LMPSoG>qJh+H}bL&O*^@%$FCQJ_@tH;&-_6rlEI)^e_( z{FQ!J?_i?y0SA5=xN1@dRb}Pbb6%+li}(IoQbj!(?MZ#=+bPf;hla*cE#{~=d?k^t8H0!%lXRgiM z7ZpUy(7Q70m^KmP6U*)vkW-Q8Ci8UMY&uSfT>^v^H>saJwT@&(r$-%(jQS0m(kGWL zFICek(OoK>i@-8E1)Y=yZJXx_6*vS59lfM*L_5=n)*J^X-s4C}_QC^l!9_w^2?c-o zLr@SZQio&<1-K`>v);qk$zVymkC2<`U7}?w#X3oERVonXCS0UEiSlfs)6NWR7C$2vy5YLE ztJBAnR-d~kC=_uh6xkiyZoxSRcEuY_z&LF9@kYQTglU^_^;oLrqy~gNXL*EAgS2yh zKh%rYIoc?;g}|HV&|tl~&4f=es!lq|OEB*EW>=n^VA2@6b(KE8m`%6pleL<~?^5VBJ*P;KbHW9B#anEU*tOs$Tzm~Ti_)ds+SG|p<6uMq6qK5KwQt@PqG4M&#q0^2FZSa%Ps2eK%>Btz{PyOK!k2XQ{e{r zlXcM!>@Tn#_T*AgrJ|Um%XDo~2>z0GQu8XzlP|hcQYnKE?Z@$VG55dueCXtz;arM# z_u1<`Wa+V>vB0b$;bCz|+Z=X0Bz=HLWZ{l@fNLn18PGT=H&TnOE8%v1v)BY$O>h5V z5%2uHP2BEdmY7S#Zd$U@W%1>B%XFz-+R6Qg?YsQ>U^rVim|vbQQVxMGD8#@Q)7)&f zG99iJnJaU64J@etu&MuW|Yt$0SOtUleSKs$>*SnR@cfF_{{oSCV;`-UM5$c1o{5Xb{JZ zb!-wdE}HAXOS(upUd;RO2LE$@_2O*Y?+&2r`llY>w}0ls{vTEO{^r%v|A%GTDu0A9 zIU)7-3g3c6?X5wslkDXkt*6iW|Je0mh$y zwpMa$#mC7wIBG6NuJNO#i}=kTxXK|2zL$&a`bjikXI71a*hC_5a6w^-VBf{3NpTz* zq|9pJY~%i)PBQ#|FvDN|{6DK!_dmo8)c>Ws;jf7F|9UI@m4E#oW(Bi69YRzfDhvZX z%Kz(Lkk)f>G_rU2dlkdKrsDo>*S`Y8z@oHyKOG{(wpXqTU41kuXJ~xF$ZCKXy7V)$Y1FqMP?ZCg_Y`J-G;Gl z+UvVN&b_!T!_Gw4poc!k$gv1=FrW)8^!Mh7qg#rnyC_Xcq@=5OibMRoKII=et|stM zgTqtnCrjBh443Ttxfn-fJN)8zZi<3sxqOR@bZwu~%{&jYpv9Bq^fX^U>+xi}W=uZbkwJM+%oW6!+|nCR=2aZePmpo^XCDxC6?U z_9j6^ni>6a&V1D^%9*bRD|YIwV5nR8HwpYJT|HFU)I_WA0BL*Tk7u)L-RV zz;Jv|sc>o!iH0{I@K$N;xkm|*gjeje5H7bRf|+dw9xBwa?CBWTB?Cdjs`LWCl^e@( zG^Q;G+F>iLX6|M5qHYIkf=<&?y^k`5As8_-2ns=WXp~^@Jj4IIdY`Z~5kaw2qWH)+ z`eZL+Ow}@17%+lcEyO&{8M&r1R+@*Dyi=uXzR~z9ZwUVhRk4UcS-o4c$r#DRF1DC8 zs01!)zwan6g@JpIBZkPR8CnvM(9`Q&YJL~|hHvw#qd()~b11ffPwMP%hoh#jj;{ej zSc^Jju-q3v02}YRGi30xzo7$4g&HUG(k%q?31^uT2NJv;dUyM=e|t4cN3vL_+ywRa zQ!K%^BG6nn66A|+VGyUsZ^_h^V-plBX{fR>N1j?ci$sUA=p}I*dVb=KJNmy3L-$~i zZ=OK>rpL*J4&z}^+xJoM&3ICyhGll1{7{FlWLzY0zTOUZ1&%C`=W-JAZ+Cj~jDkB> zXX}zSq`0^VvoGsRe#HwKJKj7k$H~?mB$yv-dhg2HDND-#5{gv~OX8J@wp{a-BB2g5 z{R57~Ook-6oS-y}OY%C;wE!Ef*{YoUTD}2gtI=vh);eXqy)whkJ08n9tYJ6Hp9GEf{a_%%gD&iBg}LXOKk4+tU2HL2wFDXIx?r8>n0RP8La zuqK_l^;`*?1Ek@0juqYX{yX3`XM%>Jc#iF1u15#PNhR1_UWozo;YTKpS&c06FkU-S z!=%EQsj;D@z<03Zq*>Oe1 zAv#x1eGu-Y5rL(1qdArFyKAT3#dy)3Y%;6@YH(E(yG5^gq8z7w33BYJK|W*F+=u2MM=Tp^ezYBmjv^jqAQ z8(H8Cjx^9`L=kjt`gDwKzN4(1w+2VX%PeIz-B;U`u7m8J1jiH@UC7&}3zG`sj&Po1VxOk>yDJ178)G)pbsIQa zY%Zh=q{&}qsk<%S)E7IUzEe~3LWN{DVNOYk9qKAdeu;vjoNcHmQF(CJB}G*ns&pFS zD$a3|9xwCLSg$2E<$~6{gCc625^fVaVxTpw2MKWGck3T0(w7DXowViv zC3=pi-G@8ob@BbeA^5t6vn01Z@F%Qh;pd+gys-ASSl4X>tv`OAC)%&yp+=C8Y~k%8AHv&LL$!vF zSeR`AJAJVab%1K!`oVRpaV{Iv%{@Xog}c{_W-oadd_acPJ zY|TFS8(HHE{Swp|TN!gbU(hQJ9Lqgi*}9BYx#!%F$la`1ykCz5-+UeFrZ<0pS_Q;= z-Jg}!(y}g*gPpJc$m@YgxCgdYYfBlt8D`$!BH!_QM)^FP!E!Dr#qm!vxtDoKk2hrM zw=~Oq6p!%D*SGA}yMwadfTZsKKL(cnI&S|r)%O2ad9wdu;r;(rR`?HBUie>Sg}*8Z z{10<7{`;=hUo(yWM_Az#FqZX42mv7RcmsqKz!q==5-tITHU{7AZEPKA9ZiADPg(1k zS<;x>8ku~od~5>{B}BwU03bm3T^INR`1l3z6?Qc<1^}d`0aU>6LIc1-umC{n=3mP- z@PJ_dOIrkl0sxFN0`~)pS_GQ`fXS)9`p5u3|A;Pd1D^KppI|v4fA0<4mjn8b-XMS+ zuz$1xF~Ht{k8=RqcT*cj8wXPxTLMNJIslu1xHR~mvjgR?HtAnY#0A`l)o4H+_^~fc z%GslU_eT!^2^tUsybcHv02B!X3<>080Jt6n00aW$FZt_hz&{|MVBiptP|z@+V1XSP z5dokeU|^u&U=R?0E(78PJPrUyf5Z<_rZy^w%R777#IX31W+KMA^!;UC+NTA)87T|FM&7|Og&jAh& z4h8&0goS}c{BO(02Jp5-{8$6PfdOxDBrqfZFW~JpFHFD|CI=#~o4}gaYAdoN7)vRu9JTL<>ob=PQV>EQv{@ zoj1{2^#{Nd0rrGgVliG+GMkEBO?frl1~^?WmWF|Ix6??{*RGLt9EQJ}5=%DJ_P0u0 zoObXjH|i6F#f$Z;Wzq>ohM9(8tm4k>Gcw2rAgS^BMVrT?-!-GPSO43k_L9rm)`5%B zY9o#cYh9P0Fp7$=yPh!0cMxHeQEYw2yET~C5|F#pQcaznCH+AnyKj9YU3z*UA*5lt zI8A&#Tuukw)VDh8P`k}x%*RZTq>iqJ@l&p`z-gjL{(W?w^GR$y`tADYJUhU@|2d!C$cH{p~E8jX<}Z5_i=noE|uT$eQQ&NJ4fe9 zr_!B>#2UmR6_HroR1-h_WeJuc4f@zBSa5fYcXl!xTsUl7_puh0FpAPIqDzkQ!k!#f zK2hZP{?vz z`~f#=vnccDTK&SP2M6p!Wp30Agw<W$y97>mrP%klk9zhbF*+yr>5KlQ?cy>!JjxFH+vi;8I3CNZ zO-ptf@nlkQ-J7vb2IpD*E4m&^Z4X;$2VaG?b=>wV;$fH4co3T2LV563hzF}r)FZDi z-hSrU!RgtuS(2p7oMpsD@lB)Nn)F2ncA-Bq=5HKrqSv=y(-yx7)s54&8e8Rx(4lOD z3?X;ML|j?Wt%_T^99JN^gmy1x7d!IGa7XGz>`-fy!DqTh7SBCn{km>LF`ATtVNhH+ z!)(F~uz6c$hznCGL+2ViNPI)8~A^8?VOWAh#;8D9^Z z#-!|WpIPTK+~eW%d=vnD$U*mg)LT)mrrNh-4Rw?VqUYj&Xhbqe3Zp1SP~?RH{;_?` z{sA`%z@M!;l-G=Q&0>fblFQ2AS2Ti|GkVj5GK}-IBPk7TVJtK@t_afa1{8Gd676{ICgPi4M1Xlj4!9mS|M@DAcL=`Zk&mK);8l95)wl z{>}U3@CN|54C?;yvFHQv`yu~0&ZGt%srsFrg8VmGm(f-9K&EJK)0g9WwEJdQt|F`X zQWH_r*P<0cI3rlfdxP@`ALPfb(}79gBF$l&>#`zI&<5<#I2Sf4uLJQX-ia^NY=R)? zINoKIo0r!-Fo!8w8ZRc^OS~$)M$=L61droBJJ?zd;x+Rk@u>AfOBMtsj7)ybskrr# zKY-OAybXPNc~&*lepk&QJab3OJ{(lfyT2+#(mB>VB}y)7{0)H&jdBt=S9GK#@*=b**SOT z<&nlJqad~>oN9alhm6YSQK;KY5}NufTVN?Kj5}ssl(dSE@+NNxQ+9&o0%OmBxX=OF z^Yi2C*M_=Bay1uwNk;V%aq?1UcGLUZbS=h z%17p3K8UVi=a+Zb>n?7^@z|LGRXG^|4giNr-E!FbR2xh6uS@?f)LEx>cE^_&4HGrp zR0(lxOwC^j*K7pKjVd=y*7$eb7Ahe6|sS-U)tZ zCBvzpy9<}g;y74euO|2ft?>_Yhg1ty!@m`Z;e)(d%7VRc$R1GHx5;IZC6HBR#QC^? z#bm`|jN2f8!iSiCaq*OBlclnOH=SaPz5Z#4H-OJd@Dirsv);c`K!3)87f*rXKW;0g zV^4aBSNyv=!ttB44T1%W`gu5EYfYkYw9nU0c1+B>svE_}=CTu(>wbvL%y3D|d;T17 z@J1h6dLnuPJ%djOVDdy@hdA@8#n{Q$ri2Z2RDL94rb0E$L+W`(^?c37l? zdZ)tr`9+Smu6UXQ?e5Bl@+-ftZIvhitxCD>8AIuF2@(WEgkWa-u-6TiCZ5{-o87_g zrq1F3PnC*CeSFxBF~{)HbGU39fjSHl%65ys0kDW&7re%Bs8AfEfibXxssV7@PD{M#6VO=4y3k08Ow*Wkl% zv01`CeDd2Btuw{Ex_fIX57{;fH<-WSDHRjE3gc|@J^;k$Hjl0}*&hG_rmaUhop*X2 zqgTSmX!OBo5LMR$Y{+@Ery?D~0_4PNBh&Xsr77pbLT|WJEZ@aO$x3Ofyn1vRO+nDJ zNRYGq=Evo;;xnfY00~(C2jFh$;R7I1P@N1cdeEy|^Sv^-c$}Q1*(TphC-&^>z%FD! zpH!xzD$d5RE&uq0BCnf_XXOw0ea!s}@PU0*UuLrTd>n6~P0$%EQ`5YDAZxqBHgETd zl@SqA9N>|5{n8!o1HudgzWh!(Lt`k{z|_og!FM}K9=7jeVe`bgh3eA|goeWJ%X>l5 z+gqP4u#^giUrCnk(XYR9UHt@l=U9Bx3<2L+7Lo|$7jaXm)C-aozyL{QDNYgx|z3o z486L23_Vz-EsGs)4tqJZFSG)K6BQq~o~mh?Yob6!&{D#Htt6!aMbXi&Pn8oD;~`(j zvFQ`1KCKErW%vJ1IAN63G9;GQkf-vZ&f5X~-Of7)EIGs|LFVcL0>>xQFGD*`i;*eg zW5w{i{nN6@DwdviH?k6RP@sBG%j-i^MsV zpRo<9#YfnFs)?EbUmblm9MNKGp1nXY%X?FvNa#`>ly{lzSoV}+bi`tc_F#?X?%ak; zxP>b->8QpYsS*|i7b2Cm$$6iK^}wc9WY#k=5u0qCxxLutqf}nMBy-x$H@YFW^@lNpvRQy|S^46vpn~ zqQfKzvC7%UPLYbIO|@HEu$4#Z@QoQSp~jQ<`(NFnRIKlg8k)+gyGAjasOy!@dp3tX zM4Q+AW|xGm+&W_tm??NW&6@0l@gz~hHF2O(BZNs(D06KOVh)+HLw3XVInBZEATwGK z2=%l0VXd-S6v#L1{@FZHs}DU?zpam!gzCQML_|9U_6yeCbcjEaaFi*bWk?SGG>M6B z0V7YYo(l55P z$EuJ^Na;v;)YXj5IiN$Thv<>df!D`^X@sReurjyZOvbo6VXuRI7yzfnQVQ9xB0O=| zM!XEGgJSJQ&Quvv9Y_I z)Z7G-Em;z^;O`W?k30qH2VK-NGMtg->H@h}F~h>ayq)Y({=OYisW0+hi{r1)key;=fEDpJx-ZA*r1){e%6P&QoRe^=B^(g zTzrcV=B;4xef>8m(DoOnQ5*SDc#2|doe+C`_d=UGl_iaT=;a$UBM^20L-ia{GPa_U zlI&T0w#CYFB363erl6Y^TBG@iUr0d&@_4jT(8VMiDZY|>&)p$=qR-vOJ!a;PQVukc zMBq^{(h~u!OJLcx@QVFh0=Ic)j(yp?WDMk)=3VM;Bnq_kvd@8WERf{!W2_stQr)#cW?2R-$U1((dCmhxfmW_>`p@FJR(^G+@{+qe zZX|2Xr4TfNZDCB0?oGg|QA2s=4Fcm9f5n2#+A|`f8ucAD7u>(==Dn$SW4zU|;kdqw zP|KF$)h_<3B1yLJ-XGO}K9+UY+(kfagY*fVZ7*dX9a;=D(QRNs_|0NiA_5-Vpn0}dU z84TY<^kNQ#6_(KaqZtCS3CzeQv%?5bAU_QK6EHBm*AXI@vo?edi4u2g@Owj^mh75E z(xX84HNOYgfnSI6Kc|)StTxrm0O7~B3jXkg1CIlzlNOuz z&J@-`g4(UgHNJ0ezSG}u50-r!sCsJhP}vW&2bnOl#0JW-~W&n%UbN2X@~4o?^So9sugVos|k3^IuNg5 zf?k2UDsN|OP)&E!BAwnN2O_ZpU(;c~YNtHV=-^oIkc5s5FCm-*TI%~7HJdzSic+fw zY1fkX5rXoB3@k+}4m=x`UT=3YTJ%S6wa%7?=2m$)&SvyUc5$TKKa`A77JJEE)$!gw z@ttNktnrGQ88V0qFgiU95utD8X@>$u3eMS{qEEq=V{@DJ2|T%ynzS*r_wU6N-IBe| zjiceNhDRiXx%{@>RG7uvy`F5%E|d`**id^|YevCKT28W(-1z=mqiR;Ieb<7^*fH$(*xGT6>AGG=eGZwLu`RR?)*fmaOrQ7Opo7Z%gZ0;T921911G$8xJ@+gq^0SYvC zaKb6+y6{duw7%adZ&2{*g`%$kQVaTZ_B*~pP7-~fo;&u>9Meu z-Ofwi1sW?kHQ++03~Z{cl$f&H>k-vgzga=8 zxdwkcmnQdK?#-zn{l@yRx>xV3tnCg5Uyk9zXNcb9gE$07h97J>g&lbbSOT<@J%fy3 z%T~Qg9B<8Kq0X7lm!dogT$>ZonW3++PS0?}%C~SCds<1RQY4>GERoMi&u!&mMYzea z7L1I9Hr^ogc9=tY55rFIoEOdDRjt^sP6mZ76F=)jvO4D8Nq&{9Gtx`gnF1p*8#)n9 zP*=TeXFAX#za^=sIEh;QLaa4Icd?$kN-|efE=%S-w2`v1@ec%8Uqz#htf&m%#qi8WIR0_-^@fH|Z{ehjD5yfYt6ii^*MA+lk%v$aLE*svA+oR9|d9n+OTIr=Wb#QgrpNmh-^zD6UNyQYOG-&+H`8o@5@mr2r!G7XU&gW zz+%9dKe|<`U^xqe+Yb+q8s8$dladu*2|e{uXP8y+$N?VQpzyLCL9FLRrcK z=4v%iYN%r@P3F8?6k)TScQfD;ox9ERPsq<_+_g!c;5)pwPW0Ax z$-A9q=gbk^v8KLAI?Z^lpvNyBq6tr5M*x9(KRkHhl;*eEtqmZcMVXtP0|bV`d}CeJ z#^lr2FgNDiMgnfMD8~q#0evG&ErLI+0c}`e2)5M(MnJ3aB164i_@?-M++cGK>9J~Y zx4A7l^eLyRH*TZWa4N`E(=}FQwL-Nz>Q=dF`J<=H)TBmO6d*g9Ww&g3*vS3aCe5pI zV>KbZ&PIVY+%FtaAYsd8CWuY0ulo71F0pq5M%k=ZNsq^VQ&+zTdJP?!o#42l9q zY?Z}U^k^7l)++}KL|VL8+5j6fs)JEf$@0HKaGQP7nXI=uQW#c|3e2Y=dZ8*U*ctl0 zcwZ5!7E-6ts5GkiI=GU1g99G#Y}BhbiAk#Y3Sn^e!IN@ZbxWaeM?L$XbG7|W1eMI3 z9R|aFo2gqyTX;si0>U)9l9F=4?x`m?hjXu}IDT4>*v)N$jG#cX{7vkE6(~^fzL&hb zU#I$8`as4=!&|Qci8gUb@D`LLEZc2SAiz4$9VGUi#j6D_hLY`$EwouYW@0NLAaymW zqp@5igxi9B!9V=rr=h=0A_2F{kQwBeG-AEQ*+t}}fq?>bK0lMkezU{CBoS?1^EkDJ ztl%qr6|S4^5Iu45eA9Ca=n$xO%#hwE{AXunz^FoY%XGHM(H-JyDEFo~hA1`q2^=h}Jo4TOz<_!mep_Q?6s@zet z!-$f^F`T?!FV>bc=WEUr%5BYUGvYV{2Ld$q`oO}r(G76}1^QrGhyp=@FXj$3VB@(n zqy^(L6*l<+sdr%Z*b>22!0dKlcy*uK=BDo>(X6bP8rsf>8O;OrCJr`F#$gaRc1PGL zE%F-|V+{SZt~YcR1L70{7zQWPn#)J)Ytyzw2m!c!DqOmmvy~vt{D|#V^Nk2ltgCY) z0bxf&l|GWHafa|mw}Qa-L4m^hQ_d=(L>9yGhk<2g5RfAwaiS7LBQ`t2S0`r)p2ae>%Rr-lB!68V5q_WcBhvFVS|v>x|u{T3TzkE6LoH$&w)J7_m&zey} zh9I`5mQ=%Z6Sw2uN2DuGH%BEbdG_@J)z2P(VXnvq#Dr)8k$-}D?G z#Hsc>5s0c@ z#vetUCb>M$8vaS8WEuq`zmaIUiUN&r zjV*!$6K_}~yVdty?qpU;Dw14fJO~^2G9RcbBi;t~EdVN>AGH zbM2|mh@;PC{lb=a)9lTJBXp(|)~|8NUXogg2tKp?Ov}`Nb8vs-X=h{qcmmVyfN@`Z z!XCY7yc|2N8moP3kOy*aTIHo);SQVJ#0WGNK7|6^a%}aDAInVwVs{vbt8q=*ybFsl z_U<$8sx%@C?rd}FoN^3!GW7vuXeBXf<pPR^PrXUl4y*?`L9DZGVZ!cy470j|PIALFQ}kX0>TzBgVFin2nu}%|r?&Q@mD7zF zN;j?zmyf08?=>&8?)LV8c&=hRmb9F|Rl|l3WR{I+4L#s(jLk(rT z*8U3Rk}JijDo?PBtgql$f*2zmu)e@f*kM)5d`2>*{kWG@kyM5(<;J3b1{YK%R4%F+ z_TUYvfyMwrc_EIgaSh2oIFm6g*Cp*cP^yCvytRi4vdqzFJ;$gdF>UGU5&dbz4f3alj1pHDl z$D+{9x=5=?D5iqX()KV~i~@K#rWOtrYts>dno?|O$35Zh-(p+HFWb&|6Qz9?KI!gZ z6G7<{nZ&HuAw~Kwyvh!T?6mQRDH3*W5qk0`wV*(4+ps}dSnm`f2KE*K1ie!}V)SXV z&bJS%v>>cRXmXmP1a-MsAvw`*Ms zH0F!|a8T{WCZwjbKQQf{I;QOO*-FU-)<-}ThZ1~Jd~(H$RC57)>=mhs7#KD5v!}Sk zyg<}Q-(=RQI&x6g?P;F^9ahZMd91hcogiYiN^cW7No)PJZHHAxxavhWl@aB}#PA4E zZ98-JD3I>d@pS~*_A&hFm~-m&uA}zPa%5(l?FE=@g6N__5ih#D;7bulo>UQrJ9$}a zlBMKS+Gp;$g?bTt?uRCtgiI_p!?iZCyJjoV;9|Xvjfep!6bQ5EaOFASo-_T?p+)@b3xP&*OsscNVeyrQ-7`6G>O zxaDIn*{XI`Rq74g5>%%M{j zS?r~5vBy+H_2;4W(xiJ&7wj@B zO&h}yHnm*lIZ1H;a!tf33{G0x)p=$P_zz?{tRHwB@kh#452f*~2NjjxU){J?6n5#| zPmRr}M()A}54)j;8lgZ~qY%3WRW*fDQ8_rD}1jAfMshosF@k>^OX%4L;f5)Y}n| zHHPtEN_5P;)bN_TDvh4I*%VNaA~nY{?JW}G27M3QoXOP3hHOu_80Ooy9$m;j7@ACe zJ%=SBx<_)rd#KCW#KXby`T958w=F}vweg#7f<2mQ#M9FomPo!`udq(m4ckXCY-zP{ z=S!`^1v)-FZrx4UrGlK^sg+?>4ARz!x`419ws;WGrtsFb z6D*|qj16SyoCnq)g^aMsFUWi!sh_iN)jlcPLE^J|@WX))TsjDDT8-pA^9*a+n;|5Q zu^@lwd56ueqm1E}tY^j@w2pOHj1Ne&>{1r+R04AjUASZ|O|BQCul}Z9nyQQuaYZ3| z(NpfqDDQiwwa-k>5=BXSzA*GHn5vJ~Zm4ZBN4i{_iY%v}erFt|%D;ZLUDwf{7Sg3E z>RO~Z%o*2 zglswA}^)CS7N^ ziYU;_QhkI=s{f%B9MhJ^^{Sh-PgBP#bkw<#8Ca9V))ZyT`k^)F$ewiq{nuqX7Y3f= z6u_Pl`#)UMCn@zuABOqAyuqkZWqEWAQVKNY6j0Q25HF43F>YHmxCPTYxj&|Lbh`B1NIOTJCBz|EX7(OVkoZPOyKiuw z=s7X2S9r`Crvu|zX!YhG=gtoGJfDv#}#Y&RN)k8)tkB_OYT-zxU- zQme&E>u$MLqqTBao&Iw8LqMc)#j6~6$n!kyjq6;7POVpLPrv7B=`?UZ5+v9A?0M_; zQ#L7AO7JByA#&MB0SC@T5}r%CFPH>pn?@N7F<$cD$06a+{jBa6bI&sqqEuQW{mnht zp>O4@j1@k!Xefm}R}{?=i@CF7IC*cP2n@b|+&Gm4dA^UB$?YsJU2st*UR-!`w%-kb zc&;GM#=~-HVV140g^>&F`qhZJUX&_3zV?)e`|*DJ>#bVk+EVK^x44uPVMKV-V}aqF zK*y-&Jh4s;ctGv@CPQDDCm$1hFEPlpKmLkNNx8g+`yc{yxG|l^)=`F>Z+hWa&u3e_ zX({sYi{XzWpiNFGk6@i)C&liV(gtCiji^0$T33=}qw$G)zc}nY$|jNg&v>`2zD{IJ zJCM#|$V-bDsRgq=b-m2FULh^@4u>zIhh}!4C_jEaoSO1mG|omi1)?DDDh7Ov!~Vu=>=ps_)x?Od7Bxv1GKr-_&w)rFc`i?K0~i`JtPhT_eo4pjB*fQ>bma zH1=f^vZX}u!N8jvxNq)bsQa-^dI!uP6&_eKvhkI6D0V3(lqXlI>^IT7TVocfoWj8w zhsOr$y(_~)lFTNB9n1~ZLwsm+M!PF=dfyFl zgR>Lc!ovZ?WHSehPKE(hHTT#ukc*(E256z|}LY>BE z6rke9PDII+3TlJgzE3~jBkc*sVM^N1Q!l++ySPJ3m60;Iy6QrL5#H633}`~%&%fRO zO^Um+MJIE(T3Np|=Q7i1>CK3QE_#*HaPu52{reKR>yI86vE9*@^SX(9R=9tK`XPZT z`CLsLm$#iyI4vtAZI^W`u_EJju*Ka`aD_98S&RP+urrjc#03^Dv$7q^*DE8dq6 ze8^D9S4y2KbRv~3g|gj#PHiqP&cXLyy{@x%{fy(-IG-okrS))&225xTIGS+ zRqaHkW8=gIXAecjQi5(LM|D$J`I{7F!If&I$zfb;`fhLZ2k+w#5j!&k$JCj3wy3iD z2CCn#jb-X;I>}X`UWTW{Dw-+-4QL>!CsgguWrzOk`-RKF0vr7X+|LHl^1rY;+pZn( z302@mO!+@^ewZf1-nJ=zOq2CQA^ojbSWZEw$V!*sO4$)BbQt#;RQXF0LcBfGBSy7z zYe-&iDpwO5THLoaNc%#?m{jx3Y4awcP>H;=^ysh zd*t(Bo|h#EnyL5fd9n7erHc-F?x}%=<>TX3SCc2*YqNi?yYzkGjaVSCWt|(?>V6d% z2Y<+4x>%UF{@hUiL+vs+vhyJz10Z=MgYa1pQS_F3@rjuVBSG%bGXvXq7SHBKVq3j- z4uYeKit4=IZwZHP1&g?k6vtcFr4*#kk1{Rb1s4@_DYolpW@HNWa9mcywZ@P3By0nx z$B~n0T9H~UC0?yFxG7h2rJhXI+W|AXF2dP>skEqu>}9HL+zXDdgK)Kz!yJTX7Y+nvio$p zCP~#_dU zk_}}_QI@bCZ@6`hCI;E;C7YM=#PV7?KpHvdLl-?pj?G)J>aMs<#`;?*9d0+;+pc$Q zxGvjfSS;;vbVN+DoedA-Z#d8RrslRAQ*&q3*1eqohl-@3w)+42W`PuAI?E8C-nf3N zH#-Lt$N$z_R-cOP_zfb^^pWf-;k7Lc4!+QAffyQUi$2=)FK1|NEjaOUx$#}kkPOgq74snp+qoo? zOlss;q|9A5Ul}%^VE?FkJ{_`hS%BQ$L7N`=DGQcu3Q^st2y;!fwo(;+l`E>N(cb>~ zy09zexU`@TrxFnv=VgiQ|t}hu0-b;LSlHHd7twYL4_`yRSk|+Hix_brp>z`sRw5 zc?$8|q{;DaBzreR?ql5Jbz>fM)HE7#1@B}kHddI2kCE7r{iMA`kwM-ST=$ZxG+}!s zcgIXp&9$%T<_UIKDUOa30oKyVWg>61mNyPeRVJ*<@lti*9{n-&FPCTnXQQwN0u$@` zzj)vHSX7tMxe#C>gRc9Wr^Wu!&|vxJGnPA0u>#0n7A_h$OT`Ru%pBm<%vI23wErH? zF|u_q`BAhRA2HCFKKwj70|5TtefdV}$v1M4`fXvN?zelz-F^@D&6&lnR53J=+_{@- zrjsWt=TK4sv(+wLRo>?EO4_sAO>Z9bw}@nee|Z1?Dqp+IC5>|J@xfqW&At6ddbG4` zles%xZ3LIfS(KEn3|?6fzEfpcpW+w(APQfin7csA3NuYSyS|Mh&^KhJ&z+8w_AIDG zwYrl`IniWk0Mw!VF}0e*TAFT@)smNGJj51DJe!Hfpzi4htA=I%*VBq{M*|sXxYS(E zE6BI|0)1fdFoR*|sJq~=WssEIFhZq9bv5hL9MbTDPcdR4_|dq-FZBhiIkns_+ZzSu zuX~3+=w%@!e$Sq3-%o!;MRx|G7Cmz2=bAS1e4FNXV(R(;`S28tP-n8%vV9kgvbVV= zja_6ww(326bvC?ezSMQ_U9|Aro$kjS?#<-_LL_R3L$)qFC^nu`d?fEKPlARp`MRzG zJCp)=B0K_z6S6`8AN-$9Z+#D3MhbYQBXHu2=y&flw6pvF&UxM#s~)M?j&Tq{lZsFV z3)K{3fi7)jUp^5Lnn>O%>b3E_IYd2PAfj=&=4vl@UE|T`grchL$;r5|&ZWKlwI%H+ z8u*8-x27re9CQX(SYt*X7AlHDDm!EX8f22QUs{y-hwJV2CXICHV2iR`#$Zr*W2zo( zBVobB@=21yVALS3sOH+$`?YB$chu2JOW-O#f%~2U)M{>_2J~&*!#Jkdv&Q5ojYnf2 zSJ;K;J^Y`)Tj0C$KJz+!84>htI9x=V9lf>;(P54|8sex;cIzXZ=jSW7K5TSaK@xAx z?@MYNGQwk;1HOrpv%P%5mTI1OZ$3;;f=T#=A#i0twlBF3yDk}=;&kBYMb zU6dcJ$tu@wPqb3H6Qf;+WQ;y^V^HYQ)~2G*(hErJ%?^BNbxB65(%9R}2As*>?OHOe z_cSwRj-w8{NE-b0;bDZXB>v3QxRUtp)t&st4gHBYx5XXbnK-XTHNxykYa#wmxu+Y> z=C_NU@2*x~FIO99Ni?3w=zQ>n$S~7Og;?h*80dtVDiSI#| z{oNrC3!E8*WhQ7Qy_SisUiNrQZkxq$=!9eUj7t=9(T>q{NYB+1B^6yKC|}$J$5Ns2 z|I?DtF0p~m112moNw6}YmB|6W|1pgH`O>*>YvW{MZOmj4rA4tGih zPM?8hfd&a&bokqA2bk6T(oXt$boF9FO2tH_Y61}G`Uen*?6+hf5Cb4|^dNA{nZ zJ3k{g7ZK%Ctgdr_C@D4bFWN86j|nJH z<%Uicwl+U1{4Zg1qe$uz!196r!}8Br|H#_U%KK@si>1TkdR(3kIL&Lo^GN^5&nux{ zh>JP7NRXVCvy=x)qX1;&<{tz-(cd!hA04lsS^4$NU%EoN0_6rTFrT z?$<@!#p?~vaq*h}jQc63b`f{+%C&Qxy3Rl2ehL6y#9h4E=NuHlq)f8lq(?w+{FyC@2O&P%rZUwD7nE#h5}VLs=lLt_vw*KE zo&N&&TW9pk0>9~$UM%_FqZ?fQ75ZHJ^PkW^Wc|)x?XLd@eXghZ@5#Mx|Au_7QF)Pk zQR(qK&x7v2k^dt`_nW0J>ONj1|8D3%t-<5*8~Hyj`y2V!kIp3z|2_Gh=f5GJ%Lra1 zUlijzw}zP4Z{+`2<7fB&n;kCV{@w&3uYSY*tqJ~qWc_<`l;6K0|NRW$@5yid{|))? zXDWYBP73%p7oQhi zBwUQPo)eDBe-ZwOy!@;^d=7n5I%JDn%D1m>|n2A>O2sEdF{=` wK_Ho$->mqbNYBOOe=fd%Prg|9Tk`+A6jzqV#QO0!%)o~&9w4mQ^y9n#14n(zH2?qr