{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Entraîner un modèle...\n", "\n", "Il y a plusieurs approches. Nous allons en présenter deux: w2v et glove.\n", "\n", "## Word2Vector\n", "\n", "[Documentation gensim](https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html)\n", "\n", "- Continous bag of words (CBOW): utiliser le contexte comme input pour prévoir le mot cible.\n", "\n", "- Skip-gram: utiliser un mot comme input pour prévoir le contexte. [Article original](https://proceedings.neurips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf)\n", "\n", "\n", "![](https://wiki.pathmind.com/images/wiki/word2vec_diagrams.png)\n", "\n", "\n", "## Skip-gram\n", "\n", "1. One hot encoding: un vecteur du même nombre des dimension que la taille du vocabulaire.\n", "\n", "$$\n", "\\begin{bmatrix}\n", " 0\\\\\n", " 0\\\\\n", " 0\\\\\n", " 1\\\\\n", " 0\\\\\n", " \\vdots \\\\\n", " 0\\\\\n", " \\end{bmatrix}\n", "$$\n", "\n", "2. une couche cachée\n", "3. un output (une série d'autres one hot encodings)\n", "\n", "Le but est d'ajuster les poids pour ensuite pouvoir représenter le mot avec la cocuhe cachée. Donc la tâche qu,on donne au réseau ne sert pas à entraîner un modèle capable ensuite de réaliser cette tâche (deviner les mots du contexte), mais cela servira simplement à représenter le mot.\n", "\n", "![](http://mccormickml.com/assets/word2vec/training_data.png)\n", "\n", "\n", "Concrètement une ligne de la couche cachée sera notre vecteur:\n", "\n", "![]()http://mccormickml.com/assets/word2vec/matrix_mult_w_one_hot.png\n", "\n", "\n", "L'idée est que si deux mots se trouvent statistiquemnt dans des contextes très similaires ils doivent être proches. " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from gensim.models import Word2Vec" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "with open('fichiers/recherche.txt', 'r') as file:\n", " recherche = file.read()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "recherche = recherche.replace('\\xa0', '')\n", "recherche = recherche.replace('\\n', ' ')\n", "sentences = [s for s in recherche.split('.')]\n", " " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "sentences = [s.split(' ') for s in sentences]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "model = Word2Vec(sentences=sentences)\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "tags": [ "output_scroll" ] }, "outputs": [ { "data": { "text/plain": [ "['de',\n", " '',\n", " 'la',\n", " 'à',\n", " 'que',\n", " 'et',\n", " 'le',\n", " 'les',\n", " 'qui',\n", " 'ne',\n", " 'pas',\n", " 'un',\n", " 'en',\n", " 'je',\n", " 'des',\n", " 'une',\n", " 'dans',\n", " 'pour',\n", " 'ce',\n", " 'plus',\n", " 'il',\n", " 'du',\n", " 'comme',\n", " 'me',\n", " 'avait',\n", " 'par',\n", " 'elle',\n", " 'se',\n", " 'lui',\n", " 'nous',\n", " 'si',\n", " 'avec',\n", " 'qu’il',\n", " 'au',\n", " 'était',\n", " 'son',\n", " 'mais',\n", " 'sa',\n", " 'vous',\n", " 'tout',\n", " 'où',\n", " 'sur',\n", " '—',\n", " 'même',\n", " 'qu’elle',\n", " 'cette',\n", " 'd’un',\n", " 'ou',\n", " 'est',\n", " 'ces',\n", " 'bien',\n", " 'on',\n", " 'M^(me)',\n", " 'a',\n", " 'Mais',\n", " 'ma',\n", " 'sans',\n", " 'mon',\n", " 'ses',\n", " 'd’une',\n", " 'faire',\n", " 'leur',\n", " 'M',\n", " 'quand',\n", " 'qu’on',\n", " 'Je',\n", " 'fait',\n", " 'c’est',\n", " 'y',\n", " 'Et',\n", " 'dont',\n", " 'été',\n", " 'chez',\n", " 'Il',\n", " 'moi',\n", " 'dit',\n", " 'aux',\n", " 'peu',\n", " 'aussi',\n", " 'être',\n", " '»',\n", " 'encore',\n", " 'j’avais',\n", " 'cela',\n", " 'moins',\n", " 'quelque',\n", " 'c’était',\n", " 'fois',\n", " 'parce',\n", " 'jamais',\n", " 'avoir',\n", " 'voir',\n", " 'Swann',\n", " 'sont',\n", " 'mes',\n", " 'dire',\n", " 'n’était',\n", " 'temps',\n", " 'vie',\n", " 'eût',\n", " 'tous',\n", " 'n’est',\n", " 'car',\n", " 'peut-être',\n", " 'faisait',\n", " 'deux',\n", " 'étaient',\n", " 'très',\n", " 'ils',\n", " 'Guermantes',\n", " 'non',\n", " 'notre',\n", " 'rien',\n", " 'qui,',\n", " 'que,',\n", " 'femme',\n", " 'qu’ils',\n", " 'moment',\n", " 'toujours',\n", " 'autre',\n", " 'seulement',\n", " 'Elle',\n", " 'devant',\n", " 'pu',\n", " 'tant',\n", " 'trop',\n", " 'chose',\n", " 'jour',\n", " 'n’avait',\n", " 'gens',\n", " 'pas,',\n", " 'toute',\n", " 'moi,',\n", " 'avaient',\n", " 'toutes',\n", " 'monde',\n", " 'après',\n", " 'eu',\n", " 'pouvait',\n", " 'celui',\n", " 'l’air',\n", " 'celle',\n", " 'peut',\n", " 'yeux',\n", " 'depuis',\n", " 'La',\n", " 'déjà',\n", " 'jeune',\n", " 'entre',\n", " 'n’y',\n", " 'leurs',\n", " 'Albertine',\n", " 'personne',\n", " 'Le',\n", " 'grand',\n", " 'quelques',\n", " 'avant',\n", " 'm’avait',\n", " 'sous',\n", " 'nom',\n", " 'plaisir',\n", " 'd’être',\n", " 'Charlus',\n", " 'assez',\n", " 'laquelle',\n", " 'qu’un',\n", " 's’il',\n", " 'elles',\n", " 'homme',\n", " 'pendant',\n", " 'C’est',\n", " 'aurait',\n", " 'cet',\n", " 'ceux',\n", " 'beaucoup',\n", " 'petit',\n", " 'presque',\n", " 'qu’à',\n", " 'ainsi',\n", " 'vers',\n", " 'semblait',\n", " 'lui,',\n", " 'souvent',\n", " 'Verdurin',\n", " 'Guermantes,',\n", " 'fut',\n", " 'fût',\n", " 'serait',\n", " 'Ce',\n", " 'j’étais',\n", " 'princesse',\n", " 'duchesse',\n", " 'cause',\n", " 'maintenant',\n", " 'soit',\n", " 'doute',\n", " 'autres',\n", " 'grande',\n", " 'chaque',\n", " 'et,',\n", " 'choses',\n", " 'qu’une',\n", " 'pourtant',\n", " 'Saint-Loup',\n", " 'côté',\n", " 'alors',\n", " 'sorte',\n", " 'On',\n", " 'là',\n", " 'disait',\n", " 'ont',\n", " 'elle,',\n", " 'tu',\n", " 'petite',\n", " 'ayant',\n", " 'première',\n", " 's’était',\n", " 'Françoise',\n", " 'jours',\n", " 'j’ai',\n", " 'grand’mère',\n", " 'parler',\n", " 'seul',\n", " 'venait',\n", " 'jusqu’à',\n", " 'nos',\n", " 'donner',\n", " 'aller',\n", " 'suis',\n", " 'Gilberte',\n", " 'vu',\n", " 'fille',\n", " 'En',\n", " 'À',\n", " 'mal',\n", " 'prendre',\n", " 'Les',\n", " 'votre',\n", " 'devait',\n", " 'contre',\n", " 'seule',\n", " 'mère',\n", " 'allait',\n", " 'venir',\n", " 'n’a',\n", " 'air',\n", " 'd’autres',\n", " 'longtemps',\n", " 'n’en',\n", " 'désir',\n", " 'ni',\n", " 'point',\n", " 'qu’elles',\n", " 'près',\n", " 'lequel',\n", " 'l’autre',\n", " 'duc',\n", " 'fort',\n", " 'vue',\n", " 'peine',\n", " 'voulu',\n", " 'premier',\n", " 'femmes',\n", " 'père',\n", " 'reste',\n", " 'passer',\n", " 'besoin',\n", " 'savoir',\n", " 'd’avoir',\n", " 'faut',\n", " 'loin',\n", " 'n’avais',\n", " 'lieu',\n", " 'mieux',\n", " 'puis',\n", " 'eux',\n", " 'personnes',\n", " 'l’avait',\n", " 'trouver',\n", " 'd’ailleurs',\n", " 'milieu',\n", " 'cœur',\n", " 'chambre',\n", " 'beau',\n", " 'Vous',\n", " 'partie',\n", " 'sais',\n", " 'visage',\n", " 'Swann,',\n", " 'soir',\n", " 'façon',\n", " 'j’aurais',\n", " 'croire',\n", " 'savait',\n", " 'jeunes',\n", " 'place',\n", " 'coup',\n", " 'Si',\n", " 'qu’en',\n", " 'dès',\n", " 'mort',\n", " 'pouvoir',\n", " 'dîner',\n", " 'quelqu’un',\n", " 'celles',\n", " 'maison',\n", " 'souvenir',\n", " 'demander',\n", " 'certains',\n", " 'tel',\n", " 'telle',\n", " 'Bloch',\n", " 'surtout',\n", " 'voulait',\n", " 'instant',\n", " 'pris',\n", " 'mot',\n", " 'donnait',\n", " 'ça',\n", " 'ton',\n", " 'reste,',\n", " 'dû',\n", " 'regard',\n", " 'hommes',\n", " 'Balbec',\n", " 'plutôt',\n", " 'heures',\n", " 'trouvait',\n", " 'Comme',\n", " 'bon',\n", " 'malgré',\n", " 's’en',\n", " '«Je',\n", " 'prince',\n", " 'auprès',\n", " 'passé',\n", " 'plus,',\n", " 'raison',\n", " 'bonne',\n", " 'vie,',\n", " 'penser',\n", " 'Car',\n", " 'fit',\n", " 'puisque',\n", " 'répondit',\n", " 'Pour',\n", " 'De',\n", " 'contraire',\n", " 'rendre',\n", " 'vrai',\n", " 'mettre',\n", " 'Odette',\n", " 'fond',\n", " 'compte',\n", " 'voix',\n", " 'tête',\n", " 'monde,',\n", " 'Un',\n", " 'part',\n", " 'amour',\n", " 'Quand',\n", " 'certaines',\n", " 'Paris',\n", " 'connaître',\n", " 'laisser',\n", " 'bien,',\n", " 'sens',\n", " 'dame',\n", " 'aucune',\n", " 'tante',\n", " 'avons',\n", " 'trois',\n", " 'genre',\n", " 'Robert',\n", " 'enfin',\n", " 'entendu',\n", " 'corps',\n", " 'Verdurin,',\n", " 'autant',\n", " 'mots',\n", " 'd’elle',\n", " 'même,',\n", " 'chercher',\n", " 'crois',\n", " 'effet,',\n", " 'l’idée',\n", " 'cru',\n", " 'aucun',\n", " 'réalité',\n", " 'Balbec,',\n", " 'tandis',\n", " 'Cambremer',\n", " 'Charlus,',\n", " 'famille',\n", " 'porte',\n", " 'Villeparisis',\n", " 'main',\n", " 'plusieurs',\n", " 'faisant',\n", " 'amis',\n", " 'l’heure',\n", " 'va',\n", " 'grands',\n", " 'pourrait',\n", " 'nouveau',\n", " 'sourire',\n", " 'lettre',\n", " 'pouvais',\n", " 'd’aller',\n", " 'lui-même',\n", " 'salon',\n", " 'paroles',\n", " 'certain',\n", " 'croyait',\n", " 'avais',\n", " 'n’eût',\n", " 'cela,',\n", " 'Nous',\n", " 'pensée',\n", " 'avez',\n", " 'n’étaient',\n", " 'ami',\n", " 'cas',\n", " 'Albertine,',\n", " 'train',\n", " 'C’était',\n", " 'd’abord',\n", " 'l’on',\n", " 'figure',\n", " 'd’Albertine',\n", " 'rester',\n", " 'force',\n", " 'nouvelle',\n", " 'effet',\n", " 'donné',\n", " 'M^(lle)',\n", " 'femme,',\n", " 'Sans',\n", " 'quoi',\n", " 'sait',\n", " 'l’un',\n", " 'nous,',\n", " 'tard',\n", " 'Ah!',\n", " 'vieux',\n", " 'te',\n", " 'autour',\n", " 'sera',\n", " 'mêmes',\n", " 'goût',\n", " 'suite',\n", " 'Dans',\n", " 'peur',\n", " 'grâce',\n", " 'étant',\n", " 'Au',\n", " 'tenait',\n", " 'connaissait',\n", " 'fin',\n", " 'savais',\n", " 'selon',\n", " 'faisaient',\n", " 'trouve',\n", " 'dis',\n", " 'Cottard',\n", " 'sentais',\n", " 'Norpois',\n", " 'parents',\n", " 'simple',\n", " 'encore,',\n", " 'montrer',\n", " 'baron',\n", " 'forme',\n", " 'soleil',\n", " 'vraiment',\n", " 'bout',\n", " 'm’a',\n", " 'voir,',\n", " '«Mais',\n", " 'Cette',\n", " 'quelle',\n", " 'm’en',\n", " 'beauté',\n", " 'trouvé',\n", " 'où,',\n", " 'autrefois',\n", " 'autres,',\n", " 'conversation',\n", " 'manière',\n", " 'Morel',\n", " 'd’autant',\n", " 'n’ai',\n", " 'voyait',\n", " 'petits',\n", " 'fois,',\n", " 'joie',\n", " 'Mais,',\n", " 'comment',\n", " 'combien',\n", " 'Tout',\n", " 'font',\n", " 'Aussi',\n", " 'propre',\n", " 'présence',\n", " 'l’esprit',\n", " 'Gilberte,',\n", " 'certaine',\n", " 'doit',\n", " 'temps,',\n", " 'filles',\n", " 'sentiment',\n", " 'comprendre',\n", " 'fallait',\n", " 'mer',\n", " 'êtes',\n", " 'tour',\n", " 'fleurs',\n", " 'dire,',\n", " 'elle-même',\n", " 'eussent',\n", " 'quel',\n", " 'rue',\n", " 'l’avais',\n", " 'relations',\n", " 'voiture',\n", " 'Saint-Loup,',\n", " 'mari',\n", " 'esprit',\n", " 'parlait',\n", " 'semble',\n", " 'maîtresse',\n", " 'là,',\n", " 'jadis',\n", " 'amie',\n", " 'justement',\n", " 'vérité',\n", " 'aussitôt',\n", " 'mille',\n", " 'Combray',\n", " 'demanda',\n", " 'tellement',\n", " '«Vous',\n", " 'heure',\n", " 'donc',\n", " 'années',\n", " 'bonheur',\n", " 'lesquels',\n", " 'voyant',\n", " 'nature',\n", " 'si,',\n", " 's’y',\n", " 'l’habitude',\n", " 'vite',\n", " 'compris',\n", " 'maître',\n", " 'Combray,',\n", " 'jour,',\n", " 'chacun',\n", " 'pauvre',\n", " 'propos',\n", " 'sentait',\n", " 'd’ailleurs,',\n", " 'n’avaient',\n", " 'Une',\n", " 'l’amour',\n", " 'celle-ci',\n", " 'Paris,',\n", " 'phrase',\n", " 'tenir',\n", " 'charme',\n", " 'sommes',\n", " 'vient',\n", " 'nuit',\n", " 'connu',\n", " 'retrouver',\n", " 'pied',\n", " 'fils',\n", " 'visite',\n", " 'caractère',\n", " 'dit:',\n", " 'voyais',\n", " 'belle',\n", " 'J’avais',\n", " 'vieille',\n", " 'ai',\n", " '«C’est',\n", " 'qu’Albertine',\n", " 'pût',\n", " 'mis',\n", " 'ici',\n", " 'donne',\n", " 'd’où',\n", " 'regarder',\n", " 'tard,',\n", " 'demandé',\n", " 'mémoire',\n", " 'disait:',\n", " 'idée',\n", " 'nouvelles',\n", " 'semblaient',\n", " 'venu',\n", " 'peux',\n", " 'd’en',\n", " 'peuvent',\n", " 'derrière',\n", " 'dehors',\n", " 'eut',\n", " 'parfois',\n", " 'ans',\n", " 'm’eût',\n", " 'dernière',\n", " 'lesquelles',\n", " 'possible',\n", " 'Or,',\n", " 'non,',\n", " 'chemin',\n", " 'rien,',\n", " 'savez',\n", " 'n’aurait',\n", " 'eux,',\n", " 'jusqu’au',\n", " 'mère,',\n", " 'parlé',\n", " 'Bergotte',\n", " 'long',\n", " 'Du',\n", " 'Ils',\n", " 'j’en',\n", " 'Françoise,',\n", " 'ait',\n", " 'salle',\n", " 'partir',\n", " 'soir,',\n", " 'voit',\n", " 'l’hôtel',\n", " 'tout,',\n", " 'Alors',\n", " 'qu’au',\n", " 'veut',\n", " 'entièrement',\n", " 'dernier',\n", " 'contraire,',\n", " 'entrer',\n", " 'appris',\n", " 'sortir',\n", " 'pays',\n", " 'aussi,',\n", " 'Mon',\n", " 'entendre',\n", " 'grandes',\n", " 'regards',\n", " 'simplement',\n", " 'absolument',\n", " 'quand,',\n", " 'Ma',\n", " 'quelquefois',\n", " 'matin',\n", " 'ensuite',\n", " 'petites',\n", " 'auquel',\n", " 'Or',\n", " 'ciel',\n", " 'd’elle,',\n", " 'êtres',\n", " 'vais',\n", " 'lendemain',\n", " 'connaissance',\n", " 'disant',\n", " 'soirée',\n", " 'valeur',\n", " 'cours',\n", " 'c’est-à-dire',\n", " '«Il',\n", " 'j’eusse',\n", " 'celui-ci',\n", " 'heureux',\n", " 'Peut-être',\n", " 'vivre',\n", " 'cheveux',\n", " 'traits',\n", " 'noms',\n", " 'd’Albertine,',\n", " 'part,',\n", " 'moments',\n", " 'bientôt',\n", " 'aimait',\n", " 'restait',\n", " 'guerre',\n", " 'dit-il',\n", " 'devoir',\n", " 'passait',\n", " 'lumière',\n", " 'grand’mère,',\n", " 'table',\n", " 'Puis',\n", " 'au-dessus',\n", " 'rire',\n", " 'nullement',\n", " 'mauvais',\n", " 'souffrance',\n", " 'dire:',\n", " 'l’être',\n", " 'seconde',\n", " 'croyais',\n", " 'idées',\n", " 'duquel',\n", " 'société',\n", " 'mauvaise',\n", " 'jouer',\n", " 'espèce',\n", " 'autre,',\n", " 'venais',\n", " 'dit,',\n", " 'amies',\n", " 'guère',\n", " 'lettres',\n", " 'plaisirs',\n", " 'D’ailleurs,',\n", " 'situation',\n", " 'véritable',\n", " 'su',\n", " 'nom,',\n", " 'laissait',\n", " 'perdu',\n", " 'mois',\n", " 'm’étais',\n", " 'livre',\n", " 'l’autre,',\n", " 'davantage',\n", " 'fus',\n", " 'oncle',\n", " 'Andrée',\n", " 'mouvement',\n", " 'docteur',\n", " 'valet',\n", " 'haut',\n", " 'vrai,',\n", " 'hasard',\n", " 'père,',\n", " 'exemple,',\n", " 'qu’avait',\n", " 'souvenirs',\n", " 'pouvaient',\n", " 'pourquoi',\n", " 'faire,',\n", " 'm’était',\n", " 'vous,',\n", " 'devenu',\n", " 'raisons',\n", " 'mais,',\n", " 'revenir',\n", " 'cependant',\n", " 'Cela',\n", " 'yeux,',\n", " 'recevoir',\n", " '(et',\n", " 'voulais',\n", " 'Jupien',\n", " 'rendait',\n", " 'veux',\n", " 'duchesse,',\n", " 'vis',\n", " 'maman',\n", " 'sentir',\n", " 'obligé',\n", " 'Quant',\n", " 'premiers',\n", " 'rose',\n", " 'attention',\n", " 'cessé',\n", " 'demandait',\n", " 'laissé',\n", " 'différent',\n", " 'alors,',\n", " 'pensais',\n", " 'doute,',\n", " 'bruit',\n", " 'l’une',\n", " 'Cottard,',\n", " 'Ainsi',\n", " 'n’ont',\n", " 'dîner,',\n", " 'marquise',\n", " 'reçu',\n", " 'paraître',\n", " 'causer',\n", " 'étions',\n", " 'douceur',\n", " 'aime',\n", " 'ensemble',\n", " 'avions',\n", " 'pensant',\n", " 'prenait',\n", " 's’est',\n", " 'afin',\n", " 'ferait',\n", " 'exactement',\n", " 'demande',\n", " 'l’ai',\n", " 'd’y',\n", " 'Ces',\n", " 'l’a',\n", " 'aimé',\n", " 'moi-même',\n", " 'mains',\n", " 'l’homme',\n", " 'impression',\n", " 'fini',\n", " 'travers',\n", " 'trouvais',\n", " 'chagrin',\n", " 'personne,',\n", " 'présenter',\n", " 'Brichot',\n", " 'd’autre',\n", " 'moins,',\n", " 'jours,',\n", " 'capable',\n", " 'vos',\n", " 'mesure',\n", " 'pensait',\n", " 'plein',\n", " 'musique',\n", " 'lui-même,',\n", " 'face',\n", " 'disais',\n", " '«Ah!',\n", " 'moment,',\n", " 'chambre,',\n", " 'Odette,',\n", " 'n’étais',\n", " 'voyage',\n", " 'nez',\n", " 'parmi',\n", " 'vois',\n", " 'paru',\n", " 'maison,',\n", " 'cherchait',\n", " 'rendu',\n", " 'exemple',\n", " 'lit',\n", " 'difficile',\n", " 'signe',\n", " 'pourrais',\n", " 'n’aurais',\n", " 'j’allais',\n", " 'douleur',\n", " 'paraissait',\n", " 'départ',\n", " 'Forcheville',\n", " 'malade',\n", " 'D’ailleurs',\n", " '—Mais',\n", " 'mariage',\n", " 'précisément',\n", " 'nombre',\n", " 'revoir',\n", " 'cas,',\n", " 'quitter',\n", " 'haute',\n", " 's’ils',\n", " 'Même',\n", " 'bras',\n", " ')',\n", " 'd’eux',\n", " 'femmes,',\n", " 'j’y',\n", " 'Berma',\n", " 'tendresse',\n", " 'dit-elle',\n", " 'mettait',\n", " 'connaissais',\n", " 'avait,',\n", " 'arriver',\n", " 'écrit',\n", " 'dix',\n", " 'portrait',\n", " 'état',\n", " 'Monsieur',\n", " 'paraît',\n", " 'fait,',\n", " 'est-ce',\n", " 'Vinteuil',\n", " 'faisais',\n", " 'minutes',\n", " 'venue',\n", " 'belles',\n", " 'général',\n", " 'n’ayant',\n", " 'tristesse',\n", " 'parle',\n", " 'l’eût',\n", " 'J’ai',\n", " 'retour',\n", " 'chapeau',\n", " 'impossible',\n", " 'parlant',\n", " 'silence',\n", " 'l’égard',\n", " 'auraient',\n", " 'rôle',\n", " 'chose,',\n", " 'pensé',\n", " 's’étaient',\n", " 'robe',\n", " 'aujourd’hui',\n", " 'fussent',\n", " 'reconnaître',\n", " 'marquis',\n", " 'vint',\n", " 'œuvre',\n", " 'déjeuner',\n", " 'm’avaient',\n", " 'bas',\n", " 'sinon',\n", " 'plaisir,',\n", " 'rentrer',\n", " 'Rachel',\n", " 'sûr',\n", " 'demi',\n", " 'connaît',\n", " 'Bloch,',\n", " 'fille,',\n", " 'fenêtre',\n", " 'moyen',\n", " 'envie',\n", " 'arrivé',\n", " 'changer',\n", " 'couleur',\n", " 'parole',\n", " 'serais',\n", " 'Villeparisis,',\n", " 'agréable',\n", " 'répondre',\n", " 'question',\n", " 'auxquels',\n", " 'faite',\n", " 'ville',\n", " 'oublié',\n", " 'manger',\n", " 'Legrandin',\n", " 'sentiments',\n", " 'monsieur',\n", " 'd’Odette',\n", " 'terre',\n", " 'curiosité',\n", " 'cinq',\n", " 'Elstir',\n", " 'satisfaction',\n", " 'mer,',\n", " 'rapport',\n", " 'mort,',\n", " 'croit',\n", " 'l’impression',\n", " 'différente',\n", " 'connais',\n", " 's’écria',\n", " 'jalousie',\n", " 'maladie',\n", " 'promenade',\n", " 'etc',\n", " 'France',\n", " 'beaux',\n", " 'manque',\n", " 'l’intelligence',\n", " 'jusque-là',\n", " 'seraient',\n", " 'Par',\n", " 'pièce',\n", " 'sommeil',\n", " 'Que',\n", " 'différentes',\n", " 'droit',\n", " 'lire',\n", " 'Madame',\n", " 'rencontré',\n", " 'roi',\n", " 'Morel,',\n", " 'tantôt',\n", " 'famille,',\n", " 'faubourg',\n", " 'pourtant,',\n", " 'génie',\n", " 'titre',\n", " 'langage',\n", " 'époque',\n", " 'Cambremer,',\n", " 'matière',\n", " 'monter',\n", " 'joues',\n", " 'tort',\n", " 'était,',\n", " 'l’eau',\n", " 'elle-même,',\n", " 'expression',\n", " 'service',\n", " 'quatre',\n", " 'faites',\n", " 'longue',\n", " 'journée',\n", " ...]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.wv.index_to_key" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "#un peu de paramètres: min_count= combien de fois un terme doit apparaître pour être mis dans le vocabulaire\n", "model_sg = Word2Vec(sentences=sentences, min_count=1, sg=1, window=10)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('tempête', 0.9400717616081238),\n", " ('Roussainville,', 0.9392350316047668),\n", " ('Montjouvain,', 0.9377919435501099),\n", " ('description', 0.9354062676429749),\n", " ('plafond', 0.9351763725280762),\n", " ('boîte', 0.9307337403297424),\n", " ('Boulogne', 0.9298250079154968),\n", " ('plénitude', 0.9287278056144714),\n", " ('date', 0.9282354712486267),\n", " ('brouillard,', 0.92800372838974)]" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_sg.wv.most_similar('madeleine')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## CBOW\n", "\n", "Le principe est juste inversé: on prend en entrée le contexte pour deviner un mot cible.\n", "\n", "Skip-Gram fonctionne bien avec de petits ensembles de données et peut mieux représenter les mots moins fréquents.\n", "\n", "CBOW s'entraîne plus rapidement que Skip-Gram, et peut mieux représenter les mots plus fréquents.\n" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "model_cbow = Word2Vec(sentences=sentences, min_count=1, sg=0, window=10)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('fuite', 0.9220989346504211),\n", " ('fraîcheur', 0.9204322099685669),\n", " ('bougie', 0.9201040863990784),\n", " ('saison', 0.9167993068695068),\n", " ('fenêtre', 0.9135265350341797),\n", " ('vision', 0.9111268520355225),\n", " ('raideur', 0.9093303680419922),\n", " ('digue', 0.9083762764930725),\n", " ('lecture', 0.9074811935424805),\n", " ('serviette', 0.9036232829093933)]" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_cbow.wv.most_similar('madeleine')" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('temps,', 0.8312168717384338),\n", " ('nationalisme', 0.6660643815994263),\n", " ('souvenir', 0.6596003770828247),\n", " ('désir', 0.6570372581481934),\n", " ('sens', 0.6536211967468262),\n", " ('plaisir', 0.6476064920425415),\n", " ('sentiment', 0.6475840210914612),\n", " ('charme', 0.6474167108535767),\n", " ('chose', 0.6332330107688904),\n", " ('pays', 0.6294322609901428)]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.wv.most_similar('temps')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Glove\n", "\n", "[Article original](https://nlp.stanford.edu/pubs/glove.pdf)\n", "\n", "Au lieu que prendre en compte le vocabulaire comme input on part de la matrice des co-occurrences des mots. Donc GloVe (Global Vectors) prend en compte les statistiques d'occurences globales et non seulement locales (comme w2v).\n", "\n", ">Unlike the matrix factorization methods, the shallow window-based methods suffer from the disadvantage that they do not operate directly on the co-occurrence statistics of the corpus. Instead, these models scan context windows across the entire corpus, which fails to take advantage of the vast amount of repetition in the data. \n", "\n", "Par exemple, admettons d'avoir la phrase:\n", "\n", ">La littérature est un art.\n", "\n", "W2v sera capable de rendre compte du fait qu'il y a un lien entre \"littérature\" et \"art\". Mais il ne saura pas profitter d'une autre information; combien de fois, dans l'ensemble du corpus, le mot \"littérature\" et le mot \"art\" se touvent ensemble?\n", "\n", "GloVe ne se requiert pas un réseau de neurones. Les vecteurs sont déterminés sur la base de la probabilité de co-occurence des mots selon la formule:\n", "\n", "$$\n", "F(w_i, w_j, \\tilde{w_k}) = \\frac{Pi_k}{Pj_k} \n", "$$" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "celltoolbar": "Edit Metadata", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.2" } }, "nbformat": 4, "nbformat_minor": 4 }