BeanDev: Mastodonhttps://plume.atsuchan.page/~/BeanDevMastodon@write.tchncs.de/atom.xml2022-06-15T08:40:41.589172+00:00<![CDATA[Multi-Instance, Multi-Account - MastodonHelper mit einem kleinen Update]]>https://write.tchncs.de/~/BeanDevMastodon/Multi-Instance,%20Multi-Account%20-%20MastodonHelper%20mit%20einem%20kleinen%20Update/2022-06-15T08:40:41.589172+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-06-15T08:40:41.589172+00:00<![CDATA[<p dir="auto">Für einen Test auf der Mastodon-Instanz, auf der ich beheimatet bin, musste ich Toots über einen bestimmten Zeitraum erstellen. Das wollte ich nicht mit meinem regulären Account machen, also habe ich einen neuen Account angelegt (und den als Bot gekennzeichnet). Die Sichtbarkeit der Toots war nicht wichtig (daher habe ich die nicht gelistet) und Follower waren auch nicht wichtig.</p>
<p dir="auto">Nun war aber der MastodonHelper zwar grundsätzlich mal dafür angedacht mehr als einen Account nutzen zu können, aber es fehlten noch die passenden Implementationen.</p>
<p dir="auto">Das habe ich mit dem <a href="https://codeberg.org/beandev/mastodon-playground/commit/77cc4eeb27ff003d18a56a71fbb6c4d85701f142" rel="noopener noreferrer">letzten Commit</a> erledigt.</p>
<p dir="auto">In der <a href="https://codeberg.org/beandev/mastodon-playground/src/commit/77cc4eeb27ff003d18a56a71fbb6c4d85701f142/mastobot-config.yaml" rel="noopener noreferrer">mastobot-config.yaml</a> kann man nun unter dem Abschnitt <code>instances</code> nicht nur <code>default</code> konfigurieren, sondern mehrere Abschnitte hinterlegen.</p>
<p dir="auto">Das kann so aussehen:</p>
<pre><code dir="auto"><span class="source"><span class="string"><span class="entity name tag">instances</span></span><span class="punctuation">:</span>
<span class="string"><span class="entity name tag">default</span></span><span class="punctuation">:</span>
<span class="string"><span class="entity name tag">url</span></span><span class="punctuation">:</span> <span class="string"><span class="entity name tag">https://social.tchncs.de</span></span>
<span class="string"><span class="entity name tag">username</span></span><span class="punctuation">:</span> <span class="constant language">~</span>
<span class="string"><span class="entity name tag">login</span></span><span class="punctuation">:</span> <span class="constant language">true</span>
<span class="string"><span class="entity name tag">mybot_on_tchncs</span></span><span class="punctuation">:</span>
<span class="string"><span class="entity name tag">url</span></span><span class="punctuation">:</span> <span class="string"><span class="entity name tag">https://social.tchncs.de</span></span>
<span class="string"><span class="entity name tag">username</span></span><span class="punctuation">:</span> <span class="string"><span class="entity name tag">mybot@myprovider.de</span></span>
<span class="string"><span class="entity name tag">login</span></span><span class="punctuation">:</span> <span class="constant language">true</span>
</span></code></pre>
<p dir="auto">Die Default Konfig bleibt, wie ihr es eingerichtet habt. Dazu können beliebig weitere Instanzen mit Usern hinzugefügt werden.</p>
<p dir="auto">Das muss man aber dann auch explizit im Python Script dann so aufrufen. Dazu gibt es den neuen Parameter instance_id (der mit <code>default</code> vorbelegt ist):</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastobot</span><span class="punctuation">.</span><span class="">MastodonHelper</span><span class="punctuation">.</span></span><span class=""><span class="variable function">open_or_create_app</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">config_yaml</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">mastobot-config.yaml<span class="punctuation string">'</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">instance_id</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">mybot_on_tchncs<span class="punctuation string">"</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">login</span><span class="keyword operator">=</span><span class="constant language">True</span>
</span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Da <code>login=True</code> gesetzt ist, wird ein Account-Login über den Username gefordert. Ihr meldet euch im Terminal also einmal an. Der Access-Token eurer Anmeldung wird dann im home-Ordner gespeichert. Das bedeutet, bei weiteren Aufrufen des Script gibt es keine Interaktion mehr: das Script meldet sich automatisch mit dem Access-Token an.</p>
<p dir="auto">Ich musste dafür auch die Dateinamen der Dateien ändern, die die <code>secrets</code> und <code>tokens</code> speichern. Das passt aber weiter zu dem bisherigen Verhalten von dem MastodonHelper.</p>
<p dir="auto">Man kann also weiterhin auch den einfachen Aufruf nutzen (was die default-Konfiguration nutzt):</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastobot</span><span class="punctuation">.</span><span class="">MastodonHelper</span><span class="punctuation">.</span></span><span class=""><span class="variable function">open_or_create_app</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">config_yaml</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">mastobot-config.yaml<span class="punctuation string">'</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">login</span><span class="keyword operator">=</span><span class="constant language">True</span>
</span><span class="punctuation">)</span></span>
</span></code></pre>
]]><![CDATA[Python-Module in Sub-Packages und wie man sich von der IDE veralbern lassen kann]]>https://write.tchncs.de/~/BeanDevMastodon/Python-Module%20in%20Sub-Packages%20und%20wie%20man%20sich%20von%20der%20IDE%20veralbern%20lassen%20kann/2022-06-05T14:00:29.452153+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-06-05T14:00:29.452153+00:00<![CDATA[<p dir="auto">Es gibt ein paar gute Tutorials (<a href="https://www.delftstack.com/de/howto/python/__init__.py-in-python/" rel="noopener noreferrer">hier</a>), die Module und Pakete erklären. </p>
<p dir="auto">Auch die Referenz-Dokumentation zeigt, wie es aufgebaut werden muss: <a href="https://docs.python.org/3/reference/import.html#regular-packages" rel="noopener noreferrer">https://docs.python.org/3/reference/import.html#regular-packages</a></p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">parent</span></span><span class="keyword operator">/</span>
<span class=""><span class="support function">__init__</span><span class="punctuation">.</span><span class="">py</span></span>
<span class=""><span class="">one</span></span><span class="keyword operator">/</span>
<span class=""><span class="support function">__init__</span><span class="punctuation">.</span><span class="">py</span></span>
<span class=""><span class="">two</span></span><span class="keyword operator">/</span>
<span class=""><span class="support function">__init__</span><span class="punctuation">.</span><span class="">py</span></span>
<span class=""><span class="">three</span></span><span class="keyword operator">/</span>
<span class=""><span class="support function">__init__</span><span class="punctuation">.</span><span class="">py</span></span>
</span></code></pre>
<p dir="auto">Arbeitet man mit einer IDE (wie zum Beispiel IntelliJ), kann man direkt aus der IDE ein Regular-Module-Package erzeugen, dass die IDE auch direkt als Modul lädt. Und dann ist die <code>__init__.py</code> Datei nicht notwendig. </p>
<p dir="auto">Wenn man das nutzt (und vergisst), wird bei einer Übergabe des Codes an andere es zu Problemen kommen.</p>
<p dir="auto">Ruft man im Ordner des ausgecheckten Codeberg Repository <code>python</code> auf, um in die Python Shell zu kommen, kann man mit</p>
<pre><code dir="auto"><span class="source"><span class="keyword operator">></span><span class="keyword operator">></span><span class="keyword operator">></span><span class=""><span class=""><span class="support function">help</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">modules<span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">…eine Liste der Module bekommen. <code>mastobot</code> wird, wegen der fehlenden <code>__init__.py</code> nicht dabei sein.</p>
<p dir="auto">Legen wir <code>__init__.py</code> in <code>mastobot</code> (und dann auch in <code>oshelper</code>), haben wir in der 4-spaltigen Liste dann auch <code>mastobot</code> und <code>oshelper</code>. Unsere playground <code>.py</code> Dateien sind ja selbst Module, also lege ich auch dort eine <code>__init__.py</code> an.</p>
<p dir="auto">Wir brauchen also ungefähr diese Struktur:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">my</span></span><span class="keyword operator">-</span><span class=""><span class="">playground</span></span><span class="keyword operator">/</span>
<span class=""><span class="">mastobot</span></span><span class="keyword operator">/</span>
<span class=""><span class="support function">__init__</span><span class="punctuation">.</span><span class="">py</span></span>
<span class=""><span class="">oshelper</span></span><span class="keyword operator">/</span>
<span class=""><span class="support function">__init__</span><span class="punctuation">.</span><span class="">py</span></span>
<span class=""><span class="support function">__init__</span><span class="punctuation">.</span><span class="">py</span></span>
<span class=""><span class="">me</span><span class="punctuation">.</span><span class="">py</span></span>
</span></code></pre>
<p dir="auto">Ich committe mal die Dateien ins <a href="https://codeberg.org/beandev/mastodon-playground" rel="noopener noreferrer">Codeberg Repo</a>, damit es nicht weiter zu Problemen kommt.</p>
]]><![CDATA[Blog-Artikel Scripte migrieren nach #Codeberg]]>https://write.tchncs.de/~/BeanDevMastodon/Blog-Artikel%20Scripte%20migrieren%20nach%20%23Codeberg/2022-05-25T19:21:12.625018+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-25T19:21:12.625018+00:00<![CDATA[<h1 dir="auto">Einleitung</h1>
<p dir="auto"><a href="https://codeberg.org/" rel="noopener noreferrer">Codeberg</a> basiert auf <a href="https://gitea.io/en-us/" rel="noopener noreferrer">gitea</a> und wird von einem Berliner Verein zur Verfügung gestellt und es ist schnell meine erste Wahl geworden. Die Funktionen sind umfangreich, ausgereift und es arbeitet, wie Plume, mit <a href="https://commonmark.org/" rel="noopener noreferrer">Common Markdown</a>. </p>
<p dir="auto">Ich werde aber nicht die bisherigen Scripte 1:1 so übernehmen, wie ich sie in den letzten Artikeln beschrieben hatte. Ich habe ein paar Refactorings vorgenommen, insbesondere in dem Bereich, wo es um den Zugriff auf die Mastodon API und OpenSearch geht. Die Konfiguration der Instanzen und Nodes, Client-IDs, Secrets und Login-Daten werden jetzt natürlich nicht mehr in den Scripten sein. Gerade Mastodon.py unterstützt sehr gut dabei, dass man von Anfang an die Konfiguration von dem Script-Code trennt. Ich hatte das nur bei den Beispielen erstmal gelassen, weil sie so schneller auszuprobieren waren. </p>
<p dir="auto">Da jetzt die Snippets auch aus dem git zu laden sind (und damit auch mal komplexer sein können), ist es klar, dass solche “Komfortfunktionen” integriert werden können, ohne auf den einfachen Charakter des Blogs verzichten zu müssen. D.h. hier bleiben die Beispiele einfach und skizzieren, wie man Lösungen anstrebt und im Repository finden sich dann besser ausformulierte Lösungen. </p>
<h1 dir="auto">Modul MastodonHelper</h1>
<p dir="auto">Damit man komplett unabhängig von den Scripten aus dem git-Repository eine eigene Konfiguration speichern kann, zudem weder im Mastodon-Web UI die App registrieren muss, noch User/Password lokal speichern muss, um in Zukunft die Scripte automatisiert ausführen zu können, kommt hier ein kleines Helper Modul, dass mit einem Methoden-Aufruf alles erledigt.</p>
<p dir="auto">Die wichtigste Funktion in dem Helper ist <code>open_or_create_app(...)</code>. Das ist ein Multitalent und ist in der Lage, die Scripte als App zu registrieren, sich entweder nur als Client-App oder User-Account anzumelden und dann die Registrierungs-Daten und den Zugriffstoken des Accounts lokal zu speichern, damit beim weiteren Aufruf weder die App erneut registriert wird noch man die Login-Daten wieder eingeben muss.</p>
<p dir="auto">Da das kaum in Worte zu fassen ist, ein Aktivitätsdiagramm (erstellt mit <plantuml.org>):</p>
<p dir="auto"><img src="https://codeberg.org/beandev/mastodon-playground/raw/branch/main/documentation/mastobot/MastodonHelper_open_or_create_app.png" alt="Activity"></p>
<p dir="auto">Man kann die Funktion in drei große Abschnitte aufteilen:</p>
<h2 dir="auto">Prepare Configuration</h2>
<p dir="auto">Der Teil der Methode übernimmt entweder die notwendigen Parameter von dem Funktions-Aufruf oder liest diese aus einer Config-Datei. Die Config-Datei wird erst im Home-Ordner gesucht, dann im Script-Ordner. Man kann auch einen absoluten Pfad angeben. Legt man aber in dem eigenen Home-Ordner die Datei `mastobot-config.yaml’ an, kann man die Beispielscripte ohne Modifikation sofort ausführen (gesetzt den Fall, man hat die Konfiguration angepasst).</p>
<p dir="auto">Alle Konfigurationsparameter werden auch grob geprüft und man bekommt ggf. Fehler um die Ohren gehauen.</p>
<h2 dir="auto">Register Application</h2>
<p dir="auto">Der Abschnitt sucht erstmal, ob wir unter dem konfigurierten Application-Name schon Client-App-Daten in eine <code>.secret</code> Datei gespeichert haben. Wenn nicht, wird eine App unter dem Namen und der gewünschten Default-Instanz registriert (siehe `mastobot-config.yaml’).</p>
<p dir="auto">Mit der erfolgreichen Registrierung, werden die <code>Client-ID</code> und das <code>Secret</code> in eine Datei gespeichert und, wie oben beschrieben, in Zukunft immer wieder verwendet. Das verhindert, dass eine App immer neu registriert wird (was Podmins der Mastodon-Instanzen zu schätzen wissen).</p>
<h2 dir="auto">Connect to Mastodon</h2>
<p dir="auto">Der dritte Abschnitt in der Methode ist das Kernstück.</p>
<p dir="auto">Erst wird geschaut, ob ein Account-Login gewünscht wird und der Access-Token des Users schon mal gespeichert wurde (das ist der linke Pfad im Aktivitätsdiagramm). Wenn ja, werden die Daten aus dem HOME-Ordner geladen und Mastodon.py meldet sich mit den Daten an.</p>
<p dir="auto">Der längere Weg ist im rechten Pfad beschrieben. Zunächst sagen wir Mastodon.py welche Client-ID und Secret genommen wird (das haben wir früher mal, oben bei <em>Register Application</em>, gespeichert). Danach prüfen wir, ob wir einen User-Account-Login wollten (über den Parameter <code>login = True</code>). Wenn ja, dann fragen wir den Benutzernamen und Password im Terminal ab (Yeah, ein UI!), loggen uns damit bei unserer Mastodon-Instanz ein, bekommen dann einen Access-Token und speichern diesen auch im Home-Ordner ab. Puh. Warum? Damit wir bei allen zukünftigen Aufrufen nicht mehr ein interaktives Login haben!</p>
<p dir="auto">Anders gesagt: Wir müssen uns nur ein einziges Mal anmelden und danach laufen alle Scripte automatisiert, ohne Interaktion, durch.</p>
<p dir="auto">Schaut euch auf Codeberg das Modul mal an: <a href="https://codeberg.org/beandev/mastodon-playground/src/commit/6797cf2bdf45018cf60219f4b69f1de4fb7fa542/mastobot/MastodonHelper.py" rel="noopener noreferrer">MastodonHelper.py</a>.</p>
<p dir="auto">Da sind noch ein paar weitere Hilfsmethoden drin. Die sollen aber das alles nur etwas strukturieren.</p>
<h1 dir="auto">me.py</h1>
<p dir="auto">Wie wird das nun verwendet? Dazu habe ich ein winziges Script geschrieben: <a href="https://codeberg.org/beandev/mastodon-playground/src/branch/main/me.py" rel="noopener noreferrer"><code>me.py</code></a>:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">json</span></span></span>
<span class=""><span class="keyword control">import</span> <span class=""><span class="">mastobot</span><span class="punctuation">.</span><span class="">MastodonHelper</span></span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastobot</span><span class="punctuation">.</span><span class="">MastodonHelper</span><span class="punctuation">.</span></span><span class=""><span class="variable function">open_or_create_app</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">config_yaml</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">mastobot-config.yaml<span class="punctuation string">'</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">login</span><span class="keyword operator">=</span><span class="constant language">True</span>
</span><span class="punctuation">)</span></span>
<span class="comment"><span class="punctuation comment">#</span> Request my account data:
</span><span class=""><span class="">me</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">me</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
<span class="comment"><span class="punctuation comment">#</span> Print out basic stuff:
</span><span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string"><span class="constant">\n</span><span class="constant">\n</span><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">Me: </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> = </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">username<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class="comment"><span class="punctuation comment">#</span> Print out everything:
</span><span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">json</span><span class="punctuation">.</span></span><span class=""><span class="variable function">dumps</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">me</span></span><span class="punctuation">,</span> <span class="variable parameter">indent</span><span class="keyword operator">=</span><span class="constant numeric">4</span><span class="punctuation">,</span> <span class="variable parameter">default</span><span class="keyword operator">=</span><span class=""><span class="support type">str</span></span></span><span class="punctuation">)</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Der obige Aufruf von <code>open_or_create_app</code> wird, wenn ihr nichts an der <code>mastobot-config.yaml</code> geändert habt, nur Fehler werfen.</p>
<p dir="auto">Also kopiert die <a href="https://codeberg.org/beandev/mastodon-playground/src/branch/main/mastobot-config.yaml" rel="noopener noreferrer">mastobot-config.yam</a> Datei in euren Home-Ordner.</p>
<p dir="auto">Tragt bei <code>name: ~</code> statt der <code>~</code> einen Namen ein, der eure App eindeutig identifiziert, zum Beispiel:</p>
<pre><code dir="auto"><span class="source"><span class="string"><span class="entity name tag">application</span></span><span class="punctuation">:</span>
<span class="string"><span class="entity name tag">name</span></span><span class="punctuation">:</span> <span class="string"><span class="entity name tag">HorstsMastodonApp</span></span>
<span class="string"><span class="entity name tag">debug</span></span><span class="punctuation">:</span> <span class="constant language">false</span> <span class="comment"><span class="punctuation comment">#</span> debug output
</span></span></code></pre>
<p dir="auto">Im Abschnitt der <code>instances</code> tragt ihr eure Mastodon-Instanz ein, wo ihr einen Account habt:</p>
<pre><code dir="auto"><span class="source"><span class="string"><span class="entity name tag">instances</span></span><span class="punctuation">:</span>
<span class="string"><span class="entity name tag">default</span></span><span class="punctuation">:</span>
<span class="string"><span class="entity name tag">url</span></span><span class="punctuation">:</span> <span class="string">https://social.example.com</span> <span class="comment"><span class="punctuation comment">#</span> your instance URL
</span></span></code></pre>
<p dir="auto">Das muss eine vollständige URL sein (und natürlich <em>nicht</em> https://social.example.com).</p>
<p dir="auto">Jetzt ruft das me.py Script auf.</p>
<p dir="auto">Ihr werdet einen Login bekommen, wo ihr eure E-Mail-Adresse und das Password eingeben müsst. Keine Angst, die Daten werden lokal nicht gespeichert. Klappt das Login damit, wird im Home-Ordner eine <code>.usertoken</code> Datei angelegt, wo nur eine ID drinsteht. Diese ID sollte <strong>niemals</strong> in andere Hände gelangen. Das ist ein Zusatzschlüssel, mit dem ihr euch einloggen könnt, ohne Benutzername und Kennwort angeben zu müssen. Das ist sozusagen eine “Hotel-RFID”-Karte zu eurem Zimmer im Mastodon-Instanz-Hotel.</p>
<p dir="auto">Wenn das alles geklappt hat, bekommt ihr ein großes JSON mit euren Account-Informationen.</p>
<p dir="auto">Ruft ihr das Script <code>me.py</code> nochmal auf, läuft es einfach durch und das JSON wird wieder angezeigt. Keine Registrierung, kein Login. </p>
<p dir="auto">Das funktioniert jetzt mit jedem Script auf eurem Rechner, dass den <code>MastodonHelper</code> mit der <code>open_or_create_app</code> Funktion und den gespeicherten Konfigurationsdaten nutzt.</p>
<p dir="auto">Also jedes Script mit dem Aufruf:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">mastobot</span><span class="punctuation">.</span><span class="">MastodonHelper</span></span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastobot</span><span class="punctuation">.</span><span class="">MastodonHelper</span><span class="punctuation">.</span></span><span class=""><span class="variable function">open_or_create_app</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">config_yaml</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">mastobot-config.yaml<span class="punctuation string">'</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">login</span><span class="keyword operator">=</span><span class="constant language">True</span>
</span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Loggt sich völlig automatisiert ein. </p>
<p dir="auto">Mit dieser Vorbereitung, werden alle weiteren Scripte, die ins git-Repository hochgeladen werden, sehr simpel im Prolog sein. </p>
<p dir="auto">Viel Spaß damit!</p>
<p dir="auto">Wie immer: Wenn ihr in eurer Bubble Menschen vermutet, die sich für solche Artikel interessieren, dann boosted! Danke :-)</p>
]]><![CDATA[Wenn sie wieder weg sind: Un-Following tracken]]>https://write.tchncs.de/~/BeanDevMastodon/Wenn%20sie%20wieder%20weg%20sind:%20Un-Following%20tracken/2022-05-22T17:40:47.741424+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-22T17:40:47.741424+00:00<![CDATA[<h1 dir="auto">Einleitung</h1>
<p dir="auto">Im <a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mastodon%20Followers%20und%20Following%20importieren.%20CircleCount%20f%C3%BCr%20Arme" rel="noopener noreferrer">letzten Artikel</a> behandelte ich das Synchronisieren von Accounts, denen ich folge und die mir folgen. Allerdings gibt es mit der Implementation das Problem, dass die Accounts nur hinzugefügt oder aktualisiert werden.</p>
<p dir="auto">Den Fall, dass wir jemanden nicht mehr folgen oder wir selbst entfolgt werden, wird mit dem letzten Script nicht erkannt.</p>
<p dir="auto">Aus psychologischer Sicht kann man sich die Frage stellen: Will ich das überhaupt wissen? Viele Social-Media-Plattformen halten sich bei dieser Frage auch bedeckt. Man wird informiert, dass man jemanden hat, der sich für die eigenen Beiträge interessiert, aber das Entfolgen passiert still. Auch Mastodon, was als Medium versucht möglichst positive Vibes zu vermitteln, verfährt so. </p>
<p dir="auto">In der Synchronisation der Accounts spiegelt sich das auch wider, aber auf einer reinen technischen Ebene. Wie erkennt man aus einer REST Schnittstelle, die darauf basiert nur existierende Daten zu lieferen, welche eben nicht mehr da sind?</p>
<p dir="auto">Mastodon hat eine Notification-API, um über Veränderung informiert zu werden. Aber die ist UI-getrieben (und dort will man Un-Follows nicht kommunizieren). Das fällt also flach. Man könnte schauen, ob sich die Anzahl verändert hat (die bekommt man aus den Account-Daten als Statistik mit). Aber das ist ungenau, denn es könnte ein Follow und Un-Follow geben und damit hat sich die absolute Anzahl nicht geändert. Bleibt also nur Brute-Force. Es müssen alle Accounts verglichen werden. D.h. man schaut, welche Account-IDs von der Mastodon-Instanz geliefert werden und vergleicht sie mit den IDs in OpenSearch. Da gibt es dann eine Differenz, die man dann als Un-Follow interpretieren kann (wenn in OpenSearch die ID existiert, aber Mastodon diese nicht mehr listet).</p>
<p dir="auto">Diese Full-Diff Vergleiche bieten eine sehr stabile Aussage über Bewegungen zwischen zwei Zeitpunkten der Abfragen, allerdings nicht, wann es genau passiert ist. Damit können wir leben. Man muss nur im Fehlerfall aufpassen, dass eine leere Antwort der Mastodon-Instanz nicht als kompletter Un-Follow aller Accounts interpretiert wird.</p>
<p dir="auto">Deswegen lösche ich auch ungern bei solchen Abgleichen Daten, die vermeintlich auf einer Remote-Instanz nicht mehr existieren. Das kann dann auf einmal zu kompletten lokalen Datenverlust führen.</p>
<h1 dir="auto">Mehr Indexe</h1>
<p dir="auto">Da ich die <code>followers</code> und <code>following</code> als aktuellen Zustand der letzten Abfrage beibehalten möchte, verschiebe ich einfach die Accounts, die meine Mastodon-Instanz nicht mehr als Verbindung listet. Dafür brauche ich zwei weitere Indexe: <code>un-followers</code> und <code>un-following</code>. Das Tolle ist, dass das <a href="https://write.tchncs.de/%7E/BeanDevOpenSearch/Index%20mit%20IndexTemplate%20erzeugen" rel="noopener noreferrer">Script zum Anlegen der Account-Indexe</a> schon so generisch ist. Da muss ich nur die zwei Namen der neuen Indexe hinzufügen:</p>
<pre><code dir="auto"><span class="source"><span class="constant language">...</span>
<span class="comment"><span class="punctuation comment">#</span> Namen der Indexe
</span><span class=""><span class="">index_names</span></span> <span class="keyword operator">=</span> <span class=""><span class="punctuation">[</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">followers<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">following<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">un-followers<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">un-following<span class="punctuation string">'</span></span></span><span class="punctuation">]</span></span>
<span class="punctuation">.</span><span class="punctuation">.</span><span class="punctuation">.</span>
</span></code></pre>
<p dir="auto">Und ja, ich wusste, dass ich das mal machen werde, deswegen hat die Schleife schon den Test eingebaut:</p>
<pre><code dir="auto"><span class="source"><span class="constant language">...</span>
<span class=""><span class="keyword control">for</span> <span class="">index_name</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">index_names</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">if</span> <span class="keyword operator">not</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">exists</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_name</span></span></span><span class="punctuation">)</span></span><span class="punctuation">:</span></span>
<span class="constant language">...</span>
</span></code></pre>
<p dir="auto">Wenn man das Script <code>osCreateFollowersIndexes.py</code>, mit den zwei zusätzlichen Index-Namen wieder aufruft, werden nur die beiden (noch nicht vorhandenen) Indexe angelegt.</p>
<p dir="auto">Nun haben wir vier Indexe, die Accounts aufnehmen können. </p>
<h1 dir="auto">Differenzen</h1>
<p dir="auto">Nein, persönliche Differenzen gab es bei den Un-Follows hoffentlich nicht, aber wir müssen die irgendwie ermittlen.</p>
<p dir="auto">Erinnern wir uns an die Methode, die unsere Accounts lädt:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">load_to_index</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index_name</span><span class="punctuation">,</span> <span class="variable parameter">initial_load</span><span class="punctuation">,</span> <span class="variable parameter">next_load</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Import </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class="constant language">None</span>
<span class=""><span class="">created</span></span> <span class="keyword operator">=</span> <span class="constant numeric">0</span>
<span class=""><span class="">updated</span></span> <span class="keyword operator">=</span> <span class="constant numeric">0</span>
<span class=""><span class="keyword control">while</span> <span class="constant language">True</span><span class="punctuation">:</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">initial_load</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">limit</span><span class="keyword operator">=</span><span class="constant numeric">80</span></span><span class="punctuation">)</span></span> <span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span> <span class="keyword operator">is</span> <span class="constant language">None</span> <span class="keyword control">else</span> <span class=""><span class=""><span class="variable function">next_load</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">page</span></span></span><span class="punctuation">)</span></span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">for</span> <span class="">account</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="">response</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">account</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class=""><span class="">response</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">result<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span> <span class="keyword operator">==</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">created<span class="punctuation string">"</span></span></span><span class="punctuation">:</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string"> New in </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">: </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">acct<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">created</span></span> <span class="keyword operator">+=</span> <span class="constant numeric">1</span>
<span class=""><span class="keyword control">elif</span></span> <span class=""><span class=""><span class="">response</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">result<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span> <span class="keyword operator">==</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">updated<span class="punctuation string">"</span></span></span><span class="punctuation variable">:</span>
<span class=""><span class="">updated</span></span> <span class="keyword operator">+=</span> <span class="constant numeric">1</span>
<span class=""><span class="keyword control">else</span><span class="punctuation">:</span></span>
<span class="keyword control">break</span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Finished "</span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">". Created </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">created</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> accounts and updated </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">updated</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Dort müssen wir uns jetzt alle IDs merken, die wir von Mastodon erhalten. Ich mache es mir einfach und schreibe die alle in eine Liste (Set wäre besser und schneller). Da ich wohl nie Millionen von Followern haben werde, sehe ich keine Speicherprobleme alle IDs auf diese Weise zu merken.</p>
<p dir="auto">Die neue Variante:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">load_to_index</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index_name</span><span class="punctuation">,</span> <span class="variable parameter">initial_load</span><span class="punctuation">,</span> <span class="variable parameter">next_load</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Import </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class="constant language">None</span>
<span class=""><span class="">created</span></span> <span class="keyword operator">=</span> <span class="constant numeric">0</span>
<span class=""><span class="">updated</span></span> <span class="keyword operator">=</span> <span class="constant numeric">0</span>
<span class=""><span class="">extracted</span></span> <span class="keyword operator">=</span> <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>
<span class=""><span class="keyword control">while</span> <span class="constant language">True</span><span class="punctuation">:</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">initial_load</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">limit</span><span class="keyword operator">=</span><span class="constant numeric">80</span></span><span class="punctuation">)</span></span> <span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span> <span class="keyword operator">is</span> <span class="constant language">None</span> <span class="keyword control">else</span> <span class=""><span class=""><span class="variable function">next_load</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">page</span></span></span><span class="punctuation">)</span></span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">for</span> <span class="">account</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="support function">id</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class=""><span class="">extracted</span><span class="punctuation">.</span></span><span class=""><span class="variable function">append</span></span> <span class="punctuation">(</span><span class=""><span class=""><span class="support function">id</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">response</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class="support function">id</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">account</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class=""><span class="">response</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">result<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span> <span class="keyword operator">==</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">created<span class="punctuation string">"</span></span></span><span class="punctuation">:</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string"> New in </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">: </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">acct<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">created</span></span> <span class="keyword operator">+=</span> <span class="constant numeric">1</span>
<span class=""><span class="keyword control">elif</span></span> <span class=""><span class=""><span class="">response</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">result<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span> <span class="keyword operator">==</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">updated<span class="punctuation string">"</span></span></span><span class="punctuation variable">:</span>
<span class=""><span class="">updated</span></span> <span class="keyword operator">+=</span> <span class="constant numeric">1</span>
<span class=""><span class="keyword control">else</span><span class="punctuation">:</span></span>
<span class="keyword control">break</span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Finished "</span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">". Created </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">created</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> accounts and updated </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">updated</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Ich bin kein Fan davon, Funktionen unnötig aufzublähen. Deswegen gibt es einen Aufruf zu einer neuen Methode, die sich dann um diese Liste der IDs auf den spezifischen Index (<code>followers</code> oder <code>following</code>) kümmert.</p>
<p dir="auto">Also noch diese Zeilen anfügen:</p>
<pre><code dir="auto"><span class="source"> <span class="comment"><span class="punctuation comment">#</span> Now we have in extracted the current active existing accounts assigned to the Mastodon-Account
</span> <span class="comment"><span class="punctuation comment">#</span> In the OS index we have maybe more. We must move the account to un-{index_name}
</span> <span class=""><span class=""><span class="variable function">move_unlisted</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class=""><span class="">extracted</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<h2 dir="auto">Verschiebebahnhof</h2>
<p dir="auto"><code>move_unlisted</code> gibt es noch nicht, also gleich mal anlegen. Da kommt auch sofort die Suche nach den Dokumenten (Accounts) rein. Das hatten wir so ähnlich schon mal in älteren Artikeln ausprobiert, allerdings machen wir es jetzt mit der Scroll API wie in dem neuen <a href="https://write.tchncs.de/%7E/BeanDevOpenSearch/Scrolled%20queries%20und%20Point-In-Time%20queries%20mit%20OpenSearch" rel="noopener noreferrer">Blog-Artikel für Scroll und PIT Abfragen</a> beschrieben:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">move_unlisted</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index_name</span><span class="punctuation">,</span> <span class="variable parameter">extracted</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">docs_listed</span></span><span class="keyword operator">=</span><span class="constant numeric">0</span>
<span class=""><span class="">docs_moved</span></span><span class="keyword operator">=</span><span class="constant numeric">0</span>
<span class=""><span class="">search_all</span></span><span class="keyword operator">=</span><span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">_source<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="constant language">False</span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">size<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="constant numeric">100</span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">query<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">match_all<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span><span class=""><span class="punctuation">}</span></span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class=""><span class="">resp</span></span><span class="keyword operator">=</span><span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">search</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">search_all</span></span><span class="punctuation">,</span> <span class="variable parameter">scroll</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">5s<span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">scroll_id</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">resp</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_scroll_id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class="keyword control">while</span> <span class=""><span class=""><span class="support function">len</span></span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">resp</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">hits<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">hits<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span><span class="punctuation">)</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">for</span> <span class="">doc</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class=""><span class="">resp</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">hits<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">hits<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">:</span></span>
<span class=""><span class="">docs_listed</span></span><span class="keyword operator">+=</span><span class="constant numeric">1</span>
<span class=""><span class="keyword control">if</span> <span class=""><span class=""><span class="support type">int</span></span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">doc</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span><span class="punctuation">)</span></span> <span class="keyword operator">not</span> <span class="keyword operator">in</span> <span class=""><span class="">extracted</span></span><span class="punctuation">:</span></span>
<span class="comment"><span class="punctuation comment">#</span> the account doc in OS is not any more listed in the Mastodon instance
</span> <span class=""><span class=""><span class="variable function">move_document</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">doc</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">index_from</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">index_to</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">un-<span class="punctuation string">"</span></span></span> <span class="keyword operator">+</span> <span class=""><span class="">index_name</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">docs_moved</span></span><span class="keyword operator">+=</span><span class="constant numeric">1</span>
<span class="comment"><span class="punctuation comment">#</span> make a request using the Scroll API
</span> <span class=""><span class="">resp</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">scroll</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">scroll_id</span> <span class="keyword operator">=</span> <span class=""><span class="">scroll_id</span></span><span class="punctuation">,</span>
<span class="variable parameter">scroll</span> <span class="keyword operator">=</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">5s<span class="punctuation string">'</span></span></span> <span class="comment"><span class="punctuation comment">#</span> length of time to keep search context
</span> </span><span class="punctuation">)</span></span>
<span class=""><span class="">scroll_id</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">resp</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_scroll_id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">clear_scroll</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">scroll_id</span><span class="keyword operator">=</span><span class=""><span class="">scroll_id</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">In "</span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">" </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">docs_listed</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> found, <span class="punctuation string">'</span></span></span>
<span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">docs_moved</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> to un-</span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">. <span class="punctuation string">'</span></span></span>
<span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">docs_listed</span></span><span class="keyword operator">-</span><span class=""><span class="">docs_moved</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> remaining<span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Ok, das ist schon harter Tobak. Vor der While-Schleife ist die Abfrage, bei der wir zugleich OpenSearch Anweisen uns eine Scroll-ID zu geben. So weit, so klar. Die While-Bedingung ist eigentlich ein: Mach so lange, wie wir Treffer bekommen. Die For-Schleife iteriert durch unsere Dokumente, die wir angefragt haben. </p>
<p dir="auto">Die einzige spannende Logik ist wirklich die Prüfung, ob die <code>doc['id']</code> sich nicht in der Liste der <code>extracted</code> befindet. Wenn das so ist, dann haben wir den Fall, dass das Dokument (also der Account) im OpenSearch Index abgelegt wurde, aber in Mastodon keine Verbindung mehr zu meinem Account hat (die <code>id</code> ist nicht in <code>extracted</code> vorhanden). Damit hätten wir einen Treffer eines “Un-Follower” oder “Un-Following”. </p>
<p dir="auto">Da mit dieses verschachtelte Monstrum an Methode nicht noch komplexer wird, erfolgt dann eben wieder ein Unteraufruf der nächsten Funktion (die es noch nicht gibt): <code>move_document (...)</code>.</p>
<p dir="auto">Der Rest der obigen Methode ist nur noch das Behandeln der Scroll-IDs und ein wenig Statistik und Infos an uns Entwickler*innen.</p>
<h2 dir="auto">Ein Dokument verschieben</h2>
<p dir="auto">Also rein in die <code>move_document</code>, die wirklich nicht mehr viel macht:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">move_document</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span></span><span class="function"><span class="punctuation parameter">:</span> <span class=""><span class="">OpenSearch</span></span></span><span class="function"><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="punctuation">,</span> <span class="variable parameter">index_from</span><span class="punctuation">,</span> <span class="variable parameter">index_to</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">doc</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">get</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_from</span></span><span class="punctuation">,</span> <span class=""><span class="support function">id</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">source</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">doc</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_source<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_to</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class="support function">id</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">source</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">delete</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_from</span></span><span class="punctuation">,</span> <span class=""><span class="support function">id</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_from</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_to</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string"> Moved </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">source</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">acct<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> to </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_to</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> <span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Sie ist sehr generisch geschrieben und kann daher mit beliebigen Dokumenten auf beliebigen Indexen ausgeführt werden. Der Aufrufer muss nur die passenden Parameter setzen. Das refresh ist etwas ineffizient, wenn wir das aus einer Schleife aufrufen, aber wir hoffen sowieso nur auf wenige Verschiebungen. </p>
<p dir="auto">Das <code>move_document</code> ist eine Kombination aus: Nimm das Dokument, indiziere es im Zielindex, lösche es im Quellindex, aktualisiere beide Indexe und schreib was Nettes in die Konsole. Fehlerbehandlung ist mal wieder ausgeklammert.</p>
<p dir="auto">Im Prinzip war das keine wirklich aufwendige Erweiterung. Etwas Komplexität kam durch den Einsatz der Scroll API. Aber es ist sinnvoll, sich Ergebnisse aus einem Datenpool grundsätzlich nur in Häppchen zu geben und ohne Scroll API wäre bei über 10.000 Dokumenten Schluss gewesen. Ob wir wirklich mal auf als 10.000 Follower haben werden… na ja, aber wir wissen nun wie es geht.</p>
<hr>
<p dir="auto">Noch ein paar Anmerkungen, bevor ich das Script mal komplett am Ende aufliste. Das Ausgeben der Informationen auf die Konsole ist eine tolle Sache, aber lebt nur diesen kurzen Augenblick und ist meistens schnell verpufft. Suchen wir explizit nach Fehlern, hilft uns das als Entwickler*innen. Tauchen aber Fehler erst im produktiven Betrieb auf, haben wir nichts in der Hand die Probleme nachzuvollziehen. </p>
<p dir="auto">Dafür gibt es überlicherweise Logging-Frameworks. Logging bedeutet, wir übergeben eine Meldung an ein Framework, dass daraus eine strukturierte Information macht und diese üblicherweise in Textdateien ablegt (für Menschen lesbar). Manche Logging-Frameworks bieten Connectoren, diese Log-Daten in Datenbanken zu schrieben, damit die durchsuchbar werden. Meistens handelt es sich dann wieder um ElasticSearch oder OpenSearch. </p>
<p dir="auto">Das könnten wir einfach verkürzen. Wir könnten einen Index für eine Arbneits-Historie anlegen und einen Index für Ausnahmen und Fehler. Ich baue das jetzt nicht ein, sondern werde dafür einen extra Artikel machen. Das ist nämlich eine ungeheuer sinnvolle Sache und man sollte das mal gemacht haben, um zu sehen, wie einfach das geht und welche Vorteile das bringt.</p>
<hr>
<h1 dir="auto">Das Script loadFollowersFollowingtoOS.py</h1>
<p dir="auto">Nun aber das komplette Script, das auch Un-Follows und Un-Following versteht:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">json</span></span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">typing</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Any</span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">mastodon</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Mastodon</span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">opensearchpy</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">OpenSearch</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">login_os</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">host</span><span class="punctuation">,</span> <span class="variable parameter">port</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class="comment"><span class="punctuation comment">#</span> Client zu dem Dev Cluster (ohne SSL, ohne Anmeldung)
</span> <span class="keyword control">return</span> <span class=""><span class=""><span class="variable function">OpenSearch</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">hosts</span><span class="keyword operator">=</span><span class=""><span class="punctuation">[</span><span class=""><span class="punctuation">{</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">host<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">host</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">port<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">port</span></span><span class="punctuation">}</span></span><span class="punctuation">]</span></span><span class="punctuation">,</span>
<span class="variable parameter">http_compress</span><span class="keyword operator">=</span><span class="constant language">True</span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> enables gzip compression for request bodies
</span> <span class="variable parameter">use_ssl</span><span class="keyword operator">=</span><span class="constant language">False</span>
</span><span class="punctuation">)</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">login_mastodon</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">url</span><span class="punctuation">,</span> <span class="variable parameter">user</span><span class="punctuation">,</span> <span class="variable parameter">password</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">Mastodon</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">client_id</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">ep*****************************************Xo<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> Eure ID
</span> <span class="variable parameter">client_secret</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">AY******************************************0s<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> das Geheimnis zur ID
</span> <span class="variable parameter">api_base_url</span><span class="keyword operator">=</span><span class=""><span class="">url</span></span>
</span><span class="punctuation">)</span></span>
<span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">access_token</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">log_in</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">username</span><span class="keyword operator">=</span><span class=""><span class="">user</span></span><span class="punctuation">,</span>
<span class="variable parameter">password</span><span class="keyword operator">=</span><span class=""><span class="">password</span></span><span class="punctuation">,</span>
<span class="variable parameter">scopes</span><span class="keyword operator">=</span><span class=""><span class="punctuation">[</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">read<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">write<span class="punctuation string">'</span></span></span><span class="punctuation">]</span></span>
</span><span class="punctuation">)</span></span>
<span class="keyword control">return</span> <span class=""><span class="">mastodon</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">move_document</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span></span><span class="function"><span class="punctuation parameter">:</span> <span class=""><span class="">OpenSearch</span></span></span><span class="function"><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="punctuation">,</span> <span class="variable parameter">index_from</span><span class="punctuation">,</span> <span class="variable parameter">index_to</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">doc</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">get</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_from</span></span><span class="punctuation">,</span> <span class=""><span class="support function">id</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">source</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">doc</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_source<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_to</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class="support function">id</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">source</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">delete</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_from</span></span><span class="punctuation">,</span> <span class=""><span class="support function">id</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_from</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_to</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string"> Moved </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">source</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">acct<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> to </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_to</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> <span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">move_unlisted</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index_name</span><span class="punctuation">,</span> <span class="variable parameter">extracted</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">docs_listed</span></span><span class="keyword operator">=</span><span class="constant numeric">0</span>
<span class=""><span class="">docs_moved</span></span><span class="keyword operator">=</span><span class="constant numeric">0</span>
<span class=""><span class="">search_all</span></span><span class="keyword operator">=</span><span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">_source<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="constant language">False</span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">size<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="constant numeric">100</span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">query<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">match_all<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span><span class=""><span class="punctuation">}</span></span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class=""><span class="">resp</span></span><span class="keyword operator">=</span><span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">search</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">search_all</span></span><span class="punctuation">,</span> <span class="variable parameter">scroll</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">5s<span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">scroll_id</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">resp</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_scroll_id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class="keyword control">while</span> <span class=""><span class=""><span class="support function">len</span></span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">resp</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">hits<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">hits<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span><span class="punctuation">)</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">for</span> <span class="">doc</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class=""><span class="">resp</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">hits<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">hits<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">:</span></span>
<span class=""><span class="">docs_listed</span></span><span class="keyword operator">+=</span><span class="constant numeric">1</span>
<span class=""><span class="keyword control">if</span> <span class=""><span class=""><span class="support type">int</span></span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">doc</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span><span class="punctuation">)</span></span> <span class="keyword operator">not</span> <span class="keyword operator">in</span> <span class=""><span class="">extracted</span></span><span class="punctuation">:</span></span>
<span class="comment"><span class="punctuation comment">#</span> the account doc in OS is not any more listed in the Mastodon instance
</span> <span class=""><span class=""><span class="variable function">move_document</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">doc</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">index_from</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">index_to</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">un-<span class="punctuation string">"</span></span></span> <span class="keyword operator">+</span> <span class=""><span class="">index_name</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">docs_moved</span></span><span class="keyword operator">+=</span><span class="constant numeric">1</span>
<span class="comment"><span class="punctuation comment">#</span> make a request using the Scroll API
</span> <span class=""><span class="">resp</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">scroll</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">scroll_id</span> <span class="keyword operator">=</span> <span class=""><span class="">scroll_id</span></span><span class="punctuation">,</span>
<span class="variable parameter">scroll</span> <span class="keyword operator">=</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">5s<span class="punctuation string">'</span></span></span> <span class="comment"><span class="punctuation comment">#</span> length of time to keep search context
</span> </span><span class="punctuation">)</span></span>
<span class=""><span class="">scroll_id</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">resp</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">_scroll_id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">clear_scroll</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">scroll_id</span><span class="keyword operator">=</span><span class=""><span class="">scroll_id</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">In "</span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">" </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">docs_listed</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> found, <span class="punctuation string">'</span></span></span>
<span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">docs_moved</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> to un-</span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">. <span class="punctuation string">'</span></span></span>
<span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">docs_listed</span></span><span class="keyword operator">-</span><span class=""><span class="">docs_moved</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> remaining<span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">load_to_index</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index_name</span><span class="punctuation">,</span> <span class="variable parameter">initial_load</span><span class="punctuation">,</span> <span class="variable parameter">next_load</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Import </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class="constant language">None</span>
<span class=""><span class="">created</span></span> <span class="keyword operator">=</span> <span class="constant numeric">0</span>
<span class=""><span class="">updated</span></span> <span class="keyword operator">=</span> <span class="constant numeric">0</span>
<span class=""><span class="">extracted</span></span> <span class="keyword operator">=</span> <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>
<span class=""><span class="keyword control">while</span> <span class="constant language">True</span><span class="punctuation">:</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">initial_load</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">limit</span><span class="keyword operator">=</span><span class="constant numeric">80</span></span><span class="punctuation">)</span></span> <span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span> <span class="keyword operator">is</span> <span class="constant language">None</span> <span class="keyword control">else</span> <span class=""><span class=""><span class="variable function">next_load</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">page</span></span></span><span class="punctuation">)</span></span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">for</span> <span class="">account</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="support function">id</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class=""><span class="">extracted</span><span class="punctuation">.</span></span><span class=""><span class="variable function">append</span></span> <span class="punctuation">(</span><span class=""><span class=""><span class="support function">id</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">response</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class="support function">id</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">account</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class=""><span class="">response</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">result<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span> <span class="keyword operator">==</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">created<span class="punctuation string">"</span></span></span><span class="punctuation">:</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string"> New in </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">: </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">acct<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">created</span></span> <span class="keyword operator">+=</span> <span class="constant numeric">1</span>
<span class=""><span class="keyword control">elif</span></span> <span class=""><span class=""><span class="">response</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">result<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span> <span class="keyword operator">==</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">updated<span class="punctuation string">"</span></span></span><span class="punctuation variable">:</span>
<span class=""><span class="">updated</span></span> <span class="keyword operator">+=</span> <span class="constant numeric">1</span>
<span class=""><span class="keyword control">else</span><span class="punctuation">:</span></span>
<span class="keyword control">break</span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Finished "</span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">". Created </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">created</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> accounts and updated </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">updated</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class="comment"><span class="punctuation comment">#</span> Now we have in extracted the current active existing accounts assigned to the Mastodon-Account
</span> <span class="comment"><span class="punctuation comment">#</span> In the OS index we have maybe more. We must move the account to un-{index_name}
</span> <span class=""><span class=""><span class="variable function">move_unlisted</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class=""><span class="">extracted</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">client</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">login_os</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">localhost<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="constant numeric">9200</span></span><span class="punctuation">)</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">login_mastodon</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">https://social.tchncs.de<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string"><email-adresse>, <span class="punctuation string">'</span></span></span><span class="keyword operator"><</span><span class=""><span class="">kennwort</span></span> <span class=""><span class="">des</span></span> <span class=""><span class="">accounts</span></span><span class="keyword operator">></span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">)<span class="invalid">
</span></span></span>
<span class="comment"><span class="punctuation comment">#</span> It's me:
</span><span class="variable parameter">me</span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">me</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">Me: </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> = </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">username<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class="comment"><span class="punctuation comment">#</span> Followers von Mastodon nach OpenSearch kopieren:
</span><span class=""><span class=""><span class="variable function">load_to_index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">followers<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">account_followers</span></span><span class="punctuation">,</span> <span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">fetch_next</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="variable function">load_to_index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">following<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">account_following</span></span><span class="punctuation">,</span> <span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">fetch_next</span></span></span><span class="punctuation">)</span></span>
</span></span></span></code></pre>
<p dir="auto">Eine Ausgabe, wie es mal bei mir ausgesehen hat (die Sternchen sind natürlich reale Daten aus Mastodon):</p>
<pre><code dir="auto">Me: ****** = beandev
Import followers
New in followers: **********@*****.****
New in followers: ******@*****.**
New in followers: *************@*****.*****
New in followers: *******@*****.**
New in followers: *********@*****.**
Finished "followers". Created 5 accounts and updated 196
Moved ********** to un-followed
In "followers" 201 found, 1 to un-followers. 200 remaining
Import following
New in following: *************@*****.*****
New in following: ************@*****.*******
New in following: ********@*****.*****
Finished "following". Created 3 accounts and updated 139
In "following" 142 found, 0 to un-following. 142 remaining
</code></pre>
<p dir="auto">Ja, also ein Account hat mich wohl verlassen… Aber jeder Abschied hat auch was Gutes. Hier war es möglich zu testen, ob die neue Logik funktioniert.</p>
<p dir="auto">Wer war das wohl?</p>
<p dir="auto"><a href="http://localhost:9200/un-followers/_search?pretty=true" rel="noopener noreferrer">http://localhost:9200/un-followers/_search?pretty=true</a></p>
<p dir="auto">Egal… </p>
<p dir="auto">Viel Spaß mit dem Script! Interessiert sich Eure Bubble auch für <a href="//write.tchncs.de/tag/Python" title="Python" rel="noopener noreferrer">#Python</a>, <a href="//write.tchncs.de/tag/Mastodon" title="Mastodon" rel="noopener noreferrer">#Mastodon</a> und <a href="//write.tchncs.de/tag/OpenSearch" title="OpenSearch" rel="noopener noreferrer">#OpenSearch</a>? Dann boosted den Artikel!</p>
]]><![CDATA[Mastodon Followers und Following importieren. CircleCount für Arme]]>https://write.tchncs.de/~/BeanDevMastodon/Mastodon%20Followers%20und%20Following%20importieren.%20CircleCount%20für%20Arme/2022-05-17T13:33:35.136333+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-17T13:33:35.136333+00:00<![CDATA[<h1 dir="auto">Einleitung</h1>
<p dir="auto">Natürlich hat das Mastodon UI schon eine recht gute Verwaltung, der Followers und Following, aber es wird spannend sein, wenn man die Informationen auch mal lokal hat. Man hat da mehr Möglichkeiten, was die Verfolgung von Veränderungen betrifft, man kann man die Interessengebiete der eigenen Bubble anschauen oder schlicht ein regelmäßiges Backup machen.</p>
<p dir="auto">Es gab im <a href="https://write.tchncs.de/%7E/BeanDevOpenSearch" rel="noopener noreferrer">OpenSearch Blog</a> bereits zwei vorbereitende Artikel, die man sich definitiv anschauen muss.</p>
<ol dir="auto">
<li><a href="https://write.tchncs.de/%7E/BeanDevOpenSearch/Index%20Templates%20in%20OpenSearch" rel="noopener noreferrer">Index Templates in OpenSearch</a></li>
<li><a href="https://write.tchncs.de/%7E/BeanDevOpenSearch/Index%20mit%20IndexTemplate%20erzeugen" rel="noopener noreferrer">Index mit IndexTemplate erzeugen</a></li>
</ol>
<p dir="auto">Mit den Index-Templates haben wir eine Vorlage angelegt, die für alle neue Indexe verwendet werden. Im zweiten Artikel greifen wir auf das Template zurück und legen zwei Indexe an: <code>followers</code> und <code>following</code>.</p>
<p dir="auto">Dazu gibt es noch diese beiden Grundlagen-Artikel:</p>
<ol dir="auto">
<li><a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mastodon%20Instanz%20API%20nutzen%20-%20Erste%20Schritte" rel="noopener noreferrer">Mastodon Instanz API nutzen - Erste Schritte</a></li>
<li><a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mit%20Python%20auf%20Mastodon%20eine%20Nachricht%20senden" rel="noopener noreferrer">Mit Python auf Mastodon eine Nachricht senden</a></li>
</ol>
<p dir="auto">In dem Einführungs-Artikel wird unsere App (also unsere Sammlung an Python Scripten) registriert. Diese Registrierungsdaten benötigen wir, da wir nicht mehr auf die Public API zugreifen. </p>
<p dir="auto">Im zweiten Artikel wird zwar beschrieben, wie man einen Toot absendet, aber auch wie man sich mit einem User-Account anmeldet. Das ist für unser kommendes Beispiel wichtig, weil wir von unserem Account die Vernetzung der Accounts lesen wollen.</p>
<p dir="auto">Jepp, ganz schön viele Bausteine, die man vorab braucht, aber ihr folgt hier einem großen Plan ;-) - alle Artikel haben ihren Sinn und werden Stück für Stück zusammengefügt.</p>
<h1 dir="auto">User-Account Anmeldung</h1>
<p dir="auto">Das Script-Präfix beginnt diesmal damit auch anders:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">json</span></span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">mastodon</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Mastodon</span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">opensearchpy</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">OpenSearch</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">login_os</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">host</span><span class="punctuation">,</span> <span class="variable parameter">port</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class="comment"><span class="punctuation comment">#</span> Client zu dem Dev Cluster (ohne SSL, ohne Anmeldung)
</span> <span class="keyword control">return</span> <span class=""><span class=""><span class="variable function">OpenSearch</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">hosts</span><span class="keyword operator">=</span><span class=""><span class="punctuation">[</span><span class=""><span class="punctuation">{</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">host<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">host</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">port<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">port</span></span><span class="punctuation">}</span></span><span class="punctuation">]</span></span><span class="punctuation">,</span>
<span class="variable parameter">http_compress</span><span class="keyword operator">=</span><span class="constant language">True</span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> enables gzip compression for request bodies
</span> <span class="variable parameter">use_ssl</span><span class="keyword operator">=</span><span class="constant language">False</span>
</span><span class="punctuation">)</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">login_mastodon</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">url</span><span class="punctuation">,</span> <span class="variable parameter">user</span><span class="punctuation">,</span> <span class="variable parameter">password</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">Mastodon</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">client_id</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">ep********************************************Xo<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> Use your registred ID
</span> <span class="variable parameter">client_secret</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">AY******************************************0s<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> Use your registred secret
</span> <span class="variable parameter">api_base_url</span><span class="keyword operator">=</span><span class=""><span class="">url</span></span>
</span><span class="punctuation">)</span></span>
<span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">access_token</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">log_in</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">username</span><span class="keyword operator">=</span><span class=""><span class="">user</span></span><span class="punctuation">,</span>
<span class="variable parameter">password</span><span class="keyword operator">=</span><span class=""><span class="">password</span></span><span class="punctuation">,</span>
<span class="variable parameter">scopes</span><span class="keyword operator">=</span><span class=""><span class="punctuation">[</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">read<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">write<span class="punctuation string">'</span></span></span><span class="punctuation">]</span></span>
</span><span class="punctuation">)</span></span>
<span class="keyword control">return</span> <span class=""><span class="">mastodon</span></span>
</span></code></pre>
<p dir="auto">Wichtig ist, dass die <a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mastodon%20Instanz%20API%20nutzen%20-%20Erste%20Schritte" rel="noopener noreferrer">Registrierungsdaten eurer App</a> da oben eingetragen werden. </p>
<p dir="auto">Um zu schauen, ob das alles klappt, mal diese Scriptzeilen anfügen:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">client</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">login_os</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">localhost<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="constant numeric">9200</span></span><span class="punctuation">)</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">login_mastodon</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">https://social.tchncs.de<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string"><your email-address><span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string"><your secret password><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class="comment"><span class="punctuation comment">#</span> It's me:
</span><span class=""><span class="">me</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">me</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">Me: </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> = </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">username<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Und wieder der Hinweis: Da oben die Mailadresse und Kennwort des persönlichen Accounts.</p>
<p dir="auto">Zusammen ausgeführt, wird eine ID und ein Benutzername ausgegeben - oder es hagelt Fehlermeldungen. Vertippt? App wirklich registriert? Bitte das erst korrigieren, dann geht es hier weiter:</p>
<h1 dir="auto">Followers laden und speichern</h1>
<p dir="auto">Im letzten Blog-Artikel haben wir zwei Indexe angelegt: followers und following. Die wollen wir nun mit unseren geliebten <a href="https://www.urbandictionary.com/define.php?term=Mutual" rel="noopener noreferrer">Mutuals</a> füllen - und zwar allen.</p>
<p dir="auto">Bei den Toots hatten wir immer eine Begrenzung der Ladevorgänge der Seiten, das machen wir nicht. Wir haben eine Endlosschleife. Mein erster Versuch sah so aus (und funktioniert auch):</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class="constant language">None</span>
<span class=""><span class="keyword control">while</span> <span class="constant language">True</span><span class="punctuation">:</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">account_followers</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">limit</span><span class="keyword operator">=</span><span class="constant numeric">80</span></span><span class="punctuation">)</span></span> <span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span> <span class="keyword operator">is</span> <span class="constant language">None</span> <span class="keyword control">else</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">fetch_next</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">page</span></span></span><span class="punctuation">)</span></span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">for</span> <span class="">account</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">followers<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">account</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="keyword control">else</span><span class="punctuation">:</span></span>
<span class="keyword control">break</span>
</span></code></pre>
<p dir="auto">Bestimmt hat jemand eine bessere Idee für einen Schleife oder Iterator, aber das ist erstmal eingängig. </p>
<p dir="auto">Allerdings müssten wir das zweimal programmieren (für followers und following), weil die Mastodon API, zwei verschiedene Methoden zum Laden nutzt. </p>
<p dir="auto">Softwareentwickler sind aber von Haus aus wirklich faul und nichts ist schlimmer als redundanter Code, wo nur eine Zeile unterschiedlich ist. Die Lösung sind Callback-Funktionen. Auch Python beherrscht das und wir basteln uns mal eine Funktion:</p>
<h1 dir="auto">Accounts laden und speichern</h1>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">load_to_index</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index_name</span><span class="punctuation">,</span> <span class="variable parameter">initial_load</span><span class="punctuation">,</span> <span class="variable parameter">next_load</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class="constant language">None</span>
<span class=""><span class="keyword control">while</span> <span class="constant language">True</span><span class="punctuation">:</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">initial_load</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">limit</span><span class="keyword operator">=</span><span class="constant numeric">80</span></span><span class="punctuation">)</span></span> <span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span> <span class="keyword operator">is</span> <span class="constant language">None</span> <span class="keyword control">else</span> <span class=""><span class=""><span class="variable function">next_load</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">page</span></span></span><span class="punctuation">)</span></span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">for</span> <span class="">account</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">account</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="keyword control">else</span><span class="punctuation">:</span></span>
<span class="keyword control">break</span>
</span></code></pre>
<p dir="auto"><code>client</code> ist unser OpenSearch Objekt, <code>index_name</code> unser Index. Dann kommen die Callback-Parameter: <code>initial_load</code> und <code>next_load</code>.</p>
<p dir="auto">Das wird auch sehr einfach aufgerufen:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class=""><span class="variable function">load_to_index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">followers<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">account_followers</span></span><span class="punctuation">,</span> <span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">fetch_next</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="variable function">load_to_index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">following<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">account_following</span></span><span class="punctuation">,</span> <span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">fetch_next</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Das ist schon ziemlich cool. </p>
<p dir="auto">Beim Import möchte ich aber sehen, was passiert. Also kommt etwas geschwätzige Statistik hinzu. Die nächste Variante:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">load_to_index</span></span></span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index_name</span><span class="punctuation">,</span> <span class="variable parameter">initial_load</span><span class="punctuation">,</span> <span class="variable parameter">next_load</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Import </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class="constant language">None</span>
<span class=""><span class="">created</span></span> <span class="keyword operator">=</span> <span class="constant numeric">0</span>
<span class=""><span class="">updated</span></span> <span class="keyword operator">=</span> <span class="constant numeric">0</span>
<span class=""><span class="keyword control">while</span> <span class="constant language">True</span><span class="punctuation">:</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">initial_load</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">me</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">limit</span><span class="keyword operator">=</span><span class="constant numeric">80</span></span><span class="punctuation">)</span></span> <span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span> <span class="keyword operator">is</span> <span class="constant language">None</span> <span class="keyword control">else</span> <span class=""><span class=""><span class="variable function">next_load</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">page</span></span></span><span class="punctuation">)</span></span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="keyword control">for</span> <span class="">account</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="">response</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">account</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="keyword control">if</span> <span class=""><span class=""><span class="">response</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">result<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span> <span class="keyword operator">==</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">created<span class="punctuation string">"</span></span></span><span class="punctuation">:</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string"> New in </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">: </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">account</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">acct<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">created</span></span> <span class="keyword operator">+=</span> <span class="constant numeric">1</span>
<span class=""><span class="keyword control">elif</span></span> <span class=""><span class=""><span class="">response</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">result<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span> <span class="keyword operator">==</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">updated<span class="punctuation string">"</span></span></span><span class="punctuation variable">:</span>
<span class=""><span class="">updated</span></span> <span class="keyword operator">+=</span> <span class="constant numeric">1</span>
<span class=""><span class="keyword control">else</span><span class="punctuation">:</span></span>
<span class="keyword control">break</span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span></span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Finished "</span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">index_name</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string">". Created </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">created</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> accounts and updated </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">updated</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Ja, das Refresh dürfen wir nicht vergessen und abschließend eine Ausgabe, was passiert ist.</p>
<p dir="auto">Ok, bekommt ihr das selber zusammengesetzt?</p>
<p dir="auto">Als Ergebnis sollte man das sehen:</p>
<pre><code dir="auto">Import followers
Finished "followers". Created 197 accounts and updated 0
Import following
Finished "following". Created 139 accounts and updated 0
</code></pre>
<p dir="auto">Ruft man das Script erneut auf:</p>
<pre><code dir="auto">Import followers
Finished "followers". Created 0 accounts and updated 197
Import following
Finished "following". Created 0 accounts and updated 139
</code></pre>
<p dir="auto">Wenn man das alle paar Tage mal macht und das Glück hat mehr Follower zu bekommen, erhöht sich die Zahl kontinuierlich.</p>
<p dir="auto">Allerdings haben wir auch ein Problem mit dieser simplen Art, von Mastodon die Accounts zu synchronisieren. Welches? Das kommt im nächsten Artikel. Ja ja, der Cliffhanger - da ist er wieder.</p>
]]><![CDATA[Mastodon Toots spielerisch verarbeiten - In Farbe und Bunt]]>https://write.tchncs.de/~/BeanDevMastodon/Mastodon%20Toots%20spielerisch%20verarbeiten%20-%20In%20Farbe%20und%20Bunt/2022-05-16T09:31:42.474321+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-16T09:31:42.474321+00:00<![CDATA[<h1 dir="auto">Einleitung</h1>
<p dir="auto">Nachdem nun das Mapping etwas in dem Toots Index mit dem letzten <a href="https://write.tchncs.de/%7E/BeanDevOpenSearch/Feinheiten%20beim%20Mappen%20mit%20OpenSearch%20-%20Teil%201/" rel="noopener noreferrer">Blog-Artikel</a> verfeinert wurde, können wir mal mit den Daten rumspielen. </p>
<p dir="auto">Ich habe nun einige Toots aus der Lokalen Timeline meiner Instanz geladen. Wenn man mehr Schleifen-Durchgänge dem Script hinzufügt oder es mit Zeitversatz aufruft, bekommt man ältere oder neuere hinzu.</p>
<p dir="auto">Mit diesem Grundstock an Statusnachrichten aus dem Fediverse ist es möglich mal ein paar “Analysen” auszuprobieren. Allerdings belasse ich es bei einer sehr spielerischen Sache, da für größere Analysen man erheblich mehr Daten benötigt. Das werden wir dann mal später durchführen. </p>
<p dir="auto">Ich lade die Toots übrigens nicht nur aus Spaß in OpenSearch. Es gibt ein paar Gründe, warum man nicht direkt auf bestimmte Datenquellen zugreift, wenn man deren Daten analysieren wird. Man spricht hier von der Bevorzugung des ELT Ansatzes. In der IT haben sich grundsätzlich zwei Paradigmen der Datenintegration etabliert. ETL (Extract, Transform, Load) und ELT (Extract, Load, Transform). Wobei es ETL schon viel länger gibt. Im ETL Prozess werden aus einer Datenquelle die Daten extrahiert (über Schnittstellen oder Dateiformate) in ein Format und in die Struktur des Zielsystems transformiert und dabei ggf. normalisiert (manchmal sogar inhaltlich angepasst), dann abschließend in das Zielsystem importiert. Mit dem ELT Workflow verändert man einen entscheidenden Schritt in der Reihenfolge und erhält in der Konsequenz einige Vorteile. Bei ELT werden die Daten extrahiert und ohne große Formatänderungen (insbesondere nicht in der Struktur, Anreicherung oder Veränderung der Attribute) in einen Datenpool (ggf. ein Queue) geladen. Aus diesem Pool werden die Daten entnommen, und dann erst Zielprozessen übergeben, die eine Transformation und Verarbeitung durchführen.</p>
<p dir="auto">Warum bevorzuge ich ELT? Es erfordert ja scheinbar etwas mehr Arbeit.</p>
<p dir="auto">ELT ist aus mehreren Gründen geboten:</p>
<ol dir="auto">
<li>Die Datenquelle ist volatil, d.h. die Daten sind ggf. zu anderen Zeitpunkten nicht mehr reproduzierbar</li>
<li>Die Datenquelle ist beim Extrahieren sehr langsam</li>
<li>Die Daten aus der Quelle werden zu schnell geliefert und eine serielle Transformation würde den Strom blockieren</li>
<li>Die Datenquelle verwendet Quotas und schränkt damit Zugriffe ein, um Ressourcen zu schonen.</li>
</ol>
<p dir="auto">In Social-Media-Plattformen treffen wir meistens gleich auf mehrere dieser Gründe und daher habe ich mir angewöhnt, Daten, die ich aus solchen Systemen verarbeiten will, vorab in OpenSearch zu laden. Bei den Experimenten überschreite ich dann nicht irgendwelche Rate-Limits (die auch in der <a href="https://docs.joinmastodon.org/api/rate-limits/" rel="noopener noreferrer">Mastodon API</a> verwendet werden). Experimente bleiben für das geladene Datenset stabil (reproduzierbar) und ich bin nicht von der Verfügbarkeit des Systems abhängig. </p>
<h1 dir="auto">Toots laden</h1>
<p dir="auto">In dem <a href="https://write.tchncs.de/%7E/BeanDevOpenSearch" rel="noopener noreferrer">Nachbar-Blog zu OpenSearch</a> habe ich das <a href="https://write.tchncs.de/%7E/BeanDevOpenSearch/Feinheiten%20beim%20Mappen%20mit%20OpenSearch%20-%20Teil%201" rel="noopener noreferrer">Mapping der Toots</a> etwas verbessert. Am Ende des Artikels, dort ist auch noch mal das Script zum Laden von Toots in den Index. Man sollte den Durchlauf der Schleife auf zum Beispiel 50 erhöhen, um mehr Status-Nachrichten zum Experimentieren zu laden:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">json</span></span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">mastodon</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Mastodon</span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">opensearchpy</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">OpenSearch</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">Mastodon</span></span> <span class="punctuation">(</span><span class="">
<span class="variable parameter">api_base_url</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">https://social.tchncs.de<span class="punctuation string">'</span></span></span>
</span><span class="punctuation">)</span></span>
<span class=""><span class="">host</span></span> <span class="keyword operator">=</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">localhost<span class="punctuation string">'</span></span></span>
<span class=""><span class="">port</span></span> <span class="keyword operator">=</span> <span class="constant numeric">9200</span>
<span class=""><span class="">client</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">OpenSearch</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">hosts</span> <span class="keyword operator">=</span> <span class=""><span class="punctuation">[</span><span class=""><span class="punctuation">{</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">host<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">host</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">port<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">port</span></span><span class="punctuation">}</span></span><span class="punctuation">]</span></span><span class="punctuation">,</span>
<span class="variable parameter">http_compress</span> <span class="keyword operator">=</span> <span class="constant language">True</span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> enables gzip compression for request bodies
</span> <span class="variable parameter">use_ssl</span> <span class="keyword operator">=</span> <span class="constant language">False</span>
</span><span class="punctuation">)</span></span>
<span class="comment"><span class="punctuation comment">#</span> Der Name des Index
</span><span class=""><span class="">index_name</span></span> <span class="keyword operator">=</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">toots<span class="punctuation string">'</span></span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Import toots<span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class="constant language">None</span>
<span class=""><span class="keyword control">for</span> <span class="variable language">_</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class=""><span class="support function">range</span></span><span class="punctuation">(</span><span class=""><span class="constant numeric">50</span></span><span class="punctuation">)</span></span><span class="punctuation">:</span></span>
<span class=""><span class="">page</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">timeline_local</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">limit</span><span class="keyword operator">=</span><span class="constant numeric">40</span></span><span class="punctuation">)</span></span> <span class=""><span class="keyword control">if</span> <span class=""><span class="">page</span></span> <span class="keyword operator">is</span> <span class="constant language">None</span> <span class="keyword control">else</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">fetch_next</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">page</span></span></span><span class="punctuation">)</span></span></span>
<span class=""><span class="keyword control">for</span> <span class="">toot</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">page</span></span><span class="punctuation">:</span></span>
<span class=""><span class="support function">id</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">toot</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">id<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class="">response</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">index</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">index_name</span></span><span class="punctuation">,</span> <span class="variable parameter">id</span><span class="keyword operator">=</span><span class=""><span class="support function">id</span></span><span class="punctuation">,</span> <span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">toot</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">client</span><span class="punctuation">.</span><span class="">indices</span><span class="punctuation">.</span></span><span class=""><span class="variable function">refresh</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index_name</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Finished<span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<h1 dir="auto">Wörter zählen</h1>
<p dir="auto">Mit den Verfeinerungen des Mappings haben wir im Content die Möglichkeit Aggregatsfunktionen auf den Text-Inhalt durchzuführen. Meine Idee ist es herauszufinden, wie häufig Wörter in all den Toots verwendet werden.</p>
<p dir="auto">OpenSearch bietet dazu eine Funktion an, die genau das macht:</p>
<p dir="auto"><code>GET {{url}}/toots/_search</code></p>
<pre><code dir="auto"><span class="source"><span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>size<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">0</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>aggregations<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>wordcount<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>terms<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>field<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>content<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>size<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>100<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>exclude<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>[0123456789]+|http.*<span class="punctuation string">"</span></span>
</span><span class="punctuation">}</span></span>
</span><span class="punctuation">}</span></span>
</span><span class="punctuation">}</span></span>
</span><span class="punctuation">}</span></span>
</span></code></pre>
<p dir="auto">Wie man sieht, muss man einen Request-Body mit der GET-Methode schicken. <code>"size": 0</code> ist komisch, bedeutet aber, dass wir an dem Ergebnis der Toots nicht interessiert sind, die bei der Suche gefunden wurden. Wir suchen über alle Toots und wollen nur Wörter zählen. Das <code>aggregations</code> Objekt deklariert, was wir berechnet haben wollen. <code>wordcount</code> ist unser frei gewählter Name (man kann nämlich auch mehrere Aggregationen durchführen, sogar verschachtelt. Da ist es gut, sprechende Namen zu vergeben). <code>terms</code> ist die Art der Aggregation (auf Term-Ebene, also auf unseren indizierten Tokens aus dem Text). Die Funktion <code>terms</code> unterstützt ein paar Parameter. Erstmal das Attribut, auf den wir aggregieren wollen (hier auf <code>content</code>). Wir sind nur an den 100 häufigsten Termen (Wort-Abschnitte, Tokens) interessiert und Ziffernfolgen oder URLs sind keine “Wörter”, die wir gezählt haben wollen.</p>
<p dir="auto">Man kann das in Postman ausführen (was ich für erste Tests immer so mache) oder auch in ein Python-Programm gießen. Da wir hier mit Python spielen, folgt gleich ein Stück Sourcecode. Ein Ergebnis in Postman sieht so aus (es ist immer gut zu wissen, wie eine Response aussieht, dann kann man einfacher dagegen programmieren):</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>took<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">155</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>timed_out<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">false</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>_shards<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>total<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">4</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>successful<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">4</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>skipped<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">0</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>failed<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">0</span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>hits<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>total<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>value<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">2001</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>relation<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>eq<span class="punctuation string">"</span></span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>max_score<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">null</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>hits<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>aggregations<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>wordcount<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>doc_count_error_upper_bound<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">18</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>sum_other_doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">25902</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>buckets<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">[</span>
<span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>key<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>mal<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">159</span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span>
<span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>key<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>esc<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">133</span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span>
<span class="invalid">.</span><span class="invalid">.</span><span class="invalid">.</span>
</span></span></span></span></span></span></span></span></code></pre>
<p dir="auto">Am Anfang wieder die Meta-Infos (155ms für die Ausführung mit 4 Shards, 2001 Toots wurden für die Aggregierung berücksichtigt, <code>hits</code> ist leer, weil wir es so wollten)</p>
<p dir="auto">Danach folgt das aggregations Objekt <code>wordcount</code> mit statistischen Infos (ja, OpenSearch ist sehr geschwätzig) und in <code>buckets</code> endlich die Ergebnisse. Das Wort <code>mal</code> kommt 159 Mal in 2001 Toots im Feld <code>content</code> vor, <code>esc</code> ganze 133 Mal, usw. usf..</p>
<p dir="auto">Cool. Wie sieht das in Python aus?</p>
<p dir="auto">Hier das <code>osWordCountOnCotnent.py</code>:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">opensearchpy</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">OpenSearch</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">login</span></span> </span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">host</span><span class="punctuation">,</span> <span class="variable parameter">port</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class="comment"><span class="punctuation comment">#</span> Client zu dem Dev Cluster (ohne SSL, ohne Anmeldung)
</span> <span class="keyword control">return</span> <span class=""><span class=""><span class="variable function">OpenSearch</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">hosts</span> <span class="keyword operator">=</span> <span class=""><span class="punctuation">[</span><span class=""><span class="punctuation">{</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">host<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">host</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">port<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">port</span></span><span class="punctuation">}</span></span><span class="punctuation">]</span></span><span class="punctuation">,</span>
<span class="variable parameter">http_compress</span> <span class="keyword operator">=</span> <span class="constant language">True</span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> enables gzip compression for request bodies
</span> <span class="variable parameter">use_ssl</span> <span class="keyword operator">=</span> <span class="constant language">False</span>
</span><span class="punctuation">)</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">word_count</span></span> </span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">aggs_wc</span></span><span class="keyword operator">=</span><span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">size<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="constant numeric">0</span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">aggregations<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">wordcount<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">terms<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">field<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">content<span class="punctuation string">"</span></span></span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">size<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">100<span class="punctuation string">"</span></span></span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">exclude<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">[0123456789]+|http.*<span class="punctuation string">"</span></span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class="keyword control">return</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">search</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">aggs_wc</span></span><span class="punctuation">,</span> <span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">client</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">login</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">localhost<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="constant numeric">9201</span></span><span class="punctuation">)</span></span>
<span class=""><span class="">words</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">word_count</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">client</span><span class="keyword operator">=</span><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class="variable parameter">index</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">toots<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">buckets</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">words</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">aggregations<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">wordcount<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">buckets<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class="keyword control">for</span> <span class="">bucket</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">buckets</span></span><span class="punctuation">:</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">bucket</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">doc_count<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"> mal </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class=""><span class="">bucket</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">key<span class="punctuation string">'</span></span></span></span><span class=""><span class="punctuation">]</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Das Ergebnis ist schön, aber leider auch etwas durchwachsen. Man wird immer noch eine Menge Allerweltswörter finden, die zwar einen Einblick in die Sprache geben, aber inhaltlich uns nicht weiterbringen. Ok, wir hatten ja auch keine spezifischen Anforderungen. Nur Wörter zählen. Aber es gibt einfach sehr viele Wörter, deren Anzahl wohl kaum jemanden interessieren wird.</p>
<p dir="auto">Man könnte nun eine riesige Liste an Excludes definieren (und letztendlich daran scheitern) oder (was auch geht) ein <code>include</code> festlegen, aber dann wird nur das gezählt, was wir selbst schon erwarten. </p>
<h1 dir="auto">Tags zählen</h1>
<p dir="auto">Aber da gibt es ja was, was sich etabliert hat, seit dem es vermutlich die Menschheit gibt: Kategorisierungen. Dieses Prinzip wird sehr einfach in sozialen Medien mit <em>Tags</em> realisiert. Um solche Tags auszuzeichnen, nutzt man als Präfix das <code>#</code> Zeichen (Hash) und daher heißen die auch Hashtags.</p>
<p dir="auto">In unserer <a href="https://write.tchncs.de/%7E/BeanDevMastodon/Anatomie%20eines%20Toots" rel="noopener noreferrer">Analyse der Toots in einem älteren Blog-Artikel</a>, konnte man sehen, dass die Mastodon API die Hashtags analysiert, extrahiert und im Content als Links formatiert und in dem Array <code>tags</code> gesondert auflistet.</p>
<p dir="auto">Und so haben wir die Informationen in unseren OpenSearch Index <code>toots</code> kopiert und mit der letzten Verfeinerung des Mappings auch zugänglich gemacht.</p>
<p dir="auto">Aber wie zählen wir nun die Häufigkeit dieser Tags? Die Aggregat-Funktion über Terme funktioniert nur in einem Feld. Das ist einer speziellen Implementation geschuldet, die extrem schnell eine zusammenhängende Wortliste durchzählt. Das im Heap der Instanzen aller OpenSearch-Nodes im Cluster und parallel über alle Shards (Apache-Lucene Prozesse). Abschließend werden die Daten dann zusammengezählt und zurückgeliefert.</p>
<p dir="auto">Wie bekommen wir also die getrennt gespeicherten, einzelnen Tags aus dem Array in ein Feld, damit die Aggregierung funktioniert? Es gibt die Funktion “nested”, die genau das tut. Sie gibt an, welches Feld in einer tiefen Struktur für eine Aggregation zusammengeführt werden soll.</p>
<p dir="auto">Als Abfrage sieht das so aus:</p>
<p dir="auto"><code>GET {{url}}\toots\_search</code></p>
<pre><code dir="auto"><span class="source"> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>size<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">0</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>aggregations<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>Nest<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>nested<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>path<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>tags<span class="punctuation string">"</span></span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>aggregations<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>tagcloud<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>terms<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>field<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>tags.name<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>min_doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">2</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>size<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>100<span class="punctuation string">"</span></span>
</span><span class="punctuation">}</span></span>
</span><span class="punctuation">}</span></span>
</span><span class="punctuation">}</span></span>
</span><span class="punctuation">}</span></span>
</span><span class="punctuation">}</span></span>
</span><span class="punctuation">}</span></span>
</span></code></pre>
<p dir="auto">Das bedeutet nichts anderes, als: Betrachte alles unter <code>tags</code> (pro Dokument) als eine Einheit und aggregiere <code>tags.name</code> als Term. Letztendlich soll das Wort mindestens zwei Mal vorkommen und wir sind an den Top 100 interessiert.</p>
<p dir="auto">Da wir nun eine verschachtelte Aggregation haben, rutschen die <code>buckets</code> eine Hierarchiestufe tiefer:</p>
<pre><code dir="auto"><span class="source"> ...
<span class="string"><span class="punctuation string">"</span>aggregations<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>Nest<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">1998</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>tagcloud<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>doc_count_error_upper_bound<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">8</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>sum_other_doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">1750</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>buckets<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">[</span>
<span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>key<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>esc<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">45</span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span>
<span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>key<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>Chatkontrolle<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">35</span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span>
<span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>key<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>twitter<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>doc_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">16</span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span>
<span class="invalid">.</span><span class="invalid">.</span><span class="invalid">.</span>
</span></span></span></span></span></span></span></span></code></pre>
<p dir="auto">Das sieht doch schon inhaltlich viel sinnvoller aus. </p>
<h1 dir="auto">Es wird bunt: Eine Tag-Cloud</h1>
<p dir="auto">Wir sind noch nicht am Ende. Es wird, wie versprochen nun bunt. Datenanalysen haben häufig den Makel, dass sie letztendlich trockene Daten produzieren, die schwer zugänglich sein können. Der Mensch ist sehr kreativ, um dieses Problem zu lösen und hat Visualisierungen entwickelt, die sich in Diagrammen widerspiegeln. Das sind aber nicht nur Plots mit x/y Achse, sondern können vielfältige Aufbereitungen bedeuten. Die wohl verspielteste Lösung ist eine Wort-Wolke. Es gibt eine <a href="https://amueller.github.io/word_cloud/" rel="noopener noreferrer">WordCloud Python Bibliothek</a>, die genau sowas erzeugt (und wirklich einen Haufen an Abhängigkeiten benötigt, aber wir werfen uns trotzdem in das Vergnügen).</p>
<p dir="auto">Damit es möglichst einfach geht, brauchen wir folgende Imports:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">opensearchpy</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">OpenSearch</span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">wordcloud</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">WordCloud</span></span>
<span class=""><span class="keyword control">import</span> <span class=""><span class="">multidict</span></span> <span class="keyword control">as</span> <span class=""><span class="">multidict</span></span></span>
<span class=""><span class="keyword control">import</span> <span class=""><span class="">matplotlib</span><span class="punctuation">.</span><span class="">pyplot</span></span> <span class="keyword control">as</span> <span class=""><span class="">plt</span></span></span>
<span class=""><span class="keyword control">import</span> <span class=""><span class="">numpy</span></span> <span class="keyword control">as</span> <span class=""><span class="">np</span></span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">PIL</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Image</span></span>
</span></code></pre>
<p dir="auto">Die notwendigen Bibliotheken mit <code>pip</code> installieren (das wird wohl mindestens wordcloud sein, ggf. mehr, wenn ihr das noch nicht habt). WordCloud wird einiges ranziehen und auf einigen OS (wie Windows) auch noch extra Downloads fordern (Visual C++).</p>
<p dir="auto">Nun das, was man erwarten würde:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">login</span></span> </span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">host</span><span class="punctuation">,</span> <span class="variable parameter">port</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class="comment"><span class="punctuation comment">#</span> Client zu dem Dev Cluster (ohne SSL, ohne Anmeldung)
</span> <span class="keyword control">return</span> <span class=""><span class=""><span class="variable function">OpenSearch</span></span><span class="punctuation">(</span><span class="">
<span class="variable parameter">hosts</span> <span class="keyword operator">=</span> <span class=""><span class="punctuation">[</span><span class=""><span class="punctuation">{</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">host<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">host</span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">port<span class="punctuation string">'</span></span></span><span class="punctuation">:</span> <span class=""><span class="">port</span></span><span class="punctuation">}</span></span><span class="punctuation">]</span></span><span class="punctuation">,</span>
<span class="variable parameter">http_compress</span> <span class="keyword operator">=</span> <span class="constant language">True</span><span class="punctuation">,</span> <span class="comment"><span class="punctuation comment">#</span> enables gzip compression for request bodies
</span> <span class="variable parameter">use_ssl</span> <span class="keyword operator">=</span> <span class="constant language">False</span>
</span><span class="punctuation">)</span></span>
<span class="function">
<span class="storage type function">def</span> <span class="entity name function"><span class="">aggregate_tags</span></span> </span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">client</span><span class="punctuation">,</span> <span class="variable parameter">index</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">aggs_tags</span></span><span class="keyword operator">=</span><span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">size<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="constant numeric">0</span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">aggregations<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">Nest<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">nested<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">path<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">tags<span class="punctuation string">"</span></span></span>
<span class="punctuation">}</span></span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">aggregations<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">tagcloud<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">terms<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">field<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">tags.name<span class="punctuation string">"</span></span></span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">min_doc_count<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="constant numeric">2</span><span class="punctuation">,</span>
<span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">size<span class="punctuation string">"</span></span></span><span class="punctuation">:</span> <span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">100<span class="punctuation string">"</span></span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class="punctuation">}</span></span>
<span class="keyword control">return</span> <span class=""><span class=""><span class="">client</span><span class="punctuation">.</span></span><span class=""><span class="variable function">search</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">body</span><span class="keyword operator">=</span><span class=""><span class="">aggs_tags</span></span><span class="punctuation">,</span> <span class="variable parameter">index</span><span class="keyword operator">=</span><span class=""><span class="">index</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Die Buckets müssen wir für WordCloud in ein MultiDict umwandeln:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">create_frequency</span></span> </span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">buckets</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">freqs</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">multidict</span><span class="punctuation">.</span></span><span class=""><span class="variable function">MultiDict</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
<span class=""><span class="keyword control">for</span> <span class="">bucket</span> <span class=""><span class="keyword control">in</span></span></span><span class=""> <span class=""><span class="">buckets</span></span><span class="punctuation">:</span></span>
<span class=""><span class=""><span class="">freqs</span><span class="punctuation">.</span></span><span class=""><span class="variable function">add</span></span> <span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">bucket</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">key<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class="punctuation">,</span> <span class=""><span class=""><span class="">bucket</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">doc_count<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span></span><span class="punctuation">)</span></span>
<span class="keyword control">return</span> <span class=""><span class="">freqs</span></span>
</span></code></pre>
<p dir="auto">Jetzt nehmen wir einfach ein Beispiel aus dem WordCloud Tutorial:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">make_wordcloud_image_simple</span></span> </span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">freqs</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">wc</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">WordCloud</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">max_words</span><span class="keyword operator">=</span><span class="constant numeric">1000</span><span class="punctuation">,</span> <span class="variable parameter">background_color</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">white<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">wc</span><span class="punctuation">.</span></span><span class=""><span class="variable function">generate_from_frequencies</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">freqs</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">plt</span><span class="punctuation">.</span></span><span class=""><span class="variable function">imshow</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">wc</span></span><span class="punctuation">,</span> <span class="variable parameter">interpolation</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">bilinear<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">plt</span><span class="punctuation">.</span></span><span class=""><span class="variable function">axis</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">off<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">plt</span><span class="punctuation">.</span></span><span class=""><span class="variable function">show</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Das rufen wir nun nacheinander auf:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">client</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">login</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">localhost<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="constant numeric">9201</span></span><span class="punctuation">)</span></span>
<span class=""><span class="">tags</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">aggregate_tags</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">client</span><span class="keyword operator">=</span><span class=""><span class="">client</span></span><span class="punctuation">,</span> <span class="variable parameter">index</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">toots<span class="punctuation string">'</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">buckets</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">tags</span></span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">aggregations<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">Nest<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">tagcloud<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span><span class=""><span class="punctuation">[</span></span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">buckets<span class="punctuation string">"</span></span></span></span><span class=""><span class="punctuation">]</span></span>
<span class=""><span class="">tag_freq</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">create_frequency</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">buckets</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="variable function">make_wordcloud_image_simple</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">tag_freq</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Wow. Unser Ergebnis:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/BECB8256-EFD8-18DF-22C6-F41104BEB6AB.png" alt="Mastodon TagCloud (simple)"></p>
<h1 dir="auto">Noch bunter: Mastodon Tag-Cloud</h1>
<p dir="auto">Es macht Spaß mit der WordCloud Bibliothek zu spielen, also setzen wir noch einen drauf. Die Bibliothek kann Word-Wolken auch in Schablonen anordnen. Was liegt da nicht näher mal ein Mamut zu nehmen:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/CBB73512-871E-00FA-38B0-E066B7DE8E46.png" alt="Mastodon Image"></p>
<p dir="auto">Der Code dafür:</p>
<pre><code dir="auto"><span class="source"><span class="function"><span class="storage type function">def</span> <span class="entity name function"><span class="">make_wordcloud_image</span></span> </span><span class="function"><span class="punctuation">(</span></span><span class="function"><span class="variable parameter">freqs</span><span class="punctuation">)</span></span><span class="function"><span class="punctuation function">:</span></span>
<span class=""><span class="">mastodon_mask</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">np</span><span class="punctuation">.</span></span><span class=""><span class="variable function">array</span></span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">Image</span><span class="punctuation">.</span></span><span class=""><span class="variable function">open</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">mastodon_mask.png<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">wc</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">WordCloud</span></span><span class="punctuation">(</span><span class=""><span class="variable parameter">max_words</span><span class="keyword operator">=</span><span class="constant numeric">1000</span><span class="punctuation">,</span> <span class="variable parameter">mask</span><span class="keyword operator">=</span><span class=""><span class="">mastodon_mask</span></span><span class="punctuation">,</span>
<span class="variable parameter">background_color</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">white<span class="punctuation string">"</span></span></span><span class="punctuation">,</span> <span class="variable parameter">contour_color</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">brown<span class="punctuation string">"</span></span></span><span class="punctuation">,</span> <span class="variable parameter">contour_width</span><span class="keyword operator">=</span><span class="constant numeric">4</span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">wc</span><span class="punctuation">.</span></span><span class=""><span class="variable function">generate_from_frequencies</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">freqs</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">plt</span><span class="punctuation">.</span></span><span class=""><span class="variable function">imshow</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">wc</span></span><span class="punctuation">,</span> <span class="variable parameter">interpolation</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">bilinear<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">plt</span><span class="punctuation">.</span></span><span class=""><span class="variable function">axis</span></span><span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">off<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="">plt</span><span class="punctuation">.</span></span><span class=""><span class="variable function">show</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
<span class="punctuation">.</span><span class="punctuation">.</span><span class=""><span class=""><span class="punctuation">.</span>
</span><span class=""><span class="variable function">make_wordcloud_image</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">tag_freq</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Das Ergebnis:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/1A370484-1E83-2D82-4F91-907754F22B32.png" alt="Mastodon TagCloud"></p>
<p dir="auto">Das war nun ein Haufen Erklärung für nur ein paar Zeilen Code. Das zeigt wie man (mit entsprechenden Bibliotheken, Frameworks und Diensten) sehr komplexe Dinge in wenigen Programmzeilen erledigen kann. Allerdings sollte man den Überblick behalten, wie man das alles orchestriert. </p>
]]><![CDATA[Anatomie eines Toots]]>https://write.tchncs.de/~/BeanDevMastodon/Anatomie%20eines%20Toots/2022-05-05T09:22:24.057697+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-05T09:22:24.057697+00:00<![CDATA[<p dir="auto"><img src="https://write.tchncs.de/static/media/E3CA901B-7DD2-81CF-27F4-41151563D55C.png" alt="Star Trek Discovery Communicator Blueprint"></p>
<p dir="auto">In der Mastodon-API nennt sich ein Toot tatsächlich <em>Status</em> und so lässt sich gleich auch schneller <a href="https://docs.joinmastodon.org/entities/status/" rel="noopener noreferrer">die notwendigen Informationen zur Struktur</a> und Inhalt eines Toot finden. </p>
<p dir="auto">Ich möchte die Dokumentation hier nicht wiederholen, sondern nur auf ein paar wichtige Bereiche eingehen.</p>
<p dir="auto">Noch mal das Beispiel aus dem <a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mit%20Python%20auf%20Mastodon%20eine%20Nachricht%20senden" rel="noopener noreferrer">letzten Blog-Artikel</a> (gekürzt):</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>id<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">108243723640515608</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>created_at<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>2022-05-04 12:42:26.372000+00:00<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>in_reply_to_id<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">null</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>in_reply_to_account_id<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">null</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>sensitive<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">false</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>spoiler_text<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span><span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>visibility<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>public<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>language<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>de<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>uri<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>https://social.tchncs.de/users/beandev/statuses/108243723640515608<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>url<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>https://social.tchncs.de/@beandev/108243723640515608<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>replies_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">1</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>reblogs_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">1</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>favourites_count<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">1</span><span class="punctuation">,</span></span>
</span></span></code></pre>
<p dir="auto">Neben den erwartbaren Attributen wie die ID, Daten der Erstellung, Sichtbarkeitseinstellungen und dem Inhalt der Nachricht selbst, finden sich einiges an Informationen, die erstmal überraschen könnten. Vor allem gibt es schon ein paar statistische Daten, wie häufig der Status favorisiert oder geboostert wurde. Daraus kann man schon mal eine Konsequenz ableiten. Wenn ein Toot über die API zu verschiedenen Zeitpunkten abgefragt wird, erhält man unterschiedliche Responses. </p>
<p dir="auto">Schauen wir uns den Content an:</p>
<pre><code dir="auto"><span class="source"> <span class="string"><span class="punctuation string">"</span>content<span class="punctuation string">"</span></span>: <span class="string"><span class="punctuation string">"</span><p>Das ist ein Test, bitte ignorieren.</p><span class="punctuation string">"</span></span>,
</span></code></pre>
<p dir="auto">Da sind HTML Tags drin! Erinnern wir uns an den Befehl, den wir verwendet hatten:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">response</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">toot</span></span> <span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">Das ist ein Test, bitte ignorieren.<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Da muss also was passiert sein. Zudem bedeutet das, es spricht nichts gegen HTML-Auszeichnungen in Nachrichten. Allerdings wird man das in keinem Micro-Blogging Client (für z.B. Mastodon) im Fediverse finden. Das ist eine Beschränkung, die andere Dienste (im Bereich des Publishing und Macro-Blogging) nicht haben (z.B. <a href="https://friendi.ca/" rel="noopener noreferrer">Friendica</a>). Aber alle diese Dienste kommunizieren miteinander auf dem gleichen Protokoll <a href="https://www.w3.org/TR/activitypub/" rel="noopener noreferrer">ActivityPub</a> und somit finden sich auf einmal HTML Tags im Content, die uns die API freundlicherweise angefügt hat. Das geht sogar noch viel weiter, wie ich noch später erläutern werde. </p>
<p dir="auto">Nicht sehr überrascht es, dass man auch die <em>Application</em> als Information geliefert bekommt. Einige Clients zeigen das zu jedem Toot direkt an:</p>
<pre><code dir="auto"><span class="source"> <span class="string"><span class="punctuation string">"</span>application<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>name<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>Spielplatz<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>website<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>https://social.tchncs.de/@beandev<span class="punctuation string">"</span></span>
</span><span class="punctuation">}</span></span>,
</span></code></pre>
<p dir="auto">Als ich das erste Mal die Response zu den Timeline-Daten und Statusnachrichten gesehen hatte, überraschte mich am meisten, für jeden einzelnen Toot die <em>Account</em>-Informationen zu bekommen:</p>
<pre><code dir="auto"><span class="source"> ...
<span class="string"><span class="punctuation string">"</span>application<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>name<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>Playground<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>website<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>https://social.tchncs.de/@beandev<span class="punctuation string">"</span></span>
</span><span class="punctuation">}</span></span>,
<span class="string"><span class="punctuation string">"</span>account<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>id<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">38900</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>username<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>beandev<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>acct<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>beandev<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>display_name<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>Aljoscha Rittner (beandev)<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>locked<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">false</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>bot<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">false</span><span class="punctuation">,</span></span>
<span class="invalid">.</span><span class="invalid">.</span><span class="invalid">.</span>
</span></span></code></pre>
<p dir="auto">Das erscheint zunächst extrem kostspielig, so viele (eigentlich redundante) Informationen mitzuliefern, ist aber in der föderierten Struktur des Fediverse begründet. Es soll sichergestellt werden, dass jeder Client alle Daten, ohne weitere Rückfragen, zur Verfügung hat. Das ist konsequent implementiert, soweit es sich um textuelle Informationen handelt, bei Daten (Bilder, Videos, Soundstreams) erhält man trotzdem nur Referenzen. Der Vorteil ist, dass es den entfernten Host (Mastodon Instanz) vor vielen Anfragen zu Detailinformationen bewahrt, die extra Roundtrips benötigen (was eine summierte hohe Latenz bedeutet und den Host zusätzlich belasten würde).</p>
<p dir="auto">Am Ende des Beispiel-Toots sehen wir noch ein paar zusätzliche Attribute:</p>
<pre><code dir="auto"><span class="source"> ...
<span class="string"><span class="punctuation string">"</span>media_attachments<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>,
<span class="string"><span class="punctuation string">"</span>mentions<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>,
<span class="string"><span class="punctuation string">"</span>tags<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>,
<span class="string"><span class="punctuation string">"</span>emojis<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>,
<span class="string"><span class="punctuation string">"</span>card<span class="punctuation string">"</span></span>: <span class="constant language">null</span>,
<span class="string"><span class="punctuation string">"</span>poll<span class="punctuation string">"</span></span>: <span class="constant language">null</span>
}
...
</span></code></pre>
<p dir="auto">Die sind für unseren ersten Beispiel-Toot leer. Aber gerade diese Attribute haben eine enorme Bedeutung, zusammen mit dem Content. </p>
<p dir="auto">Um etwas Praxis ins Spiel zu bringen, lesen wir mal einen anderen Status aus:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">json</span></span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">mastodon</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Mastodon</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">Mastodon</span></span> <span class="punctuation">(</span><span class="">
<span class="variable parameter">api_base_url</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">https://social.tchncs.de<span class="punctuation string">'</span></span></span>
</span><span class="punctuation">)</span></span>
<span class=""><span class="">response</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">status</span></span> <span class="punctuation">(</span><span class=""><span class="constant numeric">108247697988744452</span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">json</span><span class="punctuation">.</span></span><span class=""><span class="variable function">dumps</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">response</span></span><span class="punctuation">,</span> <span class="variable parameter">indent</span><span class="keyword operator">=</span><span class="constant numeric">4</span><span class="punctuation">,</span> <span class="variable parameter">default</span><span class="keyword operator">=</span><span class=""><span class="support type">str</span></span></span><span class="punctuation">)</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Im Browser sieht die Nachricht so aus:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/8C411AB0-6FDF-EE5A-3EE4-86DB87BCACEB.png" alt="Status mit Hashtag"></p>
<p dir="auto">Es gibt einen Hashtag, der als Link anzuklicken ist. Nun ist das nicht eine besondere Fähigkeit des Browser-UI. Nein, dieser Link existiert wirklich in den Daten. Der Ausschnitt aus der Response unseres obigen Aufrufs:</p>
<pre><code dir="auto"><span class="source"> <span class="string"><span class="punctuation string">"</span>content<span class="punctuation string">"</span></span>: <span class="string"><span class="punctuation string">"</span><p>Was will die neue <span class="invalid">
</span></span> <a href=\<span class="string"><span class="punctuation string">"</span>https://social.tchncs.de/tags/LucaApp<span class="constant">\"</span> <span class="invalid">
</span></span> class=\<span class="string"><span class="punctuation string">"</span>mention hashtag<span class="constant">\"</span> <span class="invalid">
</span></span> rel=\<span class="string"><span class="punctuation string">"</span>tag<span class="constant">\"</span>>#<span>LucaApp</span></a> <span class="invalid">
</span></span> nicht l\u<span class="constant numeric">0</span><span class="constant numeric">0</span>f<span class="constant numeric">6</span>sen, was die alte schon nicht konnte?</p><span class="string"><span class="punctuation string">"</span><span class="invalid">
</span></span></span></code></pre>
<p dir="auto">Da wurde also aus einer einfachen Textzeichenkette mit einer ‘<a href="//write.tchncs.de/tag/" rel="noopener noreferrer">#</a>’ Auszeichnung gleich ein ordentliches Stück HTML.</p>
<p dir="auto">Nun der Blick ans Ende der JSON Struktur:</p>
<pre><code dir="auto"><span class="source"> ...
<span class="string"><span class="punctuation string">"</span>media_attachments<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>,
<span class="string"><span class="punctuation string">"</span>mentions<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>,
<span class="string"><span class="punctuation string">"</span>tags<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">[</span>
<span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>name<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>lucaapp<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>url<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>https://social.tchncs.de/tags/lucaapp<span class="punctuation string">"</span></span>
</span><span class="punctuation">}</span></span>
<span class="punctuation">]</span></span>,
<span class="string"><span class="punctuation string">"</span>emojis<span class="punctuation string">"</span></span>: <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span>,
<span class="string"><span class="punctuation string">"</span>card<span class="punctuation string">"</span></span>: <span class="constant language">null</span>,
<span class="string"><span class="punctuation string">"</span>poll<span class="punctuation string">"</span></span>: <span class="constant language">null</span>
}
</span></code></pre>
<p dir="auto">Dort ist der Hashtag-Ausdruck als <em>tag</em> aufgelistet. Der Name des <em>tag</em> ist in den Daten nur in Kleinbuchstaben beschrieben und dazu gibt es noch ein URL-Link zur Hashtag-Timeline der Mastodon-Instanz.</p>
<p dir="auto">Die Mastodon-Instanz mit ihrer API betreibt also einen großen Aufwand, um aus den neuen Toots bestimmte Informationen zu parsen, diese in HTML umzuwandeln und zugleich diese geparsten Elemente strukturiert zur Verfügung zu stellen. </p>
<p dir="auto">Das wird auch mit <a href="//write.tchncs.de/@//" rel="noopener noreferrer">@</a> annotierte Erwähnungen gemacht, den Dateien (wie Bilder und Videos) und den auf Mastodon-Instanzen individuellen Emoticons. </p>
]]><![CDATA[Mit Python auf Mastodon eine Nachricht senden]]>https://write.tchncs.de/~/BeanDevMastodon/Mit%20Python%20auf%20Mastodon%20eine%20Nachricht%20senden/2022-05-04T12:52:31.709254+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-04T12:52:31.709254+00:00<![CDATA[<h1 dir="auto">Einführung</h1>
<p dir="auto">Im Blog Artikel zu den <a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mastodon%20Instanz%20API%20nutzen%20-%20Erste%20Schritte" rel="noopener noreferrer">ersten Schritten</a>, wurde umfangreich erklärt, wie man eine App registrieren kann. Genutzt wurde das für das erste Beispiel <a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mastodon%20Instanz%20Local%20Timeline%20lesen,%20ohne%20Anmeldung" rel="noopener noreferrer">zum Lesen einer lokalen Timeline</a> aber gar nicht. Das liegt daran, dass Mastodon drei verschiedene Ebenen der Authentifizierungen von Identitäten nutzt. <em>Public</em> bedeutet, man braucht gar keine Authentifizierung, <em>App Code</em> wird es in der Mastodon API genannt, wenn man mindestens mit einer Application registriert ist und sich einen Application Code geholt hat, mit dem man auf die API zugreifen will und letztendlich <em>User</em>, wo man als Identität einen Account benötigt und man mit diesem sich authentifiziert. </p>
<p dir="auto">Es gibt eigentlich nur einen Endpunkt in der REST API der Mastodon Instanz, die <em>App Code</em> voraussetzt. Das ist dann, wenn man einen neuen Account auf der Instanz anlegen will (was aber auch nur geht, wenn die Registrierung “offen” ist). Nahezu alle anderen Endpunkte sind entweder <em>Public</em> oder erfordern einen <em>User</em>.</p>
<h1 dir="auto">Der erste Toot</h1>
<p dir="auto">Gehen wir mal davon aus, ein Account wurde auf einer Instanz bereits registriert. Wir haben auf derselben Instanz noch unsere App registriert (wenn nicht, dann <a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mastodon%20Instanz%20API%20nutzen%20-%20Erste%20Schritte" rel="noopener noreferrer">hier</a> nachschauen), dann können wir mit wirklich wenigen Zeilen eine Nachricht an das Fediverse schicken.</p>
<p dir="auto">Der erste Teil ist wie erwartet: Unsere App-Registrierungsdaten werden als client_id und client_secret übergeben:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">json</span></span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">mastodon</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Mastodon</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">Mastodon</span></span> <span class="punctuation">(</span><span class="">
<span class="variable parameter">client_id</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">ep********************************************Xo<span class="punctuation string">'</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">client_secret</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">AY****************************************0s<span class="punctuation string">'</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">api_base_url</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">https://social.tchncs.de<span class="punctuation string">'</span></span></span>
</span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Nun brauchen wir aber den originären Login des Users, um diese Identität anzunehmen (eigentlich arbeitet nun unser Client Script im Auftrage des Users):</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">mastodon</span><span class="punctuation">.</span><span class="">access_token</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">log_in</span></span> <span class="punctuation">(</span><span class="">
<span class="variable parameter">username</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string"><die email-adresse des users>,<span class="invalid">
</span></span></span> <span class=""><span class="">password</span></span>=<span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string"><das password des users>,<span class="invalid">
</span></span></span> <span class=""><span class="">scopes</span></span>=<span class=""><span class="punctuation">[</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">read<span class="punctuation string">'</span></span></span><span class="punctuation">,</span> <span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">write<span class="punctuation string">'</span></span></span><span class="punctuation">]</span></span>
</span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Username und Password schreibt man üblicherweise nicht in Scripts, aber wir spielen ja noch rum. Besser wäre es, das Script liest sie aus den Argumenten ab oder es wird per Eingabe abgefragt.</p>
<p dir="auto">Wir brauchen den Scope ‘write’ um eine Nachricht zu senden. Das ist unglaublich einfach:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">response</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">toot</span></span> <span class="punctuation">(</span><span class=""><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">Das ist ein Test, bitte ignorieren.<span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">json</span><span class="punctuation">.</span></span><span class=""><span class="variable function">dumps</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">response</span></span><span class="punctuation">,</span> <span class="variable parameter">indent</span><span class="keyword operator">=</span><span class="constant numeric">4</span><span class="punctuation">,</span> <span class="variable parameter">default</span><span class="keyword operator">=</span><span class=""><span class="support type">str</span></span></span><span class="punctuation">)</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Das schauen wir uns mal in der Timeline an:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/24898729-EB92-21D3-C120-3B71BB9FE424.png" alt="Mastodon Toot"></p>
<p dir="auto">Perfekt, hat wohl geklappt. Wir nervten gerade ein paar tausend aktive Nutzer in der lokalen Timeline mit unserem Test.</p>
<p dir="auto">Spannend ist auch die Response des <code>toot()</code>-Aufrufs.</p>
<p dir="auto">Hier ein (stark) gekürzter Auszug:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>id<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">108243723640515608</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>created_at<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>2022-05-04 12:42:26.372000+00:00<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>visibility<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>public<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>language<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>de<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>uri<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>https://social.tchncs.de/users/beandev/statuses/108243723640515608<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>content<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span><p>Das ist ein Test, bitte ignorieren.</p><span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>reblog<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">null</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>application<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>name<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>Spielplatz<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>website<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>https://social.tchncs.de/@beandev<span class="punctuation string">"</span></span>
</span><span class="punctuation">}</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>account<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">{</span>
<span class=""><span class="string"><span class="punctuation string">"</span>id<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant numeric">38900</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>username<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="string"><span class="punctuation string">"</span>beandev<span class="punctuation string">"</span></span><span class="punctuation">,</span></span>
<span class="invalid">#</span> <span class="invalid">.</span><span class="invalid">.</span><span class="invalid">.</span>
<span class="punctuation">}</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>media_attachments<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>mentions<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>tags<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>emojis<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class=""><span class="punctuation">[</span><span class="punctuation">]</span></span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>card<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">null</span><span class="punctuation">,</span></span>
<span class=""><span class="string"><span class="punctuation string">"</span>poll<span class="punctuation string">"</span></span></span><span class=""><span class="punctuation">:</span> <span class="constant language">null</span>
</span><span class="punctuation">}</span></span>
</span></code></pre>
<p dir="auto">Man sieht aber in der Ausgabe, dass das Response-Objekt dramatisch größer ist. Lauter Attribute und vor allem eine komplette Account-Information. </p>
<p dir="auto">Tatsächlich ist Mastodon (und alle anderen Dienste im Fediverse) so geschwätzig, weil damit die Clients mit nur einer Response alle wichtigen Informationen darstellen können ohne weitere Roundtrip-Anfragen zu stellen. Das ist in einem föderierten, verteilten System erheblich einfacher, als ständig mit Nachfragen der Clients bombardiert zu werden.</p>
<p dir="auto">Abschließend sei gesagt, dass <code>toot()</code> nur für den ersten Test sinnvoll ist. Es gibt eine erheblich bessere Methode <code>status_post()</code>, die man dazu verwenden sollte (dazu aber mehr in einem anderen Artikel).</p>
]]><![CDATA[Mastodon Instanz deaktivierte Local Timeline lesen, ohne Anmeldung]]>https://write.tchncs.de/~/BeanDevMastodon/Mastodon%20Instanz%20deaktivierte%20Local%20Timeline%20lesen,%20ohne%20Anmeldung/2022-05-03T13:25:23.402075+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-03T13:25:23.402075+00:00<![CDATA[<p dir="auto">Im letzten Artikel <a href="https://write.tchncs.de/%7E/BeanDevMastodon/Mastodon%20Instanz%20Local%20Timeline%20lesen,%20ohne%20Anmeldung" rel="noopener noreferrer">Mastodon Instanz Local Timeline lesen, ohne Anmeldung</a> wurde davon ausgegangen, dass die Lokale Timeline (auch public timeline genannt), öffentlich einsehbar ist. Allerdings ist es möglich, dass Admins der Instanz das deaktiviert haben. </p>
<p dir="auto">Die API Dokumentation schreibt dazu:</p>
<blockquote dir="auto">
<p>OAuth: Public. Requires app token + read:statuses if the instance has disabled public preview.</p>
</blockquote>
<p dir="auto">Dies ist meiner Meinung nach ein Fehler in der Dokumentation. Wenn man nämlich in den <a href="https://github.com/mastodon/mastodon/pull/11802/files" rel="noopener noreferrer">git commit</a> schaut, sieht man diese Sourcecode Änderung:</p>
<pre><code dir="auto"><span class="source">
<span class=""><span class="keyword control">class</span></span><span class=""> </span><span class=""><span class="entity name"><span class="support">Api</span><span class="punctuation">::</span><span class="support">V1</span><span class="punctuation">::</span><span class="support">Timelines</span><span class="punctuation">::</span>PublicController</span></span><span class=""> <span class="punctuation"><</span></span><span class=""> </span><span class=""><span class="entity"><span class="support">Api</span><span class="punctuation">::</span>BaseController</span></span>
before_action <span class="constant"><span class="punctuation constant">:</span>require_user!</span><span class="punctuation">,</span> <span class="constant">only<span class="punctuation constant">:</span></span> <span class="punctuation">[</span><span class="constant"><span class="punctuation constant">:</span>show</span><span class="punctuation">]</span><span class="punctuation">,</span> <span class="constant">if<span class="punctuation constant">:</span></span> <span class="constant"><span class="punctuation constant">:</span>require_auth?</span>
</span></code></pre>
<p dir="auto"><code>require_auth</code> ist nur ein Check der Konfiguration, aber <code>require_user</code> macht klar, dass ein App Token nicht reicht, man muss mindestens als User angemeldet sein.</p>
<p dir="auto">Allerdings ist ein abgeschaltetes Preview extrem selten und evtl. kann man davon ausgehen, dass auch weitere Zugriffe auf die API von den Admins nicht erlaubt ist.</p>
<p dir="auto">Ob eine Instanz überhaupt das anbietet, kann man schnell so ermitteln:</p>
<ul dir="auto">
<li>Auf <a href="http://www.unmung.com/mastoview" rel="noopener noreferrer">www.unmung.com/mastoview</a> gehen (Achtung, das UI ist aus dem letzten Jahrtausend)</li>
<li>In das Feld oben links den Instanznamen eingeben (ohne Protokoll)</li>
<li>Local auswählen und dann Preview anklicken.</li>
</ul>
<p dir="auto">Dann sollte was zu sehen sein.</p>
<p dir="auto">Ich habe noch über <a href="https://instances.social/list/advanced" rel="noopener noreferrer">instances.social/list</a> gesucht und nur eine einzige Instanz mit UP gefunden, die Local Timeline Preview deaktiviert hatte. In dem Fall gibt es die Fehlermeldung:</p>
<pre><code dir="auto"> File "C:\DevTools\Core\Python39\lib\site-packages\decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "C:\DevTools\Core\Python39\lib\site-packages\mastodon\Mastodon.py", line 102, in wrapper
return function(self, *args, **kwargs)
File "C:\DevTools\Core\Python39\lib\site-packages\mastodon\Mastodon.py", line 753, in timeline_local
return self.timeline('local', max_id=max_id, min_id=min_id,
File "C:\DevTools\Core\Python39\lib\site-packages\decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "C:\DevTools\Core\Python39\lib\site-packages\mastodon\Mastodon.py", line 102, in wrapper
return function(self, *args, **kwargs)
File "C:\DevTools\Core\Python39\lib\site-packages\mastodon\Mastodon.py", line 734, in timeline
return self.__api_request('GET', url, params)
File "C:\DevTools\Core\Python39\lib\site-packages\mastodon\Mastodon.py", line 3425, in __api_request
raise ex_type(
mastodon.Mastodon.MastodonAPIError: ('Mastodon API returned error', 422, 'Unprocessable Entity', 'This method requires an authenticated user')
</code></pre>
<p dir="auto">Und die Exception macht auch klar, dass ein User (Account) mit Anmeldung notwendig ist. </p>
]]><![CDATA[Mastodon Instanz Local Timeline lesen, ohne Anmeldung]]>https://write.tchncs.de/~/BeanDevMastodon/Mastodon%20Instanz%20Local%20Timeline%20lesen,%20ohne%20Anmeldung/2022-05-03T10:57:57.995113+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-03T10:57:57.995113+00:00<![CDATA[<h1 dir="auto">Einführung</h1>
<p dir="auto">Es gab vor ein paar Tagen die Frage, ob man Timelines anderer Instanzen lesen kann, ohne einen Account zu haben. Oder anders ausgedrückt: Kann man einer kompletten Mastodon-Instanz folgen?</p>
<p dir="auto">Die Antwort ist Ja und Nein. Ja, es ist technisch möglich. Nein, mir ist kein Client bekannt, der das unterstützt (wenn wer einen Client kennt, dann würde ich mich über eine Info freuen).</p>
<p dir="auto">Wenn man in die <a href="https://docs.joinmastodon.org/api/" rel="noopener noreferrer">REST API</a> von Mastodon schaut, wird man nicht sofort sicher sein, ob sowas möglich ist. Aber es gibt ein paar Hinweise in dem <a href="https://docs.joinmastodon.org/client/intro/" rel="noopener noreferrer">API-Guide</a>, dass einige Endpunkte keine User-Authentifizierung benötigen und einige Endpunkte komplett öffentlich sind. </p>
<p dir="auto">Damit haben wir alles zusammen, dass es eine technische Lösung gibt (unabhängig davon, dass man theoretisch einen <a href="https://de.wikipedia.org/wiki/ActivityPub" rel="noopener noreferrer">ActivityPub</a> Server laufen lässt, um direkt auf die Sync-Streams zuzugreifen).</p>
<p dir="auto">Weiterhin nutze ich die <a href="https://github.com/halcy/Mastodon.py" rel="noopener noreferrer">Mastodon.py</a> Bibliothek, damit sich im Hintergrund um alles gekümmert wird. </p>
<h1 dir="auto">Abruf der lokalen Timeline</h1>
<p dir="auto">Die lokale Timeline sind bekanntermaßen die Nachrichten (Beiträge, Tröts, Toots, Status-Nachrichten), die alle Mitglieder der Instanz öffentlich gelistet haben. Es erfordert, wie schon gesagt, keine Authentifizierung eines lokalen Accounts (und damit auch kein Account) auf der Instanz, diese Timeline zu lesen.</p>
<p dir="auto">Somit ist das Script ziemlich simpel:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">json</span></span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">mastodon</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Mastodon</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">Mastodon</span></span> <span class="punctuation">(</span><span class="">
<span class="variable parameter">api_base_url</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">https://bildung.social<span class="punctuation string">'</span></span></span>
</span><span class="punctuation">)</span></span>
<span class=""><span class="">local</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">timeline_local</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">json</span><span class="punctuation">.</span></span><span class=""><span class="variable function">dumps</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">local</span></span><span class="punctuation">,</span> <span class="variable parameter">indent</span><span class="keyword operator">=</span><span class="constant numeric">4</span><span class="punctuation">,</span> <span class="variable parameter">default</span><span class="keyword operator">=</span><span class=""><span class="support type">str</span></span></span><span class="punctuation">)</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Die Methode timeline_local(), tut das, was man erwartet hätte. Es liest die letzten 20 Nachrichten der lokalen Timeline der Instanz aus.</p>
]]><![CDATA[Mastodon Instanz API nutzen - Erste Schritte]]>https://write.tchncs.de/~/BeanDevMastodon/Mastodon%20Instanz%20API%20nutzen%20-%20Erste%20Schritte/2022-05-03T09:54:24.890791+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-03T09:54:24.890791+00:00<![CDATA[<h1 dir="auto">Einführung</h1>
<p dir="auto">Mastodon hat eine sehr saubere und klare REST API, für eine Menge an Operationen. Im Prinzip alles, was man in einem Client durchführen kann (Lesen, Nachrichten schreiben, Favorisieren, Bilder hochladen, usw. usf.) lässt sich ebenfalls über die API realisieren. Allerdings erfordert das natürlich eine Authentifizierung. Ohne sich der Instanz “Bekanntzumachen”, kann man nicht auf die API zugreifen.</p>
<p dir="auto">Mastodon verwendet dafür OAuth2, was sich deutlich von einer User/Password Authentifizierung, wie es BasicAuth erlaubt, unterscheidet. Ohne tiefer ins Detail zu gehen, ist es wichtig zu wissen, dass OAuth2 verschiedene Anmelde-Workflows bietet, die stufenweise funktionieren. Man muss sich das einfach so vorstellen: Da man als Mensch ja nicht direkt mit der Maschine “Mastodon-Instanz” (dem Host) reden kann, benutzt man immer einen Stellvertreter, eine Application (der Client). Ein Client, kann eine Smartphone-App wie Tusky oder Fedilab sein, es kann das Mastodon WebUI sein oder eben auch ein Python-Script.</p>
<p dir="auto">Eine Client-App, die mit einem Host sprechen will, muss damit sich erstmal registrieren. Ansonsten akzeptiert der Host den Client nicht. Das funktioniert bei Mastodon auf zwei Arten. Entweder selbst über die API (denn über die API darf man alles machen) oder man registriert über das Mastodon WebUI eine Client Application. </p>
<p dir="auto">Dass sich eine App einfach selber registrieren darf, ist übrigens nicht selbstverständlich. Im Fediverse ist das bei dezentralen Instanzen und den vielen Client-Implementationen aber deutlich einfacher zu handhaben. Man stelle sich vor jede App (Tusky, Fedilab, Mastodon, …) müsste auf jeder Instanz in Fediverse erstmal manuell von jedem Anwender oder Entwickler registriert werden, damit man Nachrichten schreiben kann! Das wäre eine riesige Hürde, die man in einem offenen Netzwerk so nicht haben will.</p>
<p dir="auto">Wichtig ist zu wissen, dass mit der Registrierung einer App, die Mastodon Instanz sich diese Information merkt und dem Registrierenden ein paar Daten mitgibt, die man immer wieder verwenden sollte. Damit kommen wir zu wichtigen Merkmalen von OAuth2. Die App bekommt eine ID (Client-ID) und ein Geheimnis (Client-Secret) zugewiesen. Diese beiden Informationen reichen aber noch nicht, um tatsächlich auf die API zuzugreifen. Es gibt noch ein Zugangstoken. Ein Token ist eine zeitlich begrenzter Zugangsschlüssel. So wie eine Hotelkarte, mit der man nur zu den gebuchten Tagen in das Hotelzimmer darf, ist ein Zugangstoken ebenfalls auf eine bestimmte Zeit beschränkt. Ist der Token abgelaufen, muss man sich einen neuen besorgen. </p>
<p dir="auto">Das ist ein recht aufwendiger Arbeitsablauf und erfordert ein paar Schritte. Zunächst registriert man sich als Application. Dafür denkt man sich einen Namen aus und gibt eine URL der eigenen Heimat an (oder man verwendet einen speziellen Platzhalter). Dann bekommt man eine ID, ein Secret und einen Token. Mit den drei Informationen darf man dann die ersten Abfragen durchführen. Da das immer wiederkehrende Arbeiten sind, verlässt man sich dabei besser auf Bibliotheken, die das für einen machen. Für Python gibt es <a href="https://github.com/halcy/Mastodon.py" rel="noopener noreferrer">Mastodon.py</a>. Diese Bibliothek ist nicht nur ein Wrapper für die gesamte REST-API, sie kümmert sich auch um den ganzen Workflow der Registrierung und Authentifizierung. Es ist sozusagen der persönliche Hotelpage, der einem bei allem hilft, inkl. dem Öffnen der Türen.</p>
<p dir="auto">Zwar erlaubt auch Mastodon.py das direkte Registrieren per API, aber es ist sinnvoller, das über das Web-UI zu machen. In dem Fall merkt sich das UI die Registrierung und speichert die Daten für einen. Nicht jede Instanz erlaubt das, aber die meisten. </p>
<h1 dir="auto">Application Client registrieren</h1>
<h2 dir="auto">Grunddaten</h2>
<p dir="auto">Zunächst geht man über das WebUi in Einstellungen und dort auf Entwicklung:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/B05745E5-32E2-4981-2488-3D1D20FF0895.png" alt="Mastodon - Entwicklung - Anwendungen"></p>
<p dir="auto">Mit dem Button “Neue Anwendung” lässt sich nun eine Client-App registrieren:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/760B672E-FB2D-42DE-C115-BF8EF0E0AFF4.png" alt="Mastodon - Neue Anwendung - Anlegen"></p>
<p dir="auto">Als Namen denkt man sich was Nettes aus, die URL kann das eigene Instanz-Profil sein und die Weiterleitungs-URL sollte mit dem Platzhalter “urn:ietf:wg:oauth:2.0:oob” beschrieben sein. Wir schreiben Scripte und werden keine Login-Eingaben an einen Benutzer weiterleiten. </p>
<h2 dir="auto">Berechtigungen</h2>
<p dir="auto">Im untereren Bereich kann man bei “Befugnissen” festlegen, was die Client Application überhaupt alles darf. In OAuth2 nennt man das den “Scope”. Für die ersten Spielereien sollte man nur den Haken bei “read” aktiviert haben, damit man sich nicht aus Versehen was kaputt macht. Also “write” und “follow” erstmal deaktivieren. Man sieht aber, dass man sich sehr viele Einzelberechtigungen zusammenstellen kann:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/1AAA6ECB-100C-7B43-AAF6-2EFE3DC2C772.png" alt="Mastodon - Neue Anwendung - Berechtigungen"></p>
<h2 dir="auto">OAuth2 Daten der Registrierung</h2>
<p dir="auto">Nachdem man alles mit dem großen blauen Speicherbutton am Ende gesichert hat, ist erstmal Verwirrung angesagt. Nirgendwo sieht man nun die Client-ID und das Secret und auch kein Token. Dafür nun in der Liste der Anwendungen wieder auf den Namen klicken, und voilà: die OAuth2 Daten:</p>
<p dir="auto"><img src="https://write.tchncs.de/static/media/323220C9-4430-5E30-B91A-A6020CEC9A31.png" alt="Mastodon - Anwendung - OAuth2 Daten"></p>
<h1 dir="auto">Mastodon.py installieren</h1>
<p dir="auto">Ich gehe davon aus, dass Grundkenntnisse in Python existieren und vor allem eine Umgebung existiert, in Python zu entwickeln. Viele werden pip oder pip3 verwenden, also ist Mastodon.py schnell installiert.</p>
<pre><code dir="auto">pip3 install Mastodon.py
</code></pre>
<h1 dir="auto">Das erste Script</h1>
<p dir="auto">Nun die IDE der Wahl öffnen und ein superkleines Beispiel (noch nicht vollständig):</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="keyword control">import</span> <span class=""><span class="">json</span></span></span>
<span class=""><span class="keyword control">from</span></span><span class=""><span class=""> <span class=""><span class="">mastodon</span></span> <span class=""><span class="keyword control">import</span></span></span></span><span class=""></span><span class=""> <span class="">Mastodon</span></span>
<span class=""><span class="">mastodon</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="variable function">Mastodon</span></span> <span class="punctuation">(</span><span class="">
<span class="variable parameter">client_id</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">ep***************************************Xo<span class="punctuation string">'</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">client_secret</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">AY**********************************0s<span class="punctuation string">'</span></span></span><span class="punctuation">,</span>
<span class="variable parameter">api_base_url</span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">https://social.tchncs.de<span class="punctuation string">'</span></span></span>
</span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Wir brauchen “json” für bequemere Ausgaben und dann wird das installierte Mastodon.py importiert. Im nächsten Schritt wird das Mastodon-Object erzeugt und wir übergeben die client_id und das client_secret (aus dem Web-UI sind das Client-Schlüssel und Client-Secret). Die api_base muss die Instanz sein, wo wir die App registriert haben.</p>
<p dir="auto">Ups, warum nicht das Token? Wir lassen unseren Hotelpagen “Mastodon.py” den Job machen. Fehlt der Token, fragt Mastodon.py direkt nach einem neuen Token an. Etwas verschwenderisch, aber hier sinnvoll. Denn der Token läuft sowieso irgendwann ab und wir wollen ja nicht jedes Mal das Script ändern…</p>
<p dir="auto">Nun fügen wir aber noch was dem Script an:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">nodeinfo</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">instance_nodeinfo</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">json</span><span class="punctuation">.</span></span><span class=""><span class="variable function">dumps</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">nodeinfo</span></span><span class="punctuation">,</span> <span class="variable parameter">indent</span><span class="keyword operator">=</span><span class="constant numeric">4</span><span class="punctuation">,</span> <span class="variable parameter">default</span><span class="keyword operator">=</span><span class=""><span class="support type">str</span></span></span><span class="punctuation">)</span></span></span><span class="punctuation">)</span></span>
<span class=""><span class="">activity</span></span> <span class="keyword operator">=</span> <span class=""><span class=""><span class="">mastodon</span><span class="punctuation">.</span></span><span class=""><span class="variable function">instance_activity</span></span><span class="punctuation">(</span><span class="punctuation">)</span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class=""><span class=""><span class="">json</span><span class="punctuation">.</span></span><span class=""><span class="variable function">dumps</span></span><span class="punctuation">(</span><span class=""><span class=""><span class="">activity</span></span><span class="punctuation">,</span> <span class="variable parameter">indent</span><span class="keyword operator">=</span><span class="constant numeric">4</span><span class="punctuation">,</span> <span class="variable parameter">default</span><span class="keyword operator">=</span><span class=""><span class="support type">str</span></span></span><span class="punctuation">)</span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Node-Info gibt eine kleine Information der Instanz zurück, Activity eine Statistik darüber, was eigentlich so los ist. Es muss beachtet werden, dass manche Instanzen diese Abfragen nicht erlauben. Dann hagelt es Fehlermeldungen. </p>
<p dir="auto">Was abschließend noch wichtig ist. Wir haben hiermit auf öffentliche Daten der Instanz zugegriffen. Die Registrierung der Client-App (also unseres Scriptes) wurde zwar über das Web-UI (unter unserem Account) angelegt, aber wir verwendeten hier noch keinerlei Benutzerdaten. Das war für diese API Aufrufe auch nicht notwendig, weil wir keine benutzerbezogenen Aktionen ausführten. Es ist sogar möglich über den Mastodon.py Wrapper sich nur als App zu registrieren und die Informationen abzurufen. Das ist eine sehr offene Art, mit einer API umzugehen. Allerdings gibt es immer Einschränkungen. Jede Instanz kann selbst entscheiden, was abrufbar ist und auch die Häufigkeit der Abfragen ist limitiert. Fragt man zu oft einer eine Zeitperiode bestimmte Endpunkte ab, bekommt man Fehlermeldungen. Mastodon.py hat in diesem Fall verschiedene Strategien, wie es damit umgeht. Dazu aber in einem späteren Beitrag.</p>
]]><![CDATA[Einstand und Vorstellung des BeanDev Mastodon & Python Blogs]]>https://write.tchncs.de/~/BeanDevMastodon/Einstand%20und%20Vorstellung%20des%20BeanDev%20Mastodon%20&%20Python%20Blogs/2022-05-03T09:23:22.195194+00:00Aljoscha Rittnerhttps://write.tchncs.de/@/beandev/2022-05-03T09:23:22.195194+00:00<![CDATA[<p dir="auto">Vor kurzer Zeit fragte ich, ob es neben Plume noch andere bekanntere Blogsoftware (und Dienste) für das Fediverse gibt. Allerdings gibt es nicht viel, außer vielleicht WriteFreely. Sogar Plume schreibt selbst auf <a href="https://joinplu.me/" rel="noopener noreferrer">https://joinplu.me/</a>, dass die Software gerade nicht sehr aktuell ist und die Entwicklung stockt. Trotzdem erscheint mir gerade die Übersichtlichkeit der Funktionen besonders attraktiv. Zudem gibt es wohl doch ein klein wenig Entwicklungsarbeiten an Plume, da in diesem Jahr schon die Versionen 0.7.0 und 0.7.1 erschienen sind und die Integration in Mastodon verbessert.</p>
<p dir="auto">Ich brauche nicht mehr als ein wenig Markdown und etwas Syntax-Highlighting:</p>
<pre><code dir="auto"><span class="source"><span class=""><span class="">world</span></span><span class="keyword operator">=</span><span class="string"><span class="string"><span class="punctuation string">'</span></span></span><span class="string"><span class="string">Fediverse<span class="punctuation string">'</span></span></span>
<span class=""><span class=""><span class="support function">print</span> </span><span class="punctuation">(</span><span class=""><span class="storage type string">f</span><span class="string"><span class="string"><span class="punctuation string">"</span></span></span><span class="string"><span class="string">Hello </span><span class=""><span class="punctuation">{</span><span class="source"><span class=""><span class="">world</span></span></span></span><span class=""><span class="punctuation">}</span></span><span class="string"><span class="punctuation string">"</span></span></span></span><span class="punctuation">)</span></span>
</span></code></pre>
<p dir="auto">Das scheint auch gut zu funktionieren.</p>
<p dir="auto">In diesem Blog soll es etwas um Mastodon gehen. Primär aus technischer Sicht (also API, Entwicklung, Scripting usw.), aber auch News und gesellschaftliche wie soziale Themen rund ums Fediverse.</p>
<p dir="auto">Es ist auch ein Experiment, wie es sich mit Blogs im Fediverse anfühlt und wie es sich integriert. Neben dem Themenschwerpunkt Mastodon & Python, werde ich ggf. weitere Blogs mit anderen Themen aktivieren und alles mal dort sammeln, was in Toots über die Zeit verloren gehen. </p>
]]>