Merge pull request #2 from YunoHost-Apps/main

information, fail2ban configured and tested

README.md

@@ -28,7 +28,7 @@ Registered users can access a simple form to publish there small ads. The small
 
 ## Federation
 
-To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account".
+To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account"._
 
 The small add visited turns out to be a _note_ in the fediverse. It's url can be opened with your favourite fediverse client at the server you're already registered to. You then can boost the small ad like any other note you read.
 
@@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
 * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
 
 
-**Shipped version:** 0.0~ynh3
+**Shipped version:** 0.0~ynh4
 
 **Demo:** <https://flohmarkt.ween.de/>
 

README_eu.md

@@ -28,7 +28,7 @@ Registered users can access a simple form to publish there small ads. The small
 
 ## Federation
 
-To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account".
+To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account"._
 
 The small add visited turns out to be a _note_ in the fediverse. It's url can be opened with your favourite fediverse client at the server you're already registered to. You then can boost the small ad like any other note you read.
 
@@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
 * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
 
 
-**Paketatutako bertsioa:** 0.0~ynh3
+**Paketatutako bertsioa:** 0.0~ynh4
 
 **Demoa:** <https://flohmarkt.ween.de/>
 

README_fr.md

@@ -28,7 +28,7 @@ Registered users can access a simple form to publish there small ads. The small
 
 ## Federation
 
-To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account".
+To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account"._
 
 The small add visited turns out to be a _note_ in the fediverse. It's url can be opened with your favourite fediverse client at the server you're already registered to. You then can boost the small ad like any other note you read.
 
@@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
 * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
 
 
-**Version incluse :** 0.0~ynh3
+**Version incluse :** 0.0~ynh4
 
 **Démo :** <https://flohmarkt.ween.de/>
 

README_gl.md

@@ -28,7 +28,7 @@ Registered users can access a simple form to publish there small ads. The small
 
 ## Federation
 
-To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account".
+To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account"._
 
 The small add visited turns out to be a _note_ in the fediverse. It's url can be opened with your favourite fediverse client at the server you're already registered to. You then can boost the small ad like any other note you read.
 
@@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
 * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
 
 
-**Versión proporcionada:** 0.0~ynh3
+**Versión proporcionada:** 0.0~ynh4
 
 **Demo:** <https://flohmarkt.ween.de/>
 

README_zh_Hans.md

@@ -28,7 +28,7 @@ Registered users can access a simple form to publish there small ads. The small
 
 ## Federation
 
-To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account".
+To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account"._
 
 The small add visited turns out to be a _note_ in the fediverse. It's url can be opened with your favourite fediverse client at the server you're already registered to. You then can boost the small ad like any other note you read.
 
@@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
 * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
 
 
-**分发版本：** 0.0~ynh3
+**分发版本：** 0.0~ynh4
 
 **演示：** <https://flohmarkt.ween.de/>
 

conf/systemd.service

@@ -8,7 +8,7 @@ User=__APP__
 Group=__APP__
 WorkingDirectory=__INSTALL_DIR__/__APP__
 Environment="VENV_DIR=__INSTALL_DIR__/venv/"
-ExecStart=__INSTALL_DIR__/venv/bin/uvicorn --host 127.0.0.1 --port __PORT__ --reload flohmarkt.web:start
+ExecStart=/bin/bash -c "/opt/flohmarkt/venv/bin/uvicorn --host 127.0.0.1 --port 8000 --reload flohmarkt.web:start  2>&1 | /usr/bin/ts '%%Y-%%m-%%d %%H:%%M:%%S'"
 StandardOutput=append:/var/log/__APP__/__APP__.log
 StandardError=inherit
 

doc/DESCRIPTION.md

@@ -10,7 +10,7 @@ Registered users can access a simple form to publish there small ads. The small
 
 ## Federation
 
-To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account".
+To communicate with someone who published a small ad the server hints to an unregistred user _"To answer this offer please log in or create an account. OR use another fediverse-account"._
 
 The small add visited turns out to be a _note_ in the fediverse. It's url can be opened with your favourite fediverse client at the server you're already registered to. You then can boost the small ad like any other note you read.
 

doc/DEVELOPMENT.md

@@ -28,3 +28,26 @@ Once in a while we'll reach the point to tag a new `-ynhX` version for changes o
 To do so we'll push the according changes to the [flohmarkt repository at github](https://github.com/YunoHost-Apps/flohmarkt_ynh) to make upgrades available to the yunohost community.
 
 One **pitfall** doing so is that we can't rely on the yunohost CI for testing for our codeberg repository this way. If need'll be and developers would like to use yunohost as their base for active work on flohmarkt we might release another app **flohmarkt-devel_ynh** in future that closely follows the repository we use for development.
+
+#### master and main
+
+The development startet on codeberg on a branch named **main**. That worked all the way until the repository got mirrored to github to be included into the yunohost app catalog. The last change that needed to be done for flohmarkt to appear on the catalog has been to change the name of the branch to **master** for the workflows on github to recognize it for catalog inclusion.
+
+Short: for **historic** reason we use the branch **main** on codeberg and publish the versions for the catalog as **master** on github.
+
+### pushing to github
+
+* make sure the local git and the codeberg git are in sync on their main branch
+* tag a new version on codeberg `<major>.<minor>-ynh<X>`, e.g. `0.01-ynh5`. `<major>.<minor>` is the flohmarkt version. `ynhX` is the version of the integration into flohmarkt (this repo).
+  * new flohmarkt version: only `manifest.toml` changed to point to the newer source archive
+    → change `<major>.<minor>-ynh<X>` according to the new flohmarkt version
+  * changes in integration: scripts, conf files or `doc/*` changed
+    → increment `<X>` to signal a new version of the yunohost integration
+* try to push the local main branch to github which might fail
+  * there might for some reason exist an old main branch that had not been deleted after the PR to the github master branch - check carefully and delete the existing main branch
+* on github open an PR from the main branch into the master branch
+  * the PR can be tested on the CI workflow if a comment containing `!testme` is added to the PR
+* the PR will be included after
+  * it successfully ran through the CI workflow (results will show up inside the PR)
+  * it has been reviewed 
+

doc/PRE_INSTALL.md

@@ -1,16 +1,19 @@
-### Exclusive use of couchdb 
+## Warning: Exclusive use of couchdb 
 
 flohmarkt expects to install CouchDB from the Apache repository for its own, exclusive use. Installation might break already existing installs of CouchDB.
 
+**flohmarkt will probably not install if you're already running an instance of a couchdb**
+
+https://codeberg.org/flohmarkt/flohmarkt_ynh/src/commit/7721103bac61787f31a4b2f2ae695c65d4f26fc9/scripts/install#L9
 https://codeberg.org/ChriChri/flohmarkt_ynh/issues/9
 
-### Exclusive use of (sub)domain 
+## Exclusive use of (sub)domain 
 
 flohmarkt expects to bei installed on its own (sub)domain.
 
 https://codeberg.org/ChriChri/flohmarkt_ynh/issues/4 .
 
-### No integration in yunohost user database"
+## No integration in yunohost user database"
 
 flohmarkt mainanins its own user database in CouchDB. Users have to register to flohmarkt to get an account. Registration cannot be restricted to yunohost users.
 
@@ -18,17 +21,17 @@ https://codeberg.org/ChriChri/flohmarkt_ynh/issues/5 .
 
 ## removing after installation
 
-**Warning:** This will probably break any existing installation of couchdb (there's an couchdb app to install just couchdb and expose its port via nginx reverse-proxy).
+**Warning:** This might break any existing installation of couchdb (there's an couchdb app to install just couchdb and expose its port via nginx reverse-proxy and possibly other software installing a couchdb). This could happen if you installed the couchdb app after you installed flohmarkt.
 
-## read before test installation
+https://codeberg.org/flohmarkt/flohmarkt_ynh/src/commit/7721103bac61787f31a4b2f2ae695c65d4f26fc9/scripts/remove#L44
 
-**Another warning:** When installing flohmarkt on a a domain and letting it talk to other ActivityPub instances it will propagate a key associated to your domain. If you remove your flohmarkt from that domain and loose that key other instances might not want to talk to you anymore after you installed flohmarkt again on the same domain generating a new key.
+When installing flohmarkt on a a domain and letting it talk to other ActivityPub instances it will propagate a key associated to your domain. If you remove your flohmarkt from that domain and loose that key other instances might not want to talk to you anymore after you installed flohmarkt again on the same domain generating a new key.
 
-**This is really strictly for testing only - don't install on a production yunohost.**
+## list of instances
 
-## Go ahead…
+We apreciate a lot if you run an instance of flohmarkt and will publish your [instance on the wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki/flohmarkt-instances) if you [open an issue](https://codeberg.org/flohmarkt/flohmarkt/issues) asking us to do so.
 
-…test, break stuff and open issues on codeberg :) !
+If you're looking for another instance to federate with the list on the wiki is a good starting point also.
 
 # help welcome
 
@@ -36,3 +39,4 @@ You're welcome to take part by opening issues or sending pull requests. You can
 
 I also announced this work on the [yunohost forum](https://forum.yunohost.org/t/ynh-flohmarkt-flohmarkt-as-an-app-for-yunohost/28455?u=chrichri).
 
+Look at [DEVELOPMENT.md](doc/DEVELOPMENT.md) for more information.

logger.json

@@ -0,0 +1,27 @@
+{
+  "version": 1,
+  "disable_existing_loggers": false,
+  "formatters": {
+    "json": {
+      "()": "pythonjsonlogger.jsonlogger.JsonFormatter",
+      "fmt": "%(asctime)s %(name)s %(levelname)s %(message)s"
+    }
+  },
+  "handlers": {
+    "stderr": {
+      "formatter": "json",
+      "class": "logging.StreamHandler",
+      "stream": "ext://sys.stderr"
+    },
+    "stdout": {
+      "formatter": "json",
+      "class": "logging.StreamHandler",
+      "stream": "ext://sys.stdout"
+    }
+  },
+  "loggers": {
+    "uvicorn": {"handlers": ["stderr"], "level": "INFO", "propagate": false},
+    "uvicorn.error": {"level": "INFO"},
+    "uvicorn.access": {"handlers": ["stdout"], "level": "INFO", "propagate": false}
+  }
+}

manifest.toml

@@ -6,7 +6,7 @@ id = "flohmarkt"
 name = "flohmarkt"
 description.en = "A decentral federated small ads platform"
 
-version = "0.0~ynh3"
+version = "0.0~ynh4"
 
 maintainers = ["Chris Vogel"]
 
@@ -156,7 +156,8 @@ ram.runtime = "100M"
 
     [resources.apt]
     # python dependencies shall be installed in a venv using pip.
-    packages = "python3-pip python3-full curl apt-transport-https gnupg"
+	# moreutils is needed for `ts` used in systemd.service
+    packages = "python3-pip python3-full curl apt-transport-https gnupg moreutils"
     # repo for couchdb - doesn't work, yet
     # extras.couchdb.repo = "deb https://apache.jfrog.io/artifactory/couchdb-deb/ __YNH_DEBIAN_VERSION__ main"
     # extras.couchdb.key = "https://couchdb.apache.org/repo/keys.asc"

run-uvicorn.example

@@ -0,0 +1,55 @@
+/opt/flohmarkt/venv/bin/uvicorn --host 127.0.0.1 --port 8000 --reload --log-config logger.json flohmarkt.web:start
+
+Fehler:
+
+Traceback (most recent call last):
+  File "/usr/lib/python3.9/logging/config.py", line 385, in resolve
+    found = self.importer(used)
+ModuleNotFoundError: No module named 'pythonjsonlogger'
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "/usr/lib/python3.9/logging/config.py", line 543, in configure
+    formatters[name] = self.configure_formatter(
+  File "/usr/lib/python3.9/logging/config.py", line 655, in configure_formatter
+    result = self.configure_custom(config)
+  File "/usr/lib/python3.9/logging/config.py", line 470, in configure_custom
+    c = self.resolve(c)
+  File "/usr/lib/python3.9/logging/config.py", line 398, in resolve
+    raise v
+  File "/usr/lib/python3.9/logging/config.py", line 385, in resolve
+    found = self.importer(used)
+ValueError: Cannot resolve 'pythonjsonlogger.jsonlogger.JsonFormatter': No module named 'pythonjsonlogger'
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "/opt/flohmarkt/venv/bin/uvicorn", line 8, in <module>
+    sys.exit(main())
+  File "/opt/flohmarkt/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
+    return self.main(*args, **kwargs)
+  File "/opt/flohmarkt/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
+    rv = self.invoke(ctx)
+  File "/opt/flohmarkt/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
+    return ctx.invoke(self.callback, **ctx.params)
+  File "/opt/flohmarkt/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
+    return __callback(*args, **kwargs)
+  File "/opt/flohmarkt/venv/lib/python3.9/site-packages/uvicorn/main.py", line 404, in main
+    run(
+  File "/opt/flohmarkt/venv/lib/python3.9/site-packages/uvicorn/main.py", line 506, in run
+    config = Config(
+  File "/opt/flohmarkt/venv/lib/python3.9/site-packages/uvicorn/config.py", line 299, in __init__
+    self.configure_logging()
+  File "/opt/flohmarkt/venv/lib/python3.9/site-packages/uvicorn/config.py", line 411, in configure_logging
+    logging.config.dictConfig(loaded_config)
+  File "/usr/lib/python3.9/logging/config.py", line 809, in dictConfig
+    dictConfigClass(config).configure()
+  File "/usr/lib/python3.9/logging/config.py", line 546, in configure
+    raise ValueError('Unable to configure '
+ValueError: Unable to configure formatter 'json'
+
+(venv) root@yt:/opt/flohmarkt/flohmarkt# pip install python-json-logger
+Requirement already satisfied: python-json-logger in /usr/lib/python3.9/site-packages (2.0.7)
+
+Ausgabe in json will ich eigentlich auch nicht falls das in der config steht. Der Kram ist aber so dokumentiert, dass ich ihn nicht verstehe...

scripts/install

@@ -122,10 +122,7 @@ chmod 640 "/var/log/$app/$app.log"
 ynh_systemd_action --service_name=$app --action="start"
 
 # SETUP FAIL2BAN
-# no need for couchdb, because it will not listen externally
-# ynh_script_progression --message="Configuring Fail2Ban..." --weight=3
-# 
-# # Create a dedicated Fail2Ban config
-# ynh_add_fail2ban_config --logpath="/var/log/couchdb/couchdb.log" --failregex="[warning] .*couch_httpd_auth: Authentication failed for user .+ from <HOST>" --max_retry=5
+ynh_script_progression --message="Configuring Fail2Ban..." --weight=3
+ynh_add_fail2ban_config --logpath="/var/log/$app/$app.log" --failregex='INFO: +<HOST>:\d+ - "POST /token HTTP/\d+\.\d+" 403 Forbidden' --max_retry=5
 
 ynh_script_progression --message="Installation of $app completed" --last

scripts/upgrade

@@ -93,9 +93,8 @@ ynh_script_progression --message="Upgrading NGINX web server configuration..." -
 ynh_add_nginx_config
 
 # UPGRADE FAIL2BAN
-# ynh_script_progression --message="Reconfiguring Fail2Ban..." --weight=3
-# Create a dedicated Fail2Ban config
-# ynh_add_fail2ban_config --logpath="/var/log/couchdb/couchdb.log" --failregex="[warning] .*couch_httpd_auth: Authentication failed for user .+ from <HOST>" --max_retry=5
+ynh_script_progression --message="Reconfiguring Fail2Ban..." --weight=3
+ynh_add_fail2ban_config --logpath="/var/log/$app/$app.log" --failregex='INFO: +<HOST>:\d+ - "POST /token HTTP/\d+\.\d+" 403 Forbidden' --max_retry=5
 
 # start flohmarkt
 ynh_script_progression --message="Starting flohmarkt..." --weight=3