documentation: add comprehensive overview for Singularity #261
Conversation
charliemirabile
left a comment
charliemirabile
left a comment
There was a problem hiding this comment.
42 review comments later... I hope that this is helpful haha
Reminds me of the joke that the best way to figure out the answer to something on the internet is to confidently post an incorrect one.
|
|
||
| **Core Components:** | ||
|
|
||
| * **`extenginx`:** An Nginx container acting as the front-door reverse proxy. It handles SSL termination (HTTPS, SMTPS, POP3S) and routes requests to the appropriate backend services based on protocol and path. Uses a flexible snippet-based configuration. |
There was a problem hiding this comment.
strange that this one is in paragraph form while all the other containers have bullet points. It is the shortest so it isn't crazy, but bullet points to match would look nicer I think.
| * Applies patches to a central grading Git repository ([`mailman/patchset.py#L100`](mailman/patchset.py)), creating a commit and a unique tag (e.g., `<submission_id>`). | ||
| * Performs initial automated checks (e.g., whitespace errors, patch corruption). | ||
| * Records submission metadata in its database ([`mailman/db.py`](mailman/db.py)). | ||
| * **Writes initial automated feedback results to `git notes` associated with the submission commit (e.g., in `refs/notes/mailman`).** (Currently stores status in `Gradeable.status`). |
There was a problem hiding this comment.
I think that this is not going to happen in mailman even with the proposed move to git notes
| * **`git`:** Manages the central bare Git repository (`grading.git`) used for storing applied patches, feedback notes, and grading tags. | ||
| * Uses `cgit` for web interface (proxied via `orbit`). | ||
| * Provides scripts for repository creation and setup ([`git/create-repo.sh`](git/create-repo.sh), [`git/setup-repo.sh`](git/setup-repo.sh)). | ||
| * Handles Git pushes via `git-http-backend` ([`git/cgi-bin/git-receive-pack`](git/cgi-bin/git-receive-pack)), authenticated by `orbit` via Nginx/cgit proxy path. |
There was a problem hiding this comment.
the push endpoint is not accessible through nginx nor does it use orbit auth. The python web server that this container runs is exposed on a dynamic port on the host machine when the container comes up. The API is just fully unauthenticated http push, but you can only access it by running commands on the machine that hosts the containers (or using ssh to tunnel a socket from your machine to the port on the host where the ssh key based auth determines who has access i.e. only professors and graders).
| * **`submatrix`:** Integrates with a Synapse Matrix homeserver. | ||
| * Provides a custom password authentication provider ([`submatrix/orbit_auth.py`](submatrix/orbit_auth.py)) that delegates authentication checks to `orbit` ([`submatrix/orbit_auth.py`](submatrix/orbit_auth.py)). This allows users to log into Matrix with their Orbit credentials. | ||
| * Configured via [`submatrix/homeserver.yaml.template`](submatrix/homeserver.yaml.template). | ||
| * **`journal`:** A custom C component responsible for managing email data storage and access control using extended attributes (`xattr`) ([`journal/init_journal.c`](journal/init_journal.c)). It maintains a journal file that tracks the state of all emails in the mail directory, enforces per-user access limits, and provides utilities for initializing, updating, and restricting access to the email journal. |
There was a problem hiding this comment.
journal and smtp switch back to paragraph form instead of bullet points
There was a problem hiding this comment.
Once a the missing bullet point for pop is added, I would say that it is worth saying that journal integrates tightly with pop and smtp - they all agree on a common struct that represents an email in journal/email.h and journal exists to connect "fragment" files containing one or more email structs generated as mail is received by the smtp server to the master journal file that is accessed by the pop clients to view email.
| * Provides a custom password authentication provider ([`submatrix/orbit_auth.py`](submatrix/orbit_auth.py)) that delegates authentication checks to `orbit` ([`submatrix/orbit_auth.py`](submatrix/orbit_auth.py)). This allows users to log into Matrix with their Orbit credentials. | ||
| * Configured via [`submatrix/homeserver.yaml.template`](submatrix/homeserver.yaml.template). | ||
| * **`journal`:** A custom C component responsible for managing email data storage and access control using extended attributes (`xattr`) ([`journal/init_journal.c`](journal/init_journal.c)). It maintains a journal file that tracks the state of all emails in the mail directory, enforces per-user access limits, and provides utilities for initializing, updating, and restricting access to the email journal. | ||
| * **`smtp`:** A custom C implementation of an SMTP server ([`smtp/smtp.c`](smtp/smtp.c)), designed to securely receive assignment submissions via email. It handles authentication, message parsing, and storage of incoming emails and patchsets, integrating tightly with the `journal` component for access control and with `extenginx` for SSL termination and proxying. |
There was a problem hiding this comment.
I wouldn't say that it handles authentication per se. As configured here the credentials are forwarded by nginx and processed so that the server knows who is sending the mail, but the password is not checked anywhere in the smtp code. smtp relies on nginx (which in turn relies on orbit /mail_auth) to enforce access controls so that only authorized users are allowed
| * **Database Backup:** | ||
| * Identify volume paths: `podman volume inspect <volume_name>` (e.g., `singularity_orbit-db`, `singularity_mailman-db`, `singularity_denis-db`, `singularity_synapse-data`) | ||
| * Stop containers: `podman-compose down` (or stop individual services accessing the DB) | ||
| * Copy SQLite files from their volume mount points on the host. (Or use `backup/backup.sh` if it's designed for this). | ||
| * Restart containers: `podman-compose up -d` | ||
| * **Database Restore:** | ||
| * Stop containers: `podman-compose down` | ||
| * Replace SQLite files in volume mount points with backup files. (Or use `backup/restore.sh`). | ||
| * Restart containers: `podman-compose up -d` |
There was a problem hiding this comment.
we have scripts for this. These instructions are bad and should not be followed. You don't need to pull the containers down to take a backup.
| * Restart containers: `podman-compose up -d` | ||
| * **Adding/Managing Assignments:** | ||
| * Define assignment metadata and deadlines using `denis/configure.py` ([`denis/configure.py`](denis/configure.py)) which updates the `denis` database. | ||
| * Ensure corresponding email addresses/aliases are configured if needed (likely handled by mail server config based on `SINGULARITY_HOSTNAME`). |
There was a problem hiding this comment.
not needed, the email server accepts any recipient at all as long as the domain is appropriate. It is only the later stages of processing that become aware of things like student emails vs assignment aliases. Obviously you need to prevent a collision between assignment names and and student usernames, but that is easy to do if you just use e.g. _ for student names (john_smith) and - for assignment names (page-walk).
| * **Adding/Managing Assignments:** | ||
| * Define assignment metadata and deadlines using `denis/configure.py` ([`denis/configure.py`](denis/configure.py)) which updates the `denis` database. | ||
| * Ensure corresponding email addresses/aliases are configured if needed (likely handled by mail server config based on `SINGULARITY_HOSTNAME`). | ||
| * Schedule `denis` scripts (`initial.py`, `peer_review.py`, `final.py`) to run at appropriate due dates. |
There was a problem hiding this comment.
they are automatically scheduled when you reload denis after updating the configuration using the config.sh script.
| * Ensure corresponding email addresses/aliases are configured if needed (likely handled by mail server config based on `SINGULARITY_HOSTNAME`). | ||
| * Schedule `denis` scripts (`initial.py`, `peer_review.py`, `final.py`) to run at appropriate due dates. | ||
| * **Creating Git Repositories:** | ||
| * The central `grading.git` repository is typically created once during initial setup. Student-specific repos are not used in this workflow; submissions are applied to the central repo. |
There was a problem hiding this comment.
it is created automatically during container build. You should not need to touch it
| * **Cleaning Up Test Data:** | ||
| * After running `./test.sh`, containers/volumes might be modified. | ||
| * Stop containers: `podman-compose down` | ||
| * Remove potentially modified volumes: `podman volume prune` (removes *all* unused volumes) or `podman volume rm singularity_orbit-db singularity_mailman-db ...` (list specific volumes used by the project - see `podman volume ls`). ([`README.md`](README.md): Section 3) |
There was a problem hiding this comment.
use podman-compose down -v when bringing them down to automatically purge the volumes.
charliemirabile
force-pushed
the
master
branch
from
a073252 to
1a225dd
Compare
2 participants
No description provided.