You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-**Why:** The standard, feature-rich combination for modern Python testing, offering excellent DX for writing tests and robust, standard coverage reporting. ([Details](topics/06_testing-coverage.md))
-**Description:** {unittest}`unittest<>` is Python's built-in testing framework, inspired by JUnit. Tests are written in classes inheriting from `unittest.TestCase`. {coveragepy}`coverage.py<>` is the standard standalone tool for measuring code coverage.
31
+
-**Description:** {unittest}`unittest<>` is Python's built-in testing framework, inspired by JUnit. Tests are written in classes inheriting from `unittest.TestCase`. {coverage-py}`coverage.py<>` is the standard standalone tool for measuring code coverage.
32
32
-**Evaluation:**
33
33
34
34
-**Ease of Use:** Moderate. Requires significant boilerplate (class definitions, inheritance, specific method names, explicit `setUp`/`tearDown` methods). Writing simple tests is more verbose than alternatives.
35
35
-**Feature-Rich:** Moderate. Provides core testing features but lacks the advanced features and extensive plugin ecosystem of {pytest}`pytest<>` (e.g., simple functional fixtures, powerful parametrization decorators built-in).
36
-
-**Performance:** Moderate. Test execution can be slower than {pytest}`pytest<>` for large test suites due to its architecture (creating a class instance per test method). {coveragepy}`coverage.py<>` performance is generally good.
37
-
-**OS Interoperability:** Excellent. Both are foundational Python tools, highly robust across OSs. {unittest}`unittest<>` is standard library, {coveragepy}`coverage.py<>` is pure Python.
36
+
-**Performance:** Moderate. Test execution can be slower than {pytest}`pytest<>` for large test suites due to its architecture (creating a class instance per test method). {coverage-py}`coverage.py<>` performance is generally good.
37
+
-**OS Interoperability:** Excellent. Both are foundational Python tools, highly robust across OSs. {unittest}`unittest<>` is standard library, {coverage-py}`coverage.py<>` is pure Python.
38
38
-**Integration:** High (Individual). Both have CLIs easily called from Task Automation/CI. Integrating them _together_ requires explicitly wrapping `unittest` execution with `coverage run -m unittest` or using less standardized plugins compared to the {pytest-pytest-cov}`pytest<>` ecosystem. Generating standard reports like JUnit XML also often requires extra steps or third-party runners for {unittest}`unittest<>`.
-**Maturity & Stability:** Very High. Both are extremely mature, stable, battle-tested.
41
41
-**Community & Documentation:** Very High. Widely adopted, vast documentation.
42
42
@@ -53,13 +53,13 @@ We evaluated the primary testing framework and coverage tools:
53
53
-**OS Interoperability:** Excellent. Pure Python package, works reliably across OSs.
54
54
-**Integration:** Excellent. Widely supported, integrates seamlessly into editors/IDEs, {pre-commit}`pre-commit<>`, Task Automation, CI/CD. Designed for external execution via CLI.
55
55
-**Reporting:** Excellent. Provides clear terminal output. Standard support for generating JUnit XML reports (`--junitxml=...`), which is essential for CI platform integration.
56
-
-**Coverage Reporting:** Poor (Not built-in). Requires an external tool (like {coveragepy}`coverage.py<>`) and integration mechanism.
56
+
-**Coverage Reporting:** Poor (Not built-in). Requires an external tool (like {coverage-py}`coverage.py<>`) and integration mechanism.
57
57
-**Maturity & Stability:** Very High. Mature, stable, widely adopted standard for modern Python testing.
58
58
-**Community & Documentation:** Very High. Massive, active community, extensive documentation.
59
59
60
60
-**Conclusion:** The de facto standard for modern Python testing. Excels at ease of use, features, performance, and integration for the testing framework itself. Lacks built-in coverage, requiring integration with another tool.
61
61
62
-
### Option 3: {coveragepy}`coverage.py<>`
62
+
### Option 3: {coverage-py}`coverage.py<>`
63
63
64
64
-**Description:** The standard standalone tool for measuring code coverage in Python. Monitors code execution and reports on lines/branches executed.
65
65
-**Evaluation:** (Evaluated primarily as an engine, its integration is key).
@@ -76,13 +76,13 @@ We evaluated the primary testing framework and coverage tools:
76
76
77
77
### Option 4: {pytest-pytest-cov}`pytest-cov<>`
78
78
79
-
-**Description:** The official {pytest}`pytest<>` plugin that integrates {coveragepy}`coverage.py<>` seamlessly into the {pytest}`pytest<>` workflow.
79
+
-**Description:** The official {pytest}`pytest<>` plugin that integrates {coverage-py}`coverage.py<>` seamlessly into the {pytest}`pytest<>` workflow.
80
80
-**Evaluation:** (Evaluated as the integration bridge).
81
81
82
-
-**Integration with Testing & Coverage:** Excellent. Provides seamless, standard integration by adding `--cov` flags to the `pytest` command. Orchestrates running {coveragepy}`coverage.py<>` around the {pytest-pytest-cov}`pytest<>` run.
83
-
-**Accurate & Detailed Reporting:** Excellent. Leverages {coveragepy}`coverage.py<>`'s full reporting capabilities via {pytest-pytest-cov}`pytest<>` command-line arguments and config files.
84
-
-**Performance:** High (Combined). Adds minimal overhead; combined performance is driven by {pytest-pytest-cov}`pytest<>` and {coveragepy-coverage}`coverage.py<>` execution.
85
-
-**OS Interoperability:** Excellent. Pure Python plugin, inherits compatibility from {pytest-pytest-cov}`pytest<>` and {coveragepy-coverage}`coverage.py<>`.
82
+
-**Integration with Testing & Coverage:** Excellent. Provides seamless, standard integration by adding `--cov` flags to the `pytest` command. Orchestrates running {coverage-py}`coverage.py<>` around the {pytest-pytest-cov}`pytest<>` run.
83
+
-**Accurate & Detailed Reporting:** Excellent. Leverages {coverage-py}`coverage.py<>`'s full reporting capabilities via {pytest-pytest-cov}`pytest<>` command-line arguments and config files.
84
+
-**Performance:** High (Combined). Adds minimal overhead; combined performance is driven by {pytest-pytest-cov}`pytest<>` and {coverage.py}`coverage.py<>` execution.
85
+
-**OS Interoperability:** Excellent. Pure Python plugin, inherits compatibility from {pytest-pytest-cov}`pytest<>` and {coverage.py}`coverage.py<>`.
86
86
-**Callable for Workflow:** Excellent. Simply adds flags to the standard `pytest` command, easily used in Task Automation and CI.
87
87
-**Maturity & Stability:** Very High. The standard, mature, and stable plugin for {pytest-pytest-cov}`pytest<>` coverage integration.
88
88
-**Community & Documentation:** Very High. Essential part of the {pytest-pytest-cov}`pytest<>` ecosystem.
@@ -104,27 +104,27 @@ We evaluated the primary testing framework and coverage tools:
- Matrix Orchestration (for full matrix): **{nox}`Nox<>`** (invoking {pytest-pytest-cov}`pytest<>` across matrix) or optionally **{tox}`Tox<>`** (invoked by {nox}`Nox<>` for specific needs).
110
110
111
111
## Justification for the Choice
112
112
113
-
The combination of **{pytest}`pytest<>`**, **{coveragepy-coverage}`coverage.py<>`**, and **{pytest-pytest-cov}`pytest-cov<>`** is the best fit for providing robust testing and coverage capabilities in this template, complemented by **{nox}`Nox<>`** for matrix execution:
113
+
The combination of **{pytest}`pytest<>`**, **{coverage.py}`coverage.py<>`**, and **{pytest-pytest-cov}`pytest-cov<>`** is the best fit for providing robust testing and coverage capabilities in this template, complemented by **{nox}`Nox<>`** for matrix execution:
114
114
115
115
1.**Developer Experience:** {pytest-pytest-cov}`pytest<>` offers significantly **easier test writing and organization** compared to {unittest}`unittest<>`, with powerful features like fixtures and parametrization that improve test maintainability and expressiveness (addressing **Ease of Use** and **Feature-Rich**). This aligns with the **"Obvious way to do it"** for writing tests.
116
-
2.**Standards and Integration:** {pytest-pytest-cov}`pytest<>` is the de facto standard modern Python testing framework, and {coveragepy-coverage}`coverage.py<>` is the universal coverage engine. **{pytest-pytest-cov}`pytest-cov<>`** provides **seamless, standard integration** between them via a simple command-line flag (`--cov`), making combined testing and coverage easy to run and automate (addressing **Integration**).
117
-
3.**Reporting:** This combination provides **excellent standard reporting**, including JUnit XML from {pytest-pytest-cov}`pytest<>` and Cobertura XML/HTML from {coveragepy-coverage}`coverage.py<>`, which are essential for integration into CI/CD platforms (Area 13, 14) (addressing **Reporting**).
116
+
2.**Standards and Integration:** {pytest-pytest-cov}`pytest<>` is the de facto standard modern Python testing framework, and {coverage.py}`coverage.py<>` is the universal coverage engine. **{pytest-pytest-cov}`pytest-cov<>`** provides **seamless, standard integration** between them via a simple command-line flag (`--cov`), making combined testing and coverage easy to run and automate (addressing **Integration**).
117
+
3.**Reporting:** This combination provides **excellent standard reporting**, including JUnit XML from {pytest-pytest-cov}`pytest<>` and Cobertura XML/HTML from {coverage.py}`coverage.py<>`, which are essential for integration into CI/CD platforms (Area 13, 14) (addressing **Reporting**).
118
118
4.**Performance & OS Interoperability:** All chosen tools are **performant** for their tasks and **highly OS-interoperable**, working reliably across development and CI environments (addressing **Performance** and **OS Interoperability**).
119
119
5.**Matrix Testing:** While {pytest-pytest-cov}`pytest<>` itself isn't a matrix orchestrator, **{nox}`Nox<>`** (Area 12) is explicitly designed to run sessions (like our test session) across different Python versions and environments using `uv`, effectively providing the necessary matrix testing capability within the template's primary automation layer. For complex scenarios or community conventions, {nox}`Nox<>` can easily **invoke {tox}`Tox<>`**.
120
120
121
121
{unittest}`unittest<>` was discounted due to its comparative verbosity, lack of features, and less streamlined integration for testing+coverage. {tox}`Tox<>` is better suited as a matrix _runner_ called by {nox}`Nox<>` than the primary testing _framework_ itself.
122
122
123
-
By choosing this combination, the template leverages the strengths of each tool – {pytest-pytest-cov}`pytest<>` for writing tests, {coveragepy-coverage}`coverage.py<>` for coverage, {pytest-pytest-cov}`pytest-cov<>` for integration, and {nox}`Nox<>` for orchestration – to provide a robust, modern, and well-integrated testing and coverage solution.
123
+
By choosing this combination, the template leverages the strengths of each tool – {pytest-pytest-cov}`pytest<>` for writing tests, {coverage.py}`coverage.py<>` for coverage, {pytest-pytest-cov}`pytest-cov<>` for integration, and {nox}`Nox<>` for orchestration – to provide a robust, modern, and well-integrated testing and coverage solution.
124
124
125
125
## Interactions with Other Topics
126
126
127
-
-**pyproject.toml (01):** {pytest-pytest-cov}`pytest<>` and {coveragepy-coverage}`coverage.py<>` are configured via `pyproject.toml` (`[tool.pytest]`, `[tool.coverage]`) or separate config files (`.coveragerc`). Testing dependencies are managed via {uv}`uv<>` (Area 02).
127
+
-**pyproject.toml (01):** {pytest-pytest-cov}`pytest<>` and {coverage.py}`coverage.py<>` are configured via `pyproject.toml` (`[tool.pytest]`, `[tool.coverage]`) or separate config files (`.coveragerc`). Testing dependencies are managed via {uv}`uv<>` (Area 02).
128
128
-**Task Automation (12):** {nox}`Nox<>` sessions are defined to run the test suite (`uv run pytest --cov...`). This session is run across the matrix of Python versions defined in the `noxfile.py`. {nox}`Nox<>` also orchestrates {tox}`Tox<>` if needed.
129
129
-**CI Orchestration (13):** The CI pipeline runs the test sessions defined in {nox}`Nox<>` (`nox -s test`), leveraging the CI platform's matrix capabilities or relying on Nox's internal matrixing (`-p` flag). Test reports (JUnit XML) and coverage reports (Cobertura XML) are artifacts collected by CI.
130
-
-**Dev Containers (17):** {pytest-pytest-cov}`pytest<>`, {coveragepy-coverage}`coverage.py<>`, {pytest-pytest-cov}`pytest-cov<>` are installed and used within the development container for local testing.
130
+
-**Dev Containers (17):** {pytest-pytest-cov}`pytest<>`, {coverage.py}`coverage.py<>`, {pytest-pytest-cov}`pytest-cov<>` are installed and used within the development container for local testing.
Copy file name to clipboardExpand all lines: docs/topics/13_ci-orchestration.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,7 +46,7 @@ We focus on the approach of using standard CI platforms to orchestrate the templ
46
46
-**Checkout:** Standard step to get code.
47
47
-**Environment Setup:** Uses platform actions/steps to set up required Python versions (e.g., `actions/setup-python` on GitHub Actions) and cache dependencies efficiently, which is better handled by the platform than trying to manage these complex caching strategies manually in a simple Task Automation script.
48
48
-**Matrix Testing:** Combines platform matrix capabilities (OS + Python versions) with Nox's ability to run across multiple Python versions (using the Nox `python=` parameter) or specifically configured sessions to cover testing requirements reliably across combinations.
49
-
-**Reporting:** Leverages the platform's ability to collect standard reports (JUnit XML from {pytest}`pytest<>`, Cobertura XML from {coveragepy}`coverage.py<>`) generated by the Task Automation layer.
49
+
-**Reporting:** Leverages the platform's ability to collect standard reports (JUnit XML from {pytest}`pytest<>`, Cobertura XML from {coverage-py}`coverage.py<>`) generated by the Task Automation layer.
50
50
-**Status Checks:** The platform provides visual feedback (pass/fail) linked to commits/PRs based on job outcomes.
51
51
-**Adaptability:** Excellent. Switching CI platforms involves mapping the checkout, Python setup, caching, secrets, and artifact steps from the old platform to the new one, and then configuring the new platform to call the _same_`nox -s <task>` commands as before. The core Task Automation logic (`noxfile.py`) remains unchanged.
52
52
-**Value of Examples:** Very High. Providing concrete examples for popular platforms (like {github-actions}`GitHub Actions<>`) significantly speeds up user adoption of this strategy by providing a ready-to-use template configuration, demonstrating exactly how to integrate Task Automation calls, setup, matrixing, and reporting.
0 commit comments