hexdev/hydrogen
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
hexdev
2026-07-11 23:54:36 +07:00
commit b07562921b
40 changed files with 3813 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

606
HYDROGEN_CONFIGURATION.md Normal file
View File

@@ -0,0 +1,606 @@
# Hydrogen Configuration Guide
This document explains how configuration works in Hydrogen, which settings are actually used at runtime, and how configuration connects to module loading, plugin discovery, and report delivery.
## Configuration Entry Points
Hydrogen starts in `main.py`.
- `--config` (`-c`) defaults to `config.yaml` in the current working directory.
- `--path` (`-p`) defaults to the `modules/` directory in the current working directory.
- `--package-prefix` defaults to `modules`.
The configuration file is loaded by `config.py`:
```python
def load_config(path: Path) -> config.Config:
with path.open(encoding="utf-8") as f:
content = yaml.safe_load(f)
return config.Config.model_validate(content)
```
Hydrogen validates the YAML against the Pydantic model `core.schemas.config.Config` before the audit starts.
## Full Configuration Shape
Hydrogen currently expects this top-level structure:
```yaml
exclude_categories: []
fail_fast: false
dry_run: false
max_concurrency: 4
logging:
level: INFO
output: stdout
plugin_packages:
renderers:
- reporting.exporters
transports:
- reporting.transports
modules: []
strict_mode: true
allow_failures_below: medium
reports:
outputs:
- renderer:
type: json
transport:
type: file
path: reports/latest
append_extension: true
modules:
ssh:
enabled: true
```
The schema is defined in `core/schemas/config.py`.
## Top-Level Fields
### `exclude_categories`
Type: `list[str]`
Default: `[]`
This field is actively used in `core/module_loader.py`.
Hydrogen compares each module manifest's `category` value against this list:
```python
if module.manifest.category in config.exclude_categories:
logger.info("module %s was skipped due to config exclusion", module.manifest.identifier)
continue
```
Important details:
- Exclusion is based on `manifest.category`, not on the directory name.
- Exclusion happens before `build_worker()` is called.
- If multiple modules share the same category, one category entry disables all of them.
Example:
```yaml
exclude_categories:
- ssh
- tls
```
### `strict_mode`
Type: `bool`
This field is used in `core/evaluation.py` to determine the final process exit code.
Behavior:
- If a finding reaches the configured severity threshold and `strict_mode` is `true`, Hydrogen returns exit code `1`.
- Otherwise, Hydrogen returns exit code `0`.
Relevant code:
```python
if threshold_crossed and config.strict_mode:
return 1
return 0
```
This is the main switch controlling whether Hydrogen behaves as a CI gate.
### `allow_failures_below`
Type: `AuditSeverity`
Allowed values:
- `low`
- `medium`
- `high`
- `critical`
This field is also used in `core/evaluation.py`.
Hydrogen maps severities to numeric weights:
- `low` -> `1`
- `medium` -> `2`
- `high` -> `3`
- `critical` -> `4`
Then it fails the run when any finding has a weight greater than or equal to the configured threshold.
How to read the field name correctly:
- `allow_failures_below: medium` means Hydrogen tolerates only findings below `medium`.
- In practice, `medium`, `high`, and `critical` findings can trigger exit code `1` when `strict_mode: true`.
- Only `low` findings are below that threshold.
Examples:
```yaml
allow_failures_below: high
```
Meaning:
- `low` and `medium` findings are tolerated.
- `high` and `critical` findings can fail the run when `strict_mode` is enabled.
```yaml
allow_failures_below: critical
```
Meaning:
- Only `critical` findings can fail the run.
### `fail_fast`
Type: `bool`
Default: `false`
When `true`, Hydrogen stops submitting new worker tasks as soon as any module returns a `fail` status. Already-running workers are allowed to finish.
This is used in `core/runner.py`:
```python
if config.fail_fast and result.result.status is AuditStatus.FAIL:
stop_submitting = True
```
### `dry_run`
Type: `bool`
Default: `false`
When `true`, Hydrogen runs all security modules and evaluates exit codes but skips report publishing. Useful for testing module behavior without side effects.
```python
if resolved_config.dry_run:
logger.info("dry_run enabled, skipping report publish")
```
### `max_concurrency`
Type: `int`
Default: `4`
Controls the maximum number of worker threads in the `ThreadPoolExecutor`. Hydrogen runs module workers concurrently up to this limit.
### `logging`
Type: object
Controls logging behavior.
```yaml
logging:
level: INFO
output: stdout
```
Fields:
- `level`: any standard Python logging level name (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`).
- `output`: `stdout` or `stderr`.
### `plugin_packages`
Type: `PluginPackagesConfig`
This section controls where Hydrogen discovers renderers, transports, and modules.
Schema:
```yaml
plugin_packages:
renderers:
- reporting.exporters
transports:
- reporting.transports
modules:
- mycompany.hydrogen_modules
```
Fields:
- `renderers`: list of Python package names to scan for `BaseRenderer` subclasses.
- `transports`: list of Python package names to scan for `BaseTransport` subclasses.
- `modules`: list of Python package names to scan for modules (in addition to filesystem discovery).
Hydrogen uses `reporting/bootstrap.py` and `core/plugin_types.py` to discover plugins:
```python
def discover_plugins(group_config: PluginGroupConfig, base_class: type[TPlugin]):
for package_name in group_config.package_names:
for plugin_class in _discover_plugin_classes(package_name, base_class):
descriptors.append(PluginDescriptor(plugin=plugin_class(), source=...))
```
Additionally, Hydrogen discovers plugins via Python `entry_points` under the following groups:
- `hydrogen.renderers` for renderers.
- `hydrogen.transports` for transports.
- `hydrogen.modules` for modules.
This allows third-party packages installed via pip to register plugins without manual configuration.
### `reports`
Type: `ReportsConfig`
This section controls report rendering and delivery. Unlike the old single-output schema, Hydrogen now supports **multiple report outputs**.
Schema:
```yaml
reports:
outputs:
- renderer:
type: json
transport:
type: file
path: reports/audit
append_extension: true
- renderer:
type: json
transport:
type: webhook
url: https://collector.example.com/hydrogen
method: POST
payload_mode: envelope
```
Each output contains:
- `renderer`: a `PluginConfigRef` with a `type` field and optional extra fields.
- `transport`: a `PluginConfigRef` with a `type` field and optional extra fields.
Hydrogen uses it in `reporting/service.py`:
```python
def publish_many(self, report: AuditReport, outputs: list[ResolvedReportOutput]) -> list[str]:
return [self.publish(report, output) for output in outputs]
```
The `PluginConfigRef` model:
```python
class PluginConfigRef(BaseModel):
model_config = ConfigDict(extra="allow")
type: str = Field(min_length=1)
def payload(self) -> dict[str, Any]:
return dict(self.model_extra or {})
```
This means:
- The `type` field selects which renderer/transport is used.
- All other fields (the "payload") are passed to the plugin's `config_model` for validation.
- Hydrogen validates the payload using each plugin's own `config_model` Pydantic model.
### `modules`
Type: `dict[str, dict[str, Any]]`
Default: `{}`
This section stores per-module configuration.
Hydrogen looks up module settings by manifest identifier and validates them against the module's `CONFIG_MODEL`:
```python
module_config = config.modules.get(module.manifest.identifier, {})
validated_config = module.config_model.model_validate(module_config)
worker = build_worker(module, validated_config)
```
Important implications:
- The key must match `MANIFEST.identifier`, not the human-readable module name.
- If the key is missing, your module receives an empty dict, which is then validated by the module's `config_model`.
- Module configuration is validated at runtime by the module's own Pydantic config model.
- If validation fails, Hydrogen reports a `PluginRuntimeError` and continues with other modules.
Example:
```yaml
modules:
ssh:
enabled: true
test-failure: false
```
## Report Output Configuration
Each output in `reports.outputs` defines one renderer + transport pair. Hydrogen sends the rendered report through each transport.
### Renderer Config Ref
```yaml
outputs:
- renderer:
type: json
```
Fields:
- `type`: must match a registered renderer's `content_type` or one of its `aliases`.
Extra fields are passed to the renderer's `config_model` for validation.
### Transport Config Ref
```yaml
outputs:
- transport:
type: file
path: reports/security-report
append_extension: true
```
Fields:
- `type`: must match a registered transport's `transport_type`.
- Additional fields are plugin-specific (e.g., `path`, `url`, `method`, `headers`).
Extra fields are validated against the transport's `config_model`.
### Example: File Transport
```yaml
outputs:
- renderer:
type: json
transport:
type: file
path: reports/security-report
append_extension: true
```
Fields:
- `type`: must be `file`
- `path`: output file path
- `append_extension`: when `true`, Hydrogen appends the renderer's extension if the path has no suffix
Example result with the JSON renderer:
- `reports/security-report` becomes `reports/security-report.json`
### Example: Webhook Transport
```yaml
outputs:
- renderer:
type: json
transport:
type: webhook
url: https://example.internal/security-ingest
method: POST
headers:
Authorization: Bearer secret-token
timeout_seconds: 10
payload_mode: envelope
```
Fields:
- `type`: must be `webhook`
- `url`: validated as an HTTP or HTTPS URL by Pydantic
- `method`: `POST`, `PUT`, or `PATCH`
- `headers`: arbitrary string-to-string HTTP headers
- `timeout_seconds`: positive float
- `payload_mode`: `rendered` or `envelope`
Payload behavior:
- `rendered`: Hydrogen sends the renderer output directly with the renderer's `media_type` as `Content-Type`.
- `envelope`: Hydrogen sends a JSON object containing format metadata and the rendered content with `Content-Type: application/json`.
## Plugin Discovery via Entry Points
Hydrogen supports discovering plugins via Python package entry points. This is the recommended way to distribute third-party plugins.
Entry point groups:
- `hydrogen.renderers` — for custom renderers
- `hydrogen.transports` — for custom transports
- `hydrogen.modules` — for custom security modules
Example `pyproject.toml` for a third-party renderer:
```toml
[project.entry-points."hydrogen.renderers"]
markdown = "my_package.renderers:MarkdownRenderer"
```
Entry points can point to either a class (which Hydrogen instantiates) or a pre-built instance. Hydrogen validates that the result is a non-abstract subclass or instance of the expected base class.
## Example Production-Oriented Configurations
### Save a JSON Report to Disk
```yaml
fail_fast: false
dry_run: false
exclude_categories: []
logging:
level: INFO
output: stdout
strict_mode: true
allow_failures_below: high
reports:
outputs:
- renderer:
type: json
transport:
type: file
path: reports/hydrogen-audit
append_extension: true
modules:
ssh:
test-failure: false
```
### Send the Report to Multiple Destinations
```yaml
fail_fast: false
dry_run: false
exclude_categories:
- experimental
logging:
level: DEBUG
output: stdout
strict_mode: true
allow_failures_below: medium
reports:
outputs:
- renderer:
type: json
transport:
type: file
path: reports/hydrogen-audit
append_extension: true
- renderer:
type: json
transport:
type: webhook
url: https://collector.example.com/hydrogen/report
method: POST
headers:
X-Source: hydrogen
timeout_seconds: 5
payload_mode: envelope
modules:
ssh:
test-failure: false
```
## Validation and Failure Modes
Because Hydrogen validates configuration with Pydantic before execution, the following failures happen early:
- invalid enum values such as `allow_failures_below: severe`
- invalid logging output values
- missing required fields such as `reports.outputs`
Other failures happen later at runtime:
- unknown renderer names in `renderer.type` raise an error when `RendererRegistry` cannot find that name.
- unknown transport type names raise an error when `TransportRegistry` cannot find that name.
- plugin config validation errors (when a plugin's `config_model` rejects the provided payload) raise errors during config resolution in `core/runtime.py`.
- module import failures are not swallowed and can stop execution.
## Multi-Output Architecture
Hydrogen supports multiple report outputs per run. Each output defines an independent renderer + transport pair.
```yaml
reports:
outputs:
- renderer:
type: json
transport:
type: file
path: reports/audit.json
- renderer:
type: markdown
transport:
type: webhook
url: https://wiki.internal/ingest
method: POST
```
Hydrogen iterates through all outputs and publishes each one:
```python
def publish_many(self, report: AuditReport, outputs: list[ResolvedReportOutput]) -> list[str]:
return [self.publish(report, output) for output in outputs]
```
If an output fails, Hydrogen collects the error in `plugin_errors` and continues with the next output.
## Relationship Between `--path` and `--package-prefix`
This is not part of YAML, but it matters when you organize Hydrogen modules.
Hydrogen discovers modules from the filesystem path passed to `--path`, but imports them by Python package name using `--package-prefix`.
Example:
```bash
python main.py --path custom_modules --package-prefix custom_modules
```
For this to work:
- `custom_modules/` must exist.
- Each module must be a package directory with its own `__init__.py`.
- Python must be able to import `custom_modules.<module_name>`.
Additionally, modules can come from `plugin_packages.modules` in YAML and from `hydrogen.modules` entry points. All sources are combined in `discover_modules()` with deduplication.
## Internal Runtime Config
After loading and validation, Hydrogen builds a `ResolvedConfig` dataclass:
```python
@dataclass(frozen=True)
class ResolvedConfig:
exclude_categories: list[str]
allow_failures_below: AuditSeverity
strict_mode: bool
fail_fast: bool
dry_run: bool
max_concurrency: int
logging: LoggingConfig
plugin_packages: PluginPackagesConfig
reports: list[ResolvedReportOutput]
modules: dict[str, BaseModel]
```
This is the resolved internal config that modules and reporting use at runtime. Each module's config is already validated into its typed `config_model`.