Configuration
Mirror.py Configuration Guide
This document describes how to write config.json for each sync method, and
ends with an AI Assistant Protocol you can hand to an AI so it collects the
right information and emits a ready-to-use config block on its own.
config.jsonis read-only at runtime. Onlymirror setupever writes it. Runtime state (status, error counts, timestamps) lives instat.json, not here.
1. Top-level structure
{
"mirrorname": "KAIST FTP",
"hostname": "ftp.kaist.ac.kr",
"settings": { ... },
"packages": {
"<package-id>": { ... },
"<package-id>": { ... }
}
}
Key |
Type |
Description |
|---|---|---|
|
string |
Human-readable mirror name (shown in the web UI). |
|
string |
This mirror’s own FQDN. Used as the ftpsync trace host. |
|
object |
Global daemon settings (see §2). |
|
object |
Map of |
2. Global settings
"settings": {
"logfolder": "/mirror/logs",
"webroot": "/var/www/mirror",
"statusfile": "/var/www/mirror/status.json",
"statfile": "/var/lib/mirror/stat.json",
"uid": 0,
"gid": 0,
"localtimezone": "Asia/Seoul",
"errorcontinuetime": 60,
"max_runtime": "PT12H",
"maintainer": { "name": "Roul", "email": "[email protected]" },
"logger": { ... },
"ftpsync": { ... },
"plugins": {},
"socket": { "mode": "0770" }
}
Key |
Type |
Notes |
|---|---|---|
|
path |
Base directory for ftpsync tool logs. Daemon and per-package logs use |
|
path |
Web root for the status UI. |
|
path |
Web status JSON output path. |
|
path |
Persistent package state ( |
|
int |
UID/GID that sync subprocesses drop to. |
|
string |
IANA timezone name. |
|
int |
Seconds to wait before retrying after an error. |
|
string |
ISO 8601 duration; a sync running longer is killed. Below |
|
object |
|
|
object |
Log levels, formats, and file rotation (see |
|
object |
Global ftpsync defaults (see §4.2). |
|
object |
Enable-only plugin map: |
|
object |
Master control socket ownership/permissions (see §2.2). Optional. |
|
path |
Optional custom socket directory containing |
The logger block is rarely changed; copy it verbatim from config-example.json.
2.1 Global settings.ftpsync
Defaults applied to every ftpsync package; per-package options override
each key individually.
"ftpsync": {
"maintainer": "KAIST FTP Maintainers <[email protected]>",
"sponsor": "KAIST <https://ftp.kaist.ac.kr> SPARCS <https://sparcs.org>",
"country": "KR",
"location": "Daejeon",
"throughput": "10Gb",
"include": "",
"exclude": ""
}
2.2 Global settings.socket
Ownership and permissions of the master control socket (master.sock), bound
once at daemon startup. The whole block is optional.
"socket": {
"uid": 0,
"gid": 0,
"mode": "0770"
}
Key |
Type |
Default |
Notes |
|---|---|---|---|
|
int |
unset |
|
|
int |
unset |
|
|
string |
|
Octal string (e.g. |
Omit the block for the secure default: mode
0600, nochown. Restart-only. The socket is bound at startup and never re-created on reload, so changingsettings.sockettakes effect only after a daemon restart. A live reload keeps the running value and emits the warningsocket change requires daemon restart (kept current value).
For a custom settings.socket_path, use a directory, not a path ending in
.sock. Both daemons treat a .sock value as the same literal socket file,
which causes a conflict when the master and worker share that configuration.
3. Package definition (common fields)
Every package — regardless of sync method — shares this shape:
"<package-id>": {
"name": "Display Name",
"id": "<package-id>",
"href": "/path-on-web",
"synctype": "rsync",
"syncrate": "PT1H",
"link": [
{ "rel": "HOME", "href": "https://example.org/" }
],
"settings": {
"hidden": false,
"src": "rsync://upstream/module",
"dst": "/srv/ftp/example",
"options": { ... }
}
}
Field |
Type |
Description |
|---|---|---|
|
string |
Display name shown in the UI (this is the visible label). |
|
string |
Internal identifier. Must equal the map key. Must not start with |
|
string |
Web path for this mirror. |
|
string |
A registered sync plugin. Built-ins: |
|
string |
Sync interval (see §3.1). |
|
array |
List of |
|
bool |
Hide the package from the web UI when |
|
string |
Upstream source. Format depends on |
|
string |
Local destination directory. |
|
object |
Method-specific options (see §4). |
3.1 syncrate values
Value |
Meaning |
|---|---|
ISO 8601 duration |
Interval between syncs. e.g. |
|
No timed schedule; sync is triggered externally (push-based). Common for Debian ftpsync. |
|
Disabled / never auto-syncs (used by |
4. Per-method settings
Use the options documented for the selected method. Unknown-option handling
depends on the method; apt-mirror2 rejects unknown package and repository
options. Do not rely on unrecognized keys being ignored.
4.1 rsync
Incremental rsync with an optional FFTS (fast file-time-list) pre-check that skips the full sync when the upstream timestamp file is unchanged.
src: rsync URL without trailing slash, e.g.rsync://ftp.gwdg.de/pub/linux/archlinux(the module appends/automatically).dst: local directory.
Option |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable the FFTS dry-run pre-check before the full sync. |
|
string |
|
Upstream timestamp file checked by FFTS (e.g. |
|
string |
|
rsync username (sets |
|
string |
|
rsync password (sets |
|
string |
|
Flag characters to add to the default flag string. Each char must be in the whitelist |
|
string |
|
Flag characters to remove from the default flag string. No |
|
list[string] |
|
Extra |
Note: the rsync module reads
user/password. A key namedusernameis not read.Flag string: the default is
-vrltDH(since 1.3; previously-vrltDSH, i.e.-S/sparse handling was dropped;-pwas dropped earlier in 1.2, so upstream file modes are no longer mirrored by default).option_include/option_excludeadjust this string —option_includeis applied first (appends missing chars), thenoption_excluderemoves chars. To restore permission mirroring, set"option_include": "p".
"archlinux": {
"name": "ArchLinux",
"id": "archlinux",
"href": "/ArchLinux",
"synctype": "rsync",
"syncrate": "PT1H",
"link": [ { "rel": "HOME", "href": "https://archlinux.org/" } ],
"settings": {
"hidden": false,
"src": "rsync://ftp.gwdg.de/pub/linux/archlinux",
"dst": "/srv/ftp/ArchLinux",
"options": { "ffts": true, "fftsfile": "lastupdate" }
}
}
4.2 ftpsync
Debian archvsync-based mirroring. Builds an ftpsync.conf from the package and
the global settings.ftpsync defaults.
src: either a full rsync URL (rsync://host/module) or a bare host combined with thepathoption.dst: local directory (becomesTO=).syncrate: usually"PUSH"for Debian.
Option |
Type |
Description |
|---|---|---|
|
string |
rsync module path. Required when |
|
bool |
Sets |
|
string |
|
|
string |
|
|
string |
|
|
string |
Overrides global |
|
string |
Overrides global |
|
string |
Overrides global |
|
string |
Overrides global |
|
string |
Overrides global |
|
string |
|
|
string |
|
|
path |
Override ftpsync log directory (defaults to global |
"debian": {
"name": "Debian",
"id": "debian",
"href": "/debian",
"synctype": "ftpsync",
"syncrate": "PUSH",
"link": [ { "rel": "HOME", "href": "http://debian.org" } ],
"settings": {
"hidden": false,
"src": "rsync://syncproxy2.wna.debian.org/debian",
"dst": "/srv/ftp/debian",
"options": { "hub": false }
}
}
4.3 lftp
Anonymous FTP mirroring via lftp. src must be an ftp:// URL with no
credentials, query, or fragment (validated strictly).
Option |
Type |
Default |
Description |
|---|---|---|---|
|
string |
unset |
Only |
|
bool |
|
Pass |
|
list[string] |
|
|
|
list[string] |
`[”.(mirror |
notar)”, “lost+found”]` |
|
list[string] |
|
|
|
int (1–100) |
|
|
|
int (1–3600) |
|
|
"example-ftp": {
"name": "Example",
"id": "example-ftp",
"href": "/example",
"synctype": "lftp",
"syncrate": "PT6H",
"link": [],
"settings": {
"hidden": false,
"src": "ftp://ftp.example.org/pub/example",
"dst": "/srv/ftp/example",
"options": { "max_retries": 5, "net_timeout": 120 }
}
}
4.4 ubuntu
Two-stage rsync (metadata-last) tuned for Ubuntu archives, writing a trace file on success.
src: rsync URL.dst: local directory.
Option |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Write |
|
list[string] |
|
Extra rsync flags. |
|
list[string] |
|
Patterns excluded in stage 1. |
|
string |
|
rsync username. |
|
string |
|
rsync password. |
"ubuntu": {
"name": "Ubuntu", "id": "ubuntu", "href": "/ubuntu",
"synctype": "ubuntu", "syncrate": "PT6H",
"link": [ { "rel": "HOME", "href": "http://www.ubuntu.com" } ],
"settings": {
"hidden": false,
"src": "rsync://archive.ubuntu.com/ubuntu",
"dst": "/srv/ftp/ubuntu",
"options": {}
}
}
4.5 jigdo
Four-phase Debian CD mirror: template rsync, local ISO regeneration via
jigdo-mirror, a final size-only rsync, and optional trace-file creation.
src: rsync URL.dst: local directory.Required options:
jigdo_file,debian_mirror.
Option |
Type |
Default |
Description |
|---|---|---|---|
|
string |
required |
rsync URL/path to the jigdo-file index. |
|
string |
required |
Debian package mirror URL used to assemble ISOs. |
|
string |
First three CD/DVD images for i386, amd64, sparc, and source |
Extended regular expression selecting phase 2 ISO image names; see the jigdo reference for the exact default. |
|
string |
|
Extended regular expression excluding phase 2 ISO image names. |
|
int |
|
rsync |
|
bool |
|
Write trace file on success. |
|
string |
|
Relative subdir under |
|
list[string] |
|
Phase 1 exclude patterns. |
|
list[string] |
|
Phase 3 include patterns. |
|
list[string] |
|
Extra rsync flags for both rsync phases. |
|
string |
|
rsync binary path/name. |
|
string |
|
jigdo-mirror binary path/name. |
|
string |
mirror hostname |
Override hostname used in exclude patterns. |
|
string |
|
rsync credentials. |
The assembly filters are separate from final_includes, which controls ISO
files fetched through rsync in phase 3. See the
jigdo reference
for exact filter defaults and a complete package example.
4.6 bandersnatch
PyPI mirroring. The module simply runs bandersnatch mirror; the actual mirror
behavior is driven by bandersnatch’s own config file, not by settings
here. src/dst/options are not consumed by this method (still provide
src/dst for documentation/UI consistency).
Option |
Type |
Description |
|---|---|---|
— |
— |
No options are read. |
4.7 local
This server is the authoritative master for the data; there is no upstream sync.
The method only verifies dst exists and marks the package ACTIVE.
src: ignored (use"").dst: must already exist on disk.syncrate: typically"".
Option |
Type |
Description |
|---|---|---|
— |
— |
No options are read. |
4.8 debmirror
Mirror a Debian-style APT archive using the external debmirror command.
Install it on the worker host; rsync transport also requires rsync.
src: archive URL usinghttp,https,ftp,rsync, orfile. Afile:URL requires an absolute path and no authority other thanlocalhost.dst: absolute local destination path.
Option |
Type |
Default |
Description |
|---|---|---|---|
|
string or list[string] |
discovered |
Distributions to mirror; comma-separated strings are accepted. |
|
string or list[string] |
discovered |
Archive components to mirror. |
|
string or list[string] |
discovered |
Binary architectures to mirror. |
|
bool |
|
Include source packages. |
|
bool |
|
Verify Release signatures. |
|
absolute path or list[path] |
unset |
Trusted local keyrings; configure them when signature verification is enabled. Missing keyrings produce warnings. |
|
string |
|
One of |
Omitted selections are discovered on each run from listings and Release
metadata. Signatures are verified by default; check_gpg: false disables
verification, and ignore_release_gpg: true allows unverified metadata when
signature verification fails. Automatic discovery refuses to remove an existing
distribution that disappears from a listing; set dist explicitly when
intentionally narrowing the mirror.
See the debmirror reference for all transport overrides, content filters, installer options, timeouts, signature controls, and a complete package example.
4.9 apt-mirror2
Mirror multiple regular or flat APT repositories as one scheduled package.
Install the apt-mirror2 Python extra in the environment used by the daemon
and worker. The daemon checks that it is installed before delegating the job.
The worker runs python -m apt_mirror from apt-mirror==16; it does not use the
unrelated Perl apt-mirror executable. Signature verification requires gpgv.
src: package display URL only; actual sources are inoptions.config.dst: non-root absolute destination without whitespace,$, or#.
Package option |
Type |
Default |
Description |
|---|---|---|---|
|
list[object] |
required |
Non-empty repository list. Every item requires |
|
bool |
|
Default for downloading source indexes and packages; items can override it. |
|
positive int |
|
Download concurrency shared by the package. |
|
positive int or string |
unlimited |
Aggregate bytes per second, optionally suffixed with |
Repository option |
Type |
Default |
Description |
|---|---|---|---|
|
string |
required |
Unique HTTP, HTTPS, or anonymous FTP repository URL without credentials, queries, fragments, or path traversal. |
|
string |
required |
Safe relative directory below the package destination. Item destinations must not overlap. |
|
string or list[string] |
discovered |
Distributions, or flat paths ending in |
|
string or list[string] |
discovered |
Components for regular repositories; not accepted for flat repositories. |
|
string or list[string] |
discovered |
Binary architectures for regular repositories; not accepted for flat repositories. |
|
bool |
package |
Include source indexes and packages for this item. |
|
bool |
|
Verify metadata during discovery and native mirroring. |
|
absolute path or list[path] |
required when |
Trusted local keyrings for this item. Keys are not downloaded or shared between items. |
Unknown package and repository options are rejected. Release metadata is
required even when signature verification is explicitly disabled. Omitted
selections are discovered each run; automatic discovery refuses to remove an
existing distribution that disappears from a listing. Use explicit dist
selections when intentionally narrowing the mirror.
See the apt-mirror2 reference for full URL and path restrictions, discovery, cleanup behavior, and a complete package example.
5. AI Assistant Protocol
Paste this section (or the whole document) to an AI assistant. It tells the assistant exactly what to ask for and how to produce a config block.
Role: You generate a mirror.py package config block from information the
user provides. Follow these steps:
Identify the sync method. Map the user’s intent to one
synctype:rsync URL + “skip when unchanged” / timestamp file ->
rsync(setffts/fftsfile).Debian, push-based ->
ftpsync(syncrate: "PUSH").Selected Debian-style APT archive with discovery or filters ->
debmirror.Multiple regular or flat APT repositories in one job ->
apt-mirror2.Ubuntu archive ->
ubuntu.Debian CD/ISO with jigdo ->
jigdo.Anonymous
ftp://source ->lftp.PyPI ->
bandersnatch.This host is the origin / no upstream ->
local.
Collect the required inputs for that method:
Always: display
name,id(kebab-case, equals the map key),href,src,dst,syncrate, and any referencelinks.rsync: whether FFTS is used, an optional
fftsfilename, and credentials if any.ftpsync: rsync URL or (host +
path);hub; anyINFO_*overrides.debmirror: archive URL, optional
dist/section/archselections, whether source packages are needed, and local trusted keyrings for signature checks.apt-mirror2: a non-empty
configrepository list with each item’s actualsrc, relativedst, selections, and local keyrings unlesscheck_gpgis explicitly disabled. The packagesrcis only a display URL.lftp: confirm
srcis anftp://URL; retry/timeout/excludes if any.ubuntu: trace on/off; extra excludes if any.
jigdo:
jigdo_fileanddebian_mirror(both required).bandersnatch: note the separate bandersnatch config file is required.
local: confirm
dstexists;srcempty.
Apply naming conventions exactly as the user states them. If the user specifies a display spelling (e.g. “represent it as ArchLinux”), use that spelling for
name,href, anddstas appropriate, while keepingidin kebab-case.Pick a sane
syncrateif the user did not give one:PT1Hfor most rsync/lftp mirrors,"PUSH"for Debian ftpsync,""forlocal. State the assumption.Validate before output:
idequals the map key, does not start with_, and is not a reserved name (get,items,keys,values,to_dict).srcstrips any trailing slash for rsync.Only options documented for the chosen method, including its linked full reference, appear in
options.Use
user/password(notusername) for credentials.
Output the package block as a JSON snippet ready to paste under
"packages". If the user is creating a full standalone file, wrap it with the top-level structure and copy thesettingsblock from an existing config. After the JSON, briefly list any assumptions you made.Do not invent options. If the user asks for behavior no option supports, say so instead of fabricating a key.
Minimal input the AI should ask for if missing: sync method (or enough to
infer it), src, dst, display name, and desired syncrate.