Configuration

Mirror.py Configuration Guide

This document describes how to write config.json for each sync method, and ends with an AI Assistant Protocol you can hand to an AI so it collects the right information and emits a ready-to-use config block on its own.

config.json is read-only at runtime. Only mirror setup ever writes it. Runtime state (status, error counts, timestamps) lives in stat.json, not here.


1. Top-level structure

{
    "mirrorname": "KAIST FTP",
    "hostname": "ftp.kaist.ac.kr",
    "settings": { ... },
    "packages": {
        "<package-id>": { ... },
        "<package-id>": { ... }
    }
}

Key

Type

Description

mirrorname

string

Human-readable mirror name (shown in the web UI).

hostname

string

This mirror’s own FQDN. Used as the ftpsync trace host.

settings

object

Global daemon settings (see §2).

packages

object

Map of package-id to a package definition (see §3).


2. Global settings

"settings": {
    "logfolder": "/mirror/logs",
    "webroot": "/var/www/mirror",
    "statusfile": "/var/www/mirror/status.json",
    "statfile": "/var/lib/mirror/stat.json",
    "uid": 0,
    "gid": 0,
    "localtimezone": "Asia/Seoul",
    "errorcontinuetime": 60,
    "max_runtime": "PT12H",
    "maintainer": { "name": "Roul", "email": "[email protected]" },
    "logger": { ... },
    "ftpsync": { ... },
    "plugins": {},
    "socket": { "mode": "0770" }
}

Key

Type

Notes

logfolder

path

Base directory for ftpsync tool logs. Daemon and per-package logs use logger.fileformat and logger.packagefileformat.

webroot

path

Web root for the status UI.

statusfile

path

Web status JSON output path.

statfile

path

Persistent package state (stat.json).

uid / gid

int

UID/GID that sync subprocesses drop to. 0 (root) triggers a warning — set non-root.

localtimezone

string

IANA timezone name.

errorcontinuetime

int

Seconds to wait before retrying after an error.

max_runtime

string

ISO 8601 duration; a sync running longer is killed. Below PT6H warns; PT12H+ recommended.

maintainer

object

{ "name", "email" }.

logger

object

Log levels, formats, and file rotation (see config-example.json).

ftpsync

object

Global ftpsync defaults (see §4.2).

plugins

object

Enable-only plugin map: { "<name>": {"enabled": bool} }. Use {} when unused. Per-plugin config lives in <name>.json next to config.json.

socket

object

Master control socket ownership/permissions (see §2.2). Optional.

socket_path

path

Optional custom socket directory containing master.sock and worker.sock. Configure the same directory for both daemons and restart both after changes. CLI commands normally discover the running master socket from runtime metadata.

The logger block is rarely changed; copy it verbatim from config-example.json.

2.1 Global settings.ftpsync

Defaults applied to every ftpsync package; per-package options override each key individually.

"ftpsync": {
    "maintainer": "KAIST FTP Maintainers <[email protected]>",
    "sponsor": "KAIST <https://ftp.kaist.ac.kr> SPARCS <https://sparcs.org>",
    "country": "KR",
    "location": "Daejeon",
    "throughput": "10Gb",
    "include": "",
    "exclude": ""
}

2.2 Global settings.socket

Ownership and permissions of the master control socket (master.sock), bound once at daemon startup. The whole block is optional.

"socket": {
    "uid": 0,
    "gid": 0,
    "mode": "0770"
}

Key

Type

Default

Notes

uid

int

unset

chown owner of master.sock. Omit (or no block) to leave ownership unchanged.

gid

int

unset

chown group of master.sock. Omit to leave unchanged.

mode

string

"0600"

Octal string (e.g. "0770"), not an int. Controls who may connect to the daemon socket.

Omit the block for the secure default: mode 0600, no chown. Restart-only. The socket is bound at startup and never re-created on reload, so changing settings.socket takes effect only after a daemon restart. A live reload keeps the running value and emits the warning socket change requires daemon restart (kept current value).

For a custom settings.socket_path, use a directory, not a path ending in .sock. Both daemons treat a .sock value as the same literal socket file, which causes a conflict when the master and worker share that configuration.


3. Package definition (common fields)

Every package — regardless of sync method — shares this shape:

"<package-id>": {
    "name": "Display Name",
    "id": "<package-id>",
    "href": "/path-on-web",
    "synctype": "rsync",
    "syncrate": "PT1H",
    "link": [
        { "rel": "HOME", "href": "https://example.org/" }
    ],
    "settings": {
        "hidden": false,
        "src": "rsync://upstream/module",
        "dst": "/srv/ftp/example",
        "options": { ... }
    }
}

Field

Type

Description

name

string

Display name shown in the UI (this is the visible label).

id

string

Internal identifier. Must equal the map key. Must not start with _ or collide with reserved names (get, items, keys, values, to_dict).

href

string

Web path for this mirror.

synctype

string

A registered sync plugin. Built-ins: rsync, ftpsync, lftp, bandersnatch, ubuntu, jigdo, local, debmirror, apt-mirror2.

syncrate

string

Sync interval (see §3.1).

link

array

List of { "rel", "href" } reference links (e.g. upstream HOME).

settings.hidden

bool

Hide the package from the web UI when true.

settings.src

string

Upstream source. Format depends on synctype (see §4).

settings.dst

string

Local destination directory.

settings.options

object

Method-specific options (see §4). {} when none.

3.1 syncrate values

Value

Meaning

ISO 8601 duration

Interval between syncs. e.g. PT10M (10 min), PT1H (1 h), PT6H, P1D (1 day), P1DT2H (1 day 2 h).

"PUSH"

No timed schedule; sync is triggered externally (push-based). Common for Debian ftpsync.

""

Disabled / never auto-syncs (used by local).


4. Per-method settings

Use the options documented for the selected method. Unknown-option handling depends on the method; apt-mirror2 rejects unknown package and repository options. Do not rely on unrecognized keys being ignored.

4.1 rsync

Incremental rsync with an optional FFTS (fast file-time-list) pre-check that skips the full sync when the upstream timestamp file is unchanged.

  • src: rsync URL without trailing slash, e.g. rsync://ftp.gwdg.de/pub/linux/archlinux (the module appends / automatically).

  • dst: local directory.

Option

Type

Default

Description

ffts

bool

false

Enable the FFTS dry-run pre-check before the full sync.

fftsfile

string

""

Upstream timestamp file checked by FFTS (e.g. lastupdate, fullfiletimelist-rocky). Optional; an omitted or empty value checks the source and destination roots.

user

string

""

rsync username (sets USER env). Leave empty for anonymous.

password

string

""

rsync password (sets RSYNC_PASSWORD env).

option_include

string

""

Flag characters to add to the default flag string. Each char must be in the whitelist vrlptDHaznhPxWENcimub; no -, spaces, or control chars. Chars already present are skipped. e.g. "p" re-adds permission preservation.

option_exclude

string

""

Flag characters to remove from the default flag string. No -, spaces, or control chars. e.g. "H" drops hard-link preservation.

exclude

list[string]

[]

Extra --exclude=<pattern> rules, appended after the built-in --exclude=*.~tmp~. Items must be strings without control characters.

Note: the rsync module reads user/password. A key named username is not read.

Flag string: the default is -vrltDH (since 1.3; previously -vrltDSH, i.e. -S/sparse handling was dropped; -p was dropped earlier in 1.2, so upstream file modes are no longer mirrored by default). option_include / option_exclude adjust this string — option_include is applied first (appends missing chars), then option_exclude removes chars. To restore permission mirroring, set "option_include": "p".

"archlinux": {
    "name": "ArchLinux",
    "id": "archlinux",
    "href": "/ArchLinux",
    "synctype": "rsync",
    "syncrate": "PT1H",
    "link": [ { "rel": "HOME", "href": "https://archlinux.org/" } ],
    "settings": {
        "hidden": false,
        "src": "rsync://ftp.gwdg.de/pub/linux/archlinux",
        "dst": "/srv/ftp/ArchLinux",
        "options": { "ffts": true, "fftsfile": "lastupdate" }
    }
}

4.2 ftpsync

Debian archvsync-based mirroring. Builds an ftpsync.conf from the package and the global settings.ftpsync defaults.

  • src: either a full rsync URL (rsync://host/module) or a bare host combined with the path option.

  • dst: local directory (becomes TO=).

  • syncrate: usually "PUSH" for Debian.

Option

Type

Description

path

string

rsync module path. Required when src is a bare host; overrides the URL path otherwise.

hub

bool

Sets HUB=. Default false.

user

string

RSYNC_USER. Only emitted when both user and password are present.

password

string

RSYNC_PASSWORD.

email

string

MAILTO= for ftpsync notifications.

maintainer

string

Overrides global INFO_MAINTAINER.

sponsor

string

Overrides global INFO_SPONSOR.

country

string

Overrides global INFO_COUNTRY.

location

string

Overrides global INFO_LOCATION.

throughput

string

Overrides global INFO_THROUGHPUT.

arch_include

string

ARCH_INCLUDE (defaults to global include).

arch_exclude

string

ARCH_EXCLUDE (defaults to global exclude).

logdir

path

Override ftpsync log directory (defaults to global logfolder).

"debian": {
    "name": "Debian",
    "id": "debian",
    "href": "/debian",
    "synctype": "ftpsync",
    "syncrate": "PUSH",
    "link": [ { "rel": "HOME", "href": "http://debian.org" } ],
    "settings": {
        "hidden": false,
        "src": "rsync://syncproxy2.wna.debian.org/debian",
        "dst": "/srv/ftp/debian",
        "options": { "hub": false }
    }
}

4.3 lftp

Anonymous FTP mirroring via lftp. src must be an ftp:// URL with no credentials, query, or fragment (validated strictly).

Option

Type

Default

Description

list_options

string

unset

Only "-a" is accepted (lists hidden files).

scan_all_first

bool

false

Pass --scan-all-first to lftp mirror.

exclude_x

list[string]

["\\.in\\..*\\."]

-x regex excludes. Setting this enables custom-exclude mode.

exclude_X

list[string]

`[”.(mirror

notar)”, “lost+found”]`

exclude

list[string]

[]

--exclude patterns.

max_retries

int (1–100)

3

net:max-retries.

net_timeout

int (1–3600)

60

net:timeout seconds.

"example-ftp": {
    "name": "Example",
    "id": "example-ftp",
    "href": "/example",
    "synctype": "lftp",
    "syncrate": "PT6H",
    "link": [],
    "settings": {
        "hidden": false,
        "src": "ftp://ftp.example.org/pub/example",
        "dst": "/srv/ftp/example",
        "options": { "max_retries": 5, "net_timeout": 120 }
    }
}

4.4 ubuntu

Two-stage rsync (metadata-last) tuned for Ubuntu archives, writing a trace file on success.

  • src: rsync URL. dst: local directory.

Option

Type

Default

Description

trace

bool

true

Write <dst>/project/trace/<host> on success.

extra_rsync_args

list[string]

[]

Extra rsync flags.

stage1_excludes

list[string]

["Packages*","Sources*","Release*","InRelease"]

Patterns excluded in stage 1.

user

string

""

rsync username.

password

string

""

rsync password.

"ubuntu": {
    "name": "Ubuntu", "id": "ubuntu", "href": "/ubuntu",
    "synctype": "ubuntu", "syncrate": "PT6H",
    "link": [ { "rel": "HOME", "href": "http://www.ubuntu.com" } ],
    "settings": {
        "hidden": false,
        "src": "rsync://archive.ubuntu.com/ubuntu",
        "dst": "/srv/ftp/ubuntu",
        "options": {}
    }
}

4.5 jigdo

Four-phase Debian CD mirror: template rsync, local ISO regeneration via jigdo-mirror, a final size-only rsync, and optional trace-file creation.

  • src: rsync URL. dst: local directory.

  • Required options: jigdo_file, debian_mirror.

Option

Type

Default

Description

jigdo_file

string

required

rsync URL/path to the jigdo-file index.

debian_mirror

string

required

Debian package mirror URL used to assemble ISOs.

jigdo_include

string

First three CD/DVD images for i386, amd64, sparc, and source

Extended regular expression selecting phase 2 ISO image names; see the jigdo reference for the exact default.

jigdo_exclude

string

".*kfreebsd.*"

Extended regular expression excluding phase 2 ISO image names.

timeout

int

7200

rsync --timeout (seconds).

trace

bool

true

Write trace file on success.

trace_path

string

project/trace

Relative subdir under dst for the trace file.

template_excludes

list[string]

["*.iso"]

Phase 1 exclude patterns.

final_includes

list[string]

["*businesscard*.iso","*netinst*.iso","i386/**.iso"]

Phase 3 include patterns.

extra_rsync_args

list[string]

[]

Extra rsync flags for both rsync phases.

rsync_bin

string

rsync

rsync binary path/name.

jigdo_mirror_bin

string

jigdo-mirror

jigdo-mirror binary path/name.

hostname

string

mirror hostname

Override hostname used in exclude patterns.

user / password

string

""

rsync credentials.

The assembly filters are separate from final_includes, which controls ISO files fetched through rsync in phase 3. See the jigdo reference for exact filter defaults and a complete package example.

4.6 bandersnatch

PyPI mirroring. The module simply runs bandersnatch mirror; the actual mirror behavior is driven by bandersnatch’s own config file, not by settings here. src/dst/options are not consumed by this method (still provide src/dst for documentation/UI consistency).

Option

Type

Description

—

—

No options are read.

4.7 local

This server is the authoritative master for the data; there is no upstream sync. The method only verifies dst exists and marks the package ACTIVE.

  • src: ignored (use "").

  • dst: must already exist on disk.

  • syncrate: typically "".

Option

Type

Description

—

—

No options are read.

4.8 debmirror

Mirror a Debian-style APT archive using the external debmirror command. Install it on the worker host; rsync transport also requires rsync.

  • src: archive URL using http, https, ftp, rsync, or file. A file: URL requires an absolute path and no authority other than localhost.

  • dst: absolute local destination path.

Option

Type

Default

Description

dist

string or list[string]

discovered

Distributions to mirror; comma-separated strings are accepted.

section

string or list[string]

discovered

Archive components to mirror.

arch

string or list[string]

discovered

Binary architectures to mirror.

source

bool

false

Include source packages.

check_gpg

bool

true

Verify Release signatures.

keyring

absolute path or list[path]

unset

Trusted local keyrings; configure them when signature verification is enabled. Missing keyrings produce warnings.

cleanup

string

"postcleanup"

One of postcleanup, precleanup, or nocleanup.

Omitted selections are discovered on each run from listings and Release metadata. Signatures are verified by default; check_gpg: false disables verification, and ignore_release_gpg: true allows unverified metadata when signature verification fails. Automatic discovery refuses to remove an existing distribution that disappears from a listing; set dist explicitly when intentionally narrowing the mirror.

See the debmirror reference for all transport overrides, content filters, installer options, timeouts, signature controls, and a complete package example.

4.9 apt-mirror2

Mirror multiple regular or flat APT repositories as one scheduled package. Install the apt-mirror2 Python extra in the environment used by the daemon and worker. The daemon checks that it is installed before delegating the job. The worker runs python -m apt_mirror from apt-mirror==16; it does not use the unrelated Perl apt-mirror executable. Signature verification requires gpgv.

  • src: package display URL only; actual sources are in options.config.

  • dst: non-root absolute destination without whitespace, $, or #.

Package option

Type

Default

Description

config

list[object]

required

Non-empty repository list. Every item requires src and dst.

source

bool

false

Default for downloading source indexes and packages; items can override it.

nthreads

positive int

8

Download concurrency shared by the package.

limit_rate

positive int or string

unlimited

Aggregate bytes per second, optionally suffixed with k or m, such as "20m".

Repository option

Type

Default

Description

src

string

required

Unique HTTP, HTTPS, or anonymous FTP repository URL without credentials, queries, fragments, or path traversal.

dst

string

required

Safe relative directory below the package destination. Item destinations must not overlap.

dist

string or list[string]

discovered

Distributions, or flat paths ending in /, such as "./".

section

string or list[string]

discovered

Components for regular repositories; not accepted for flat repositories.

arch

string or list[string]

discovered

Binary architectures for regular repositories; not accepted for flat repositories.

source

bool

package source

Include source indexes and packages for this item.

check_gpg

bool

true

Verify metadata during discovery and native mirroring.

keyring

absolute path or list[path]

required when check_gpg is true

Trusted local keyrings for this item. Keys are not downloaded or shared between items.

Unknown package and repository options are rejected. Release metadata is required even when signature verification is explicitly disabled. Omitted selections are discovered each run; automatic discovery refuses to remove an existing distribution that disappears from a listing. Use explicit dist selections when intentionally narrowing the mirror.

See the apt-mirror2 reference for full URL and path restrictions, discovery, cleanup behavior, and a complete package example.


5. AI Assistant Protocol

Paste this section (or the whole document) to an AI assistant. It tells the assistant exactly what to ask for and how to produce a config block.

Role: You generate a mirror.py package config block from information the user provides. Follow these steps:

  1. Identify the sync method. Map the user’s intent to one synctype:

    • rsync URL + “skip when unchanged” / timestamp file -> rsync (set ffts/fftsfile).

    • Debian, push-based -> ftpsync (syncrate: "PUSH").

    • Selected Debian-style APT archive with discovery or filters -> debmirror.

    • Multiple regular or flat APT repositories in one job -> apt-mirror2.

    • Ubuntu archive -> ubuntu.

    • Debian CD/ISO with jigdo -> jigdo.

    • Anonymous ftp:// source -> lftp.

    • PyPI -> bandersnatch.

    • This host is the origin / no upstream -> local.

  2. Collect the required inputs for that method:

    • Always: display name, id (kebab-case, equals the map key), href, src, dst, syncrate, and any reference links.

    • rsync: whether FFTS is used, an optional fftsfile name, and credentials if any.

    • ftpsync: rsync URL or (host + path); hub; any INFO_* overrides.

    • debmirror: archive URL, optional dist/section/arch selections, whether source packages are needed, and local trusted keyrings for signature checks.

    • apt-mirror2: a non-empty config repository list with each item’s actual src, relative dst, selections, and local keyrings unless check_gpg is explicitly disabled. The package src is only a display URL.

    • lftp: confirm src is an ftp:// URL; retry/timeout/excludes if any.

    • ubuntu: trace on/off; extra excludes if any.

    • jigdo: jigdo_file and debian_mirror (both required).

    • bandersnatch: note the separate bandersnatch config file is required.

    • local: confirm dst exists; src empty.

  3. Apply naming conventions exactly as the user states them. If the user specifies a display spelling (e.g. “represent it as ArchLinux”), use that spelling for name, href, and dst as appropriate, while keeping id in kebab-case.

  4. Pick a sane syncrate if the user did not give one: PT1H for most rsync/lftp mirrors, "PUSH" for Debian ftpsync, "" for local. State the assumption.

  5. Validate before output:

    • id equals the map key, does not start with _, and is not a reserved name (get, items, keys, values, to_dict).

    • src strips any trailing slash for rsync.

    • Only options documented for the chosen method, including its linked full reference, appear in options.

    • Use user/password (not username) for credentials.

  6. Output the package block as a JSON snippet ready to paste under "packages". If the user is creating a full standalone file, wrap it with the top-level structure and copy the settings block from an existing config. After the JSON, briefly list any assumptions you made.

  7. Do not invent options. If the user asks for behavior no option supports, say so instead of fabricating a key.

Minimal input the AI should ask for if missing: sync method (or enough to infer it), src, dst, display name, and desired syncrate.