← Back to smarthome overview

YouTube Archiving: From TubeArchivist and TubeSync to yt-playlist-sync

Keeping offline copies of YouTube channels worth rewatching started with two self-hosted tools running side by side on the Synology NAS. Neither turned out to be the right fit, so both were replaced by a custom build: yt-playlist-sync.

What ran before

TubeArchivist and TubeSync approached the same goal from different directions. TubeArchivist provided a full archive with search and metadata, while TubeSync focused on pushing playlists into a media server library. Both ran via Docker Compose on docker-host-nas, and for a while both ran in parallel, because neither could replace the other.

TubeArchivist

TubeArchivist self-hosts a searchable archive of downloaded YouTube content (Elasticsearch, Redis, and the TubeArchivist app itself). It's used to archive channels worth rewatching, including tutorials, long-form content, and anything that might disappear behind copyright strikes or channel deletions. Two things stayed unsatisfying:

Playlist detection. New playlists on subscribed channels were not always recognized by the app, so the videos in them were simply skipped during automatic downloads. The workaround was submitting videos and playlists by hand through the built-in submission tool, which works but defeats the purpose of automated archiving. This issue was never resolved.
A heavy stack on a NAS kernel. Newer Elasticsearch releases need a kernel feature (SECCOMP) that Synology's DSM kernel doesn't compile in, so Elasticsearch had to be version-pinned. Containers sometimes failed to join the same Docker network even when defined in one Compose file, app upgrades had to follow a very specific step-by-step path because skipping versions breaks the database migration, and the initial indexing of large channels took a long time.

TubeSync

TubeSync takes the simpler route: it syncs YouTube playlists straight into a media server library without the archive UI. Everything is manual, playlists have to be added by URL, and there is no discovery of new content at all. It's a pure download-and-organize tool, reliable for a fixed list of playlists but impossible to leave alone.

The result

Both tools were never really convincing. TubeArchivist had the better archive but missed playlists, TubeSync downloaded reliably but needed constant curation, and keeping both felt like running two half-solutions. Shutting one of them down was never an option because each covered what the other lacked.

yt-playlist-sync

yt-playlist-sync is a self-hosted service built to replace both. It watches the public playlists of one YouTube channel and downloads them as video with yt-dlp to a NAS share. No Google login, no API keys, only public playlists.

Discovery every hour. New playlists are found every 50 to 70 minutes and downloaded right away.
Nightly incremental sync. Regular playlists are synced once per night, with a per-playlist download archive so only new videos are fetched.
Oneshot playlists. Playlists whose title contains a keyword (default setlist) are downloaded once and never touched again afterwards, with a status, an error summary and a retry button for failed runs.
It never deletes anything. Playlists that disappear from YouTube are only marked as removed.
One download at a time, with sleeps between videos and between jobs, to stay polite towards YouTube.
A small web UI with status, sync and oneshot tabs, plus a playlist detail page with cover, thumbnail gallery, metadata from the local .info.json files, placeholder rows for missing videos and a built-in player that streams the downloaded files.
Operational guards: Healthchecks pings, a pause on HTTP 429 or 403, a self-updating yt-dlp with an atomic swap, a weekly image rebuild and a nightly backup of the database and the download archives.

It runs as a Docker container from the public image ghcr.io/skoelle/yt-playlist-sync:latest on port 8048, with three volumes: /config for the SQLite database and archives, /data for the downloads, and /backup for the nightly copies.

Migration: export from both, import into the new tool

Both existing libraries were moved over without re-downloading anything. Each exporter writes the folder layout yt-playlist-sync itself downloads into (<Playlist title> [<playlist id>]/NN - <Title> [<video id>].mp4), together with a manifest.json, yt-dlp style .info.json sidecars, covers and the download archive files. Nothing is deleted or overwritten, and every step starts with a dry run.

Export from TubeArchivist

python -m app.ta_export pulls playlists, videos and metadata from the TubeArchivist REST API and pairs them with the media files on disk. It runs as a container on the NAS itself with the TubeArchivist folder mounted read-only, so the videos never travel over the network twice. Run order: report only, then metadata only, then the full copy.

.venv/bin/python -m app.ta_export --dry-run --metadata-only   # report only
.venv/bin/python -m app.ta_export --metadata-only             # metadata, no videos
.venv/bin/python -m app.ta_export                             # full copy

Export from TubeSync

TubeSync has no JSON API, so python -m app.ts_export opens its SQLite database read-only, pairs the rows with the files below its downloads/ folder and writes exactly the same tree. The report, the manifests and the rules are identical to the TubeArchivist export.

.venv/bin/python -m app.ts_export --dry-run --metadata-only   # report only
.venv/bin/python -m app.ts_export                             # full copy

Import into yt-playlist-sync

The export is only a folder tree until the app knows about it. python -m app.ta_import reads the manifest.json files and writes playlists, video entries and download archives into the database, offline, without YouTube and without either old tool. The app has to be stopped first, and a preview comes free with --dry-run.

docker compose run --rm yt-playlist-sync python -m app.ta_import --dry-run
docker compose run --rm yt-playlist-sync python -m app.ta_import

Imported playlists become oneshot, so the nightly sync never touches them.
Local files decide the state: done only when every entry has a file on disk, otherwise idle, so missing videos can be fetched later with a retry.
Entries are upserted, never removed, and nothing on disk is moved or deleted.
Every local video gets a line in the yt-dlp download archive, so later sync runs don't re-check it.
For playlists that exist twice, python -m app.forget clears a playlist (report first, apply on request, folder moved to quarantine, backup beforehand) so the import starts from a clean state.

How it fits into the backup strategy

Without Elasticsearch and Redis there is much less state to protect. Downloaded media files still ride on the NAS backup volumes, and the application state is two small things that the service backs up itself every night at 00:30 with rotation.

Data	Included in NAS backup?
Downloaded media files	Yes, via the NAS's own backup volumes
SQLite database (`/config`)	Yes, nightly copy into `/backup`, kept copies are rotated
yt-dlp download archives (`/config`)	Yes, nightly tarball into `/backup`, rotated as well

What it took

The basic functions that were actually needed were up and running after a single day of SPEC-driven development: write the specification first, then build against it. Within one week, done next to the day job, I migrated everything from the old tools, added extra features (exports, importer, maintenance CLI, gallery, backups) and brought the service into a state that is good enough to run in production. TubeArchivist and TubeSync are switched off.