RustyGet docs · rewrite in progress
Source

rustyget::docs::differences

Overview

Differences from NZBGet

RustyGet started from NZBGet 27.0 and the work of pull request 850. This page lists what differs, area by area. The sections after the table cover the details.

At a glance

Area NZBGet RustyGet
Language C++ C++ moving to Rust module by module; Rust port status lists the modules ported so far
An article missing on every server Lost; par2 repair or failure Fetched from a matching file in another copy of the release, then checked by its place in the file (for yEnc articles), and, where they're available, by its checksum (with the default CrcCheck=yes) and par2
Duplicates Spare downloads, tried one after another One fleet: copies lend each other articles, bytes, and par2 volumes
Proving two copies hold the same files Not done Par2 checksums prove twins, the exact same post. Alts, the same video in another container, are matched by the video's bytes
Not enough par2 blocks Repair fails Repairs with a twin's recovery volumes
Holes after the download Left to par2 Patched with verified bytes from duplicates, across RAR, ZIP, 7-Zip, splits, and encryption
A download that can't complete Runs to the end, then fails over HealthCheck=dupe fails over within seconds, and a probe catches dead copies before they start
Finding more copies Only what feeds and clients send DupeSearch queries NZBHydra2, checks each copy on your servers, and queues the live ones
Clients sending many copies One append per copy One appendfleet call; RustyGet checks every copy and picks the best
Health of a backup Unknown until it downloads Measured, shown with its score, and used to skip dead copies
Configuration 155 options 23 more options and a new HealthCheck value, off unless the configuration file turns them on; the template that ships with RustyGet turns on live repair and early failover
API NZBGet API NZBGet API plus appendfleet, the TLS cipher test, and new fields; unsafe methods need POST; may diverge later
Queue files NZBGet formats NZBGet formats, with RustyGet's data in a trailer that NZBGet ignores
Bug fixes About 100 fixes to inherited bugs, 96 of them reported upstream
Unpacking External unrar and 7z programs Built in: RAR, 7-Zip, ZIP, and tar archives extract in Rust, with no external programs
Build and test workflows Builds for every platform on each push Builds and tests on Linux x86-64 and macOS for every pull request and push to main

Duplicate fleets

Feature Setting Summary
Article fallback DupeArticleFallback=article Missing articles come from matching files in other copies, matched by par2 content, name, or structure, and checked against the download's par2 block checksums when the download's par2 set describes the file.
Cutover and lead rotation Part of article fallback After three recoveries in a file, RustyGet asks the lead duplicate first; after three misses in a row, the next duplicate leads.
Twins and alts Automatic with Rust par2 file lists prove which copies are the exact same post (twins). Copies that hold the same video file in a different container are alts.
par2 from a twin Automatic When recovery blocks run short, RustyGet fetches and uses the fewest twin recovery volumes that cover the damage.
Stream repair DupeArticleFallback=stream Byte-level repair from duplicates after the download, with same-byte, whole-file, cross-packaging, and cross-encryption methods.
Live repair DupeArticleFallback=live Stream repair during the download.
Borrow first DupeBorrowFirst=yes Duplicates repair missing articles and damaged files during the download, before the download's own par2 files; par2 then checks the result and repairs the rest.
Compressed donors DupeStreamDecompress=yes Extracts a compressed duplicate to repair from its contents.
Repair time limit DupeStreamTimeout Stops a stream repair that stalls, or that would take longer than downloading again.
Early failover HealthCheck=dupe Fails over as soon as a download can't complete, by failure counts, projected health, or par2 block damage.
Dead-copy probe HealthCheck=dupe Ten articles checked on every server before a download starts, when a backup waits.
Failed-article recheck Article fallback or HealthCheck=dupe After a failure, RustyGet asks for a sample of the failed articles again and retries them once if most now exist.
Donor search DupeSearch=yes Finds, checks, and queues more copies through NZBHydra2.
Fleet submission appendfleet A client sends up to 50 copies in one call.

See Duplicate fleets and Donor search.

Changes to inherited duplicate handling

  • Failover skips a backup whose NZB file is gone, a backup found dead, and a copy of a posting that already failed.
  • Among backups with equal scores, the most complete one goes first.
  • A title already on disk isn't downloaded again, and once one copy is on disk, the other copies stop.
  • With HealthCheck=dupe, a forced download fails over to its backups like a scored one.
  • When no backup qualifies, the log gives the reason for each one.
  • The duplicate mode of a download survives a restart.

Options

Twenty-two new options are in INCOMING NZBS, after DupeCheck: DupeArticleFallback, DupeStreamDecompress, DupeStreamTimeout, DupeBorrowFirst, DupeSearch, DupeSearchUrl, DupeSearchApiKey, DupeSearchDelay, DupeSearchDryRun, DupeSearchMaxDonors, DupeHealthPercent, DupeHealthMin, DupeHealthMax, DupeBodyChecks, DupeHealthBudget, DupeMinAlive, DupeFastDonors, DupePrimaryScore, DupeSizeTolerance, DupeBodyPercent, DupeCheckConcurrency, and DupeCheckConnections. Repair from duplicates is on by default, in the configuration file and when an option is left out of it: DupeArticleFallback=live, DupeStreamDecompress=yes and DupeBorrowFirst=yes. HealthCheck gains the value dupe, now its default, and ParScan=dupe, DirectUnpack=yes and DirectRename=yes are the defaults too. DupeSearch stays off by default. A fleet ranks its copies TWINs first (by health), then ALTs whose packaging the byte-level repair can read, then the rest, and duplicates are asked for missing data in the same order. One new option is in SECURITY: JsonP, off by default, turns on answers to JSON-P requests, which NZBGet always answered.

Defaults

A RustyGet setting has two defaults: the value in the configuration template, rustyget.conf in the repository, which a new installation starts from, and the built-in value that applies when the file leaves the option out. Configuration options shows the template's values.

The template differs from NZBGet's in these values:

Option NZBGet RustyGet template
DupeArticleFallback live
DupeStreamDecompress yes
DupeBorrowFirst yes
DirectUnpack no yes
HealthCheck park dupe
ArticleCache 0 500
WriteBuffer 0 1024
ParBuffer 16 500
ContinuePartial yes no
ControlUsername nzbget rustyget
LockFile, LogFile nzbget.lock, nzbget.log rustyget.lock, rustyget.log
Server1.Active, Server1.Connections yes, 8 no, 20

The built-in values changed too, for an option that the file leaves out: ArticleCache=500, ParBuffer=250, WriteBuffer=1024, ContinuePartial=yes, DirectRename=yes, ReorderFiles=yes, Unpack=yes, DirectUnpack=yes, UnpackCleanupDisk=yes, NzbCleanupDisk=yes, ExtCleanupDisk=.par2, .sfv, _brokenlog.txt, InterDir=${MainDir}/inter, and ControlUsername=rustyget. The built-in duplicate-fleet settings stay off, so a configuration file carried over from NZBGet doesn't turn them on.

UnrarCmd and SevenZipCmd left the template because unpacking is built in. The extension options, ScriptDir, Extensions, ScriptOrder, ScriptPauseQueue, ShellOverride, EventInterval, and CategoryN.Extensions, left too: RustyGet builds in what NZBGet needed extensions for, such as nested unpacking, permission fixes, and donor search. See Built in instead of extensions.

Option parsing also changed: a scheduler task with an unknown command is reported and skipped instead of running as PauseDownload.

New startup warnings point out risky combinations, such as donor search without DupeCheck or with fewer than three article retries.

See Configuration options for every option.

API

  • New method appendfleet, the TLS cipher test methods tlscipherstart, tlscipherstatus, and tlsciphercancel, and new fields DupeRecoveredArticles, DupeRecoveredBytesLo, DupeRecoveredBytesHi, DupeRecoveredHoles, and LiveRepairing.
  • New stage STREAM_REPAIRING.
  • New NZB parameters, including DupeKind, DupeCopyKind, DupeFiles, DupeAlive, and DupeFleet.
  • Methods that change the system or reach out to the network need POST. Plain GET runs only the methods that read.
  • JSON-P is off unless option JsonP turns it on. Then it runs only the methods that read, with the restricted user's access, so user names, passwords, and API keys come back as ***. In NZBGet, JSON-P ran every method with full access, and any web page could read the answers.
  • system.multicall respects the caller's access level.
  • Outgoing HTTP requests, such as NZB, RSS, and indexer fetches, identify as nzbget/26.3 and add the header UserAgentExtended: RustyGet/0.1, so indexers that expect NZBGet keep working.
  • Out-of-range numbers are rejected, saveconfig refuses malformed input, and several decoding and output bugs are fixed.

See API and compatibility.

Web interface

  • Fleet tags such as PRIMARY-3fa9c2, TWIN-3fa9c2, and ALT-81d0e4 on downloads and history items. Copies with the same fingerprint hold the same files.
  • A score badge with each backup's measured health.
  • A Repair badge during live repair, and a dupe repair post-processing status.
  • Recovered from duplicates in the statistics dialog.
  • History shows twin or alt instead of dupe where known.
  • Escaping of names, categories, and messages that NZBGet printed as HTML, and http or https links only.
  • Optimize TLS ciphers in the news-server settings measures the ciphers each encrypted server supports and saves the fastest one for each server.
  • Fixes to keyboard shortcuts, the filter box, upload errors, and size totals.

Security and robustness

  • DupeSearchApiKey, and any option with an API key in its name, is protected like a password: the restricted user and JSON-P see ***.
  • File cleanup never follows symbolic links.
  • NZB parsing refuses XML entity declarations.
  • The control port limits concurrent connections and keeps running when a thread or a connection fails.
  • State files accept lines of any length. In NZBGet, a line over 1,024 characters can crash the program.
  • The lock file is deleted only by the process that owns it.
  • Scripts get close-on-exec pipes, full-length output lines, and a correct exit code when a signal kills them.

Bug fixes

RustyGet fixes about 100 inherited bugs. 96 of them are reported to the NZBGet project, as its issues 940 to 1046. Highlights:

  • Downloads: 451 errors count as missing, 502 login refusals move on, malformed headers move to the next server, a connection that times out in the middle of an article is closed instead of reused, yEnc data can't overrun its range, and uuencoded parts no longer pass through the article cache.
  • Post-processing: ParScan=extended and ParScan=dupe find extra blocks again, split files join correctly, a failed direct unpack no longer deletes the download folder, ParTimeLimit counts from the right moment, RAR-rename works per folder, and archives with encrypted headers parse.
  • Queue and history: concurrent appends, names with newlines or slashes, long names, merges, splits, and edits of missing items all behave.
  • Files: moves across file systems and copies that fail leave no partial file behind.
  • Feeds and URLs: fixes to dates in RFC 822 format, redirects, protocol-relative URLs, and the retry loop.
  • Clock changes: the scheduler, quotas, and maintenance handle a clock that moves backward.

Build

  • rust/rust-toolchain.toml pins Rust 1.99.0, and just build compiles the Rust crate and links the rustyget binary.
  • PAR2 verification, streaming repair, post-download repair, and par-renaming use the par-rs crate, and archive extraction, including decompressed donors for stream repair, uses the zipfast and zipview crates. RustyGet needs no par2, unrar, or 7z program.
  • A required build check compiles and tests RustyGet on Linux x86-64 and macOS for every pull request and every push to main. The inherited workflows for packages are gone.