Overview
Differences from NZBGet
RustyGet started from NZBGet 27.0 and the work of pull request 850. This page lists what differs, area by area. The sections after the table cover the details.
At a glance
| Area | NZBGet | RustyGet |
|---|---|---|
| Language | C++ | C++ moving to Rust module by module; Rust port status lists the modules ported so far |
| An article missing on every server | Lost; par2 repair or failure | Fetched from a matching file in another copy of the release, then checked by its place in the file (for yEnc articles), and, where they're available, by its checksum (with the default CrcCheck=yes) and par2 |
| Duplicates | Spare downloads, tried one after another | One fleet: copies lend each other articles, bytes, and par2 volumes |
| Proving two copies hold the same files | Not done | Par2 checksums prove twins, the exact same post. Alts, the same video in another container, are matched by the video's bytes |
| Not enough par2 blocks | Repair fails | Repairs with a twin's recovery volumes |
| Holes after the download | Left to par2 | Patched with verified bytes from duplicates, across RAR, ZIP, 7-Zip, splits, and encryption |
| A download that can't complete | Runs to the end, then fails over | HealthCheck=dupe fails over within seconds, and a probe catches dead copies before they start |
| Finding more copies | Only what feeds and clients send | DupeSearch queries NZBHydra2, checks each copy on your servers, and queues the live ones |
| Clients sending many copies | One append per copy |
One appendfleet call; RustyGet checks every copy and picks the best |
| Health of a backup | Unknown until it downloads | Measured, shown with its score, and used to skip dead copies |
| Configuration | 155 options | 23 more options and a new HealthCheck value, off unless the configuration file turns them on; the template that ships with RustyGet turns on live repair and early failover |
| API | NZBGet API | NZBGet API plus appendfleet, the TLS cipher test, and new fields; unsafe methods need POST; may diverge later |
| Queue files | NZBGet formats | NZBGet formats, with RustyGet's data in a trailer that NZBGet ignores |
| Bug fixes | About 100 fixes to inherited bugs, 96 of them reported upstream | |
| Unpacking | External unrar and 7z programs |
Built in: RAR, 7-Zip, ZIP, and tar archives extract in Rust, with no external programs |
| Build and test workflows | Builds for every platform on each push | Builds and tests on Linux x86-64 and macOS for every pull request and push to main |
Duplicate fleets
| Feature | Setting | Summary |
|---|---|---|
| Article fallback | DupeArticleFallback=article |
Missing articles come from matching files in other copies, matched by par2 content, name, or structure, and checked against the download's par2 block checksums when the download's par2 set describes the file. |
| Cutover and lead rotation | Part of article fallback | After three recoveries in a file, RustyGet asks the lead duplicate first; after three misses in a row, the next duplicate leads. |
| Twins and alts | Automatic with Rust | par2 file lists prove which copies are the exact same post (twins). Copies that hold the same video file in a different container are alts. |
| par2 from a twin | Automatic | When recovery blocks run short, RustyGet fetches and uses the fewest twin recovery volumes that cover the damage. |
| Stream repair | DupeArticleFallback=stream |
Byte-level repair from duplicates after the download, with same-byte, whole-file, cross-packaging, and cross-encryption methods. |
| Live repair | DupeArticleFallback=live |
Stream repair during the download. |
| Borrow first | DupeBorrowFirst=yes |
Duplicates repair missing articles and damaged files during the download, before the download's own par2 files; par2 then checks the result and repairs the rest. |
| Compressed donors | DupeStreamDecompress=yes |
Extracts a compressed duplicate to repair from its contents. |
| Repair time limit | DupeStreamTimeout |
Stops a stream repair that stalls, or that would take longer than downloading again. |
| Early failover | HealthCheck=dupe |
Fails over as soon as a download can't complete, by failure counts, projected health, or par2 block damage. |
| Dead-copy probe | HealthCheck=dupe |
Ten articles checked on every server before a download starts, when a backup waits. |
| Failed-article recheck | Article fallback or HealthCheck=dupe |
After a failure, RustyGet asks for a sample of the failed articles again and retries them once if most now exist. |
| Donor search | DupeSearch=yes |
Finds, checks, and queues more copies through NZBHydra2. |
| Fleet submission | appendfleet |
A client sends up to 50 copies in one call. |
See Duplicate fleets and Donor search.
Changes to inherited duplicate handling
- Failover skips a backup whose NZB file is gone, a backup found dead, and a copy of a posting that already failed.
- Among backups with equal scores, the most complete one goes first.
- A title already on disk isn't downloaded again, and once one copy is on disk, the other copies stop.
- With
HealthCheck=dupe, a forced download fails over to its backups like a scored one. - When no backup qualifies, the log gives the reason for each one.
- The duplicate mode of a download survives a restart.
Options
Twenty-two new options are in INCOMING NZBS, after DupeCheck: DupeArticleFallback, DupeStreamDecompress, DupeStreamTimeout, DupeBorrowFirst, DupeSearch, DupeSearchUrl, DupeSearchApiKey, DupeSearchDelay, DupeSearchDryRun, DupeSearchMaxDonors, DupeHealthPercent, DupeHealthMin, DupeHealthMax, DupeBodyChecks, DupeHealthBudget, DupeMinAlive, DupeFastDonors, DupePrimaryScore, DupeSizeTolerance, DupeBodyPercent, DupeCheckConcurrency, and DupeCheckConnections. Repair from duplicates is on by default, in the configuration file and when an option is left out of it: DupeArticleFallback=live, DupeStreamDecompress=yes and DupeBorrowFirst=yes. HealthCheck gains the value dupe, now its default, and ParScan=dupe, DirectUnpack=yes and DirectRename=yes are the defaults too. DupeSearch stays off by default. A fleet ranks its copies TWINs first (by health), then ALTs whose packaging the byte-level repair can read, then the rest, and duplicates are asked for missing data in the same order. One new option is in SECURITY: JsonP, off by default, turns on answers to JSON-P requests, which NZBGet always answered.
Defaults
A RustyGet setting has two defaults: the value in the configuration template, rustyget.conf in the repository, which a new installation starts from, and the built-in value that applies when the file leaves the option out. Configuration options shows the template's values.
The template differs from NZBGet's in these values:
| Option | NZBGet | RustyGet template |
|---|---|---|
DupeArticleFallback |
live |
|
DupeStreamDecompress |
yes |
|
DupeBorrowFirst |
yes |
|
DirectUnpack |
no |
yes |
HealthCheck |
park |
dupe |
ArticleCache |
0 |
500 |
WriteBuffer |
0 |
1024 |
ParBuffer |
16 |
500 |
ContinuePartial |
yes |
no |
ControlUsername |
nzbget |
rustyget |
LockFile, LogFile |
nzbget.lock, nzbget.log |
rustyget.lock, rustyget.log |
Server1.Active, Server1.Connections |
yes, 8 |
no, 20 |
The built-in values changed too, for an option that the file leaves out: ArticleCache=500, ParBuffer=250, WriteBuffer=1024, ContinuePartial=yes, DirectRename=yes, ReorderFiles=yes, Unpack=yes, DirectUnpack=yes, UnpackCleanupDisk=yes, NzbCleanupDisk=yes, ExtCleanupDisk=.par2, .sfv, _brokenlog.txt, InterDir=${MainDir}/inter, and ControlUsername=rustyget. The built-in duplicate-fleet settings stay off, so a configuration file carried over from NZBGet doesn't turn them on.
UnrarCmd and SevenZipCmd left the template because unpacking is built in. The extension options, ScriptDir, Extensions, ScriptOrder, ScriptPauseQueue, ShellOverride, EventInterval, and CategoryN.Extensions, left too: RustyGet builds in what NZBGet needed extensions for, such as nested unpacking, permission fixes, and donor search. See Built in instead of extensions.
Option parsing also changed: a scheduler task with an unknown command is reported and skipped instead of running as PauseDownload.
New startup warnings point out risky combinations, such as donor search without DupeCheck or with fewer than three article retries.
See Configuration options for every option.
API
- New method
appendfleet, the TLS cipher test methodstlscipherstart,tlscipherstatus, andtlsciphercancel, and new fieldsDupeRecoveredArticles,DupeRecoveredBytesLo,DupeRecoveredBytesHi,DupeRecoveredHoles, andLiveRepairing. - New stage
STREAM_REPAIRING. - New NZB parameters, including
DupeKind,DupeCopyKind,DupeFiles,DupeAlive, andDupeFleet. - Methods that change the system or reach out to the network need POST. Plain GET runs only the methods that read.
- JSON-P is off unless option
JsonPturns it on. Then it runs only the methods that read, with the restricted user's access, so user names, passwords, and API keys come back as***. In NZBGet, JSON-P ran every method with full access, and any web page could read the answers. system.multicallrespects the caller's access level.- Outgoing HTTP requests, such as NZB, RSS, and indexer fetches, identify as
nzbget/26.3and add the headerUserAgentExtended: RustyGet/0.1, so indexers that expect NZBGet keep working. - Out-of-range numbers are rejected,
saveconfigrefuses malformed input, and several decoding and output bugs are fixed.
Web interface
- Fleet tags such as
PRIMARY-3fa9c2,TWIN-3fa9c2, andALT-81d0e4on downloads and history items. Copies with the same fingerprint hold the same files. - A score badge with each backup's measured health.
- A Repair badge during live repair, and a dupe repair post-processing status.
- Recovered from duplicates in the statistics dialog.
- History shows
twinoraltinstead ofdupewhere known. - Escaping of names, categories, and messages that NZBGet printed as HTML, and
httporhttpslinks only. - Optimize TLS ciphers in the news-server settings measures the ciphers each encrypted server supports and saves the fastest one for each server.
- Fixes to keyboard shortcuts, the filter box, upload errors, and size totals.
Security and robustness
DupeSearchApiKey, and any option with an API key in its name, is protected like a password: the restricted user and JSON-P see***.- File cleanup never follows symbolic links.
- NZB parsing refuses XML entity declarations.
- The control port limits concurrent connections and keeps running when a thread or a connection fails.
- State files accept lines of any length. In NZBGet, a line over 1,024 characters can crash the program.
- The lock file is deleted only by the process that owns it.
- Scripts get close-on-exec pipes, full-length output lines, and a correct exit code when a signal kills them.
Bug fixes
RustyGet fixes about 100 inherited bugs. 96 of them are reported to the NZBGet project, as its issues 940 to 1046. Highlights:
- Downloads: 451 errors count as missing, 502 login refusals move on, malformed headers move to the next server, a connection that times out in the middle of an article is closed instead of reused, yEnc data can't overrun its range, and uuencoded parts no longer pass through the article cache.
- Post-processing:
ParScan=extendedandParScan=dupefind extra blocks again, split files join correctly, a failed direct unpack no longer deletes the download folder,ParTimeLimitcounts from the right moment, RAR-rename works per folder, and archives with encrypted headers parse. - Queue and history: concurrent appends, names with newlines or slashes, long names, merges, splits, and edits of missing items all behave.
- Files: moves across file systems and copies that fail leave no partial file behind.
- Feeds and URLs: fixes to dates in RFC 822 format, redirects, protocol-relative URLs, and the retry loop.
- Clock changes: the scheduler, quotas, and maintenance handle a clock that moves backward.
Build
rust/rust-toolchain.tomlpins Rust 1.99.0, andjust buildcompiles the Rust crate and links therustygetbinary.- PAR2 verification, streaming repair, post-download repair, and par-renaming use the
par-rscrate, and archive extraction, including decompressed donors for stream repair, uses thezipfastandzipviewcrates. RustyGet needs nopar2,unrar, or7zprogram. - A required
buildcheck compiles and tests RustyGet on Linux x86-64 and macOS for every pull request and every push tomain. The inherited workflows for packages are gone.