Difference between revisions of "NKit/UserGuide"

Latest revision as of 10:54, 20 September 2019

Welcome to the NKit user guide

This guide aims to cover NKit's features and options from a user point of view. Please refer to the Glossary and FAQ if something isn't clear. Feel free to ask anything else in the GBA Temp NKit App thread. Thanks for reading.

Key Uses

There are two key uses of NKit:

Recovery: Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
Convert: Converts anything as is. When converting to the NKit Format the source data is preserved so that it can be restored back to the source iso

NKit Apps

There are 2 types of apps, command line and GUI. All apps use the NKit.dll.

Note for Devs: The NKit.dll contains all the processing logic. The apps below contain very little code and demonstrate how to use it.

Command Line Apps

ConvertToISO.exe - converts a source file to iso. This can be wbfs (scrubbing is preserved), iso.dec (junk is recreated) or NKit [iso/gcz] (source data is restored back to iso)
ConvertToNKit.exe - preserves the source image data as nkit [iso/gcz]. This can also be used to preserve scrubbed, hacked or bad dumps etc. For WBFS and iso.dec when restoring back to iso with ConvertToIso.exe the iso will contain all data that was in the original WBFS / iso.dec (just as .iso)
RecoverToISO.exe - attempts to recover GameCube and Wii images to ISO
RecoverToNKit.exe - attempts to recover GameCube and Wii images and convert to nkit.iso or nkit.gcz
RecoveryExtract.exe - saves recovery files from images. CRC's in The NKit.dll.config are used to save extracted files in to a 'Redump' or 'Other' folder.

All commandline apps have the same usage. A list of space separated files and masks. Paths can be relative or absolute. The apps will scan all inputs for files and collate them into a unique set. They are processed for multipart files (rars, split etc) then sorted and processed.

Examples

Example	Description
`RecoverToNKit test\.`	Relative path, all files
`RecoverToNKit c:\test\.`	Absolute path, all files
`RecoverToNKit "test path\."`	Path with spaces
`RecoverToNKit "backup image.iso"`	Specific file with spaces
`RecoverToNKit *.r??`	All rar files (including parts r00, r01 etc)
`RecoverToNKit *.wbf?`	All wbfs files (including wbfs + wbf1)
`RecoverToNKit path`	All files in a relative folder
`RecoverToNKit path\.`	All files in a relative folder:

GUI Apps

NKitExtractionApp.exe - extract recovery files or extract files from the image filesystems
NKitProcessingApp.exe - recover and convert files to iso / nkit [iso/gcz]. It has a drop down of Modes that performs the NKit conversions (covered by the first 4 command line apps above)

The configuration options in the GUI apps override the equivalent settings in the NKit.dll.config file.

This user guide refers to conversions throughout which are types of processing offered by NKit. The command line apps refer to them - ConvertToISO, ConvertToNKit, RecoverToISO and RecoverToNKit.

Feature Overview

The features in this section are used across the conversions in NKit.

NKit Format

ConvertToNKit and RecoverToNKit will output to the NKit Format. This format is the smallest GameCube and Wii image format. Both GameCube and Wii images are playable in Dolphin, but only GameCube is supported by hardware.

The NKit Format is non-lossy and supports clean / Redump images as well as scrubbed and hacked images. Some corrupt and bad images are supported too, although these can error due to invalid fst.bin modifications.

There are 2 NKit output formats, raw (iso) and compressed (gcz). Both were designed with 1:1 preservation, size and playability in mind. Hardware support for Wii was sacrificed for smaller image sizes. If hardware support is required then WBFS is currently the best available and recoverable to Redump where not hacked or corrupt.

System	Format	Hardware Supported	Dolphin Supported	Restorable 1:1	Notes
GameCube	nkit.iso	Yes	Yes	Yes	Same as compacted GameCube iso
GameCube	nkit.gcz	No	Yes	Yes	GCZ is Dolphin's own block seekable compression format
Wii	nkit.iso	No	Yes	Yes	RVT-H format only playable in Dolphin
Wii	nkit.gcz	No	Yes	Yes	RVT-H in GCZ playable in Dolphin only

NKit Format images contain the bare minimum of data. All junk and scrubbing is removed. Non-uniform (NonJunk) data is also preserved. Wii encryption and hashes are fully recreatable and removed. Meaning any remaining data is as compressible as possible.

Dat Files

NKit relies heavily on Dat files. The Redump Dat files (GameCube and Wii) hold the checksums of all the good images. NKit uses this information to help recover from modified images. Without these files NKit can only unscrub images and can't know if they're modified in other ways.

Multiple Dat files can be placed in the folders. NKit will load the one with latest modified date

Mask Renaming Output Files

Output images that match Redump or Other Dat file entries can be renamed to a mask. The masks allow processed images to be moved in to separate folder for Redump Matched, Custom Matched or Match Fail.

There are various masks that can be used in the renaming:

Item	Name	Comment
`%src`	Source image path	Path of the source image
`%nmo`	Original filename
`%nmd`	Disc Title	Read from the header of the disc
`%nmg`	GameTDB Title	Title from the GameTDB dat file
`%nmm`	Matched Name	Redump or Custom Dat matched entry - falls back to %nmg then %nmo
`%dno`	Disc No.	E.g. (Disc 2) - Only added if not in the filename already. Not output if (Disc 1)
`%rev`	Revision No.	E.g. (Rev 1) - Only added if not in the filename already. Not output if (Rev 0)
`%ver`	Revision No.	E.g. (v1.01) - Only added if not in the filename already. Not output if (v1.00)
`%ext`	File Extension	Does not contain leading .
`%crc`	CRC32	Output file CRC32
`%md5`	MD5	Only calculated if the config is set to Calculate Hashes
`%sha`	SHA1	Only calculated if the config is set to Calculate Hashes
`%id4`	Disc ID4	The Disc ID
`%id6`	Disc ID6	The Disc ID and Maker Code
`%id8`	Disc ID8	The Disc ID, MakerCode, Disc No. and Disc Revision (Disc No and Revision are as Hex bytes e.g. 0000)

An example of a working mask is %pth\GameCube\%nmm.%ext you may choose to use something like %pth\NewFolder\%nmm %id8 %crc.%ext. This will add [] around the values.

Recovery Files

The concept of recovery files is that by saving the bare minimum of information from Redump good images. NKit can recover many modified images.

GameCube other tools that modify these images scrub and move (sometimes reorder) the files in the filesystem. NKit recovery files are the appldr.bin (a handful are used across all images) and fst.bin (unique per image, NKit stores some header info in these files also).
Wii - other tools remove unused blocks or scrub them by wiping unused blocks with 00 or FF bytes. It is common for Non-Data Partitions are also removed to save space. NKit recovery files consist of the channel and update partitions.

NKit can extract the recovery files from images. Running this on a set of random Wii images may extract Update partitions that can be used to recover other images that have had them removed. The update partitions are shared across images. There's only a handful of unique ones.

Brute Force Recovery

In addition to the recovery files, NKit can attempt to force a match by checking various Header CRCs and Update Partition CRCs (for Wii only). Once a match is found the modifications can be applied to the output file and the CRC will match the Dat entry.

NKit does this by CRCing sections of files as it processes them. Upon completing a conversion other CRCs calculated combined to attempt to make a match. Header modifications include Region and for Wii - swapping the Update partition CRC and using the Data Partition data-header for the disc header.

Summary Log

The summary log is an output file used to track conversions. It holds stores the following columns:

TimeStamp, App, System, ReadResult, OutputResult, OutputCrc, OutputID4, RedumpMatch, RedumpName, InputSize, OutputSize, FullSize, InputFilename, OutputFilename, MD5, SHA1, Passes, SecondsElapsed, ErrorMessage

This file should be used to evaluate the results of a large conversion before deleting source files. The data values are tab separated for pasting in to your favourite spreadsheet app.

Conversions

NKit can convert to and from various formats. To simplify recovering and converting to and from the formats a multi pass system will process the image through the various stages.

Configuration

The NKit.dll uses a configuration file name NKit.dll.config. This file contains base configuration. There are multiple sections or which the first 3 contain items the user will be most interested in editing:-

<appSettings> - General settings
<gamecube> - GameCube Redump and recovery settings. Including some lookup data
<wii> - Wii Redump and recovery settings. Including some lookup data

AppSettings

Item	Description
SearchSubfolders	Search sub folders when finding images to process
WaitForKeyAfterProcessing	Wait for newline entry before exiting a commandline app
Path	All paths are relative to the working folder - global substitutions %exe=exe path, %pth=Path value. A base path that can be used in other paths with %pth, paths can be absolute or relative to the working folder - not supported by apps above
TempPath	Temp working folder, absolute or relative to current working directory - best to be the same drive as the output
SummaryLog	Summary log path, a log of all conversions with image details - check this file before deleting source files etc
EnableSummaryLog	Turn the SummaryLog on or off
FullVerify	Verify the output file by converting it back to iso in memory to ensure the CRC is valid
CalculateHashes	Calculates additional MD5 and SHA1 hashes. This is required to output them in to the summary log or use them in the rename file masks
NkitFormat	Save NKit as iso or gcz
NkitReencode	Force a full reencode when converting nkit to nkit. If false when converting nkit.iso to nkit.gcz, only the gcz coversion is performed
MaskRename	Rename output images to the masks in the gamecube and wii config sections, else use the source path and filename
TestMode	Convert images, but delete them when complete - View SummaryLog for results
RecoveryMatchFailDelete	Recovery Only - delete failed matches, false will rename them to the FailMatchRenameToMask - they may be corrupt
DeleteSource	Delete the source file if FullVerify returns VerifySuccess (FullVerify setting must be true) TestMode=true cancels this setting
OutputLevel	Logging level when processing 0=Silent, 1=Info, 2=Detailed
DatPathNameGameTdbMask	Path and mask to GameTDB file - latest file is used

GameCube / Wii

GameCube and Wii share a log of the same settings. Depending on which image is being processed the corresponding section is referenced.

Item	Description
DatPathRedumpMask	Path and mask to redump dat file - latest file is used
DatPathCustomMask	Path and mask to custom dat file - latest file is used
RedumpMatchRenameToMask	Used when an output file's CRC matches and entry in the Redump dat. Move and Rename the file to this mask
CustomMatchRenameToMask	Used when an output file's CRC matches and entry in the Custom dat. Move and Rename the file to this mask
MatchFailRenameToMask	Used when an output file's CRC fails to match any dat. Move and Rename the file to this mask
RecoveryFilesPath	Path to Redump recovery files. Recovery extracted files are stored here if the file's CRC is known
OtherRecoveryFilesPath	Path to Other recovery files. Recovery extracted files are stored here if the file's CRC is not known
NkitUpdatePartitionRemoval*	Remove the update partition when converting to nkit, the CRC of the removed partition is stored in the NKit header for restoration
NkitRecoveryFilesPath*	Path to Wii NKit extracted update partition when extracted by ConvertToNKit with NkitUpdatePartitionRemoval set to true

* Wii section only

Getting Started

The apps will work out of the box and a default configuration is provided, but in order to Recover images and match to Redump you will need to do the following:-

Download the Dat files from Redump and place in the Dats folders
Download the GameTDB dat and place in Dats\GameTdb folder
Place any Recovery files in the Recovery\Redump folders
Edit the NKit.dll.config file

Base Configuration

You may prefer to change the default configuration before using the apps. The following sections highlight some alternative configurations.

Smallest Image Output Configurations

NKit Format	External Compression	Configuration
nkit.iso	7-zip	`NkitFormat="iso"`
nkit.gcz	No	`NkitFormat="gcz"`

Smallest Wii Output Configurations

There are a few factors to consider. The table below shows the configurations that create the smallest Wii images from Smallest to Largest. The relevant configuration options are also listed.

Removed Update partitions are stored in the Redump Recovery folder if they don't already exist. If they are not known they are stored in the NKitExtracted Recovery folder. The Removed Update Partition CRC is stored in the NKit header so it can be safely restored.

NKit Format	External Compression	Update Partition Removed	Configuration
nkit.iso	7-zip	Yes	`NkitFormat="iso", NkitUpdatePartitionRemoval="true"`
nkit.gcz	No	Yes	`NkitFormat="gcz", NkitUpdatePartitionRemoval="true"`
nkit.iso	7-zip	No	`NkitFormat="iso", NkitUpdatePartitionRemoval="false"`
nkit.gcz	No	No	`NkitFormat="gcz", NkitUpdatePartitionRemoval="false"`

External compression must be applied after NKit has completed. GCZ cannot match 7-zip. GCZ is a block seekable compression that is supported by Dolphin.

Convert To NKit and Delete Source

FullVerify="true" - This must be enabled and return VerifySuccess
DeleteSource="true" - This is defaulted to false, Test Mode cancels this

Convert To NKit and Audit The Results

TestMode="true" - Process and remove the output file
EnableSummaryLog="true" - Audit the results of the files processed
MaskRename="true" - Audit which Dat entry matches
CalculateHashes="true" - true if auditing Summary to contain the MD5 and SHA1 of the output file
FullVerify="true" - true if the auditing should verify the output CRC is correct

@@ Line 71: / Line 71: @@
 | GameCube || nkit.gcz || No || Yes || Yes || GCZ is Dolphin's own block seekable compression format
 |-
-| Wii || nkit.iso || No (Yes with Dev Kit?) || Yes || Yes || RVT-H format only playable in Dolphin or a Dev Kit (Untested)
+| Wii || nkit.iso || No || Yes || Yes || RVT-H format only playable in Dolphin
 |-
 | Wii || nkit.gcz || No || Yes || Yes || RVT-H in GCZ playable in Dolphin only
@@ Line 84: / Line 84: @@
 =====Mask Renaming Output Files=====
-Images output that match Redump or Other Dat file entries can be renamed to a mask. The masks allow processed images to be moved in to separate folder for Redump Matched, Custom Matched or Match Fail.
+Output images that match Redump or Other Dat file entries can be renamed to a mask. The masks allow processed images to be moved in to separate folder for Redump Matched, Custom Matched or Match Fail.
 There are various masks that can be used in the renaming:
@@ Line 121: / Line 121: @@
 | <code>%id8</code> || Disc ID8 || The Disc ID, MakerCode, Disc No. and Disc Revision (Disc No and Revision are as Hex bytes e.g. 0000)
 |}
+An example of a working mask is <code>%pth\GameCube\%nmm.%ext</code> you may choose to use something like <code>%pth\NewFolder\%nmm %id8 %crc.%ext</code>. This will add <code>[]</code> around the values.
 =====Recovery Files=====
@@ Line 126: / Line 128: @@
 * '''GameCube''' other tools that modify these images scrub and move (sometimes reorder) the files in the filesystem. NKit recovery files are the appldr.bin (a handful are used across all images) and fst.bin (unique per image, NKit stores some header info in these files also).
 * '''Wii''' - other tools remove unused blocks or scrub them by wiping unused blocks with 00 or FF bytes. It is common for Non-Data Partitions are also removed to save space. NKit recovery files consist of the channel and update partitions.
+NKit can extract the recovery files from images. Running this on a set of random Wii images may extract Update partitions that can be used to recover other images that have had them removed. The update partitions are shared across images. There's only a handful of unique ones.
 =====Brute Force Recovery=====
 In addition to the recovery files, NKit can attempt to force a match by checking various Header CRCs and Update Partition CRCs (for Wii only). Once a match is found the modifications can be applied to the output file and the CRC will match the Dat entry.
-NKit does this by CRCing sections of files as it processes them. Upon completing a conversion other CRCs calculated combined to attempt to make a match. Header modifications include Region and for Wii swapping the Update partition CRC and using the Data Partition data-header for the disc header.
+NKit does this by CRCing sections of files as it processes them. Upon completing a conversion other CRCs calculated combined to attempt to make a match. Header modifications include Region and for Wii - swapping the Update partition CRC and using the Data Partition data-header for the disc header.
 =====Summary Log=====