Difference between revisions of "NKit/UserGuide"
(→Wii - Smallest) |
(→Wii - Smallest) |
||
| Line 221: | Line 221: | ||
| nkit.gcz || No || No || <code>NkitFormat="gcz", NkitUpdatePartitionRemoval="false"</code> | | nkit.gcz || No || No || <code>NkitFormat="gcz", NkitUpdatePartitionRemoval="false"</code> | ||
|} | |} | ||
| + | External compression must be applied after NKit has completed. GCZ cannot match 7-zip. GCZ is a block seekable compression that is supported by Dolphin. | ||
Revision as of 11:35, 12 September 2019
Welcome to the NKit user guide
This guide aims to cover NKit's features and options from a user point of view. Please refer to the Glossary and FAQ if something isn't clear. Feel free to ask anything else in the GBA Temp NKit App thread. Thanks for reading.
Key Uses
There are two key uses of NKit:
- Recovery: Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
- Convert: Converts anything as is. When converting to the NKit Format the source data is preserved so that it can be restored back to the source iso
NKit Apps
There are 2 types of apps, command line and GUI. All apps use the NKit.dll.
Note for Devs: The NKit.dll contains all the processing logic. The apps below contain very little code and demonstrate how to use it.
Command Line Apps
- ConvertToISO.exe - converts a source file to iso. This can be wbfs (scrubbing is preserved), iso.dec (junk is recreated) or NKit [iso/gcz] (source data is restored back to iso)
- ConvertToNKit.exe - preserves the source image data as nkit [iso/gcz]. This can also be used to preserve scrubbed, hacked or bad dumps etc. For WBFS and iso.dec when restoring back to iso with ConvertToIso.exe the iso will contain all data that was in the original WBFS / iso.dec.
- RecoverToISO.exe - attempts to recover GameCube and Wii images to .iso
- RecoverToNKit.exe - attempts to recover GameCube and Wii images and convert to nkit.iso or nkit.gcz
- RecoveryExtract.exe - saves recovery files from images. CRC's in The NKit.dll.config are used to save extracted files in to a 'Redump' or 'Other' folder.
All commandline apps have the same usage. A list of space separated files and masks. Paths can be relative or absolute. The apps will scan all inputs for files and collate them into a unique set. They are processed for multipart files (rars, split etc) then sorted and processed.
Examples (all standard command line)
| Example | Description |
|---|---|
RecoverToNKit test\*.* |
Relative path, all files |
RecoverToNKit c:\test\*.* |
Absolute path, all files |
RecoverToNKit "test path\*.*" |
Path with spaces |
RecoverToNKit "backup image.iso" |
Specific file with spaces |
RecoverToNKit *.r?? |
All rar files (including parts r00, r01 etc) |
RecoverToNKit *.wbf? |
All wbfs files (including wbfs + wbf1) |
RecoverToNKit path |
All files in a relative folder |
RecoverToNKit path\*.* |
All files in a relative folder: |
GUI Apps
- NKitExtractionApp.exe - extract recovery files or extract files from the image filesystems
- NKitProcessingApp.exe - recover and convert files to iso / nkit [iso/gcz]. It has a drop down of option that performs the tasks of the first 4 commandline apps above
The configuration options in the GUI apps override the equivalent settings in the NKit.dll.config file.
This user guide refers to conversions throughout which are types of processing offered by NKit. The command line apps refer to them - ConvertToISO, ConvertToNKit, RecoverToISO and RecoverToNKit.
Feature Overview
The features in this section are used across the conversions in NKit.
Dat Files
NKit relies heavily on Dat files. The Redump Dat files (GameCube and Wii) hold the checksums of all the good images. NKit uses this information to help recover from modified images. Without these files NKit can only unscrub images and can't know if they're modified in other ways.
Multiple Dat files can be placed in the folders. NKit will load the one with latest modified date
Mask Renaming Output Files
Images output that match Redump or Other Dat file entries can be renamed to a mask. The masks allow processed images to be moved in to separate folder for Redump Matched, Custom Matched or Match Fail.
There are various masks that can be used in the renaming:
| Item | Name | Comment |
|---|---|---|
%src |
Source image path | Path of the source image |
%nmo |
Original filename | |
%nmd |
Disc Title | Read from the header of the disc |
%nmg |
GameTDB Title | Title from the GameTDB dat file |
%nmm |
Matched Name | Redump or Custom Dat matched entry - falls back to %nmg then %nmo |
%dno |
Disc No. | E.g. (Disc 2) - Only added if not in the filename already. Not output if (Disc 1) |
%rev |
Revision No. | E.g. (Rev 1) - Only added if not in the filename already. Not output if (Rev 0) |
%ver |
Revision No. | E.g. (v1.01) - Only added if not in the filename already. Not output if (v1.00) |
%ext |
File Extension | Does not contain leading . |
%crc |
CRC32 | Output file CRC32 |
%md5 |
MD5 | Only calculated if the config is set to Calculate Hashes |
%sha |
SHA1 | Only calculated if the config is set to Calculate Hashes |
%id4 |
Disc ID4 | The Disc ID |
%id6 |
Disc ID6 | The Disc ID and Maker Code |
%id8 |
Disc ID8 | The Disc ID, MakerCode, Disc No. and Disc Revision (Disc No and Revision are as Hex bytes e.g. 0000) |
Recovery Files
The concept of recovery files is that by saving the bare minimum of information from Redump good images. NKit can recover many modified images.
- GameCube other tools that modify these images scrub and move (sometimes reorder) the files in the filesystem. NKit recovery files are the appldr.bin (a handful are used across all images) and fst.bin (unique per image, NKit stores some header info in these files also).
- Wii - other tools remove unused blocks or scrub them by wiping unused blocks with 00 or FF bytes. It is common for Non-Data Partitions are also removed to save space. NKit recovery files consist of the channel and update partitions.
Brute Force Recovery
In addition to the recovery files, NKit can attempt to force a match by checking various Header CRCs and Update Partition CRCs (for Wii only). Once a match is found the modifications can be applied to the output file and the CRC will match the Dat entry.
NKit does this by CRCing sections of files as it processes them. Upon completing a conversion other CRCs calculated combined to attempt to make a match. Header modifications include Region and for Wii swapping the Update partition CRC and using the Data Partition data-header for the disc header.
Summary Log
The summary log is an output file used to track conversions. It holds stores the following columns:
TimeStamp, App, System, ReadResult, OutputResult, OutputCrc, OutputID4, RedumpMatch, RedumpName, InputSize, OutputSize, FullSize, InputFilename, OutputFilename, MD5, SHA1, Passes, SecondsElapsed, ErrorMessage
This file should be used to evaluate the results of a large conversion before deleting source files. The data values are tab separated for pasting in to your favourite spreadsheet app.
Conversions
NKit can convert to and from various formats. To simplify recovering and converting to and from the formats a multi pass system will process the image through the various stages.
Configuration
The NKit.dll uses a configuration file name NKit.dll.config. This file contains base configuration. There are multiple sections or which the first 3 contain items the user will be most interested in editing:-
-
<appSettings>- General settings -
<gamecube>- GameCube Redump and recovery settings. Including some lookup data -
<wii>- Wii Redump and recovery settings. Including some lookup data
AppSettings
| Item | Description |
|---|---|
| SearchSubfolders | Search sub folders when finding images to process |
| WaitForKeyAfterProcessing | Wait for newline entry before exiting a commandline app |
| Path | All paths are relative to the working folder - global substitutions %exe=exe path, %pth=Path value. A base path that can be used in other paths with %pth, paths can be absolute or relative to the working folder - not supported by apps above |
| TempPath | Temp working folder, absolute or relative to current working directory - best to be the same drive as the output |
| SummaryLog | Summary log path, a log of all conversions with image details - check this file before deleting source files etc |
| EnableSummaryLog | Turn the SummaryLog on or off |
| FullVerify | Verify the output file by converting it back to iso in memory to ensure the CRC is valid |
| CalculateHashes | Calculates additional MD5 and SHA1 hashes. This is required to output them in to the summary log or use them in the rename file masks |
| NkitFormat | Save NKit as iso or gcz |
| NkitReencode | Force a full reencode when converting nkit to nkit. If false when converting nkit.iso to nkit.gcz, only the gcz coversion is performed |
| MaskRename | Rename output images to the masks in the gamecube and wii config sections, else use the source path and filename |
| TestMode | Convert images, but delete them when complete - View SummaryLog for results |
| RecoveryMatchFailDelete | Recovery Only - delete failed matches, false will rename them to the FailMatchRenameToMask - they may be corrupt |
| DeleteSource | Delete the source file if FullVerify returns VerifySuccess (FullVerify setting must be true) TestMode=true cancels this setting |
| OutputLevel | Logging level when processing 0=Silent, 1=Info, 2=Detailed |
| DatPathNameGameTdbMask | Path and mask to GameTDB file - latest file is used |
GameCube / Wii
GameCube and Wii share a log of the same settings. Depending on which image is being processed the corresponding section is referenced.
| Item | Description |
|---|---|
| DatPathRedumpMask | Path and mask to redump dat file - latest file is used |
| DatPathCustomMask | Path and mask to custom dat file - latest file is used |
| RedumpMatchRenameToMask | Used when an output file's CRC matches and entry in the Redump dat. Move and Rename the file to this mask |
| CustomMatchRenameToMask | Used when an output file's CRC matches and entry in the Custom dat. Move and Rename the file to this mask |
| MatchFailRenameToMask | Used when an output file's CRC fails to match any dat. Move and Rename the file to this mask |
| RecoveryFilesPath | Path to Redump recovery files. Recovery extracted files are stored here if the file's CRC is known |
| OtherRecoveryFilesPath | Path to Other recovery files. Recovery extracted files are stored here if the file's CRC is not known |
| NkitUpdatePartitionRemoval* | Remove the update partition when converting to nkit, the CRC of the removed partition is stored in the NKit header for restoration |
| NkitRecoveryFilesPath* | Path to Wii NKit extracted update partition when extracted by ConvertToNKit with NkitUpdatePartitionRemoval set to true |
* Wii section only
Getting Started
The apps will work out of the box, however some needs to take place to set up the Redump matching and Recovery files if you're looking to recover images to match Redump Dat entries. The previous sections cover the features mentioned in this section.
- Download the Dat files from Redump and place in the Dats folders
- Download the GameTDB dat and place in Dats\GameTdb folder
- Place any Recovery files in the Recovery\Redump folders
- Edit the NKit.dll.config file
Configuration Options
Wii - Smallest
There are a few factors to consider. The table below shows the configurations that create the smallest Wii images from Smallest to Largest. The relevant configuration options are also listed.
Removed Update partitions are stored in the Redump Recovery folder if they don't already exist. If they are not known they are stored in the NKitExtracted Recovery folder.
| NKit Format | External Compression | Update Partition Removed | Configuration |
|---|---|---|---|
| nkit.iso | 7-zip | Yes | NkitFormat="iso", NkitUpdatePartitionRemoval="true"
|
| nkit.gcz | No | Yes | NkitFormat="gcz", NkitUpdatePartitionRemoval="true"
|
| nkit.iso | 7-zip | No | NkitFormat="iso", NkitUpdatePartitionRemoval="false"
|
| nkit.gcz | No | No | NkitFormat="gcz", NkitUpdatePartitionRemoval="false"
|
External compression must be applied after NKit has completed. GCZ cannot match 7-zip. GCZ is a block seekable compression that is supported by Dolphin.
