From WikiTemp, the GBAtemp wiki
(→‎NKit Apps)
(→‎NKit Format)
 
(55 intermediate revisions by the same user not shown)
Line 2: Line 2:
 
Welcome to the NKit user guide
 
Welcome to the NKit user guide
  
−
This guide aims to cover NKit's features and options from a user point of view. Please refer to the Glossary and FAQ if something isn't clear. Feel free to ask anything else in the GBA Temp NKit App thread. Thanks for reading.
+
This guide aims to cover NKit's features and options from a user point of view. Please refer to the [[NKit/Glossary|Glossary]] and [[NKit/FAQ|FAQ]] if something isn't clear. Feel free to ask anything else in the GBA Temp NKit App thread. Thanks for reading.
  
 
==Key Uses==
 
==Key Uses==
−
There are two key used of NKit:
+
There are two key uses of NKit:
−
* Recovery: Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
+
* '''Recovery''': Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
−
* Convert: Converts anything to the NKit Format and preserves the data so that it can be restored back to the source iso
+
* '''Convert''': Converts anything as is. When converting to the NKit Format the source data is preserved so that it can be restored back to the source iso
  
 
==NKit Apps==
 
==NKit Apps==
−
There are 2 types of apps, command line and GUI. All apps use the NKit.dll.  
+
There are 2 types of apps, command line and GUI. All apps use the NKit.dll.
  
−
  Note for Devs: The NKit.dll contains all the processing logic. Reference it to add NKit processing to any apps. The apps below all contain very little code and are simple examples how to use it.
+
  Note for Devs: The NKit.dll contains all the processing logic. The apps below contain very little code and demonstrate how to use it.
  
 
===Command Line Apps===
 
===Command Line Apps===
 
* '''ConvertToISO.exe''' - converts a source file to iso. This can be wbfs (scrubbing is preserved), iso.dec (junk is recreated) or NKit [iso/gcz] (source data is restored back to iso)
 
* '''ConvertToISO.exe''' - converts a source file to iso. This can be wbfs (scrubbing is preserved), iso.dec (junk is recreated) or NKit [iso/gcz] (source data is restored back to iso)
−
* '''ConvertToNKit.exe''' - preserves the source image data as nkit [iso/gcz]. This can also be used to preserve scrubbed, hacked or bad dumps etc. For WBFS and iso.dec when restoring back to iso with ConvertToIso.exe the iso will contain all data that was in the original WBFS / iso.dec.
+
* '''ConvertToNKit.exe''' - preserves the source image data as nkit [iso/gcz]. This can also be used to preserve scrubbed, hacked or bad dumps etc. For WBFS and iso.dec when restoring back to iso with ConvertToIso.exe the iso will contain all data that was in the original WBFS / iso.dec (just as .iso)
−
* '''RecoverToISO.exe''' - attempts to recover GameCube and Wii images to .iso
+
* '''RecoverToISO.exe''' - attempts to recover GameCube and Wii images to ISO
 
* '''RecoverToNKit.exe''' - attempts to recover GameCube and Wii images and convert to nkit.iso or nkit.gcz
 
* '''RecoverToNKit.exe''' - attempts to recover GameCube and Wii images and convert to nkit.iso or nkit.gcz
 
* '''RecoveryExtract.exe''' - saves recovery files from images. CRC's in The NKit.dll.config are used to save extracted files in to a 'Redump' or 'Other' folder.
 
* '''RecoveryExtract.exe''' - saves recovery files from images. CRC's in The NKit.dll.config are used to save extracted files in to a 'Redump' or 'Other' folder.
 +
 +
All commandline apps have the same usage. A list of space separated files and masks. Paths can be relative or absolute. The apps will scan all inputs for files and collate them into a unique set. They are processed for multipart files (rars, split etc) then sorted and processed.
 +
 +
=====Examples=====
 +
{| class="wikitable"
 +
|-
 +
! Example !! Description
 +
|-
 +
| <code>RecoverToNKit test\*.*</code> || Relative path, all files
 +
|-
 +
| <code>RecoverToNKit c:\test\*.*</code> ||  Absolute path, all files
 +
|-
 +
| <code>RecoverToNKit "test path\*.*"</code> ||  Path with spaces
 +
|-
 +
| <code>RecoverToNKit "backup image.iso"</code> ||  Specific file with spaces
 +
|-
 +
| <code>RecoverToNKit *.r??</code> ||  All rar files (including parts r00, r01 etc)
 +
|-
 +
| <code>RecoverToNKit *.wbf?</code> ||  All wbfs files (including wbfs + wbf1)
 +
|-
 +
| <code>RecoverToNKit path</code> ||  All files in a relative folder
 +
|-
 +
| <code>RecoverToNKit path\*.*</code> ||  All files in a relative folder:
 +
|}
  
 
===GUI Apps===
 
===GUI Apps===
 
* '''NKitExtractionApp.exe''' - extract recovery files or extract files from the image filesystems
 
* '''NKitExtractionApp.exe''' - extract recovery files or extract files from the image filesystems
−
* '''NKitProcessingApp.exe''' - recover and convert files to iso / nkit [iso/gcz]. It has a drop down of option that performs the tasks of the first 4 commandline apps above
+
* '''NKitProcessingApp.exe''' - recover and convert files to iso / nkit [iso/gcz]. It has a drop down of Modes that performs the NKit conversions (covered by the first 4 command line apps above)
 +
 
 +
The configuration options in the GUI apps override the equivalent settings in the NKit.dll.config file.
 +
 
 +
This user guide refers to '''conversions''' throughout which are types of processing offered by NKit. The command line apps refer to them - ConvertToISO, ConvertToNKit, RecoverToISO and RecoverToNKit.
 +
 
 +
==Feature Overview==
 +
The features in this section are used across the conversions in NKit.
 +
 
 +
====NKit Format====
 +
ConvertToNKit and RecoverToNKit will output to the [[NKit/NKitFormat|NKit Format]]. This format is the smallest GameCube and Wii image format. Both GameCube and Wii images are playable in Dolphin, but only GameCube is supported by hardware.
 +
 
 +
The NKit Format is non-lossy and supports clean / Redump images as well as scrubbed and hacked images. Some corrupt and bad images are supported too, although these can error due to invalid fst.bin modifications.
 +
 
 +
There are 2 NKit output formats, raw (iso) and compressed (gcz). Both were designed with 1:1 preservation, size and playability in mind. Hardware support for Wii was sacrificed for smaller image sizes. If hardware support is required then WBFS is currently the best available and recoverable to Redump where not hacked or corrupt.
 +
 
 +
{| class="wikitable"
 +
|-
 +
! System !! Format !! Hardware Supported !! Dolphin Supported !! Restorable 1:1 !! Notes
 +
|-
 +
| GameCube || nkit.iso || Yes || Yes || Yes || Same as compacted GameCube iso
 +
|-
 +
| GameCube || nkit.gcz || No || Yes || Yes || GCZ is Dolphin's own block seekable compression format
 +
|-
 +
| Wii || nkit.iso || No || Yes || Yes || RVT-H format only playable in Dolphin
 +
|-
 +
| Wii || nkit.gcz || No || Yes || Yes || RVT-H in GCZ playable in Dolphin only
 +
|}
 +
 
 +
NKit Format images contain the bare minimum of data. All junk and scrubbing is removed. Non-uniform (NonJunk) data is also preserved. Wii encryption and hashes are fully recreatable and removed. Meaning any remaining data is as compressible as possible.
 +
 
 +
=====Dat Files=====
 +
NKit relies heavily on Dat files. The Redump Dat files (GameCube and Wii) hold the checksums of all the good images. NKit uses this information to help recover from modified images. Without these files NKit can only unscrub images and can't know if they're modified in other ways.
 +
 
 +
Multiple Dat files can be placed in the folders. NKit will load the one with latest modified date
 +
 
 +
=====Mask Renaming Output Files=====
 +
Output images that match Redump or Other Dat file entries can be renamed to a mask. The masks allow processed images to be moved in to separate folder for Redump Matched, Custom Matched or Match Fail.
 +
 
 +
There are various masks that can be used in the renaming:
 +
{| class="wikitable"
 +
|-
 +
! Item !! Name !! Comment
 +
|-
 +
| <code>%src</code> || Source image path || Path of the source image
 +
|-
 +
| <code>%nmo</code> || Original filename ||
 +
|-
 +
| <code>%nmd</code> || Disc Title || Read from the header of the disc
 +
|-
 +
| <code>%nmg</code> || GameTDB Title || Title from the GameTDB dat file
 +
|-
 +
| <code>%nmm</code> || Matched Name || Redump or Custom Dat matched entry - falls back to %nmg then %nmo
 +
|-
 +
| <code>%dno</code> || Disc No. || E.g. (Disc 2) - Only added if not in the filename already. Not output if (Disc 1)
 +
|-
 +
| <code>%rev</code> || Revision No. || E.g. (Rev 1) - Only added if not in the filename already. Not output if (Rev 0)
 +
|-
 +
| <code>%ver</code> || Revision No. || E.g. (v1.01) - Only added if not in the filename already. Not output if (v1.00)
 +
|-
 +
| <code>%ext</code> || File Extension || Does not contain leading .
 +
|-
 +
| <code>%crc</code> || CRC32 || Output file CRC32
 +
|-
 +
| <code>%md5</code> || MD5 || Only calculated if the config is set to Calculate Hashes
 +
|-
 +
| <code>%sha</code> || SHA1 || Only calculated if the config is set to Calculate Hashes
 +
|-
 +
| <code>%id4</code> || Disc ID4 || The Disc ID
 +
|-
 +
| <code>%id6</code> || Disc ID6 || The Disc ID and Maker Code
 +
|-
 +
| <code>%id8</code> || Disc ID8 || The Disc ID, MakerCode, Disc No. and Disc Revision (Disc No and Revision are as Hex bytes e.g. 0000)
 +
|}
 +
 
 +
An example of a working mask is <code>%pth\GameCube\%nmm.%ext</code> you may choose to use something like <code>%pth\NewFolder\%nmm %id8 %crc.%ext</code>. This will add <code>[]</code> around the values.
 +
 
 +
=====Recovery Files=====
 +
The concept of recovery files is that by saving the bare minimum of information from Redump good images. NKit can recover many modified images.
 +
* '''GameCube''' other tools that modify these images scrub and move (sometimes reorder) the files in the filesystem. NKit recovery files are the appldr.bin (a handful are used across all images) and fst.bin (unique per image, NKit stores some header info in these files also).
 +
* '''Wii''' - other tools remove unused blocks or scrub them by wiping unused blocks with 00 or FF bytes. It is common for Non-Data Partitions are also removed to save space. NKit recovery files consist of the channel and update partitions.
 +
 
 +
NKit can extract the recovery files from images. Running this on a set of random Wii images may extract Update partitions that can be used to recover other images that have had them removed. The update partitions are shared across images. There's only a handful of unique ones.
 +
 
 +
=====Brute Force Recovery=====
 +
In addition to the recovery files, NKit can attempt to force a match by checking various Header CRCs and Update Partition CRCs (for Wii only). Once a match is found the modifications can be applied to the output file and the CRC will match the Dat entry.
 +
 
 +
NKit does this by CRCing sections of files as it processes them. Upon completing a conversion other CRCs calculated combined to attempt to make a match. Header modifications include Region and for Wii - swapping the Update partition CRC and using the Data Partition data-header for the disc header.
 +
 
 +
=====Summary Log=====
 +
The summary log is an output file used to track conversions. It holds stores the following columns:
 +
TimeStamp, App, System, ReadResult, OutputResult, OutputCrc, OutputID4, RedumpMatch, RedumpName, InputSize, OutputSize, FullSize, InputFilename, OutputFilename, MD5, SHA1, Passes, SecondsElapsed, ErrorMessage
 +
 
 +
This file should be used to evaluate the results of a large conversion before deleting source files. The data values are tab separated for pasting in to your favourite spreadsheet app.
 +
 
 +
==Conversions==
 +
NKit can convert to and from various formats. To simplify recovering and converting to and from the formats a multi pass system will process the image through the various stages.
 +
 
 +
[[File:NKitConversionsKey.jpg|right]]
 +
[[File:NKitConversions.jpg|framed|center]]
 +
 
 +
==Configuration==
 +
The NKit.dll uses a configuration file name NKit.dll.config. This file contains base configuration. There are multiple sections or which the first 3 contain items the user will be most interested in editing:-
 +
* <code><appSettings></code> - General settings
 +
* <code><gamecube></code> - GameCube Redump and recovery settings. Including some lookup data
 +
* <code><wii></code> - Wii Redump and recovery settings. Including some lookup data
 +
 
 +
====AppSettings====
 +
{| class="wikitable"
 +
|-
 +
! Item !! Description
 +
|-
 +
| SearchSubfolders || Search sub folders when finding images to process
 +
|-
 +
| WaitForKeyAfterProcessing || Wait for newline entry before exiting a commandline app
 +
|-
 +
| Path || All paths are relative to the working folder - global substitutions %exe=exe path, %pth=Path value. A base path that can be used in other paths with %pth, paths can be absolute or relative to the working folder - not supported by apps above
 +
|-
 +
| TempPath || Temp working folder, absolute or relative to current working directory - best to be the same drive as the output
 +
|-
 +
| SummaryLog || Summary log path, a log of all conversions with image details - check this file before deleting source files etc
 +
|-
 +
| EnableSummaryLog || Turn the SummaryLog on or off
 +
|-
 +
| FullVerify || Verify the output file by converting it back to iso in memory to ensure the CRC is valid
 +
|-
 +
| CalculateHashes || Calculates additional MD5 and SHA1 hashes. This is required to output them in to the summary log or use them in the rename file masks
 +
|-
 +
| NkitFormat || Save NKit as iso or gcz
 +
|-
 +
| NkitReencode || Force a full reencode when converting nkit to nkit. If false when converting nkit.iso to nkit.gcz, only the gcz coversion is performed
 +
|-
 +
| MaskRename || Rename output images to the masks in the gamecube and wii config sections, else use the source path and filename
 +
|-
 +
| TestMode || Convert images, but delete them when complete - View SummaryLog for results
 +
|-
 +
| RecoveryMatchFailDelete || Recovery Only - delete failed matches, false will rename them to the FailMatchRenameToMask - they may be corrupt
 +
|-
 +
| DeleteSource || Delete the source file if FullVerify returns VerifySuccess (FullVerify setting must be true) TestMode=true cancels this setting
 +
|-
 +
| OutputLevel || Logging level when processing 0=Silent, 1=Info, 2=Detailed
 +
|-
 +
| DatPathNameGameTdbMask || Path and mask to GameTDB file - latest file is used
 +
|}
 +
====GameCube / Wii====
 +
GameCube and Wii share a log of the same settings. Depending on which image is being processed the corresponding section is referenced.
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Item !! Description
 +
|-
 +
| DatPathRedumpMask || Path and mask to redump dat file - latest file is used
 +
|-
 +
| DatPathCustomMask || Path and mask to custom dat file - latest file is used
 +
|-
 +
| RedumpMatchRenameToMask || Used when an output file's CRC matches and entry in the Redump dat. Move and Rename the file to this mask
 +
|-
 +
| CustomMatchRenameToMask || Used when an output file's CRC matches and entry in the Custom dat. Move and Rename the file to this mask
 +
|-
 +
| MatchFailRenameToMask || Used when an output file's CRC fails to match any dat. Move and Rename the file to this mask
 +
|-
 +
| RecoveryFilesPath || Path to Redump recovery files. Recovery extracted files are stored here if the file's CRC is known
 +
|-
 +
| OtherRecoveryFilesPath || Path to Other recovery files. Recovery extracted files are stored here if the file's CRC is not known
 +
|-
 +
| NkitUpdatePartitionRemoval'''*''' || Remove the update partition when converting to nkit, the CRC of the removed partition is stored in the NKit header for restoration
 +
|-
 +
| NkitRecoveryFilesPath'''*''' || Path to Wii NKit extracted update partition when extracted by ConvertToNKit with NkitUpdatePartitionRemoval set to ''true''
 +
|}
 +
'''*''' Wii section only
 +
 
 +
==Getting Started==
 +
 
 +
The apps will work out of the box and a default configuration is provided, but in order to Recover images and match to Redump you will need to do the following:-
 +
# Download the Dat files from Redump and place in the Dats folders
 +
# Download the GameTDB dat and place in Dats\GameTdb folder
 +
# Place any Recovery files in the Recovery\Redump folders
 +
# Edit the NKit.dll.config file
 +
 
 +
===Base Configuration===
 +
You may prefer to change the default configuration before using the apps. The following sections highlight some alternative configurations.
 +
 
 +
====Smallest Image Output Configurations====
 +
{| class="wikitable"
 +
|-
 +
! NKit Format !! External Compression !! Configuration
 +
|-
 +
| nkit.iso || 7-zip || <code>NkitFormat="iso"</code>
 +
|-
 +
| nkit.gcz || No || <code>NkitFormat="gcz"</code>
 +
|}
 +
 
 +
====Smallest Wii Output Configurations====
 +
 
 +
There are a few factors to consider. The table below shows the configurations that create the smallest Wii images from Smallest to Largest. The relevant configuration options are also listed.
 +
 
 +
Removed Update partitions are stored in the Redump Recovery folder if they don't already exist. If they are not known they are stored in the NKitExtracted Recovery folder. The Removed Update Partition CRC is stored in the NKit header so it can be safely restored.
 +
 
 +
{| class="wikitable"
 +
|-
 +
! NKit Format !! External Compression !! Update Partition Removed !! Configuration
 +
|-
 +
| nkit.iso || 7-zip || Yes || <code>NkitFormat="iso", NkitUpdatePartitionRemoval="true"</code>
 +
|-
 +
| nkit.gcz || No || Yes || <code>NkitFormat="gcz", NkitUpdatePartitionRemoval="true"</code>
 +
|-
 +
| nkit.iso || 7-zip || No || <code>NkitFormat="iso", NkitUpdatePartitionRemoval="false"</code>
 +
|-
 +
| nkit.gcz || No || No || <code>NkitFormat="gcz", NkitUpdatePartitionRemoval="false"</code>
 +
|}
 +
External compression must be applied after NKit has completed. GCZ cannot match 7-zip. GCZ is a block seekable compression that is supported by Dolphin.
 +
 
 +
====Convert To NKit and Delete Source====
 +
 
 +
* <code>FullVerify="true"</code> - This must be enabled and return VerifySuccess
 +
* <code>DeleteSource="true"</code> - This is defaulted to false, Test Mode cancels this
 +
 
 +
====Convert To NKit and Audit The Results====
 +
 
 +
* <code>TestMode="true"</code> - Process and remove the output file
 +
* <code>EnableSummaryLog="true"</code> - Audit the results of the files processed
 +
* <code>MaskRename="true"</code> - Audit which Dat entry matches
 +
* <code>CalculateHashes="true"</code> - true if auditing Summary to contain the MD5 and SHA1 of the output file
 +
* <code>FullVerify="true"</code> - true if the auditing should verify the output CRC is correct

Latest revision as of 10:54, 20 September 2019

Welcome to the NKit user guide

This guide aims to cover NKit's features and options from a user point of view. Please refer to the Glossary and FAQ if something isn't clear. Feel free to ask anything else in the GBA Temp NKit App thread. Thanks for reading.

Key Uses

There are two key uses of NKit:

  • Recovery: Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
  • Convert: Converts anything as is. When converting to the NKit Format the source data is preserved so that it can be restored back to the source iso

NKit Apps

There are 2 types of apps, command line and GUI. All apps use the NKit.dll. 

Note for Devs: The NKit.dll contains all the processing logic. The apps below contain very little code and demonstrate how to use it.

Command Line Apps

  • ConvertToISO.exe - converts a source file to iso. This can be wbfs (scrubbing is preserved), iso.dec (junk is recreated) or NKit [iso/gcz] (source data is restored back to iso)
  • ConvertToNKit.exe - preserves the source image data as nkit [iso/gcz]. This can also be used to preserve scrubbed, hacked or bad dumps etc. For WBFS and iso.dec when restoring back to iso with ConvertToIso.exe the iso will contain all data that was in the original WBFS / iso.dec (just as .iso)
  • RecoverToISO.exe - attempts to recover GameCube and Wii images to ISO
  • RecoverToNKit.exe - attempts to recover GameCube and Wii images and convert to nkit.iso or nkit.gcz
  • RecoveryExtract.exe - saves recovery files from images. CRC's in The NKit.dll.config are used to save extracted files in to a 'Redump' or 'Other' folder.

All commandline apps have the same usage. A list of space separated files and masks. Paths can be relative or absolute. The apps will scan all inputs for files and collate them into a unique set. They are processed for multipart files (rars, split etc) then sorted and processed.

Examples
Example Description
RecoverToNKit test\*.* Relative path, all files
RecoverToNKit c:\test\*.* Absolute path, all files
RecoverToNKit "test path\*.*" Path with spaces
RecoverToNKit "backup image.iso" Specific file with spaces
RecoverToNKit *.r?? All rar files (including parts r00, r01 etc)
RecoverToNKit *.wbf? All wbfs files (including wbfs + wbf1)
RecoverToNKit path All files in a relative folder
RecoverToNKit path\*.* All files in a relative folder:

GUI Apps

  • NKitExtractionApp.exe - extract recovery files or extract files from the image filesystems
  • NKitProcessingApp.exe - recover and convert files to iso / nkit [iso/gcz]. It has a drop down of Modes that performs the NKit conversions (covered by the first 4 command line apps above)

The configuration options in the GUI apps override the equivalent settings in the NKit.dll.config file. 

This user guide refers to conversions throughout which are types of processing offered by NKit. The command line apps refer to them - ConvertToISO, ConvertToNKit, RecoverToISO and RecoverToNKit.

Feature Overview

The features in this section are used across the conversions in NKit.

NKit Format

ConvertToNKit and RecoverToNKit will output to the NKit Format. This format is the smallest GameCube and Wii image format. Both GameCube and Wii images are playable in Dolphin, but only GameCube is supported by hardware.

The NKit Format is non-lossy and supports clean / Redump images as well as scrubbed and hacked images. Some corrupt and bad images are supported too, although these can error due to invalid fst.bin modifications.

There are 2 NKit output formats, raw (iso) and compressed (gcz). Both were designed with 1:1 preservation, size and playability in mind. Hardware support for Wii was sacrificed for smaller image sizes. If hardware support is required then WBFS is currently the best available and recoverable to Redump where not hacked or corrupt.

System Format Hardware Supported Dolphin Supported Restorable 1:1 Notes
GameCube nkit.iso Yes Yes Yes Same as compacted GameCube iso
GameCube nkit.gcz No Yes Yes GCZ is Dolphin's own block seekable compression format
Wii nkit.iso No Yes Yes RVT-H format only playable in Dolphin
Wii nkit.gcz No Yes Yes RVT-H in GCZ playable in Dolphin only

NKit Format images contain the bare minimum of data. All junk and scrubbing is removed. Non-uniform (NonJunk) data is also preserved. Wii encryption and hashes are fully recreatable and removed. Meaning any remaining data is as compressible as possible.

Dat Files

NKit relies heavily on Dat files. The Redump Dat files (GameCube and Wii) hold the checksums of all the good images. NKit uses this information to help recover from modified images. Without these files NKit can only unscrub images and can't know if they're modified in other ways.

Multiple Dat files can be placed in the folders. NKit will load the one with latest modified date

Mask Renaming Output Files

Output images that match Redump or Other Dat file entries can be renamed to a mask. The masks allow processed images to be moved in to separate folder for Redump Matched, Custom Matched or Match Fail.

There are various masks that can be used in the renaming:

Item Name Comment
%src Source image path Path of the source image
%nmo Original filename
%nmd Disc Title Read from the header of the disc
%nmg GameTDB Title Title from the GameTDB dat file
%nmm Matched Name Redump or Custom Dat matched entry - falls back to %nmg then %nmo
%dno Disc No. E.g. (Disc 2) - Only added if not in the filename already. Not output if (Disc 1)
%rev Revision No. E.g. (Rev 1) - Only added if not in the filename already. Not output if (Rev 0)
%ver Revision No. E.g. (v1.01) - Only added if not in the filename already. Not output if (v1.00)
%ext File Extension Does not contain leading .
%crc CRC32 Output file CRC32
%md5 MD5 Only calculated if the config is set to Calculate Hashes
%sha SHA1 Only calculated if the config is set to Calculate Hashes
%id4 Disc ID4 The Disc ID
%id6 Disc ID6 The Disc ID and Maker Code
%id8 Disc ID8 The Disc ID, MakerCode, Disc No. and Disc Revision (Disc No and Revision are as Hex bytes e.g. 0000)

An example of a working mask is %pth\GameCube\%nmm.%ext you may choose to use something like %pth\NewFolder\%nmm %id8 %crc.%ext. This will add [] around the values.

Recovery Files

The concept of recovery files is that by saving the bare minimum of information from Redump good images. NKit can recover many modified images.

  • GameCube other tools that modify these images scrub and move (sometimes reorder) the files in the filesystem. NKit recovery files are the appldr.bin (a handful are used across all images) and fst.bin (unique per image, NKit stores some header info in these files also).
  • Wii - other tools remove unused blocks or scrub them by wiping unused blocks with 00 or FF bytes. It is common for Non-Data Partitions are also removed to save space. NKit recovery files consist of the channel and update partitions.

NKit can extract the recovery files from images. Running this on a set of random Wii images may extract Update partitions that can be used to recover other images that have had them removed. The update partitions are shared across images. There's only a handful of unique ones.

Brute Force Recovery

In addition to the recovery files, NKit can attempt to force a match by checking various Header CRCs and Update Partition CRCs (for Wii only). Once a match is found the modifications can be applied to the output file and the CRC will match the Dat entry.

NKit does this by CRCing sections of files as it processes them. Upon completing a conversion other CRCs calculated combined to attempt to make a match. Header modifications include Region and for Wii - swapping the Update partition CRC and using the Data Partition data-header for the disc header.

Summary Log

The summary log is an output file used to track conversions. It holds stores the following columns: 

TimeStamp, App, System, ReadResult, OutputResult, OutputCrc, OutputID4, RedumpMatch, RedumpName, InputSize, OutputSize, FullSize, InputFilename, OutputFilename, MD5, SHA1, Passes, SecondsElapsed, ErrorMessage

This file should be used to evaluate the results of a large conversion before deleting source files. The data values are tab separated for pasting in to your favourite spreadsheet app.

Conversions

NKit can convert to and from various formats. To simplify recovering and converting to and from the formats a multi pass system will process the image through the various stages.

NKitConversionsKey.jpg
NKitConversions.jpg

Configuration

The NKit.dll uses a configuration file name NKit.dll.config. This file contains base configuration. There are multiple sections or which the first 3 contain items the user will be most interested in editing:-

  • <appSettings> - General settings
  • <gamecube> - GameCube Redump and recovery settings. Including some lookup data
  • <wii> - Wii Redump and recovery settings. Including some lookup data

AppSettings

Item Description
SearchSubfolders Search sub folders when finding images to process
WaitForKeyAfterProcessing Wait for newline entry before exiting a commandline app
Path All paths are relative to the working folder - global substitutions %exe=exe path, %pth=Path value. A base path that can be used in other paths with %pth, paths can be absolute or relative to the working folder - not supported by apps above
TempPath Temp working folder, absolute or relative to current working directory - best to be the same drive as the output
SummaryLog Summary log path, a log of all conversions with image details - check this file before deleting source files etc
EnableSummaryLog Turn the SummaryLog on or off
FullVerify Verify the output file by converting it back to iso in memory to ensure the CRC is valid
CalculateHashes Calculates additional MD5 and SHA1 hashes. This is required to output them in to the summary log or use them in the rename file masks
NkitFormat Save NKit as iso or gcz
NkitReencode Force a full reencode when converting nkit to nkit. If false when converting nkit.iso to nkit.gcz, only the gcz coversion is performed
MaskRename Rename output images to the masks in the gamecube and wii config sections, else use the source path and filename
TestMode Convert images, but delete them when complete - View SummaryLog for results
RecoveryMatchFailDelete Recovery Only - delete failed matches, false will rename them to the FailMatchRenameToMask - they may be corrupt
DeleteSource Delete the source file if FullVerify returns VerifySuccess (FullVerify setting must be true) TestMode=true cancels this setting
OutputLevel Logging level when processing 0=Silent, 1=Info, 2=Detailed
DatPathNameGameTdbMask Path and mask to GameTDB file - latest file is used

GameCube / Wii

GameCube and Wii share a log of the same settings. Depending on which image is being processed the corresponding section is referenced.

Item Description
DatPathRedumpMask Path and mask to redump dat file - latest file is used
DatPathCustomMask Path and mask to custom dat file - latest file is used
RedumpMatchRenameToMask Used when an output file's CRC matches and entry in the Redump dat. Move and Rename the file to this mask
CustomMatchRenameToMask Used when an output file's CRC matches and entry in the Custom dat. Move and Rename the file to this mask
MatchFailRenameToMask Used when an output file's CRC fails to match any dat. Move and Rename the file to this mask
RecoveryFilesPath Path to Redump recovery files. Recovery extracted files are stored here if the file's CRC is known
OtherRecoveryFilesPath Path to Other recovery files. Recovery extracted files are stored here if the file's CRC is not known
NkitUpdatePartitionRemoval* Remove the update partition when converting to nkit, the CRC of the removed partition is stored in the NKit header for restoration
NkitRecoveryFilesPath* Path to Wii NKit extracted update partition when extracted by ConvertToNKit with NkitUpdatePartitionRemoval set to true

* Wii section only

Getting Started

The apps will work out of the box and a default configuration is provided, but in order to Recover images and match to Redump you will need to do the following:-

  1. Download the Dat files from Redump and place in the Dats folders
  2. Download the GameTDB dat and place in Dats\GameTdb folder
  3. Place any Recovery files in the Recovery\Redump folders
  4. Edit the NKit.dll.config file

Base Configuration

You may prefer to change the default configuration before using the apps. The following sections highlight some alternative configurations.

Smallest Image Output Configurations

NKit Format External Compression Configuration
nkit.iso 7-zip NkitFormat="iso"
nkit.gcz No NkitFormat="gcz"

Smallest Wii Output Configurations

There are a few factors to consider. The table below shows the configurations that create the smallest Wii images from Smallest to Largest. The relevant configuration options are also listed. 

Removed Update partitions are stored in the Redump Recovery folder if they don't already exist. If they are not known they are stored in the NKitExtracted Recovery folder. The Removed Update Partition CRC is stored in the NKit header so it can be safely restored.
NKit Format External Compression Update Partition Removed Configuration
nkit.iso 7-zip Yes NkitFormat="iso", NkitUpdatePartitionRemoval="true"
nkit.gcz No Yes NkitFormat="gcz", NkitUpdatePartitionRemoval="true"
nkit.iso 7-zip No NkitFormat="iso", NkitUpdatePartitionRemoval="false"
nkit.gcz No No NkitFormat="gcz", NkitUpdatePartitionRemoval="false"

External compression must be applied after NKit has completed. GCZ cannot match 7-zip. GCZ is a block seekable compression that is supported by Dolphin.

Convert To NKit and Delete Source

  • FullVerify="true" - This must be enabled and return VerifySuccess
  • DeleteSource="true" - This is defaulted to false, Test Mode cancels this

Convert To NKit and Audit The Results

  • TestMode="true" - Process and remove the output file
  • EnableSummaryLog="true" - Audit the results of the files processed
  • MaskRename="true" - Audit which Dat entry matches
  • CalculateHashes="true" - true if auditing Summary to contain the MD5 and SHA1 of the output file
  • FullVerify="true" - true if the auditing should verify the output CRC is correct
Retrieved from "https://wiki.gbatemp.net/wiki/NKit/UserGuide?oldid=70755"