From WikiTemp, the GBAtemp wiki
(Brute Force Recovery)
(GUI Apps)
(2 intermediate revisions by the same user not shown)
Line 2: Line 2:
 
Welcome to the NKit user guide
 
Welcome to the NKit user guide
  
This guide aims to cover NKit's features and options from a user point of view. Please refer to the Glossary and FAQ if something isn't clear. Feel free to ask anything else in the GBA Temp NKit App thread. Thanks for reading.
+
This guide aims to cover NKit's features and options from a user point of view. Please refer to the [[NKit/Glossary|Glossary]] and [[NKit/FAQ|FAQ]] if something isn't clear. Feel free to ask anything else in the GBA Temp NKit App thread. Thanks for reading.
  
 
==Key Uses==
 
==Key Uses==
Line 48: Line 48:
 
* '''NKitExtractionApp.exe''' - extract recovery files or extract files from the image filesystems
 
* '''NKitExtractionApp.exe''' - extract recovery files or extract files from the image filesystems
 
* '''NKitProcessingApp.exe''' - recover and convert files to iso / nkit [iso/gcz]. It has a drop down of option that performs the tasks of the first 4 commandline apps above
 
* '''NKitProcessingApp.exe''' - recover and convert files to iso / nkit [iso/gcz]. It has a drop down of option that performs the tasks of the first 4 commandline apps above
 +
 +
The configuration options in the GUI apps override the equivalent settings in the NKit.dll.config file.
  
 
  This user guide refers to '''conversions''' throughout which are the names of the command line apps - ConvertToISO, ConvertToNKit, RecoverToISO and RecoverToNKit.
 
  This user guide refers to '''conversions''' throughout which are the names of the command line apps - ConvertToISO, ConvertToNKit, RecoverToISO and RecoverToNKit.
Line 172: Line 174:
 
| DatPathCustomMask || Path and mask to custom dat file - latest file is used
 
| DatPathCustomMask || Path and mask to custom dat file - latest file is used
 
|-
 
|-
| RedumpMatchRenameToMask || Used when an output file's CRC matches and entry in the Redump dat. Move and Rename the file to this mask.
+
| RedumpMatchRenameToMask || Used when an output file's CRC matches and entry in the Redump dat. Move and Rename the file to this mask
 
|-
 
|-
| CustomMatchRenameToMask || Used when an output file's CRC matches and entry in the Custom dat. Move and Rename the file to this mask.
+
| CustomMatchRenameToMask || Used when an output file's CRC matches and entry in the Custom dat. Move and Rename the file to this mask
 
|-
 
|-
| MatchFailRenameToMask || Used when an output file's CRC fails to match any dat. Move and Rename the file to this mask.
+
| MatchFailRenameToMask || Used when an output file's CRC fails to match any dat. Move and Rename the file to this mask
 
|-
 
|-
| RecoveryFilesPath || Path to Redump recovery files. Recovery extracted files are stored here if the file's CRC is known.
+
| RecoveryFilesPath || Path to Redump recovery files. Recovery extracted files are stored here if the file's CRC is known
 
|-
 
|-
| OtherRecoveryFilesPath || Path to Other recovery files. Recovery extracted files are stored here if the file's CRC is not known.
+
| OtherRecoveryFilesPath || Path to Other recovery files. Recovery extracted files are stored here if the file's CRC is not known
 
|-
 
|-
| NkitUpdatePartitionRemoval'''*''' || Remove the update partition when converting to nkit, the CRC of the removed partition is stored in the NKit header for restoration.
+
| NkitUpdatePartitionRemoval'''*''' || Remove the update partition when converting to nkit, the CRC of the removed partition is stored in the NKit header for restoration
 
|-
 
|-
 
| NkitRecoveryFilesPath'''*''' || Path to Wii NKit extracted update partition when extracted by ConvertToNKit with NkitUpdatePartitionRemoval set to ''true''
 
| NkitRecoveryFilesPath'''*''' || Path to Wii NKit extracted update partition when extracted by ConvertToNKit with NkitUpdatePartitionRemoval set to ''true''
 
|}
 
|}
 
'''*''' Wii section only
 
'''*''' Wii section only

Revision as of 00:11, 12 September 2019

Welcome to the NKit user guide

This guide aims to cover NKit's features and options from a user point of view. Please refer to the Glossary and FAQ if something isn't clear. Feel free to ask anything else in the GBA Temp NKit App thread. Thanks for reading.

Key Uses

There are two key used of NKit:

  • Recovery: Rebuild and repair images to match Redump verified images. This is aided by use of Recovery files that can be extracted from good sources. Recovery uses these files to rebuild the iso and repair modified disc headers, unscrub blocks, reinsert missing Wii partitions, move and reorder GameCube filesystems
  • Convert: Converts anything to the NKit Format and preserves the data so that it can be restored back to the source iso

NKit Apps

There are 2 types of apps, command line and GUI. All apps use the NKit.dll.

Note for Devs: The NKit.dll contains all the processing logic. Reference it to add NKit processing to any apps. The apps below all contain very little code and are simple examples how to use it.

Command Line Apps

  • ConvertToISO.exe - converts a source file to iso. This can be wbfs (scrubbing is preserved), iso.dec (junk is recreated) or NKit [iso/gcz] (source data is restored back to iso)
  • ConvertToNKit.exe - preserves the source image data as nkit [iso/gcz]. This can also be used to preserve scrubbed, hacked or bad dumps etc. For WBFS and iso.dec when restoring back to iso with ConvertToIso.exe the iso will contain all data that was in the original WBFS / iso.dec.
  • RecoverToISO.exe - attempts to recover GameCube and Wii images to .iso
  • RecoverToNKit.exe - attempts to recover GameCube and Wii images and convert to nkit.iso or nkit.gcz
  • RecoveryExtract.exe - saves recovery files from images. CRC's in The NKit.dll.config are used to save extracted files in to a 'Redump' or 'Other' folder.

All commandline apps have the same usage. A list of space separated files and masks. Paths can be relative or absolute. The apps will scan all inputs for files and collate them into a unique set. They are processed for multipart files (rars, split etc) then sorted and processed.

Examples (all standard command line)

Example Description
RecoverToNKit test\*.* Relative path, all files
RecoverToNKit c:\test\*.* Absolute path, all files
RecoverToNKit "test path\*.*" Path with spaces
RecoverToNKit "backup image.iso" Specific file with spaces
RecoverToNKit *.r?? All rar files (including parts r00, r01 etc)
RecoverToNKit *.wbf? All wbfs files (including wbfs + wbf1)
RecoverToNKit path All files in a relative folder
RecoverToNKit path\*.* All files in a relative folder:

GUI Apps

  • NKitExtractionApp.exe - extract recovery files or extract files from the image filesystems
  • NKitProcessingApp.exe - recover and convert files to iso / nkit [iso/gcz]. It has a drop down of option that performs the tasks of the first 4 commandline apps above

The configuration options in the GUI apps override the equivalent settings in the NKit.dll.config file.

This user guide refers to conversions throughout which are the names of the command line apps - ConvertToISO, ConvertToNKit, RecoverToISO and RecoverToNKit.

Feature Overview

The features in this section are used across the conversions in NKit. They are described here along with the configuration items that they use. The configuration items are listed in the section below.

Dat Files

NKit relies heavily on Dat files. The Redump Dat files (GameCube and Wii) hold the checksums of all the good images. NKit uses this information to help recover from modified images. Without these files NKit can only unscrub images and can't know if they're modified in other ways.

Mask Renaming Output Files

Images output that match Redump or Other Dat file entries can be renamed to a mask. The masks allow processed images to be moved in to separate folder for Redump Matched, Custom Matched or Match Fail.

There are various masks that can be used in the renaming:

Item Name Comment
%src Source image path Path of the source image
%nmo Original filename
%nmd Disc Title Read from the header of the disc
%nmg GameTDB Title Title from the GameTDB dat file
%nmm Matched Name Redump or Custom Dat matched entry - falls back to %nmg then %nmo
%dno Disc No. E.g. (Disc 2) - Only added if not in the filename already. Not output if (Disc 1)
%rev Revision No. E.g. (Rev 1) - Only added if not in the filename already. Not output if (Rev 0)
%ver Revision No. E.g. (v1.01) - Only added if not in the filename already. Not output if (v1.00)
%ext File Extension Does not contain leading .
%crc CRC32 Output file CRC32
%md5 MD5 Only calculated if the config is set to Calculate Hashes
%sha SHA1 Only calculated if the config is set to Calculate Hashes
%id4 Disc ID4 The Disc ID
%id6 Disc ID6 The Disc ID and Maker Code
%id8 Disc ID8 The Disc ID, MakerCode, Disc No. and Disc Revision (Disc No and Revision are as Hex bytes e.g. 0000)
Recovery Files

The concept of recovery files is that by saving the bare minimum of information from Redump good images. NKit can recover many modified images.

  • GameCube other tools that modify these images scrub and move (sometimes reorder) the files in the filesystem. NKit recovery files are the appldr.bin (a handful are used across all images) and fst.bin (unique per image, NKit stores some header info in these files also).
  • Wii - other tools remove unused blocks or scrub them by wiping unused blocks with 00 or FF bytes. It is common for Non-Data Partitions are also removed to save space. NKit recovery files consist of the channel and update partitions.
Brute Force Recovery

In addition to the recovery files, NKit can attempt to force a match by checking various Header CRCs and Update Partition CRCs (for Wii only). Once a match is found the modifications can be applied to the output file and the CRC will match the Dat entry.

NKit does this by CRCing sections of files as it processes them. Upon completing a conversion other CRCs calculated combined to attempt to make a match. Header modifications include Region and for Wii swapping the Update partition CRC and using the Data Partition data-header for the disc header.

Summary Log

The summary log is an output file used to track conversions. It holds stores the following columns:

TimeStamp, App, System, ReadResult, OutputResult, OutputCrc, OutputID4, RedumpMatch, RedumpName, InputSize, OutputSize, FullSize, InputFilename, OutputFilename, MD5, SHA1, Passes, SecondsElapsed, ErrorMessage

This file should be used to evaluate the results of a large conversion before deleting source files. The data values are tab separated for pasting in to your favourite spreadsheet app.

Conversions

NKit can convert to and from various formats. To simplify recovering and converting to and from the formats a multi pass system will process the image through the various stages.

NKitConversionsKey.jpg
NKitConversions.jpg

Configuration

The NKit.dll uses a configuration file name NKit.dll.config. This file contains base configuration. There are multiple sections or which the first 3 contain items the user will be most interested in editing:-

  • <appSettings> - General settings
  • <gamecube> - GameCube Redump and recovery settings. Including some lookup data
  • <wii> - Wii Redump and recovery settings. Including some lookup data

AppSettings

Item Description
SearchSubfolders Search sub folders when finding images to process
WaitForKeyAfterProcessing Wait for newline entry before exiting a commandline app
Path All paths are relative to the working folder - global substitutions %exe=exe path, %pth=Path value. A base path that can be used in other paths with %pth, paths can be absolute or relative to the working folder - not supported by apps above
TempPath Temp working folder, absolute or relative to current working directory - best to be the same drive as the output
SummaryLog Summary log path, a log of all conversions with image details - check this file before deleting source files etc
EnableSummaryLog Turn the SummaryLog on or off
FullVerify Verify the output file by converting it back to iso in memory to ensure the CRC is valid
CalculateHashes Calculates additional MD5 and SHA1 hashes. This is required to output them in to the summary log or use them in the rename file masks
NkitFormat Save NKit as iso or gcz
NkitReencode Force a full reencode when converting nkit to nkit. If false when converting nkit.iso to nkit.gcz, only the gcz coversion is performed
MaskRename Rename output images to the masks in the gamecube and wii config sections, else use the source path and filename
TestMode Convert images, but delete them when complete - View SummaryLog for results
RecoveryMatchFailDelete Recovery Only - delete failed matches, false will rename them to the FailMatchRenameToMask - they may be corrupt
DeleteSource Delete the source file if FullVerify returns VerifySuccess (FullVerify setting must be true) TestMode=true cancels this setting
OutputLevel Logging level when processing 0=Silent, 1=Info, 2=Detailed
DatPathNameGameTdbMask Path and mask to GameTDB file - latest file is used

GameCube / Wii

GameCube and Wii share a log of the same settings. Depending on which image is being processed the corresponding section is referenced.

Item Description
DatPathRedumpMask Path and mask to redump dat file - latest file is used
DatPathCustomMask Path and mask to custom dat file - latest file is used
RedumpMatchRenameToMask Used when an output file's CRC matches and entry in the Redump dat. Move and Rename the file to this mask
CustomMatchRenameToMask Used when an output file's CRC matches and entry in the Custom dat. Move and Rename the file to this mask
MatchFailRenameToMask Used when an output file's CRC fails to match any dat. Move and Rename the file to this mask
RecoveryFilesPath Path to Redump recovery files. Recovery extracted files are stored here if the file's CRC is known
OtherRecoveryFilesPath Path to Other recovery files. Recovery extracted files are stored here if the file's CRC is not known
NkitUpdatePartitionRemoval* Remove the update partition when converting to nkit, the CRC of the removed partition is stored in the NKit header for restoration
NkitRecoveryFilesPath* Path to Wii NKit extracted update partition when extracted by ConvertToNKit with NkitUpdatePartitionRemoval set to true

* Wii section only