Wednesday, December 09, 2015

How To Boot an NVME SSD from a Legacy BIOS (non UEFI) w/ Intel 750 Series SSD

This tutorial is about LEGACY BIOSes.

This is something i've been wanting to try for a very long time, since I have an older X58 motherboard, which is still my main system, and I'm using a Core i7 950 OC'ed to 3.8GHz.

EVERYONE said it couldn't be done. That is no longer true, because I have proven it myself!
It involves a program/bootloader called Tianocore, which has mostly been used for as part of "Clover" to boot Hackintoshes in EFI mode. But it will serve us so we can boot Windows in EFI mode, so the NVME drive is able to be detected at the boot stage, prior to Windows loading.

Background Info (scroll down to see solution, if you are lazy)

Only a UEFI compatible with NVME (something like UEFI version 2.3.1 and later) along with a "efi driver"/module will allow the computer to POST->BOOT->OS from an NVME drive.
There are other websites explaining how to mod your UEFI if you need to add this support into an older UEFI.

  • The Legacy BIOS sees the card as a generic undefined "SCSI Card" in the boot devices list, but this will not work.
  • The Windows 10 OS (what I used) - going back to Windows 8.1 (not 8) - come included with a "Standard NVME Controller driver" which lets Windows use the drive as a data drive ( D:\ for example).
  • The Windows 10 Installer sees the drive immediately but refuses to let you install to it because of a double-checking routine which queries the BIOS to make sure your system WOULD be bootable, so it doesn't waste your time installing to something impossible to boot from. (This is quite annoying in this case, but we can work around it by any number of ways, Cloning is one... Unrelatedly, also of note, this BIOS double-check also applies to Hot-plugging a SATA drive (for me anyway, as my BIOS has no options for supporting Hot Plug detection, even though the SATA Controllers do.)
  • The Intel 750 Series PCI-e NVME SSD has an Option ROM flash chip on the card. Dumping the ROM from linux gets us:

    Image 1:

      PCI Expansion ROM Header:

        Signature: 0x55aa (Ok)

        CPU unique data: 0x2b 0x00 0xf1 0x0e 0x00 0x00 0x0b 0x00

                         0x64 0x86 0x01 0x00 0x00 0x00 0x00 0x00

        Pointer to PCI Data Structure: 0x001c


      PCI Data Structure:

        Signature: 0x50434952 'PCIR' (Ok)

        Vendor ID: 0x8086

        Device ID: 0x0953
        Vital Product Data:  0x0000
        PCI Data Structure Length: 0x001c (28 bytes)
        PCI Data Structure Revision: 0x03
        Class Code: 0x000000 (Legacy Device)
        Image Length: 0x002b blocks (22016 bytes)
        Revision Level of Code/Data: 0x0000
        Code Type: 0x03 (Intel EFI (unofficial))
        Last-Image Flag: 0x80 (last image in rom)
        Reserved: 0x0000
      Parsing of platform specific data not available for this image

Some things to note: Class Code: Legacy Device, well, yes it is a Legacy device, you can use it just fine as long as you aren't booting to it. Code Type: Intel EFI. and then the last line is the most telling (especially when you compare it to a ROMdump of something like a network card) - reveals that this boot ROM is only applicable to EFI platforms (not BIOS). Therefore we cannot just mod/add the vendor device ID 8086_0953 to the BIOS tables to get it to boot.
Theoretically, if we could find a legacy BIOS based option ROM for this card, we could re-flash the card with that as a solution however no such thing exists and likely never will.

Some History on Windows, and the Boot Process

I thought by using a 2nd visible SATA HDD to initiate the MBR boot instruction, then calling BOOTMGR to obtain the List of Windows Installations (where you push F8 for safe mode etc) would be enough, but this also falls dead in its tracks.... The process goes like this: POST->BIOS->MBR->BOOTMGR->winload.exe->failure. Winload.exe is what loads NTOSKRNL.exe and the HAL.dll and all of the drivers classified as "Boot" (like hard drive controllers). The problem is that it FAILS to reach the point of loading any drivers. I thought I could trick the BCD Editor with tricky manual editing (bcdedit.exe) but no amount of editing will allow winload.exe to see an NVME drive as a valid path (even though Windows has the driver, its just too early in the process, and this native code is relying heavily on the BIOS).

This is an entry for a default regular MBR/BIOS install of Windows:

Windows Boot Loader
identifier              {current}
device                  partition=C:
wath                    \Windows\system32\winload.exe
description             Windows 10
locale                  en-us
inherit                 {bootloadersettings}
allowedinmemorysettings 0x15000075
osdevice                partition=C:
systemroot              \Windows
resumeobject            {fdd8d97c-9d4e-11e5-9c8b-bcaec519e8ac}
nx                      OptIn
bootmenupolicy          Standard 

You can see "device" = C: and path = winload.exe. Then you also see "osdevice" = C: and systemroot = \Windows. Changing one or both to D: or anything else (many hours of trial and error).... does not work.
BCD configuration only accepts drive letters, or paths in the form \Device\HarddiskVolume2, which is IMMEDIATELY converted into a drive letter (if mounted). But actually in the background, whats happening is that the Volume Serial Number/partition signature (an 8 or 16 digit hex number on the partition's logical sector 0) is being located and stored. This is why many times if you clone a drive, because some cloning programs decide its best to change the signature, and then do not correctly update the boot config to match. It's not just "Boot from Drive C:" or "boot from partition 1".

No amount of trickery with bootloaders, chainloaders, GRUB, or modifying windows's settings/ registry will help with this.

SOLUTION: 10 Steps to Take to Really Make it work

0. Start off with your NVME drive initialized as MBR, not GPT. (I have not tested GPT.) 
1. Create a small FAT32 partition like ~100MB on the NVME drive as partition 1. (not under 32MB or FAT32 won't work - i think windows has some larger-than32mb but less-than100mb abnormal minimum limit before it refuses to format, just stick to 100MB).  Name it something like "EFI BOOT" for good measure. Assign it a drive letter to be able to access it. Make it Active.
          C:\> diskpart.exe
     DISKPART> list disk
     DISKPART> select disk 4
     DISKPART> clean      (start fresh: removes any partitions/the MBR. skip if needed)
     DISKPART> convert mbr (starting fresh: initializes disk as MBR. skip if needed)
     DISKPART> create partition primary SIZE=100
     DISKPART> list partition
     DISKPART> select partition 1
     DISKPART> format fs=fat32 label=efi_boot
     DISKPART> assign letter=K
     DISKPART> active
     DISKPART> exit
2. Clone a partition already containing Windows to the available space, as partition 2. Since the installer is fail-tastic. Theoretically you can install a fresh version, but I did not do this, and you would need a UEFI copy of the windows installation media to launch the installer in UEFI mode.
Now you need to make sure the registry on the new cloned disk has C: pointing to the NVME disk's Disk Signature. Some cloning programs do this automatically.  HKLM\SYSTEM\MountedDevices\ holds this key. The best way to do this is to clone an OS that has seen the NVME drive that you just initialized. (since initializing resets the disk signature).  If you have an old image you want to re-use, export the Reg-key of the NVME drive's letter from the running OS, and save it so you can add it in the next step.
2a. Editing the registry ON the cloned drive: This requires manually loading the cloned drive's SYSTEM hive in a running OS.  Then load regedit.exe, click the HKLM key, click File|Load Hive..., type any name. Now navigate to  HKLM\SYSTEM\MountedDevices\ , Then you can delete the existing entry for C:, find the letter that the NVME drive's windows partition existed as, and  rename that key/letter so its the new C:. (or import the reg-key you exported) Then unload the reg-hive (it auto-saves). 
3. Create the boot files on the FAT32 partition to boot the newly cloned NTFS windows partition. Get an admin command prompt ready and type:
bcdboot L:\Windows /s K: /h UEFI
where L:\Windows is the NVME drive's windows directory and K: is the FAT32 partition on the NVME drive. (dont worry that L: is not called C:, this part does not obey the \MountedDevices list you just edited)
4. Your'e going to want some kind of MBR master boot sector on this drive I think (I forget how I did this. Maybe bcdboot puts one.).
5. Download Tianocore (the entire zip file) from here
This is an open source EFI "emulator" . Extract it somewhere making sure the pathname has no spaces in it.
6. They intend for you to create a USB Flash drive to boot with, which you should do first, to test it. (you can worry about creating something more permanent like a 2nd internal disk later - see bottom of this page). Run:
          CreateUSB.cmd M:
where M: is the letter of your USB flash drive. Do NOT put this on the NVME drive, that would be dumb. You want to trigger the NVME boot partition from something thats ACTUALLY bootable, in this case a USB drive.
6a. Plug and replug it. If it shows up as Non-Formatted, Thats not right. I had issues with this. Doing Diskpart.exe: clean , followed by, convert mbr , will start fresh. to clear out any latent crap in the MBR or just try a new flash drive.
6b. USB Creation Step 2, run:
          CreateUSB.cmd M: UDK_X64
6c. Make the partition active. (diskpart or diskmgmt.msc)
7. Test the USB Flash drive. Boot off it. Make sure you see "Tianocore" logo, and a little menu hopefully. You can poke around in here, but really no configuration is needed.
8. The NVME drive should be triggered and show a Windows Logo or blue windows screen now. Windows will not load fully however. (INACCESSIBLE_BOOT_DEVICE) You have to press F8 and boot into Safe Mode First. I'm not exactly sure why that is necessary, but it is. (I've repeated the process twice to be sure.) I feel like safe mode resets something in the registry that I dont know about.
9. Reboot, off the USB flash drive, and now Tianocore should boot, and then Windows should boot into normal mode. Perfect!
10. DONE!

Now you probably want to know how to make it so you don't have a USB stick hanging out of your system for the rest of its natural life. you are going to need a SECOND FAT32 partition on a SATA disk, and it is going to have to be Partition 1. I have not figured out a way to allow this partition to be at the end of the disk to save effort of resizing the partition. The best partition management tool i've found is "MiniTool Partition Wizard Home Edition 9" available for free, full featured. They will resize very fast, safely, and in windows without rebooting, even resizing to make space in the FRONT. Once you have that small (again like 100MB FAT32) partition on a bootable SATA disk, the hard part comes. The damn CreateUSB.cmd script relies on some programs called BootSectImage.exe and GenBootSect.exe and they modify the MBR and the PBR of a USB flash drive. Examining the source code of these files reveals that they should be compatible with IDE disks, but yet my testing reveals they do not work on SATA HARD DRIVES at least in Windows. You should try first running them from a NATIVE DOS prompt to see if they are treated as Legacy IDE in DOS and it allows to work. If not, well, you have to cheat. Run the following procedures,editing the first line to point to the path that you extracted the Tianocore Zip file to. Pseudo commands and actual commands are mixed because I have not automated this as of yet to set this up.

set UEFI_DUET=E:\750_Project\tiano\

set WIN_BIN_DIR=%UEFI_DUET%\Windows_Binaries


<STEP 1: Patch the PBR>
#back up the desired SATA HDD FAT32 
partition's existing boot sector (sector 2048) manually to a file called Fat32-2048.bin with secinspect.exe or other tool. #
# #


SECINSPECT.EXE -backup Q: MY_SECTORS\Fat32-2048.bin 2048 1
%WIN_BIN_DIR%\BOOTSECTIMAGE.EXE --verbose -g MY_SECTORS\Fat32-2048.bin MY_SECTORS\newBS32.bin -f
# now write this newbs32.bin back to 2048 and 2054(backup) #

<STEP 2: Patch the MBR>
The MBR at sector 0 contains two sections, from hex addresses 0-1B8, the boot code. From that point on, exists the partition table which you do not want to mess with. You can either Hex-Edit the MBR and copy and paste the code from the file, (which is what I did as a learning process) or do a similar procedure:

SECINSPECT.EXE -backup Q: MY_SECTORS\MBR-sec0.bin 0 1

You now have done step 1 and 2 . Proceed to Step 3 (only copies files):
          CreateUSB.cmd Q: UDK_X64

This should be all thats required to get the equivalent of the USB disk onto your HDD. Now everything is done. Enjoy. Simple huh :) Hope you could understand this all, I put a lot of time into it. email me @ mrlithium NOT@AT if you need help.

Friday, September 04, 2015

How to Split a Matroska MKV file without re encode or any dumb software

The command line is:
"C:\Program Files\MKVToolNix\mkvmerge.exe" -o NewFile.mkv -v --split 3G --link "TheVeryLargeFileYouWantToSplit.mkv"

For a 4 gig file on an SSD, it took a total of 44 seconds.

You can split by size, time, chapters, precise frames, etc.... (note that in my example it would be 3G for 3 Gigabytes) or you can use (01:00:00 for 1 hour chunks)

The crucial option here is the --link option which is dryly explained below, but it basically means that the file looks, acts, feels like you never actually split it, but now it exists on the hard drive in multiple files, that are referencing each other. Your player software will continue on with no gaps, jumps, and the chapters and elapsed time intact. (hopefully you are using MPC-HC & LAV codecs on your computer).
(the -v is for verbose = prints status of what its doing )

Using the program MediaInfo, you can open the MKV file you want to split, and read the version of MKVMerge that was used to create it, and its recommended to use at least that version to edit/split/link it, in case its using some new features (probably not - and if not, you can use older mkvmerge versions to split newer files). Also - 
Its possible that you can download too new of a version of MKVToolNix that is incompatible with your player, as sometimes the latest version will write an incompatible header for older players/boxes.  I'm using because i've tested all my devices on it, and using version 8.3.0 which is the latest version as of this blog post, might break small things in edge case scenarios and I won't know until its too late when I go to try to play it back (So i've heard). So test on standalone bluray players/Stream boxes, and if you experience problems on playback, try to download the next major old version, unless you need some new feature, at the cost of retro compatibility. The changelogs are here (and really quite hard to read, but CTRL+F search for "Released v" which you can also use to search builds by date). 


Multiple ways to split:

 as per the --help switch of mkvmerge.exe:

 File splitting, linking, appending and concatenating (more global options):
  --split <d[K,M,G]|HH:MM:SS|s>
                           Create a new file after d bytes (KB, MB, GB)
                           or after a specific time.
  --split timecodes:A[,B...]
                           Create a new file after each timecode A, B
  --split parts:start1-end1[,[+]start2-end2,...]
                           Keep ranges of timecodes start-end, either in
                           separate files or append to previous range's file
                           if prefixed with '+'.
  --split parts-frames:start1-end1[,[+]start2-end2,...]
                           Same as 'parts:', but 'startN'/'endN' are frame/
                           field numbers instead of timecodes.
  --split frames:A[,B...]
                           Create a new file after each frame/field A, B
  --split chapters:all|A[,B...]
                           Create a new file before each chapter (with 'all')
                           or before chapter numbers A, B etc.
  --split-max-files <n>    Create at most n files.


Copied from the documentation file in "C:\Program Files\MKVToolNix\doc\en\mkvmerge.html", Section 11:

11. File linking

Matroska(tm) supports file linking which simply says that a specific file is the predecessor or successor of the current file. To be precise, it's not really the files that are linked but the Matroska(tm) segments. As most files will probably only contain one Matroska(tm) segment the following explanations use the term 'file linking' although 'segment linking' would be more appropriate.
Each segment is identified by a unique 128 bit wide segment UID. This UID is automatically generated by mkvmerge(1). The linking is done primarily via putting the segment UIDs (short: SID) of the previous/next file into the segment header information. mkvinfo(1) prints these SIDs if it finds them.
If a file is split into several smaller ones and linking is used then the timecodes will not start at 0 again but will continue where the last file has left off. This way the absolute time is kept even if the previous files are not available (e.g. when streaming). If no linking is used then the timecodes should start at 0 for each file. By default mkvmerge(1) does not use file linking. If you want that you can turn it on with the --link option. This option is only useful if splitting is activated as well.
Regardless of whether splitting is active or not the user can tell mkvmerge(1) to link the produced files to specific SIDs. This is achieved with the options --link-to-previous and --link-to-next. These options accept a segment SIDin the format that mkvinfo(1) outputs: 16 hexadecimal numbers between 0x00 and 0xff prefixed with '0x' each, e.g. '0x41 0xda 0x73 0x66 0xd9 0xcf 0xb2 0x1e 0xae 0x78 0xeb 0xb4 0x5e 0xca 0xb3 0x93'. Alternatively a shorter form can be used: 16 hexadecimal numbers between 0x00 and 0xff without the '0x' prefixes and without the spaces, e.g. '41da7366d9cfb21eae78ebb45ecab393'.
If splitting is used then the first file is linked to the SID given with --link-to-previous and the last file is linked to the SID given with --link-to-next. If splitting is not used then the one output file will be linked to both of the twoSIDs.


Side Note as a reason for creating this blog post: 

-You can not just use any old file cutter utility to chop the file in half and then repair it.
-If you search for how to do this online, you get about 100 horrible freeware,shareware, malware laden and general JUNK video editor programs. So annoying....

Monday, August 17, 2015

Bring your own device + Virtualization / KVM idea, to combine two computers into one seamless software experience.

I just thought of an incredible Idea that would make me a millionaire if i was able to prove the concept
imagine putting your entire computer on a USB flash drive, like live-USB basically
and then you bring it to someones house and you plug it in
this is the current method which is clunky and its either Theirs or Yours, and requiring reboots
Now imagine if its not ONLY storage - but has blue-tooth,WiFi, and a CPU, like a compute stick
barely enough hardware to make it work.
and there's a "synergy" interface between the two devices
and you can basically run programs from your computer ON theirs
no reboots. It would run inside a host app.
the hurdle is to like remove all the software barriers of doing such a thing so anyone can do it
and then find the proper device to really help this idea thrive.
i mean, I can attempt such a thing today with a great deal of prep-work and manual configuration
All that's required is to create the automatic interoperability software

a better way to explain it would be:
 "a blue-tooth enabled computer that pairs to someone else's computer" with software to remove barriers between the two, so you could use BOTH computers at once
not all that far fetched IMO!
it would rely heavily on virtualization and sandboxing just to keep things secure
essentially, a virtualization, KVM, sandbox, VM host App 
the appearance of x86 phones really make this sound possible to me
the phone only needs a WiFi chip, a cpu, minimal amount of ram to run a hypervisor, ~256gb-1TB+ storage to fit everything you want to bring with you
and then blue-tooth to initiate the pairing. which then hands-off to WiFi
you open the VM/sandbox app on the host PC, initiate the blue-tooth pairing process, and then your hard drives are made available, the host reads your icons, programs, registry, documents, and sorts EVERYTHING out (not as hard as it sounds)

phase 2 of the idea would be getting the mobile device powerful enough to where you could comfortably do the reverse - probably 5 years away
Nobody has thought of this probably, because laptops are self sufficient, do not need pairing, and phones are typically ARM and largely incompatible and off in their own ecosystem and seen as an impossible task which even microsoft nor apple cannot unify or accomplish.
however, i do believe we're closer to this on Windows 10 + x86 than Apple is with OSX + IOS, and that independent developers can even be the one to prove this before the big guys take over.

who knows, in a few years, our addiction to carrying around phones, and making phone calls, and calling the thing a "phone" will be gone, and we just call it a computer.....
and theyd be able to basically have this like "cellular chip" that plugs into a flap on the back to enable phone calls. that may be taking the concept too far too soon but there you have it!
you could even implement a docking station to connect @ Thunderbolt 3's 40Gbps or USB 3.1 10gbps + Displayport, and improve pairing and connectivity speeds.

Tuesday, January 20, 2015

Stop External USB Hard Drive from Auto-Playing in Windows 7

The instructions on the internet are very long, and confusing and for some reason this can only be done through the registry.

To disable it one drive letter at a time:


This chart above is important and the key to this process. It is in binary code. 
Copy it into notepad for editing. Under the letter you want to disable, change it from 0 to 1.
0 means enabled. 1 means disabled.

  1. Open calculator. Click the menu "View | Programmer".
  2. On the left of the 4 bubbles, select "Bin" (for binary). 
  3. Now copy and paste the binary numbers from notepad into calc.exe
  4. Now click the bubble "Hex" (to convert to hex).
  5. Copy the result (or remember).
Now open regedit.exe and navigate to: 


On the right side, you should see a bunch of stuff already there.

Click the menu "Edit | New > DWORD (32-bit) Value"
Give it the name: NoDriveAutoRun
Enter the value from calc.exe (and make sure the bubble is on Hex)

To disable it for one Type of drive:

There is another method that disables it for an entire type:
(Removable (USB flash or HD) / Non-Removable (HDD) / Network / CD-DVD / RAMdisk).

The process is the same as above, in the same registry location but the entry is called:

Using the chart below, change the appropriate column from 0 to 1, and follow steps 1-5 shown above.


Bit numberBit patternDrive type
200000100Disk that can be removed from the drive (such as a floppy disk or USB flash drive)
300001000Disk that cannot be removed from drive (usually a hard disk)
400010000Network drive
500100000CD/DVD drive
601000000RAM disk
Notice the 3rd column is Bit #2 because binary starts with 0. Bit #1 does not seem to exist.
(not listed above, Bit #0 and Bit #7 controls Disables AutoRun on drives of unknown type.)

The default is 0x00000091 (91 Hex) or "145" (Dec) or "1001 0001" (Binary)
Meaning by default, unknown types and network drives are disabled, and everything else is enabled.

Cheat Sheet:
91AutoRun is enabled for CD/DVD drives and USB flash drives
95AutoRun is disabled for USB flash drives
B1AutoRun is disabled for CD/DVD drives
B5AutoRun is disabled for CD/DVD drives and USB flash drives

To disable it for everything:

Follow the process for "type" and use the same key ( NoDriveTypeAutoRun )

If you want to disable all the types listed above:
Use the bit-mask "01111100" in binary, or "7C"(Hex) or "124" (Dec)

If you want to disable EVERYTHING (even drives with an unknown type):
The bit-mask "11111111" in binary, or "FF" (Hex) and "255" (Dec) 

Just fill in this hex code into the NoDriveTypeAutoRun entry.
Theoretically Autoplay would never show up again.

Sub-Note:  The NoDriveTypeAutoRun subkey value in registry has higher priority than the setting in the "Control Panel > AutoPlay Options".
So if AutoRun for a drive is disabled via NoDriveTypeAutoRun then you can not enable it by using the Control Panel. 

Side Note: Apparently, you can edit the same location in HKEY_LOCAL_MACHINE to make it machine wide, instead of per-user.

Saturday, January 10, 2015

SB X-Fi Titanium HD JP2 pinout

Sound Blaster X-Fi Titanium HD
The expansion port is labeled JP2.

The old audigy/sb Live AUD_EXT Pinout (above right) does not match this sound card!!!! This image is provided for curiosity only.

A standard 40 wire IDE cable is able to be plugged into the JP2 port, which I used to make testing the individual pins easy. Pin 1 (the Pink wire) corresponds to the marked arrow.

Voltages of JP2 in Millivolts:

 1  -  0
 2  -  23 mv
 3  -  0
 4  -  0
 5  -  23
 6  -  0
 7  -  3210
 8  -  23
 9  -  0
10  -  0
11  -  24
12  -  123-139
13  -  0
14  -  0
15  -  1460
16  -  23
17  -  168
18  -  50-80
19  -  0
20  -  26
21  -  0
22  -  0
23  -  3230
24  -  2500
25  -  1466
26  -  24
27  -  25
28  -  3210
29  -  24
30  -  0
31  -  24
32  -  24
33  -  1295
34  -  22
35  -  1520
36  -  23
37  -  1636
38  -  30
39  -  50-80
40  -  23