UEFI Booting NVME on x58 Project w/ Tianocore by mrlithium (draft) 4-29-2018 - Round Two

Using a TianoCore DUET USB stick to Boot Windows 10 in EFI mode from NVME SSDs such as Samsung PM961 on Legacy Platform X58 Chipset.

Already confirmed working on Asus Sabertooth X58 motherboard and Intel 750 Series SSDs approx 2 years ago.
Since then, a nice person named AudioCricket has developed a cleaner tutorial, (see AudioCricket blog article on Booting NVME ) which made me want to revise my own process, as he credits me with thanks for knowing it was even possible. You should read his also, its got some good details.

This tutorial is meant to give you bootability on extremely high speed NVME drives. If you are not using a high speed drive, or its in a wrong slot, you may not see much benefit. The whole boot process workaround itself, from the Tianocore DUET USB, takes up a valuable 10-15 seconds at bootup. You will pay a longer price on boot, but your Windows OS will be booted onto this faster drive, and there is no substitute for that. You already could use these drives as secondary storage partitions with no configuration, but thats just boring.
Linux Note: that if you are using Linux, all my info is largely irrelevant. Linux takes entirely another approach to booting.

My Process involves using the TianoCore DUET software and UEFI bootloader to chain boot load EFI Windows from a MBR USB stick.

To redirect your BIOS Boot -> MBR of USB stick -> virtual/emulated UEFI environment (Tianocore DUET) -> load open source NVME driver -> NVME GPT / EFI Windows System boot.
1 USB Stick for TianoCore UDK_X64  (USB 3.0 maybe needs an XHCI driver)
1 USB Stick for Windows 10 Setup ISO
Executing a few commands at the shell prompt
Running the Windows Setup Installer off a 2nd USB stick, using that to lay down new partition information and blank your NVME SSD.
Installing a new copy of Windows to the NVME SSD, with its own EFI partition and GPT layout. This is the best way to get the EFI boot working.
Booting NVME cannot be done without UEFI and booting EFI must be done for Windows (winload.efi). You cannot have Windows boot MBR (winload.exe)
At this point you will have two Windows Installs, and able to dual boot to the new NVME OS drive by inserting the Tianocore USB stick and booting

Boot process summarized: (Thanks to AudioCricket for his excellent description)

1. Computer powers on
2. Legacy BIOS boots / initializes
3. Tianocore DUET boots from the USB stick
4. Tianocore DUET executes UEFI Shell
5. UEFI Shell loads generic NvmExpressDxe-64.efi driver and remaps system block devices
6. UEFI Shell now sees Windows 10 hidden FAT partition DIRECTLY on the NVMe
7. UEFI Shell executes Windows 10 UEFI bootloader from that hidden NVMe FAT partion directly (\EFI\Boot\Bootx64.efi image)
8. Windows 10 is now booting and is in full control of your computer (under UEFI mode, of course)

Tianocore USB Stick Creation:

You will need the TianoCore DUET download package and stuff 

The USB stick needs to be formatted with "HP USB Disk Storage Format Tool" as opposed to Windows, so just download that. There is a reason for needing that stupid program, The MBR and DBR / BPB boot sectors need to be written as if it was a HDD not a removable (floppy) - err don't ask - :) After you reformat it, unplug/replug it.

Start up an administrative command prompt, and cd \tianocore_uefi_duet_installer-master
or, go into the location of CreateUSB.cmd  

Replace J: with the appropriate letter for your USB stick. (double check with HP USB Tool).

.

CreateUSB.cmd J:

Make sure no errors / unsavory messages are shown,
then unplug/replug it, and run this:

CreateUSB.cmd J: UDK_X64

This copies MOST of the files. You still need to manually copy over the file Drivers\NvmExpressDxe-64.efi this can be obtained here: or you can compile your own (coming soon/later)

UEFI Interactive Shell Usage:

From now on, rebooting from the Tianocore USB will launch into a shell. It acts kind of like DOS or Linux does, a simple command shell. Here you can explore and decide what commands to run:
Hit enter after every command. Theres also tab completion. You can use dir to look around.
Green colored files are executable and just typing their name should run them, for example *.efi
FS1: for drives (like C:) and \ for directories.
FS0: should be your booted Tianocore USB drive
FS1: should be the Windows USB stick
FS2: should be the NVME SSD drive (not visible yet)
Run these two important commands:
load \EFI\Drivers\NvmExpressDxe-64.efi  (loads the .efi program for detecting NVME devices - from this root pathname on the current prompts' FS device name)
map -r  (this will make new FS2: alias, etc)
Theres also "BLK" devices that correspond; you can ignore these besides to make sure they do exist for the right USB SATA or PCIe location.

A "startup.nsh" script can also be created, to AutoRun on boot, to automate the above commands in their final form or for anything else special that you may need or want.
Startup.nsh lives at \EFI\Boot\ directory and should just be auto-executed from there.

To boot the Windows Installer on the 2nd USB stick:

FS1:\bootmgr.efi
or
FS1:\efi\boot\bootx64.efi

Follow the Windows Setup, "Install Now" and choose "Custom Upgrade" so you can pick the right drives UnPartitioned Space (If you already attempted this, its best to Delete partition and go back to UnPartitioned Space and then say yes to "Windows may create some additional system Partitions". If it complains they are not in the correct order, "Delete" and hit OK on "UnPartitioned Space" again. The installer's Partitioner should be able to see your NVME drive immediately and without any drivers needing to be loaded.  Installation is seamless from here on out, other than a few reboots. Once the Windows Installer accepts your partition layout, it lays down the EFI partition on your NVME drive,  and starts "Copying Files...".  Meaning, across reboots, you need to reboot it from the NVME drive, not the Windows Install USB stick. While you're still technically doing the Windows Install, everytime the system reboots, this is the command you want to enter at the shell prompt to continue the installation. 

To boot Windows on the NVME drive:

FS2:\EFI\Microsoft\Boot\bootmgfw.efi

FS2 should be the alias for your NVME PCIe SSD drive device, (if not- double check, find it out).
This Runs the Microsoft EFI boot manager - that supports NVME boot.
Then Microsoft takes over, and the rest is magic. Now your system is fully working.
You would want to put this command in "startup.nsh" so its automatic. Keep in mind the FS drive letters numbers may get shifted, even due to just USB sticks.

Possible Secondary Optional or Advanced Steps:

Update Driver ? - replace the Microsoft stornvme.inf driver with a Samsung NVME driver (or whichever one, Intel?).

Using USB 3.0 to boot may require: \EFI\Drivers\XhciDxe-64.efi
Eliminating the need for a permanent USB Stick by copying the partition to any other internal hard drive in your system, and making room for it and also being able to dual boot it.
Not actually installing windows, just laying the boot partition down and then re-copying an Image of windows back over to the new SSD.
Small = ~4 Few megabytes. = Can be downloaded as a Disk Image *.dd, and use RUFUS to restore it back. Should be able to replicate Tianocore USB sticks for backups/disasters.
Editing the Windows BCD to get Dual Boot or Multiple Partitions combined or something else.
Remove un-wanted drive letters of the other system after bootup.

Super Fast NVME PCIe SSD's on Legacy X58 Platform - other Practical Considerations:

*** PCI-e version revision and link width ***
PCI-e 2.0 was the max for X58, and as of today we are up to PCI-e 3.0 which doubles bandwidth by 2x.
Theres also certain slots that are PCI-e 1.0 slots (the bottom slot on Asus Sabertooth is 1.0 and the 2nd GPU x16 slot is the only next best highest speed 2.0 x4 slot :/
Benchmark overall speed drastically improved when switching slots, from around 750-770MB/s over PCI-express 1.0 x4 to 1450-1600MB/s on PCI-express 2.0 x4. (With the Samsung PM961 SSD, and tested under CrystalDiskMark and AS-SSD.)
Note this is nowhere near the drives theoretical maximum, that would requires PCI-e 3.0 and should be hitting somewhere over 2-3000 MB/s.
Also note that most slots are only physically, but not electrically, the full width of PCI-e lanes. Make sure your slot is valid.
It is advised to check in a program like HWInfo64 or AIDA64 at your Motherboard slot DMI information and for sure confirm the devices are detecting at proper revision and width. (CrystalDiskMark will say the link speed for PCI-e devices now so thats handy). If not, Power Management settings could be configured to do Link State Power Management or whatever thats called.

UltraDefrag Mod 7.0.4 alpha genBTC edition

UltraDefrag 7.04 (UDefrag) - mod by genBTC

Compiled from C++ source, and edited.  Requires WxWidgets.

Integrated StopGap software - requires Boost libs

New Build system devised for use in Visual Studio . Supports 2017 and CMAKE on Windows.

Supports Building with MSVC or MINGW G++.

Now includes Files List tab.
Also Supports finding files by LCN cluster, finding LCN cluster by filename, right click on file list.

- Development has been put on hold, and there is a great deal of work involved in this project.

Windows 10 SysAdmin : Maintenance Ideas and Tasks for Power Users - edit 1

As I write this, I'm thinking of the multiple ways Windows needs to be managed after its installed.

Lots of people only know 3 steps. #1.Install, #2.Run WindowsUpdates, and #3.Re-Format when its broken. Thats not my style.

Before we even install we have choices to make: Enterprise LTSB Version of Windows.
The current version is Enterprise LTSB 2016 and this removes all the extraneous Modern Apps and allows the blocking of telemetry easier, and applies some group policy updates.
Regular Windows 10 Pro 1709 has more features tightly integrated that are harder to rip out after installing.

The Windows 10 OS has many subsystems which need to be understood as seperate parts that are all interconnected. Understanding what to do with them and why is the mission of this blogpost.

Task Manager: 
Processes (PID, CPU Time, RAM usage) / Performance Counters, Services
This should be the first place you should start looking.

Filesystem Permissions / ACLs' / Security Descriptors / SIDs
the built-in named Administrator account should be disabled, and a new Administrative one with a new name should be Created, and a strong password that you can remember be given to it. This password should not be something re-used as the local Windows SAM Security database is highly insecure and these credentials can be easily read or reset by anyone with physical access.
Your Daily Driver user account should be a standard user, and only use the new Admin account and password when you need to UAC escalate priveleges. This follows the concept of Principle of Least Priveleges.

Group Policy / User Accounts / User Groups / Active Directory (Domain)
Now that you're a standard user and you can hardly do anything, you may feel tempted to just re-add yourself to the Administrator group or PowerUser group or BackupOperator group. These groups accounts come provided with certain pre-defined rights pre-assigned. Its very hard to tell what does what, and there is some advanced configuration located in GPedit.msc \ \ Computer Configuration \ Windows Settings \ Security Settings \ Local Policies \ User Rights Assignment & Security Options.
This brings us to group policy: Theres a ton of stuff in here that you are free to look through and customize.
The major trick to run is "Windows Restricted Functionality Limited Baseline" profile available from external download from Microsoft.  That will run a few scripts and lock the system down, even 1 step too far in fact. It will over-ambitiously enable "Disable Automatic Root Certificate Updates".  Reason we need to change this is, you need your OS Certificates stored certs to update or you will be unable to browse most websites (even in Chrome). Go into gpedit.msc and d this and reverse that entry (to "Not configured"). It is under computer configuration, Double-click Administrative Templates, double-click System, double-click Internet Communication Management, and then click Internet Communication settings. Any hyper paranoid person can individually manage their certificate store and this is another topic.


Windows Registry - regedit - hives
HKEY_CURRENT_USER = %USERPROFILE%\NTuser.dat (@ C:\Users\Username)
HKEY_LOCAL_MACHINE  = %WINDIR%\config\SYSTEM (@ C:\Windows\system32)

Task Scheduler Service - Automatic (important)
Task Scheduler Engine: C:\Windows\System32\Tasks
Files exist in this directory and can be managed with: taskschd.msc
Security Permissions are required for any Tasks running as NETWORK SERVICE, etc.
Individual Tasks:
My specific Win10.10240 Install has 143 Tasks , about half of which I was able to disable manually.
They have "Description" fields written by Microsoft with actually valid information to decipher what these mystery tasks actually do. Note Some are blank, and Note some tasks being disabled cause Event Log Errors. This leads me to the next point.

The Event Viewer: eventvwr.msc
Application, Security, Setup, System = Default Logfiles
These are useful to skim through on their own. You can diagnose most of your errors this way.
But, Microsoft has also hidden thousands of other logfiles under the \Microsoft directory (a lot of which are useless, blank, disabled) and you can dig down to obtain further info on anything in specific, an example as such is "DeviceSetupManager" - to look at the list of every device that was plugged into your computer.
You can then Filter the list by "Event ID" and there is even a whole .XML for customizing the filters, saving them as a Custom Log, and referring back to that in the future.


"Programs & Features" > "Turn Windows features on or off" (list of toggleboxes)
Disabling Internet Explorer 11 will prevent it from being run, however it can never be truly gone, as its provided in DLL form as shdocvw.dll and mshtml.dll. These cannot be force-deleted as the number of programs that used the Windows Shell API (SHLWAPI.dll which indirectly references those IE dlls) to make outgoing internet connections will fail. This is something other people have noticed as having security impacts: https://packetstormsecurity.com/files/136702/Microsoft-Internet-Explorer-11-DLL-Hijacking.html
100% disable the "SMB 1.0/CIFS File Sharing Support" this is a vulnerable component in SMB filesharing (Samba support) for backwards compatible to legacy windows or legacy linux systems. Read: Old. Nothing will be impacted and this was in fact even recommended by Microsoft recently.
You will want to look through the list and Disable Everything EXCEPT the .NET environment, Windows Powershell 2.0 and Windows Process Activation Service.

"Uninstall or change a program"
This list of programs comes directly from the registry. Environment variables are set mapped to the proper directory there also.
Windows Installer Service
C:\Windows\Installer\
C:\ProgramData\ (%ProgramData%)
C:\Program Files\Common Files (%CommonProgramFiles%)
C:\Program Files and C:\Program Files (x86)  = Administrator (%ProgramFiles% and %ProgramFiles(x86)%)e
C:\Users\Username\AppData\Local  = User Local (%LOCALAPPDATA%)
C:\Users\Username\AppData\Roaming = User Roaming (%APPDATA%)

Windows Updates:
Windows Update Service , Background Intelligent Transfer Service , Device Setup Manager (drivers)
C:\Windows\SoftwareDistribution\ (safe to delete and auto-rebuild, instructions online)
This should be disabled with the settings in the program "O&O Shutup 10" available free.
To install updates, you need to do it manually.
Step 1 - visit the list of updates and find the right version: https://support.microsoft.com/en-us/help/4093112
Step 2 - at the bottom of that, click "Microsoft Update Catalog" and follow the download links.
Windows Update Service needs to be re-enabled for the Cumulative Update .msi Packs to install, or it will say some Error like "Could not run this update" If you see "This was not compatible with your machine, you picked the wrong one"

Modern  / Metro / Tile Apps:
C:\Program Files\WindowsApps
C:\ProgramData\Microsoft\Windows\AppRepository
Tile Data Model Server - updates modern start menu's tiles.

TaskBar
Shell Infrastructure Host ( sihost.exe ) - The black GUI overlay of systray and Modern GUI desktop/taskbar app-windows. Disabling this will cause some problems like not being able to start Calculator or Display Settings (desktop ms-settings:display has no assosciated program)
SIH client (server-initiated healing) sihclient.exe - detect and fix system components that are vital to automatic updating of Windows and Microsoft software (Constantly Goes Online) - disable in TaskScheduler
Cortana:
This can be slightly disabled with O&O Shutup 10 and thats all you can do. Other ways to force rip Cortana out can result in OS breaking.
Remove-AppxPackage -name *Cortana* (dont do this unless you follow another guide)

Startup:
HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
Win10 Task Manager Startup tab does not show those entries, they are different.

PowerShell:
Remote-Signed Security Policy - necessary to run scripts from the internet, BUT can/should be toggled by admin On/Off as needed instead of leaving it as default on.
All the Internal Components in Windows can be managed by the appropriate PowerShell Cmdlets.
There is Tab Completion, and a Get-Help command which downloads useful help from the web.

Windows Firewall:
Windows Firewall with Advanced Security = On by Default, Leave on, VERY IMPORTANT.
You should be micro-managing this. This is your lifeline to prevent intrusion and exfiltration.
This deserves its own whole topic but I will try to keep it short. The default config comes with hardly anything blocked. The Restricted Traffic Baseline does a better job at it, however most Outbound conections are still Unblocked. Don't trust your Win10 OS. It makes random connections out, and you want to set a default rule of Block ALL Outgoing, Block ALL Incoming. And then set up whitelists for every program you use. Most people hate using Windows Firewall because of the poor GUI, and the tedium of adding a rule for every single program. But new programs have come out called "Windows Firewall Control" and "Windows Firewall Notifier" that pop up Notifications everytime an unknown Outgoing connection is made and you will be prompted (normal windows firewall only prompts you for Incoming). For reference, I have 842 rules and I try to keep them logically organized and up to date. These programs are very powerful and inventing your own system and Getting used to this fully will take several months.

Windows Defender:
Unless you are Admin'ing someone elses computer, I personally see no use for this; definition based scanning has a poor success rate and consistently takes a ton of resources and it bogs down hard drive accesses because its literally scanning every single file you access while Real Time Protection is running. MSMPENG.exe is the process name, and related ones, and you can check to see the resource usage in Task Manager... I just force disable it with "O&O Shutup 10" and disabling the tasks in Task Scheduler  - \Microsoft\Windows Defender
Following the rest of my tips will keep you informed of when your system is going wrong, you should be able to prevent yourself from getting virused and malwared, and you dont need this waste of space built-in program and its weak detection heuristics and false positives.

Drivers:
Device Manager : View | Show Hidden Devices
Make sure nothing weird is latent.
You can delete any grayed out devices that you know you don't need.
Device Drivers and System Drivers and System Services
Serviwin.exe from NirSoft: Gives you access to the list of System Drivers and Services.

System Drivers:

Services:

Data Backup

Strategy : 

1. Onsite Backups: When a server crashes or fails, it is helpful to have data backups on hand for easy restoration. It’s a cliché, but time is indeed money. Onsite backups are often faster to restore than cloud backups and almost always faster than offsite tape backups.

2. Offsite Backups: Onsite backups are valuable, but they cannot be counted on alone. Should something disastrous happen to the data center, it could also damage any backups you have in the building. For that reason, it is always wise to have copies of your backups offsite where they can be accessed manually or through the cloud.

3. Optimized Backup Schedule: Backups are not a one and done process. Key data in your data center must be regularly and consistently backed up according to a clear and organized schedule. Check out our blog article on just a few backup rotation schemes for more information.

4. Backup Testing: Backups need to be tested and need to be tested regularly. In addition, the IT staff must be trained on how to access and restore their data backups as quickly as possible. A backup that fails or a team that is unable to restore the backup quickly undermines the company’s investment in a backup solution in the first place.

5. Organized Storage System: Mostly applying to tape-based backup solutions, the storage repository for backups and labeling system must be clear and organized. The team cannot commit extra time digging through box after box of tape looking for a specific backup from a specific date several years ago

https://www.gomindsight.com/blog/5-must-haves-data-backup-strategy/

Practicality (my strategy):

Nightly backup of the entire C drive and boot partition, with Macrium Reflect. To a compressed non-linear Macrium image *.mrimg , for dynamic incremental restore using Volume Shadow Copy . 

However using Macrium seems to cause errors restoring Chrome, it detects a change in the filesystem and corrupts the userprofile unless you use the Account Sync to Cloud UserProfile feature. Restoring sector-by-sector instead is extremely slow, but also more accurate. 

Sector Backups (like with linux dd) also have the benefit of being more robust for data-recovery in disaster situations where part of the image may be unreadable, you may still be able to read parts of the direct disk image. A NTFS FileSystem can even be mounted and manually restored from a linux recovery console.

Windows Account User Profile is backed up to FreeNAS ZFS + DeDupe and Snapshotted nightly.

Some Documents and Pictures are backed up too, but not much.

All the rest of my stuff is stored on FreeNas on a ZFS RAIDZ-1 or ZFS Mirrored SMB share

Data needs to be categorized into: 3 Copies, 2 Copies, 1 Copy. Anything with 1 copy is accepting the risk of total loss. The 3rd copy is for the most important stuff and should be in a totally different format, and physical location.

Programs:

Macrium Reflect Pro

Easeus Todo Backup Home

Acronis Backup 11.7 Workstation 

Windows Block Level Backup Engine Service

System Recovery Partition , WinRe.wim

NextCloud / OwnCloud

Backblaze / Carbonite / Crashplan / Mozy / etc

My Other Redirected Websites

My Other Redirected Websites

http://autotrimps.site
http://mrlithium.site

My Actual Other Website

https://genr8.pw

https://genbtc.github.io (Github Pages)
https://mrlithium.blogspot.com (this page - Blogger)

I am currently setting up a bunch of servers for all those things, in the meantime they all just link to my resume. More to come.

Coursera Courses - Learning

Data Science
10 courses | Johns Hopkins University
Build foundational data science skills in this hands-on introduction to R, machine learning, and more.

Business Analytics
5 courses | The Wharton School of the University of Pennsylvania
Make data-driven marketing, HR, finance, and operations decisions.

Machine Learning
4 courses | University of Washington
Fit models, analyze data, and build an end-to-end practical machine learning applications.

Statistics with R
5 courses | Duke University
Perform frequentist and Bayesian statistical inference and modeling in R.

Business Foundations
6 courses | The Wharton School of the University of Pennsylvania
Develop basic literacy and skills in marketing, finance, accounting, and operations.

Introduction to Project Management - Principles and Practices
4 courses | UC Irvine
Coordinate and scope complex projects; fufill PMI education requirements.

Digital Marketing
6 courses | University of Illinois at Urbana-Champaign
Strategic planning and analytics for digital marketing channels.

Leading People and Teams
5 courses | University of Michigan
Inspire, motivate, and influence individuals and groups.

Python - Popular in Computer Science
5 courses | University of Michigan
Get a beginner-friendly introduction to programming and processing data in Python.

Java Programming and Software Engineering Fundamentals
5 courses | Duke University
Learn how to build scalable products in this introduction to software development.

Data Engineering on Google Cloud Platform
5 courses | Google
Gain practical, hands-on machine learning and big data experience.

Full Stack Web and Multiplatform Mobile App Development
5 courses | Hong Kong University of Science and Technology
Master frameworks like Bootstrap and Angular to be a versatile web developer.

XKCD #386 - Somebody is wrong on the internet.

Permanent link to this comic: https://xkcd.com/386/

My Mood Right now

https://genbtc.github.io/

I've set up a private website @ https://genbtc.github.io/
- Right now its hosted on Github Pages using their free domain, but:
- Host it on DigitalOcean for $5/mo - My Referral Link
- or Host it on Vultr for as low as $2.50/mo - My Referral Link
- NameCheap has actually $1 a year domains

My Guide to DigitalOcean Products:

These are the servers I recommend in green, and if you are wondering you can skip the ones in red.



- Get the products above on DigitalOcean for $5/mo - My Referral Link

also

Vultr  Has a special going on right now for Bare Metal Instances that are pretty heavy duty for darn good prices:

Thats 32GB of ram, 50TB data, an 8 core CPU, 2x 240GB SSD and I can assure you they are fast.

Or this is what the regular products are:

And luckily they are in a buncha areas too:

- or Host it on Vultr for as low as $2.50/mo - My Referral Link

New Posts

I'm going to start posting more, And I need to remember How much fun I have when I write these blog posts.
-Thanks to anyone who reads them!