Monday, November 13, 2017

Compiling a Node.JS app into a .EXE using PKG step by step walkthrough and hints about zeit pkg and completing dynamic requires using package.JSON

I have a Node.JS app thats very bloated on disk (lot of node.js dependencies) and a huge file tree.
In this guide, we will turn that from:























Using:

https://github.com/zeit/pkg - Node.JS Binary Compiler

"This command line interface enables you to package your Node.js project into an executable that can be run even on devices without Node.js installed."

Start up a command prompt with Node installed and run:
npm install -g pkg
The actual command to compile is one such as:
pkg discordbot --targets latest-win-x64 --debug

Usage:
pkg = this command (npm node command)
entrypoint = the folder of the node.js app, or . for current dir - must find package.json
--targets = nodeRange-platform-arch
    nodeRange node${n} or latest
    -platform freebsd, linux, macos, win
    -arch x64, x86, armv6, armv7
--debug = There is very little output, compilation succeeds but may have omitted resources.

Now is the fun part, you get to find out which errors you had and how to fix them:
The first problem is:
package.json

A) You need to know the basics of JSON.
B) You need additional information in there to make sure it works.
C) Files are going to have to be declared manually, as the tool is not dynamic require aware. This means such things like:
const requireStructure = name => require(`../../structures/${name}`);
or:
function createNpmDependenciesArray (packageFilePath) {
    var p = require(packageFilePath);
...
Aren't going to work.

First step is find package.json in your root tree of your app package dir, and open it in a text editor. Duplicate your "main" line as a "bin" line:
  "main": "discord_bot.js",
  "bin": "discord_bot.js"
This is a sample config:

{
  "name": "DiscordBot",
  "version": "0.1.2",
  "description": "Bot for Discord app modded by genBTC",
  "readme": "README.md",
  "maintainers": [],
  "author": "",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/genBTC/DiscordBot.git"
  },
  "license": "GPL-2.0",
  "dependencies": {
    "8ball": "^1.0.6",
    ...
    "youtube-node": "1.2.x"
  },
  "pkg": {
     "scripts" : ["src/client/websocket/packets/handlers/*.js",
...
        "src/client/websocket/packets/handlers/Ready.js",
...
        "src/client/actions/*.js",
        "src/client/actions/ActionsManager.js"
        ]
    }
}
JSON tip: Note the comma seperated [ list ] of files in "quotes", and with FORWARD / slashes as the path seperator. You can also use * wildcards.

Before we know what files we need to add, we need to compile the program (as shown before with --debug). You can also pipe the output using > result.txt to save it as a text file and look at it later.

C:\Software>DiscordBot.exe
Starting DiscordBot
Node version: v9.0.0
Discord.js version: 10.0.1
pkg/prelude/bootstrap.js:1172
      throw error;
      ^

Error: Cannot find module './handlers/Ready'
1) If you want to compile the package/file into executable, please pay attention to compilation warnings and specify a literal in 'require' call. 2) If you don't want to compile the package/file into executable and want to 'require' it from filesystem (likely plugin), specify an absolute path in 'require' call using process.cwd() or process.execPath.
    at Function.Module._resolveFilename (module.js:540:15)
    at Function.Module._resolveFilename (pkg/prelude/bootstrap.js:1269:46)
    at Function.Module._load (module.js:470:25)
    at Module.require (module.js:583:17)
    at Module.require (pkg/prelude/bootstrap.js:1153:31)
    at require (internal/module.js:11:18)
    at WebSocketPacketManager.register (C:\snapshot\discordbot\node_modules\discord.js\src\client\websocket\packets\WebSocketPacketManager.js:54:21)
    at new WebSocketPacketManager (C:\snapshot\discordbot\node_modules\discord.js\src\client\websocket\packets\WebSocketPacketManager.js:18:10)
    at new WebSocketManager (C:\snapshot\discordbot\node_modules\discord.js\src\client\websocket\WebSocketManager.js:24:26)
    at new Client (C:\snapshot\discordbot\node_modules\discord.js\src\client\Client.js:63:15)
This tells us that it couldnt find "handlers/Ready", and the faulty node_module is "discord.js".
Also, we can open the files indicated in blue and examine the .js scripts for what kind of stuff is being dynamically included, usually we can find a list of stuff such as:

So, we dig around in that dir, and find handlers/Ready (and all the other ones).
If you want to be precise you can:
Open up another command prompt to generate a listing of these files by :
dir /b *.js > list.txt
Then you can use NotePad++ to manipulate the text (add the rest of the path and " ", and form them into the list of files that needs to be passed to package.json.) Hopefully you are fast at this, because there is going to be a lot of files.
Otherwise you can suffice to say you found out that you want to add the entire
   "pkg": {
     "scripts" : ["src/client/websocket/packets/handlers/*.js",
        "src/client/actions/*.js"
        ]
    }

Process Explained:

You are going to repeat the process of Compiling, Looking at the compiler's debug output, Running the .exe and Looking at the binary's output. Then from that - analyze which files need to be included. You want to predict ahead of time and grab entire directories at once.

You can compare the results files, of the before and after - they should indicate success in adding.
C:\Software>DiscordBot.exe
Starting DiscordBot
Node version: v9.0.0
Discord.js version: 10.0.1
logging in with token
(node:49936) [DEP0013] DeprecationWarning: Calling an asynchronous function without callback is deprecated.
Using gateway wss://gateway.discord.gg/?encoding=json&v=6
Connecting to gateway wss://gateway.discord.gg/?encoding=json&v=6
Connection to gateway opened
Identifying as new session
Logged in! Serving in 2 servers
pkg/prelude/bootstrap.js:1172
      throw error;
      ^

Error: ENOENT: no such file or directory, scandir 'C:\Software\resources\default_app\plugins'
1) If you want to compile the package/file into executable, please pay attention to compilation warnings and specify a literal in 'require' call. 2) If you don't want to compile the package/file into executable and want to 'require' it from filesystem (likely plugin), specify an absolute path in 'require' call using process.cwd() or process.execPath.
    at Object.fs.readdirSync (fs.js:924:18)
    at Object.fs.readdirSync (pkg/prelude/bootstrap.js:776:35)
    at getDirectories (C:\snapshot\discordbot\plugins.js:5:15)
....
We've gotten further. This tells us we've logged in now, but it can't find the plugins directory. This is a directory right under our main app dir, and includes
some JSON and non-JSON files. resources\default_app is the internal name of the .exe's file-structure. So we need to include "plugins" into the exe. For this we can use Assets:

"assets" : ["plugins/*"]


If when you try to run the app, you get NPM trying to download and install stuff, its because it wasnt listed in your package.json dependencies. Just go in and manually add it.

After all this, it works!


Thanks for reading. GenBTC here, signing out. Happy Computing. This has been a 2017 genBTC Production.

Sunday, October 15, 2017

C++ Code Log - activities that I've been learning or doing the past month #1

Things learned:
Zenwinx/Udefrag needs only ntdll.lib no semicolon after. W/ NODEFAULTLIB on.
STUXNET = winapi getprocaddress loadmodule dll on NTDLL kernel WINAPI32 functions.
Depends.exe dependency walker
lib.exe /list:DLL/LIB | findstr strtofind
Assembly: WebAssembly + Emscripten
Use godbolt.org for Compiler Explorer to analyse assembly code.
CPP React
CopperStone w/e
libfat/reactos

Things from the past:
LIB+DLL = Linked Libraries, Shared (DLLs)
LIB+EXP = Static Libraries.
Static Vs Shared. MT vs MD  vs Debug ( MTd vs MDd)
WXWidgets 3.1 Compilation in 8 configs = 32/64 * Debug/Release * Shared/Static
make sure you create the directories right. Its advised to open up the vc($PlatformToolsetVersion)</wxCompilerPrefix> which was actually mentioned in the readme.
WinDDK/WinSDK/NTNDK
MSVC 2010 + SDK 7
2013 = C+11
2015 = C+14
2017 = C+17 
uCRT/uCRTD/MSVCRT/vcruntime.lib

VC Project Files. 
SLN basically does nothing 
VCXPROJ does it all. Edit it carefully, its like XML but less tolerant. 
Things have to go in the right order.
There is also an inheritance system. That is the parent and you can import other .props files in that, based on the order they will inherit in various rates.
The MSVC dropdowns are finicky and you need to realize theres inheritance there, and clearing is not the same as defaulting.
Also theres multiple configurations, and sometimes brings up the wrong config page.
You can set up additonal property pages with the "Property Manager" window. 

There is a utility called SLN2CMake and now VS2017 supports CMake. That way you can convert over.

Intricacies:
/d2noftol3 on CL.exe Compiler switch to disable remade SSE2 functions. Otherwise Linker issue without the ucrt.
https://stackoverflow.com/questions/19556103/how-to-get-vs2013-to-stop-generating-calls-to-dtol3-dtoui3-and-other-funct/31691179#31691179
Its a SSE primitive of Cvtds2dsi?
Windows uses 16-bit(2 byte) chars. Linux is 32-bit. use w_char or wchar_t

C++ Frameworks
Boost165 (Boost Optional. Boost Threading)
Intel Threading Building Blocks https://www.threadingbuildingblocks.org/
QT
CopperSpice
WX
Nana is a library for cross-platform GUI programming in modern C++ style (on Windows/Linux)
LANGUAGE C++11/14
COMPILER Any Standard C++ compiler(Visual C++ 2013, GCC/MinGW and Clang)


Github Projects Cloned or Starred


C++ Videos watched:

CppCon 2016: Dan Gohman “C++ on the Web: Let's have some serious fun." (WebAssembly)
https://www.youtube.com/watch?v=jXMtQ2fTl4c
Unicode Strings: Why the Implementation Matters
https://www.youtube.com/watch?v=ysh2B6ZgNXk
CopperSpice - C++ GUI Framework
https://www.youtube.com/watch?v=LIiwBNvTllk

Friday, July 21, 2017

Compiling FSV - the Jurassic Park program - on Windows 10 with MSYS2

Download the Source Code From https://github.com/genbtc/fsv/archive/master.zip
Github Repository @ https://github.com/genbtc/fsv
If this is what you are going after: http://fsv.sourceforge.net/screenshots/
You came to the right place.

Compiling FSV (File System Visualizer) for Windows takes a bit of effort. This is the tutorial. You need to set up the "MSYS2" development environment, which you can consider similar to Cygwin, if you dont know, it basically sets up a linux environment, and entire filesystem, with windows compatible .exe's compiled of everything, on your Windows machine under a subdirectory like C:\Software\MSYS32

INSTRUCTIONS:
STEP 1 - MSYS

On my machine, I dont like making too many root folders, so I make a C:\Software or C:\Code or whatever name, to put all my dev environment stuff in. Dont make the name too long and MAKE SURE IT DOESNT HAVE SPACES.

Download this version of MSYS2: msys2-i686-20161025.exe (its the latest 32 bit version)
It is available from sourceforge. https://sourceforge.net/projects/msys2/files/Base/i686/msys2-i686-20161025.exe/download

Install it, Pointing the installer to your C:\Software\MSYS32 directory.

We want the "launch after install" button when its done. (otherwise launch it from "C:\Software\msys32\mingw32.exe")
Note: the terminology MSYS, MSYS2, MSYS32 are all the same thing.
Note: However, now we want mingw32.exe (gray icon) don't confuse it with msys2.exe (purple icon)
You should now have this running:

Note: The following commands start with $ to indicate its a command at a command prompt. Dont retype it...

$ pacman -Syu

Hit enter to install the updates. After its done, it asks you to terminate it but you have to click the X until End Process shows and force end it. Also make sure pacman.exe is not running in task manager. Close it if it is.
Note: This requires internet access. If you are using Windows Firewall, you are going to want to disable it during the process because theres just too many .exe's that needs access to manually allow all of them. Dont get stuck on this step trying to do that...

Once again, open back up the MINGW32 shell ("C:\Software\msys32\mingw32.exe") and run the same command again:

$ pacman -Syu

Continue Installing Stuff, these are development packages:

$ pacman -S base-devel unzip mingw-w64-i686-toolchain mingw-w64-i686-gtk2 mingw-w64-i686-freeglut mingw-w64-i686-gtkglext

It will ask you some things, use the default, just press enter each time. Then it will take a while.
base-devel = provides autoconf/automake/pkg-config/tons of stuff.
toolchain = provides GCC,G++ stuff to compile.
GTK2 = a pre-requisite for the user interface (window dressings, menus etc).
Freeglut = an open source version of GLUT which is an API to OpenGL.
GTKGLext = the OpenGL extensions for GTK which we also need.

INSTRUCTIONS:
STEP 2 - Source Code

Download the Source Code From https://github.com/genbtc/fsv/archive/master.zip
(It has to be my repository, because that is where the changes for the Win32 port are)
Extract the FSV source code .zip file from github to your Home dir @ C:\Software\msys32\home\...\ so it lives in C:\Software\msys32\home\...\fsv-master\ (where ... is your username) and has the src/ subdirectory underneath.
Note: the screenshots show fsv-orig (but yours can be fsv-master or fsv)

STEP 2a - Compiling the source code of a gtkglarea-release (pre-req)
Execute the following:
Extract the source of gtkglarea-release-2-0-0.zip to gtkglarea-release-2-0-0 by typing:

$ cd fsv
$ unzip gtkglarea-release-2-0-0.zip

This package provides the required source and the library, and is not available through pacman, so I have provided it in original unmodified form.

Execute the following:

$ cd gtk[TAB](to tab complete)[ENTER]
$ ./autogen.sh

It will run, auto-generate, and auto-execute the configure script for you.
You should see this now:


$ make

You will see a whole ton of stuff scrolling past, this is the actual compiler GCC being run on all the source files. It shows some errors about libtool, but no problem.


$ make install

This will copy the files outputted from "make": libgtkgl-2.0-1.dll and libgtkgl-2.0.a (the linux version of a .dll) to a linux system-wide directory (I could have just provided this file but you wanted to compile stuff)



$ cd ../
$ ./autogen.sh

autogen.sh is the program that checks the autoconf file (configure.ac) and automake file (Makefile.am) input files and parses them into a ./configure script and ./Makefile script
You should see this now:


Execute the configure script.

$ ./configure



If this produces some kind of error, instead of ending happily, refer to Troubleshooting Error #1 below.


$ make



It will compile a ton of stuff, and then end unceremoniously.
But its done now! Now you can install the program by typing:

$ cp fsv.exe /mingw32/bin/

And run the program by typing:
$ fsv







Note: Errors? Read Troubleshooting Below.

Troubleshooting:
ERROR 1:

You did not compile or install the gtkglarea-release-2-0-0.zip properly. Re-read the instructions.
As a last resort, you can override that check by removing the word gtkgl-2.0 from "C:\Software\fsv\configure.ac" (find and delete only that word) and rerunning ./autogen.sh and rerunning ./configure. HOWEVER since you need this package to continue, I doubt that will help.

ERROR2:

                                      OR
ERROR3:

If it says the libgtkgl-2.0-1.dll is missing or you receive the message: " The procedure entry point g_malloc_n could not be located in the dynamic link library C:\..\libgtkgl-2.0-1.dll "
Make sure you place your fsv.exe INSIDE the MSYS binary dll directory (ex: C:\Software\msys32\mingw32\bin\ )
Then you can just make a shortcut to it, and run that.
Alternatively, you can add this C:\Software\msys32\mingw32\bin\ directory to your system environment PATH variable, in System settings as shown:

Alternate to this, you can also create a launcher .bat file. such as "LaunchFSV.bat",  file as a launcher. This sets the required Windows' System Environment "PATH" variable. When you are inside MSYS, it sets the PATH for you, so you can launch it from there no problem by typing ./fsv or ./fsv.exe , But when you try to launch it from Windows, you get libgtkgl-2.0-1.dll errors because it cannot find the all the .dlls we installed through MSYS (in addition to that specific one).
LaunchFSV.bat
PATH=%PATH%;C:/Software/msys32/mingw32/bin;./;
start fsv.exe


ERROR 4:
Disregard this error. It has no impact on the final outcome, and can safely be ignored.

ERROR 5:
You downloaded the source from the wrong repository. You need the genBTC repository (mine), because that is where the win32 fork changes are located. https://github.com/genbtc/fsv and https://github.com/genbtc/fsv/archive/master.zip
OTHER ERRORS NOT LISTED:
Report any bugs on the github repo. Everything will be handled through Github.

Using FSV:
It takes a CONSIDERABLE amount of time for the program to traverse the entire directory structure, (at anywhere from 300-6000 per second) and it does not display much until it does. For your first run, you should point it to a folder with a small amount of files and subdirectories.  FSV.exe takes path arguments, and the bat file does too, so you can put the .BAT file in your Windows SendTo folder, and then right click any directory and Send To LaunchFSV.bat, and it will launch starting in that dir.


Theres a bug in the "Change Root" command, 
You have to delete everything in the bottom text box, THEN click OK, for it to work. It will open to what it says in: "Selection". If you fail to do this, it will just rescan whatever dir you're in, and that sucks. I will work on fixing this soon.

SCREENSHOTS:

This is how it looks in Tree View:
This is how it looks in Map View:
This is how it looks in Disc View (very experimental):


Enjoy. I have not done anything significant enough to claim a copyright on this project, please refer to the original owner's repository and license documentation @ https://github.com/mcuelenaere/fsv

Thursday, July 13, 2017

FSV Filesystem Navigator - from Jurassic Park - SGI IRIX Sun Solaris clone - running on Windows

Ive been working on porting this 3D Filesystem Viewer or FSV from Linux to Windows. It looks like this and was made by this guy: https://github.com/jtsiomb/fsnav

(little sample of errors to clear)






For some reason i decided to try to download this one https://github.com/mcuelenaere/fsv because it has a lot more to it, like a whole menu/window/status/UI :) and it was wayyy harder to compile, in fact its not even fully working, theres a few issues preventing me from distributing it. But as you can probably tell, the look is just NOT there, its all wrong, you cant move well. Its a pretty bad program, even if it was for all intents and purposes the most real "clone" of the original SGI code. Which I do have and its from 1992-03-19 which is where the visual aesthetic of the blue Sky, grass, desert, ocean, space, indigo themes come from. Nobody elses code really has that. And it probably has more stuff, but i dont think i have the actual source available, just half of it.


Coded by
-Abei Villafane