2020-01-07

hadoop is not dead

https://tech.marksblogg.com/is-hadoop-dead.html

When you hear "No one needs big data", look over the CV of the speaker. An African telecoms carrier going through amazing levels of growth is not going to reach out to a greenfield JavaScript Web Developer and ask them if they could help architect their data platform and optimise their billing calculations. You might find a lot of internally-hosted web applications in an Airline's headquarters but when it comes to analysing PBs of aircraft telemetry for predictive maintenance there might not be any PHP developers on that project.

Companies producing PBs of data often have a queue of experienced consultants and solutions providers at their door. I've rarely seen anyone plucked out of web development by their employer and brought into the data platform engineering space; it's almost always a lengthy, self-retraining exercise.


Disks are catching up in speeds as well. There was a 4 x 2 TB PCIe 4.0 NVMe SSD card announced recently that could read and write at 15 GB/s. The price point of the PCIe 4.0 NVMe drives will be very competitive with RAM and provide non-volatile storage. I can't wait to see an HDFS cluster with some good networking using those drives as it'll demonstrate what an in-memory data store with non-volatile storage with the rich, existing tooling of the Hadoop ecosystem looks like.



The migration to a columnar storage solution for this client brought down those infrastructure costs by two orders-of-magnitude and sped up querying times by two orders-of-magnitude.



2019-10-31

the minifi can not be installed as a service or the service do not start

error in the log file of the minifi, when trying to run install-service.bat in order to run minifi as a service

Exception in thread "main" on: .\conf\config.yml (The system cannot find the path specified)


solution:
edit bootstrap.conf and edit the config file folder as a full path  (see below) 
Double the \ if you are on windows !

# The location for the configuration file
# When running as a Windows service use the full path to the file
nifi.minifi.config=C:\\minifi-0.5.0\\conf\\config.yml


from the manual
https://nifi.apache.org/minifi/system-admin-guide.html


Running as a Windows Service

MiNiFi can run as a Windows service. To do so, you must modify the conf/bootstrap.conf to set absolute paths for some properties. The properties are:
  • lib.dir (e.g. C:\\minifi-x.x.x\\lib)
  • conf.dir (e.g. C:\\minifi-x.x.x\\conf)
  • nifi.minifi.config (e.g. C:\\minifi-x.x.x\\conf\\config.yml)
You can now install the MiNiFi service by running the install-service.bat script. To remove the service run the delete-service.bat file.
The minifi.exe in MiNiFi bin directory is used to run MiNiFi Windows service. The bundled one is for 64 bit architecture and requires 64 bit JRE. If you have to use 32 bit JRE for some reason, you need to replace the minifi.exe file with the one for 32 bit to make MiNiFi service runs successfully. To do so, go to Commons Daemon project download page, download the binary (e.g. commons-daemon-1.1.0-bin.zip), extract it and replace bin/minifi.exe by copying commons-daemon-x.x.x-bin/prunsrv.exe into MiNiFi bin directory as minifi.exe to overwrite the 64 bit exe with the 32 bit one.

2019-10-25

microstrategy list of graphical commands

Please notice that you need to open your “<HOME_PATH>/bin” to find these files: 
MicroStrategy Command Manager – mstrcmdmgrw
MicroStrategy Configuration Wizard – mstrcfgwiz
MicroStrategy Connectivity Configuration Wizard – mstrconnectwiz
MicroStrategy DB Query Tool – mstrdbquerytool
MicroStrategy Diagnostics and Performance Logging – mstrdiag
MicroStrategy Health Center Configuration Wizard – mstrsuppappcfgwiz
MicroStrategy Health Center Center Console – mstrsuppappgui
MicroStrategy License Manager – mstrlicmgr
MicroStrategy ODBC Administrator – mstrodbcadx
MicroStrategy Service Manager – mstrsvcmgr


2019-09-30

postgresql partitions partition of table from to inclusive

When creating a range partition, the lower bound specified with FROM is an inclusive bound, whereas the upper bound specified with TO is an exclusive bound.

So in fact in
CREATE TABLE parsel_2014_10 PARTITION OF parsel_part 
     FOR VALUES FROM ('2014-10-01') TO ('2014-10-31');
the date 2014-10-31 is not included in this partition, and hence nowhere in all your partition tables (and same for all ending dates)

6 doctors

“Six Best Doctors in the World…
  • Sunlight
  • Rest
  • Exercise
  • Diet
  • Self Confidence
  • Friends”