Companies producing PBs of data often have a queue of experienced consultants and solutions providers at their door. I've rarely seen anyone plucked out of web development by their employer and brought into the data platform engineering space; it's almost always a lengthy, self-retraining exercise.
Disks are catching up in speeds as well. There was a 4 x 2 TB PCIe 4.0 NVMe SSD card announced recently that could read and write at 15 GB/s. The price point of the PCIe 4.0 NVMe drives will be very competitive with RAM and provide non-volatile storage. I can't wait to see an HDFS cluster with some good networking using those drives as it'll demonstrate what an in-memory data store with non-volatile storage with the rich, existing tooling of the Hadoop ecosystem looks like.
The migration to a columnar storage solution for this client brought down those infrastructure costs by two orders-of-magnitude and sped up querying times by two orders-of-magnitude.
Running the (Segment) Space Advisor - on a Partitioned Table - Here is a quick demo on running the Segment Space Advisor manually I need to start with the ADVISOR privilege $sqlplus SQL*Plus: Release 184.108.40.206.0 Produc...
Acum o zi