some serious flaws in accordance with efficiency and space utilization. For example, when a student registers for more than one course, then details of the student has to be entered for every course he registers. This can be overcome by dividing the data across multiple related tables. Figure 2.12 represents the data in the above table is divided among multiple related tables with unique primary and foreign keys.
Relational tables have attributes that uniquely identify each row. The attributes which uniquely identify the tuples are called primary key. StudentId is the primary key, and hence its value should be unique. Attribute in one table that references to the primary key in another table is called foreign key. CourseId in RegisteredCourse is a foreign key, which references to CourseId in the CoursesOffered table.
Table 2.1 Student course registration database.
Figure 2.12 Data divided across multiple related tables.
Relational databases become unsuitable when organizations collect vast amount of customer databases, transactions, and other data, which may not be structured to fit into relational databases. This has led to the evolution of non‐relational databases, which are schema‐less. NoSQL is a non‐relational database and a few frequently used NoSQL databases are Neo4J, Redis, Cassandra, and MongoDb. Let us have a quick look at the properties of RDBMS and NoSQL databases.
2.4.1 RDBMS Databases
RDBMS is vertically scalable and exhibits ACID (atomicity, consistency, isolation, durability) properties and support data that adhere to a specific schema. This schema check is made at the time of inserting or updating data, and hence they are not ideal for capturing and storing data arriving at high velocity. The architectural limitation of RDBMS makes it unsuitable for big data solutions as a primary storage device.
For the past decades, relational database management systems that were running in corporate data centers have stored the bulk of the world’s data. But with the increase in volume of the data, RDBMS can no longer keep pace with the volume, velocity, and variety of data being generated and consumed.
Big data, which is typically a collection of data with massive volume and variety arriving at a high velocity, cannot be effectively managed with traditional data management tools. While conventional databases are still existing and used in a large number of applications, one of the key advancements in resolving the problems with big data is the emergence of modern alternate database technologies that do not require any fixed schema to store data; rather, the data is distributed across the storage paradigm. The main alternative databases are NoSQL and NewSQL databases.
2.4.2 NoSQL Databases
A NoSQL (Not Only SQL) database includes all non‐relational databases. Unlike RDBMS, which exhibits ACID properties, a NoSQL database follows the CAP theorem (consistency, availability, partition tolerance) and exhibits the BASE (basically, available, soft state, eventually consistent) model, where the storage devices do not provide immediate consistency; rather, they provide eventual consistency. Hence, these databases are not appropriate for implementing large transactions.
The various types of NoSQL databases, namely, Key‐value databases, document databases, column‐oriented databases, graph databases, were discussed in detail in Section 2.3. Table 2.2 shows examples of various types of NoSQL databases.
Table 2.2 Popular NoSQL databases.
| Key‐value databases | Document databases | Column databases | Graph databases |
|---|---|---|---|
| Redis | MongoDB | DynamoDB | Neo4j |
| Riak | CouchDB | Cassandra | OrientDB |
| SimpleDB | RethinkDB | Accumulo | ArangoDB |
| BerkeleyDB Oracle | MarkLogic | Big Table | FlockDB |
2.4.3 NewSQL Databases
NewSQL databases provide scalable performance similar to that of NoSQL systems combining the ACID properties of a traditional database management system. VoltDB, NuoDB, Clustrix, MemSQL, and TokuDB are some of the examples of NewSQL database.
NewSQL databases are distributed in nature, horizontally scalable, fault tolerant, and support relational data model with three layers: the administrative layer, transactional layer, and storage layer. NewSQL database is highly scalable and operates in shared nothing architecture. NewSQL has SQL compliant syntax and uses relational data model for storage. Since it supports SQL compliant syntax, transition from RDBMS to the highly scalable system is made easy.
The applications targeting these NewSQL systems are those that execute the same queries repeatedly with different inputs and have a large number of transactions. Some of the commercial products of NewSQL databases are briefed below.
2.4.3.1 Clustrix
Clustrix is a high performance, fault tolerant, distributed database. Clustrix is used in applications with massive, high transactional volume.
2.4.3.2 NuoDB
NuoDB is a cloud based, scale‐out, fault tolerant, distributed database. They support both batch and real‐time SQL queries.
2.4.3.3 VoltDB
VoltDB is a scale‐out, in‐memory, high performance, fault tolerant, distributed database. They are used to make real‐time decisions to maximize business value.
2.4.3.4 MemSQL
MemSQL is a high performance, in‐memory, fault tolerant, distributed database. MemSQL is known for its blazing fast performance and used for real‐time analytics.
2.5 Scaling Up and Scaling Out Storage
Scalability is the ability of the system to meet the increasing demand for storage capacity. A system capable of scaling delivers increased performance and efficiency. With the advent of the big data era there is an imperative need to scale data storage platforms to make them capable of storing petabytes of data. The storage platforms can be scaled in two ways:
Scaling‐up (vertical scalability)
Scaling‐out (horizontal scalability)
Scaling‐up. The vertical scalability adds more resources to the existing server to increase its capacity