How it Works

How it Works

Datera Elastic Data Fabric delivers elastic block storage for enterprises and service providers in a single software platform. With unprecedented scale-out to handle today’s digital workloads, it is the only data management solution uniquely built for on-premises clouds that delivers enterprise application performance at web-scale economics and operations.

Multiple Generations and Types of Hardware

 

Data Plane: High Performance and Low Latency

Each Datera node is built on a direct-on-disk, log-structured key/value store that was designed from ground up to support the raw power of next-generation flash technologies on any standard x86 server. This key/value store forms multiple storage media tiers: high performance flash (NVMe), capacity flash (SAS/SATA) or capacity drives (HDDs), variously into all-flash or hybrid nodes.

To deliver low write latencies, each Datera key/value store incorporates persistent memory (NVDIMMs, NVRAM, and soon 3D Xpoint) that absorbs writes “instantly,” coalesces them and then de-stages them sequentially to the flash tier. On hybrid nodes, the flash tier itself serves as a write-back cache, further coalescing writes, and streaming them down to the capacity tier.

Datera pools the various data tiers in mix-and-match nodes via a novel lockless scale-out coherency protocol. This protocol can multi-path data across any number of nodes in the data center, thereby aggregating massive amounts of read/write bandwidth to support the scale and speed needed for modern real-time data processing applications.

The unique Datera design combines the best of both worlds: the operational flexibility of software-defined scale-out storage on commodity x86 servers, and the excellent performance and latencies of traditional storage arrays. On hybrid nodes, it even does it at the cost of capacity drives (HDDs).

The Datera data plane has the full enterprise feature set expected today:

Datera is a major contributor to the Linux kernel, with a keen focus on IO optimizations across the data path to drive low latency for a broad range of standard x86 servers. For instance, Datera authored the industry-standard Linux-IO (LIO) Target Driver stack that adds a full suite of block storage protocols to Linux, including iSCSI, iSER, Fibre Channel, FCoE, and soon NVMe-OF.

This allows Datera to connect through its own industry-standard block storage protocols to any traditional or cloud native workloads, with zero host-side footprint.

Control Plane: Intelligent Scalability

Datera has decoupled scalability from performance with its a fully symmetrically distributed control plane. Its simplified deployment and operational model, in conjunction with standards-based data protocols, eliminates the need of any external control or data elements, such as separate metadata or controller nodes, external protocol gateways, or host-side proxies.

The Datera data infrastructure can seamlessly scale up or down with the addition or removal of nodes. Deployments can start small and inexpensively, and gradually grow with evolving needs by adding mix-and-match nodes , including rolling technology upgrades.

Datera continuously re-balances all system elements and automatically migrates all data, to guarantee continuous application access with zero downtime.

 

Management Plane: Intelligent Automation

  • Application preferences are codified as intents in templates (or profiles or manifests), which the policy engine translates into dynamic policy sets (performance, cost, QoS, availability, replication, snapshots, security, access, authentication, etc.), and adapts them according to their current tenancy and roles at any given point in time.
  • The diverse capabilities of each node are abstracted and dynamically exported through idempotent service agents that run on each node.
  • The policy engine is network-aware to learn the changing operational constraints in the datacenter, such as network topology, availability zones, power domains, latency cliffs, migrating application endpoints, etc.

At the core of the Datera system is a policy engine that automatically optimizes all elements of the data infrastructure by continuously adapting to changing application requirements, hardware capabilities and operational constraints in the data center.

Based on its understanding of applications, node capabilities and data center constraints, the Datera policy engine continuously optimizes all elements in the data infrastructure, across all applications, tenants and environments. As intents are invariant, they bring seamless portability, scalability and composability to stateful applications across data centers and clouds.

The powerful Datera policy engine can easily fit data to applications across the data center, rather than forcing it into rigid handcrafted application silos, always place it on the most cost-efficient nodes (incl. flash, hybrid or any combination), and coherently manage its full lifecycle. The result is ground-breaking scalable automation and price/performance elasticity that brings the data agility that is key to any successful cloud project.

Every aspect of the system is fully programmable through a rich RESTful API. Based on its REST API, Datera provides multiple language API bindings, including a python-base SDK and a powerful command line interface (CLI), to simplify integration and accelerate adoption.

Datera seamlessly integrates with all relevant workload orchestration frameworks: VMWare, Openstack, Docker, Kubernetes, Mesosphere/DC-OS and Cloudstack.

The Datera data infrastructure dynamically composes all its resources on the fly to deliver a rich spectrum of application needs: performance, quality of service, security, availability, and reliability. As businesses adapt to evolving needs, Datera provides a single data infrastructure that can support them all.