Skip links

Accelerate Development With a Virtual Data Pipeline

The term «data pipe» refers a series processes that gather raw information and convert it into an app-friendly format. Pipelines can be real-time or batch. They can be installed on-premises or in the cloud. their tools are commercial or open source.

Data pipelines are like physical pipelines that carry water from a stream into your home. They transfer data from one layer to another (data lakes or warehouses) the same way physical pipes bring water from the river to a house. This allows analytics and insights to be derived from the data. In the past, transferring the data required manual processes such as daily uploads and long wait time for insights. Data pipelines replace manual processes and enable companies to transfer data more efficiently and without risk.

Accelerate development with a virtual data pipeline

A virtual data pipeline provides large infrastructure savings in terms of storage costs in the datacenter as well as remote offices, as well as equipment, network and management costs involved in deploying non-production environments like test environments. Automating data refresh, masking, and access control based on roles, as well as integration and customization of databases, can help to reduce time.

IBM InfoSphere Virtual Data Pipeline is a multicloud copy-management solution that separates testing and development environments from production infrastructures. It uses patented snapshot and changed-block tracking technology to capture application-consistent copies of databases and other files. Users can instantly create masked, near-instant virtual copies of databases from VDP to VMs and mount them in non-production environments so that they can begin testing within minutes. This is particularly beneficial for accelerating DevOps and agile methods as also speeding time to market.

Leave a comment