Data Analysis Technology

Data is everywhere. In itself, data does not translate to anything meaningful. We know that the true value is not in the data per se but in the knowledge hidden in it.

Success of transforming data to meaningful information is the first step. Identifying valuable data patterns, conducting effective data analysis, deriving useful information from meaningful sets of data - these are the key aspects of data engineering.

There are several key issues that need to be addressed by innovative approaches to large scale data Integration – some of which are listed below.

• Integrating large diverse communities and devices that store and deliver data.

• Providing robust and reliable infrastructure within the budgetary constraints.

• Handling of real-time updates from continuous and discrete data streams and distributed data entry points.

• Supporting evolutionary growth of adaptive systems.

• Constructing self-assembling and self-maintaining systems.

• Maintaining data consistency, especially in the case of large data sets.

Challenge

Most current development architectures that support the core infrastructure sectors are heavily tied to the database sources and architectures underneath. This poses serious problems for the consumers of data - both software consumers and final consumers. 

On the software consumer side, the interfaces that handle the data tend to be arbitrarily complex because of the non-standard mechanisms. 

To the final consumers, the lack of isolation of data, schema, viewer and processing logic creates very inflexible and difficult user interfaces. Even data stored in databases have abstraction issues due to the variety of software decisions designers make. 

Data processing methodologies present another challenge. Due to the variety of processing patterns (compounded by the variety of data representations) consistent view and processing of information across multiple sources turns out to be difficult.

Solution

To address the overall challenges discussed above, a broker architecture that provides the transparency in data access and processing methodology, as well as conformance to current state-of-the-art standards can provide a powerful development paradigm for consumers of data.  It also can significantly aid in intensive applications of knowledge discovery applications and advanced decision support systems.