When data is scattered across multiple databases with different file formats, most IT administrators will immediately think of data integration solutions to solve corporate data mining operations. Data integration is the process of bringing together all of the organizations data into a single view that allows new manipulations and reporting previously not available.
Combinations of flat file or relational databases needing to be merged with data stored in spreadsheets, word processing files, letters or emails, and calendars never used to be possible without expending significant person time manually collating the information, sometimes many times over if the data aged and needed to be recompiled for monthly and quarterly reports.
Data integration software natively understands multiple data sources and can transform data using standard database tools, sometimes by directly accessing the source file, at other times thru the use of wrapper software that queries the source and then converts the data for use in other applications. Most data integration solutions will require extensive customizing owing to complexity of the task.
Most corporate data from databases and other software applications are stored in proprietary file formats that are rarely compatible with standards based formats such as XML. Lacking interoperability they are difficult to deploy, and create significant challenges for the organization in their efforts to integrate data and move towards a standards-based future.
Investing a great deal of time and cost in switching enterprise software when in fact existing applications already contain all of the data needed can be better achieved by adopting a data integration solution. Solutions that offer ease of querying thru simpler user interfaces and require less training are gaining in popularity.
Currently the most popular data integration products require teams of programmers, costly server installations, and often a great deal of ongoing maintenance. Enterprise information can be very complex and require a lot of storage, resulting in huge queries. Integration products therefore need to scale to the enterprise level consistently and with relative ease.
As the market for data integration software matures visual tools are being developed by many of the smaller specialist vendors at prices far below those of the established market leaders. Visual integration tools allow new solutions without the overhead of a team of experts. Once the software has been installed a small group of trained personnel are able to drag and drop data fields and create new relationships with ease.
The advantages to the corporation are many, costs may be reduced, training can be simplified, and new reports designed on as needed basis allowing greater use of the raw data across an ever larger range of departments. Previously the preserve of the strategic planning and marketing departments, data integration solutions are now finding greater application within manufacturing, logistics, legal, and many other departments not normally associated with accounting and CRM data.
With the move to standards-based file formats such as XML and the significant use of Linux powered servers and database applications open source solutions for data integration are entering the market, offering attractive new possibilities.