• subscribe
August 24, 2004 12:00 AM

Using DTS to Extract Data from Multiple Database Systems

SQL Server Pro
InstantDoc ID #43632

My company is running several different database systems, including Oracle and Ingres. To produce financial reports, we need to make a central repository for the relevant reporting data, which we'll extract from these multiple data warehouses. I was thinking of using SQL Server as the repository and Data Transformation Services (DTS) to perform the extraction. Would that approach work? Also, I read that the new SQL Server 2005 Express is limited to storing databases with a maximum size of 4GB each. Does SQL Server 2000 also have this database-size limit?

DTS is an effective extraction, transformation, and loading (ETL) tool, but it relies heavily on the quality of the OLE DB providers you use to read from the heterogeneous systems you mentioned. Many customers have found that it can sometimes be faster to unload data to delimited text files on each host, ship the text files to SQL Server by using a file-transfer tool (or make them available to DTS as a network share), then use DTS to load them into SQL Server. DTS, which uses bulk-load interfaces, is highly optimized for loading data into SQL Server. However, if I were loading data into Oracle, I'd probably use DTS to run a SQL*Loader command task to get maximum throughput (and I'm sure IBM has a similar batch data load API).

As for your question about the new SQL Server 2005 Express, a replacement for Microsoft SQL Server Desktop Engine (MSDE), that database is designed largely for organizations that need a database engine to package with an application. MSDE's size limit per database is even smaller—2GB. But the full versions of SQL Server 2000 and the upcoming SQL Server 2005 don't have the database-size limit you noted. So if you're storing more than 2GB of data, you should consider SQL Server 2000 Standard Edition, which also gives you a set of database management tools that you aren't licensed to use with MSDE. And if you buy a server CPU-based license, you can connect as many users as your hardware will allow, up to the 32,727 limit.



ARTICLE TOOLS

Comments
  • Eryck
    5 years ago
    Oct 12, 2007

    Up to now , there has not been a tool made for extracting data out of Oracle that has turned out more efficient by Fast Extract for Oracle distributed by a company called IRI. its benchmarks show the extraction of a 50,000,000 rows table just under 9 minutes. The flat file formats is known to be the most adequate for data transformation and data manipulation.

  • a
    7 years ago
    Nov 12, 2005

    fastreader is at http://www.wisdomforce.com

  • a
    7 years ago
    Nov 12, 2005

    You are right about taht. Unloading data from Oracle with DTS is significant slow and adds heavy load on Oracle. We used fastreader from wisdomforce to extract data from Oracle into flat ascii dump files and then loaded with bcp into sql server. fastreader also creates on fly control files for bcp along with the script for load import into sql server

You must log on before posting a comment.

Are you a new visitor? Register Here
  • SP1?
    I know there is a SP1 for SQL 2008 R2 available....and there is a "feature pack" as well... ...
  • SQL database mirroring
    I have SQL Server 2008 R2 Enterprise 64bit on Windows 2008 R2 Enterprise 64bit.  Each SQL Server has...
  • Dell Compellent Disk Drive
    Does anybody has experience with Dell Compellent Disk Drive? Basically, this system manages all disk...
  • Sql server performance tuning
    I need to find a tool that help me to optimize sql server,queries,improve the performance and solve ...