[Jeff Reed]

Good Feedback thank you Kathleen. Part of my initial design considerations were to minimize re-coding of existing systems that dynamically loaded files based on NAACCR version from flat files. but it would make sense to load the XML directly and transform from there. Last time I had to load XML data we had to revert back to flat or delimited files due to performance issues that may not be as big of a problem with current SW/HW. And no butting out your in too deep now …

[Jeff Reed]

Many thanks to all for the great resource links provided. I have always been a big fan of “borrowing” code where available and will be sure to develop as generically as possible to help share the wealth.

My next steps will be to review open source tools as Rich suggested, review XMLExchangePlus, thank you Kathleen, and dig into the Java logic, thank you Fabian. Should be able to get some good design and architecture ideas from those sources.

At this point I am thinking of developing C# class libraries for generic table definitions and integrating the code for converting XML to flat file’s for “bulk loading”. I am thinking the flat files could be “|” delimited or fixed length format with wrappers to load the flat files into the respective databases. This not not rule out using the tools suggested above.

The second step of post load ETL processing would be more DB specific, Oracle in my case, as the data is validated and normalized for relational DB storage. We should be able to stick to ANSI standard SQL to help with different DB platforms.

Many thanks to all, will keep you posted, I feel the excitement building!

Jeff

[Author]

Posts