[rant]
The more I do, and the more I work back towards the world of Tim Bray's 'back-door technology', the more I come to the conclusion that Ted Neward is only half right about O/R Mapping being Java's Vietnam. I think he's only half right because I don't think he took it far enough. The database is a sacred pile of mud That Shall Not Be Touched in the Java world. On TSS every O/R Mapping discussion gets slammed with "the dba (all hail) would never let you do X." Bull shit. Denormalization is only allowed if the DBA holds your hand and signs off. Queries have to be optimizable by the DBA. SQL is the one true god, and the DBA is the path to him (the SQL god is way to insecure to be a female god).
The SQL God spaketh to the DeeBeeAie through a Burning Frog (okay, TOAD, but frogs are special too) and he said "Thou Shalt Profit By Worshipping Me." He explained that "Math Supports Me, So I Am Correct And Provable And That Is Good Because Ye Dirty Programmers Gave Up On Mathematical Provability When Ye Rejected LISP And ML And Haskell And All That Is Good In Maths." Most people repeating the story leave off the second part of that sentence and just say "relational algebra is based on math, so it is good, unlike all those dirty imprecise programming languages that let you do things like pointers."
I think I mentioned before that I got in trouble once for talking about creation myths. I got in trouble because I called the creation stories of a living religion "myths", and the person I was speaking to was a priest. He kindly pointed out that it is only a myth when no one believes it anymore, until then it is Religion or Truth depending on whether you believe it or not. We laughed about this. I digress.
This is religion now. Sorry. I have met some really bright people who Believe in relational algebra, and for the most part they don't particularly like the current implementations commercially available. I find that amusing, and sad. I have met other really bright people who believe in the Database, and hate that people are allowed to do things to its pristine and perfect structure. I've met other smart people who have this totally bizarre idea that the database is a tool to help them solve a problem, nothing more, nothing less. I sort of agree with them.
The Database, compared to the database, is sacred because it is the integration point for everything. The industry was convinced, for better or worse, that Programmers Are Stupid And Do Stupid Things But Databases Are Pure And DBAs Are Perfect about the time that 4GL was popular. Sort of makes sense, in retrospect, and probably was quite true. The Database was the center of the world, and was mysterious, and special, and had special incantations and priests to care for it and feed it. Applications existed to speak to the database, sometimes (blasphemy!) by speaking EssQueueEll directly, but (orthodixically) through the Stored Procedure written by the DeeBeeEh (how do you phonetically spell out "A" anyway?).
Pop quiz: is application integration at the database level the "Right Way" ?
The answer is, of course, "it depends." For a decade and a half it has been either the "right way" or the standard way for a lot of development. This has lead to it being pretty easy to do. Embedded databases are not ENTERPRISE or SCALABLE or SOMETHING-ELSE-THAT-SOUNDS-IMPORTANT. Well, actually... anyway. Externalizing the database isn't a bad thing. In fact it's really good. And taking care of it, and building a correct data model are really good ideas.
But don't worship the bloody thing! Lift your head up from behind the pew in front of you and look at it. Write out some theses and pin them to Oracle's door. Yeah, relational databases do a lot right, but they are tools that are supposed to help you do your job, not be some opaque monstrous thing requiring Holy Intermediaries to intervene on your behalf.
JDBC is crap, sorry, have to say it. It was mostly specced out to expose database features, not make programming easy. Entity Beans are a nice exploratory attempt at making a convenient API, which was a good prototype which got pushed too far. Transparent O/R mapping, the current sacred frog, is another nice shot, I think, but I have yet to see an implementation which isn't painful (including OJB which I find to be the least painful Java one, even if it isn't spelled H-I-B-E-R-N-A-T-E or E-J-A-Y-B-E-R-N-A-T-E) -- (if you need thousands of lines of proprietary XML scripting, or a tool to generate it for you and leave you cursing, to make it work, it is painful). Externalized mapping (iBatis being the current flavor of choice) is a step in the right direction, but is still painful. It continues to worship from below, not celebrate, and has thousands of lines of xml scripting to boot.
The problems all come back to "the database is sacred, and hard, and only the special elite few are allowed to understand it." Give it up. Celebrate the database. It exists for one purpose -- to hold state for you while you reboot, or in case you don't have enough memory to keep it all available yourself. It has lots of gewgaws to make you think it exists to do more (oh its New-cue-ler, erm, I mean Atomic) which, shockingly, ain't that hard to do. Any API which talks to it and holds ANY state and is reliable is also New-Cue-Ler (yep, that includes your Entity Beans, your O/R Mapper, your JDBC abstraction layer, your iBatis, etc). Holy bats, you mean going New-Cue-Ler is not np-complete. You can even get different levels of Nuclear Armament, ones where people can see your state, or where you copy your state and explode if someone changes it before you let go, or... the list goes on. It isn't in the easy category, but sure isn't in the hard category. It is like writing a payroll system, complex, yes, but well understood and not technically difficult, heck, there are whole books written on it.
I am not belittling the importance of maintaining a strong nuclear arsenal. I mean, the only reason we haven't finished conquering the remaining oil producing nations is because they support read-committed isolation. But don't worship your nuclear arsenal, use it. Make a few radioactive parking lots for your SUV driving users. Er, wait, the metaphor is getting totally out of hand here...
A good database is an amazing tool. Stop fighting it. Stop pretending it isn't there. Stop making appeasements to it through specially anointed go-betweens. Use the services it provides to do what they do best -- hold the state which you need to have held. Store it in the way that makes your job easier. Programmer time, despite the best attempts of a lot of sales folk, is still more expensive than database licenses (okay, programmer time of programmers whose work you won't have to pay some other, much much more expensive, programmers to fix, anyway). Go ahead and get your hands dirty -- normalize your schema, denormalize your schema, use dynamically generated schemas (GASP!).
If you are worried about correct behavior, write tests.
If you are worried about integration, expose an integration API via messaging or SOAP.
If you are worried about your job, produce useful software.
That last one is really important. Think about it some. If your job is to spend the IT budget, by all means, worship your database. If you job is to produce useful software, stop fighting and learn to love the bomb. Oops, bizarre metaphor snuck back in.
Maybe copying ODBC and sticking a J in front wasn't the best approach, who knows. I wasn't on the expert group, and cannot whip out the "Right Way" right now; JDBC/ODBC/DBI/etc have all provided lots of good use for years, just like assembly language did (thank you someone on TSS for that reference, not sure who it was or I'd credit you). It is easy to second guess, so I should shut up, but I'd dearly love to have convenient access to the relation constructs in the database from the application language. Give me a handle on a Table, or a ForeignKey =) I'd still even use O/R mapping solutions when they were the best option, I promise!
[/rant]