Commentary on the datastore benchmark
I’ve gotten some excellent feedback on the benchmark I posted a few days ago, titled “Hibernate is faster than I thought it was,” from Hibernate supporters who were a little upset, from GigaSpaces people who had some optimization tips, from people who actually prefer accurate benchmarks…
All valid, actually. So I’m rebuilding the speed tests, for version three. I have version two, and it has some illuminating data as well (as well as speeding up almost all of the operations!) but it’s still lacking some crucial pieces, so instead of building on a flawed base, I’m re-architecting the whole dadgum thing.
The next version will have:
- More datastores supported. JDBC (via Spring’s JdbcTemplate), MongoDB, internal and external GigaSpaces, Hibernate (embedded and external DB), Java Serialization (yes, Java Serialization, which was the first one I implemented to test out the architecture) and any others I can throw together; the new project structure makes this kinda easy.
- Multithreaded access to the datastores.
- Cache key misses (the key generation phase generates keys that should not exist).
- DB-optimal models (I’m willing to contort the DAOs to optimize for the specific datastore, which was required for MongoDB.)
I can tell you from revision two of the benchmarks: Hibernate still does well, but MongoDB and GigaSpaces still fly past it, with embedded GigaSpaces being the fastest, and MongoDB showing some very good times. We’ll see how well that goes in the next round.
I’m still leaving transactions in there, and yes, this is a throughput test; one comment pointed out that I was testing Hibernate cache quite a bit (rather than Hibernate itself), to which I say: right on, brother. I’m not configuring the cache all that much; I’m turning on ehcache and leaving it at that.
Turning off the cache would be horribly unfair to Hibernate, and that violates the whole idea of seeing how it performs with sort-of-fake data.
I’m nowhere near done.
- I have the Java Serialization done, and most of the JDBC DAO (still haven’t finished query by example, and I’m still testing the rest of it.)
- The data model isn’t complete yet; I still haven’t added attributes to the model. I’m still debating whether to make them searchable or not, which would translate the model to an RDF-like container. It will also slaughter performance for things that manage relationships externally (i.e., Hibernate, JDBC.) It also violates the spirit of the original purpose (which was to model external procedure calls as messages) so I’m okay with not making them searchable, but I’m pretty sure people would want it.
More as I finish more of the benchmarks.
Hibernate is faster than I thought it was.
I was playing around with a speed test of Gigaspaces XAP, and got some pretty good numbers out of it – but I couldn’t escape the thought that I was missing something. Then I had it: a comparison point!
If GigaSpaces is X, then… it’s X. But if we compare it to something else – a Y, if you will – then we get a better sense of whether it’s actually fast or not.
Timings capture only the time spent in the DAO, as captured by a dynamic proxy. Therefore, a given test might take four minutes, but yield only 60 seconds’ worth of measurements, because none of the non-DAO stuff was measured. The idea was to isolate any testing code away from the actual focus of the speed test, which was: “How fast do read/write/delete operations run under simple transactions, when connecting to external servers?”
The tests are serial – one operation at a time, over and over again. This is way not real-world. I know it, you should know it, if you take these benchmarks as anything other than a “hmm, that’s interesting” data point, a yeti will hunt you down and eat your toes.
So I created a simple 1:M mapping, an Owner to OwnerAttribute model, and created mappings for Hibernate and GigaSpaces. (Same object, just annotated for both GigaSpaces and Hibernate.) The test ran over four basic operations in bulk, using transactions on every operation, based around a common DAO interface.
When I first ran the tests, I was horrified: Hibernate was so freakin’ bleedin’ farglin’ bleepin’ slow SLOW SLOW! Amazingly slow. Horrifyingly slow. I isolated the cause to two reasons: session acquisition and object model. The session acquisition surprised me: I was acquiring one session and using it for all operations, which I thought would be faster.
After all, it takes time to acquire a session, does it not? Boy, was I wrong.
This is why you test your assumptions, people. Having the DAO get a new session from the factory every call sped things up dramatically. Score one for Hibernate, one point lost for me.
The object model was a stickier problem. The test creates a number of attributes for the Owner object and stores them as well.
When I had OwnerAttribute stored as an external record (@OneToMany, etc.) Hibernate crawled.
Crawled bad.
Crawled “Are you reusing the session, Joe?” bad.
So then I thought about the object model. In the real-world situation this models, I don’t ever search for owners by attributes; the actual attributes are more like function call parameters. So there was no reason to have them stored externally, in either the Hibernate or GigaSpaces model.
Therefore I stored them as a Serializable in the Owner class. This simplified the object model by a lot (the DAO for GigaSpaces would have had to compensate for an Owner being an aggregate) and also, well, sped things up for Hibernate.
The tradeoffs: I couldn’t search by attribute (again, not part of the real application), Hibernate sped up by leaps and bounds, and the GigaSpaces DAO implementation was far simpler than it would have been.
What of those is valuable to you is, well, up to you. Personally, I find the DAO implementation more expensive up front, and the speed benefits are worth it.
The tests themselves: 100000 writes, 100000 serial reads (in order by PK), 100000 random reads (Collections.shuffle() on the PKs), 100000 serial deletes (i.e., destructive reads), 100000 random deletes.
Hibernate did way better than I thought it would. Here’s one view of the data:
| GigaSpaces | Hibernate | |
|---|---|---|
| Serial Writes | 0.522ms | 2.483ms |
| Serial Reads | 0.238 | 0.668 |
| Serial Deletes | 0.613 | 2.652 |
| Random Reads | 0.247 | 0.854 |
| Random Deletes | 0.606 | 3.641 |
Now, Hibernate was talking to an external Derby instance; GigaSpaces was talking to an external GigaSpaces container. Some outliers on Hibernate slowed it down a little; the actual perceptive numbers are probably better than you see here.
But this is still really good. I’d expected Hibernate to be in the 6-8 millisecond range, and it was way faster than I thought.
I’m including the source code to the test, if you’re interested. Again, this data isn’t worth making a real conclusion about – you need to test your own object models – but it’s interesting.
Art: Malach
This is more art. “Malach” – מלאך – is “angel” in Hebrew.
Malach, angel
This is a piece based on a design created maybe twenty-five years ago, in the throes of a depressive cycle. I found it uplifting then, and somewhat uplifting now. (My eyes are older and more jaundiced.)
It is in the shape of a cross, I suppose, which is an odd result – it wasn’t intentional, but reflected a winged messenger with arms wide open, wings suspended above the arms. No halo, thank goodness.
This one “moves” a little, I guess. I’m happy with it.
Worth reading: “Patterns of Failure” from TheDailyWTF
Posted by dreamreal in Programming on 4 March, 2010
“Patterns of Failure” is a semi-essay from TheDailyWTF that actually points out some useful information from developers.
We human beings are quite remarkable at recognizing patterns. Take clouds, for example. A cloud dog looks nothing like a real dog, yet no matter how hard we try, once we see the dog in the cloud, that’s all we can see.
While this ability has clear evolutionary advantages, it’s often a disservice in today’s modern world. Pattern recognition yields many false positives, leading towards Gamblers’ Fallacy, prejudice, and can even extend to really poorly-written software.
Now, recognizing patterns at the micro level (i.e., code) is almost always a Good Thing. Code often does repeat itself, and consolidating repetitive code into subroutines tends to help throughout development and especially when it comes to maintenance. The real problem – and the one behind the aforementioned systems – is recognizing patterns at the macro/application level.
I like TDWTF; it can be trite (“Look! Dumb code in real life!”) but sometimes has some funny or interesting moments.
This one is interesting.
Art: Comet
This is a drawing I made of, well, something flying. I called it a comet; it’s not a comet, obviously. A comet wouldn’t have this much definition, and the intent is that the object looks like it’s going up and to the right – whereas a comet with a tail like this would be going down and to the left.
I used pastels and, well, my fingers to blur. I don’t particularly like the color usage all that much.
I love art of almost all kinds, but I’m not visually-oriented, so this kind of art is always pushing it for me. Plus, I kinda suck at it so far; I don’t know how to communicate with visuals yet. This moves, physically (sort of), but doesn’t move the viewer at all.
I wonder why I resist repairing myself.
Posted by dreamreal in Psychology, general on 23 February, 2010
I was reading an essay concerning left vs. right sources of authority and something stood out: the author referred to Dr. Charles Krauthammer’s opinion on stem cell research and factored in Dr. Krauthammer’s paralysis.
I didn’t know any of this. The paragraph in question that caught my eye:
Take, for example, Krauthammer’s position on embryonic stem cell research. Though Krauthammer is secular, he opposed creating human embryos for the sake of stem cell research — a position all the more noteworthy in light of his own paralysis as a result of a spinal cord injury incurred as a young man.
Now, this is a nuanced paragraph. The opposition is for specifically creating human embryos for stem cell research, not stem cell research itself; Wikipedia summarizes Krauthammer’s opinion on stem cells as being pro-stem-cell research using discarded human embryos with restrictions in applications.
I think it’s fair to say that Mr. Prager is exploiting nuance for his readers. He was not wrong – but result is that one thinks Charles Krauthammer is opposed to stem cell research when what he actually is opposing is embryo farming (which I am defining here as entities acquiring embryos for no purpose other than stem cell research. I know, this isn’t nuanced either. But at least I’m honest about it.)
Anyway, this got me thinking on a tangent.
One of the defining aspects of me is my cleft palate. Bilateral cleft palate, double harelip, the works; I have pretty much one of the more severe forms of the defect. (And you wonder why I can’t sing…)
Nowadays, kids born with this problem go to a doctor almost immediately and are repaired. It’s one operation, usually a fairly short recovery time, little impact on the rest of their lives. (This assumes availability of the proper medical care, of course.)
When I was born, though, nothing of the sort happened. I guess survival rates were pretty low; in decades prior I probably would have been left at an orphanage to die, or abandoned on a hill for the gods (depending on when and where we’re talking about.)
However, my parents decided to go for it; I ended up spending a lot of time at Duke University Hospital, undergoing thirty-seven major operations for reconstruction (and construction.) That’s my count, so I could be off by some, I guess, but I’m not aware of any deviations.
But it wasn’t finished. Kids today get reconstructed at birth, and their growth cycle is part of the reconstruction. For me, though, I got put into a holding pattern until my face stopped growing…
… and I never finished. When I was 13 or so, my step-father asked me if I wanted to be who I was, or change who I was, which was an unfair question to someone so young (especially me, really.)
I chose to remain who I was. No more surgery, no more recovery, no more missing six weeks of class, no more spending months unable (forbidden) to say a word, no more of any of it.
This was a mistake.
But now… I have the opportunity to undergo reconstruction (at forty, I’d think my growth is pretty much done) but I don’t want to.
It would be nice to not have some of the physical burdens; I still get stared at by little kids, I still have to remind myself how to speak clearly, I still can’t whistle, for goodness’ sake – but do I want to undergo reconstruction, change who I stare at in the mirror?
No.
And I don’t really understand why. I tried to look this up, but I couldn’t find anything; maybe I’m just looking the wrong way.
I wish I understood myself better.
Suspension of Disbelief well worth reading
Posted by dreamreal in Psychology, general on 23 February, 2010
Suspension of Disbelief is worth reading. It’s stuff like this that makes Slashdot worthwhile, even if it’s rare.
I suspect the artificial extension of childhood grew out of the fact that because modern jobs are more complicated than they used to be, we need more years of schooling before we can go out and compete in the workforce. The fallacy there, though, is that just because we need more years of schooling, doesn’t mean that the natural age of “human maturity” has gone up. So we end up with 17-year-olds having to go to court to establish their right to criticize their teachers on their own time.
Good stuff, thought-provoking stuff.
Lay off, people.
Mystery is a bad thing in communication.
Posted by dreamreal in Psychology, general on 20 February, 2010
I hate you people who use voicemail to say “call me.” I also hate you morons who say “hey, can you come by? I want to talk to you” in the office. All of you need to have a baseball bat applied to your grey matter.
Both statements are really common, and they’re offensive. If you have something you want someone to listen to later, give them what it was you wanted them to listen to.
“Your grandmother is sick.” That’s a good voice mail message (with sad content, but still.)
“We need to talk about changing your work hours.” That’s a good inter-office request for a meeting. (Maybe not with positive implications, but still!)
I get burned by this crap all the time. When someone says “hey, I need to talk to you,” my mind immediately runs through possibilities: did I do something wrong? Did something bad happen? Did someone die? Am I getting a pay cut? Is the company shutting down? Am I getting fired? Is someone sick? Is there a rumor going around that’s negative and affects me?
WHAT IS IT?!?!
It might be nothing, right? It might be neutral or even be positive. But I’d far rather be surprised by good news than bad, so I naturally and innately run through possibilities just in case.
See, this happened at work one time too. I initiated a meeting. When I did so, I didn’t say “Hey, I want to talk to you about something,” I went to my manager and said “I would like to talk to you about this,” where “this” was a specific subject. He immediately said, “well, you should talk to your immediate super about it,” which was fine and correct.
Do you see what’s happened right there? If I’d left off the actual subject and request, he wouldn’t have known and we’d have wasted a meeting.
So I went to the guy I was supposed to talk to about it. We then set up a meeting to talk about it a few days later.
But he screwed me over, badly: he assumed my subject was the discussion. Thirty seconds in, he tells me “no,” and hands me a piece of paper that formed the basis of his saying “no” – a “performance improvement plan.”
Note that we didn’t talk about it. He assumed a whole bunch of things about what I might have wanted, and didn’t think that maybe I had thought about it myself. So I went in thinking “this might be better for all of us if we all agree” and he didn’t even listen to me.
I was mad at him about it, too; insulted, but that’s secondary. He got bit by the other side of this stupid “hey, call me” culture (and bit me in return.) He was so used to people stringing others along that when someone did not string him along, he figured that’s all there was to it.
“Surely if there was more to it, he’d have said so initially instead of requesting a meeting in which we would talk.”
That’s a completely flawed assumption. It’s sort of understandable if you accept the circumstances that created it: a one-hit culture where nothing is allowed to be deep, and nothing carries forward.
It’s madness. This lack of depth is cruel and stupid.
One of the “performance improvement plan” points was that I didn’t respond quickly enough on the phone, okay? But the only times that I didn’t respond on the phone were when I didn’t get the phone call – they had this tendency to call me when I was in transit and thus unavailable, you understand.
So when I landed, if they mentioned something actionable, I took care of it. The week I got the “performance improvement plan,” there was this exact circumstance: they called me at boarding time, and left a message that something was down and had a problem. So when I landed, at 2:30 the next morning I was fixing it.
That had happened other times, too: they’d call after hours and if I couldn’t fix it then, I was in the office before anyone else was, to correct things.
But the perception was that I was unresponsive, formed by the very moment of leaving a message, and when they wrote up the “performance improvement plan,” they didn’t think, “hey, you know, THIS WEEK he actually responded when we wouldn’t have bothered.”
They just wrote up the plan. Since it was written, they didn’t want me to respond to anything; it was already committed to paper.
All because people think “hey, call me” and “let’s meet soon” with vague message payloads is normal.
Total crap.
Twitter and Facebook – and texting – are my downfall.
It’s not like I’m addicted to Facebook – I’m rarely there – and I’m not even on Twitter. The problem is that the media concept espoused by immediate-gratification written networks is exactly wrong for someone like me.
See, I’m a reader. I inhale four hundred pages in a night. When I worked for TheServerSide.com, I had a thousand entries come through a day, and actually read a few hundred of those.
As a reader, I’m also a writer. I love to read, and as a result, I am a pale imitation of those writers for whom I have admiration. It leads to me saying things precisely, subtly, in some ways elegantly.
Okay, “elegantly” is arguable. But it’s my blog and I’ll use “elegant” if I want to.
Therein lies the problem. Twitter, facebook, even SMS — all want short, concise, easy-to-scan text.
There’s no room for “it’s not like I’m addicted to facebook.” There is room for “i h8 facebookin yawl.” One communicates actual intent. The other gets read.
It’s starting to kill me, actually. I wrote an email to my coworkers today that had this sentence in it: “I have little doubt you’re right,” in reference to a VPN issue being on my end of things and not theirs.
Yet the response was something like “You have a right to your opinion, but here’s how the VPN works, and here’s why we think it’s on your end.”
Yet I was agreeing with the person in the first place. I have a right to my opinion, of course, but why protest when my opinion matches yours? The reason, of course, is that I didn’t say “ur right” but that I did say “I have little doubt you’re right.”
All he read was “I doubt you,” and therefore got the exact opposite of what I’d said and intended him to understand.
Again: killing me professionally. Apparently I’m the one who looks at things, who thinks about how things work, who thinks about how to express things. The one. Everyone else wants 140-character summaries, with more emphasis on succinctness rather than correctness.
I’m doomed.
This Ain’t Jazz
“This Ain’t Jazz” is a piece I put together partly because I really wanted to play, like, real drums.
See, here’s the thing: I started on drums, back in 1980 or so, in the Wildwood Middle School Guitar Group, which basically played a set of hits, based on the skills of the students at the time.
I picked up drums because I was sure that beating on things had to be easier than putting your fingers in the right places all the time, and I sure couldn’t sing then any more than I can sing now. And I liked hitting things.
I’m still convinced that all of these things are true. Especially the “I like hitting things” bit.
Anyway, my house burned down around Christmas of 1984 or so, and what I happened to have with me in Miami was a cheapo Casio keyboard and an entry-level acoustic guitar. So without any other instruments to play… I became a guitarist.
BTW, as I write this, I’m actually weeping. I looked up the date based on the deaths of Sherri Baxley and Michael Baxley, who were killed in an auto accident, and rereading the obituary is very emotional. I loved both of them.
When my house burned down, I was relieved to find that nobody was hurt – that was my main concern, right? Because I’m a decent person and all, you know, the worldly possessions – including my rather impressive library and my Commodore 64 – didn’t matter as much. When I got back to Tallahassee, talking to a friend of mine, that was my main point: “Sure, my house burned down, but nobody got hurt, right?”
… to which the person with whom I was speaking said, “Oh… you haven’t heard?”
Ouch. Ouch then, and ouch now, and the pain is just as raw now as it was then.
(Give me a minute, okay?)
So, back to “This Ain’t Jazz.”)
I was kicking around in Banjo Mart (a friend’s name for Guitar Center) and played a djembe for a bit. I can’t have a real drum kit around, because I play way too loud and they take up a lot of room, but a djembe is so… neat, that it caught my imagination.
A friend of mine, Steve Peterson, rather graciously let me borrow one (as I’ve recounted elsewhere), and I’ve been trying a few times to see if I could record it.
Tangent! I was listening to Jaco Pastorius’ eponymous album, and something about the rhythms sounded good in my head and in my ears and stuff. Jaco used a lot of latin rhythms, which I can’t honestly claim (or play) but the idea of the drums as harmonic tones made me think…
And “This Ain’t Jazz” is what came of it. It’s not jazz, right? Even so, it owes a lot of its construction to a jazz motif.
I hope you like it. Thanks to my use of some really deep tones, it’s probably worth putting on some decent speakers if you want to listen to it at all; it’s the djembe (three tracks), the bass synth, a sort of symphonic piano patch, and a single guitar, with a very simple motif.