Originally published at An Atypical Life. Please leave any comments there.

what was i doing this past week at night and on the weekend?

new backyard slate walk

Originally published at An Atypical Life. Please leave any comments there.

I wanted to post a public thanks to the great minds over in freenode #couchdb, including Paul Davis and Jan Lehnardt for helping me learn more about CouchDB, and helping me investigate the performance issue I posted about last time. In fact, they both posted on planet couchdb with some thoughts about benchmarking in general.

I wanted to share my reason for benchmarking CouchDB in the manner I did. The fact is that it was entirely personal in nature, as in trying to make my own dataset and code as fast as possible. I’m not trying to generate pretty graphs for my management, for publication or for personal gain. This work comes out of having to load and reload fairly large datasets from scratch on a regular basis as a part of my research methodology. I need a large dataset to get meaningful results out of the rest of my system, and I was not content to wait for an hour or two each time my system bulk loaded the content.

So the suggestions provided - not using random UUIDs to help CouchDB balance the b+-tree, correctly using bulk insert, and redoing my code to use the official couchdb interface (instead of a hacked-up version using raw PUTs and POSTs) helped a lot.

It turned out that the spike that I was seeing (see last post) disappeared when I randomized the order of incrementing that variable, so much so that 3 randomized runs show almost no peaks. However, when I “scroll” through that variable (increasing the batch size) I still see the peak around a batch size of 3k.

Trying the software on another platform (AMD x64-3500 Debian lenny with a 3ware hardware RAID array, as opposed to a 4-core Mac Pro with only a single local disk) revealed that the peak shifted to a different value, a much higher one.

Lesson? Always benchmark your own application against your own data, and tweak until you’re satisfied. Or, in the words immortalized at Soundtracks Recording Studio, New Trier High School, “Jiggle the thingy until it clicks.”

I suspect suspected that Jan’s article ranting about benchmarking was at least in part stimulated by my experiences as shared over IRC. (I was wrong - see the comments.) They must have seemed somewhat inscrutable — why would someone care so much about something most database-backed applications will do rarely, as compared to reads, queries and updates? My application right now has a very particular set of criteria for performance (repeatable high performance bulk load) that is not likely anyone’s standard use case but my own. Nor is it going to be a worthwhile effort on the developers’ part to spend a bundle of effort optimizing this particular scenario.

That said, Jan also is calling for people to start compiling profiling suites that “…simulate different usage patterns of their system.” With this research, I don’t have the weight of a corporation who is willing to agree on “…a set of benchmarks that objectively measure performance for easy comparison,” but I can at least contribute my use case for use by others. Paul Davis’ benchmark script looks quite a bit like what I’m doing, except the number of documents is larger by a factor of 100 (~2mil here) and the per-document size is smaller by a factor of 25 (100-200 bytes here). Knowing the time it takes to insert and to run a basic map/reduce function on fairly similar data is a great place to start thinking about performance considerations in my application.

Oh, and knowing the new code on the coming branches will get me a performance increase of at least 2x with no effort on my part is the icing on the cake.

Thdavisp. Thjan.

Originally published at An Atypical Life. Please leave any comments there.

Based on a tip from my university colleague Chris Teplovs, I started looking at CouchDB for some analytics code I’ve been working on for my graduate studies. My experimental data set is approximately 1.9 million documents, with an average document size of 256 bytes. Documents range in size from approximately 100 to 512 bytes. (FYI, this represents about a 2x increase in size from the raw data’s original form, prior to the extraction of desired metadata.)

I struggled for a while with performance problems in initial data load, feeling unenlightened by other posts, until I cornered a few of the developers and asked them for advice. Here’s what they suggested:

1. Use bulk insert. This is the single most important thing you can do. This reduced the initial load time from ~8 hours to under an hour, and prevents the need to compact the database.

Baseline time: 42 minutes, using 1,000 documents per batch.

2. Don’t use the default _id assigned by CouchDB. It’s just a random ID and apparently really slows down the insert operation. Instead, create your own sequence; a 10-digit sequential number was recommended. This bought me a 3x speedup and a 6x reduction in database size.

Baseline time: 12 minutes, again using 1,000 documents per batch.

Using 1,000 documents per batch was a wild guess, so I decided it was time to run some tests. Using a simple shell script and GNU time, I generated the following plot of batch size vs. elapsed time:

Strange bulk insert performance under CouchDB 0.9.0

The more-than-exponential growth at the right of the graph is expected; however, the peak around 3,000 documents per batch is not. I was so surprised by the results that I ran the test 3 times - and got consistent data. I’m currently running a denser set of tests between 1,000 and 6,000 documents per batch to qualify the peak a bit better.

Are there any CouchDB developers out there who can comment? You can find me on the #couchdb freenode channel as well.

Originally published at An Atypical Life. Please leave any comments there.

The Cascata delle Marmore waterfall.

Just like the subject says, the Italy vacation photos are online.

Originally published at An Atypical Life. Please leave any comments there.

Dragging myself to consciousness, I gasped for air. The images of  gravestones and memorials of the intellectual elite, festooned with working mainframe key punches and proofs of famous mathematical theorem in honour of their contributions to society, still lingered. I could still feel the dirt being shoveled on top of me prematurely, as I struggled to break free of my restraints. The bottoms of my lungs burned like the teenage mistake of inhaling deeply from a clove cigarette. Still, it burned less than the stinging sensation of my sub-conscious clawing through a thin layer of conceit I’d previously put up in my life to hide the twin holes of fear and shame.

What a terrible metaphor. I’ll start again.

Lately I’ve been working harder on my doctoral research, in the hopes that this may be my best chance to leave something of value behind in this world. I’ve chosen not to raise children, and judging from the superior job my friends J., D., N. and S. are doing with theirs, I made the right choice. I then look at my friends with some life calling they’ve dedicated themselves to since childhood, and wonder how different my life would have been if I only could have settled on one thing to do, rather than insisting on being a polymath.

Then I think of the reality: it’s a huge conceit to pretend that anyone will remember me at all 5 years after my death, let alone 500. The odds against that are so low that they’re unthinkable. But why the terror of it happening, if it’s so likely? And how does this square against my personal philosophy that it’s the ideas that count, not the people?

OK, so I’ll substitute idea for name, and the good feeling I would get from knowing I helped other people far into the future. It’s still a huge conceit to pretend that anyone will remember my ideas at all 5 years after I come up with them, let alone 500. So why fear the inevitable? And why put so much pressure on myself to achieve something that’s rare (and, most likely, out of my hands)?

Interestingly, I don’t fear death. I think I overcame that one many years ago when I struggled with some of the other demons in my life, and came out on top. Perhaps this is my chance to overcome this ridiculous idea as well. There was definitely a time at which I really bought into the idea that helping just one person was as good as helping a whole flotilla - and I’ve definitely helped at least one person in an immense way. What changed?

Now, after talking it out online, I’m fairly sure it was the dawning realization, though not quite ecological, that the resources I have burned (money, time and patience of those smarter than I, and yes, non-renewables like jet fuel and natural gas) are above the average. I also have been given a lot of unique and interesting opportunities, and gained a lot of special skills. It was then I decided that I had to do everything I could to put all of that good stuff back into the world, in as many ways as possible. Anything else is just conspicuous consumption. I owe it to everyone else to do as good as a job as I can to pay things back.

The thing is, my life is almost all about paying it back in one way or another. I’ve had my phases of acting spoiled, but after being forced (almost at knife-point) to volunteer my time in college, helping others has become a sort of addiction, perhaps even an unhealthy one. My job for many years now has nothing but helping other people get their jobs done more effectively. I teach, I learn (then re-teach what I learn), I research (and then teach what I find out), and I volunteer my time when I know full well I really should be taking it for myself. I helped keep a household of friends going when no others could really make ends meet. I’ve helped those in need find money for surgery, and given them the emotional strength necessary to pull through. It’s never felt like an obligation. But it’s never felt like it’s enough.

I treat that feeling of emptiness inside as telling me that I still have more work to do, more in me to give to others. Perhaps it is only a twisted redirection of guilt and shame, a hope at becoming immortal in some sense. But I don’t work hard only to see my name in lights (especially since it never happens), or predicate it only on the knowledge I’ll gain something out of it. It’s because I like doing a good job, because it does feel good to know I’ve accomplished something concrete, like presenting my first paper in several years at a conference. I’ll do it even if I’m ignored, or if someone else claims the credit for my work. (Unfortunately this sometimes makes me a poor businesswoman.) And, more importantly than anything else, I know that when I find I am doing something that causes harm to someone else, I change how I act. This is my own private version of walking softly; I have yet to figure out how to correctly carry a big stick, so I walk with my hands in my pockets instead.

And yet I still have dreams like this one tonight that wake me at 3:30 and prevent me from going back to sleep, and keep me writing uncontrollably. What am I missing? Am I acting reasonably? Should I be doing more? Less? Something different? I know I’m not the only person who has felt this way, but I also know my attention span is so poor right now that I can’t think of where to start researching to look at motivations of the great, the noble, the weak and those of the despicable monsters.I need a sanity check (and maybe a kick in the ass) so I can move forward. I refuse to sink into solipsistic musings, but a little introspection every now and again can’t hurt!

Originally published at An Atypical Life. Please leave any comments there.

tonight i went overboard and deep fried things. yanno, when the oil is hot, ya gotta use it, right? my beer batter included sleeman’s cream ale, flour, and two kinds of bacon salt.

besides the fairly mundane broccoli and onions, i also deep fried local organic cheese curds. they are now my new favourite vs. mozzarella sticks.

but the real amazing item were the deep fried bounty bars, in the same batter (yes with bacon salt). these things have no right to taste this good. seriously. if you live in the US try finding bounty bars (coconut enrobed in dark chocolate), they’re in many places now. if not you will have to use the inferior mounds bars.

Originally published at An Atypical Life. Please leave any comments there.

today I was offline the entire day. I taught my friend dys everything i’ve learned about digital performer and my mc mix controller. verdict: good. want to play with it together and make music.

yes this is a bit lame for you to read, but it was very satisfying for me to do. i like helping other people. the end.

Originally published at An Atypical Life. Please leave any comments there.

done in one take: chick corea’s Eternal Child. Instrument: Yamaha TX816. Effects by Lexicon and MOTU.

Originally published at An Atypical Life. Please leave any comments there.

can’t publish this one - but i created some awesome reports in a virtual machine for my employer today. really slique. very cool. cognos-based. anything more and i’d have to kill you. ;)

Originally published at An Atypical Life. Please leave any comments there.

been fighting my mac for weeks now, with constant freezes, hangs, system-wide crashes or video corruption from 1-100 minutes after reboot. seems i followed some bad advice in the past and turned on something i shouldn’t have. so, my thing for today is sharply worded advice:

Do not enable QuartzGL (2D acceleration) on your Mac Pro. To check that QuartzGL is off, open Terminal, paste in this line and press Return:

sudo defaults write /Library/Preferences/com.apple.windowserver QuartzGLEnabled -boolean NO

Reboot to make this take effect. Voila, no more annoying crashes. You’re welcome.

Originally published at An Atypical Life. Please leave any comments there.

If you’ve had trouble posting on my blog since I opened up comments, you should be able to do so now. I’ve deleted all registered users - so, if you were registered before, now you shouldn’t be forced to log in just to comment.

Originally published at An Atypical Life. Please leave any comments there.

Still burned out on music. So I made chocolate chip cookies, from this NY Times recipe. I had to make a few substitutions because of what I had on hand:

• Whole wheat flour instead of regular flour (still used the cake flour)
• Dark brown sugar instead of light brown sugar
• President’s Choice chocolate chips instead of the gourmet ones suggested

Delicious!

Originally published at An Atypical Life. Please leave any comments there.

Randomly, Mr. Neilathotep and I both thought of Barney Miller at the same time. Freaky, but good inspiration.

Here is my rendition. I’ve been spending rather longer on these tracks than 20-40 minutes…too much of a perfectionist, I guess…

Originally published at An Atypical Life. Please leave any comments there.

I got a bit burned out on forcing myself to do a song a day. So today was a day of organization. I finally cleaned up and posted photos of the erection of my garage. Big thanks to everyone who helped! When I find the photos of the siding and roofing I’ll put those up too.

Full gallery of construction photos here.

Originally published at An Atypical Life. Please leave any comments there.

The classic, a tribute to Jaco Pastorius.

Download. (mp3, 1:45, 2.2MB)

Instruments: Steinway, Rhodes, Minimoog, brushed drums, double bass arco, fretless bass. Effects by Lexicon.

Originally published at An Atypical Life. Please leave any comments there.

I had a terrible scare tonight. None of my Digital Performer (my DAW) projects from before I moved to my new Mac would open. It suddenly felt like I’d lost 5+ years worth of musical experimentation.

After panicing a bit, I did a whole lot of research, and came up with this process. It’s slow, but it works. And it’s a thing for today since no one else has ever written it all up in one place before.

1. Go to the Terminal and change directories to your project, for example: cd Waynemanor/DP\ Projects/Barracuda\ Project/ (If use of a UNIX command prompt and escaping spaces are new to you, you may want to read through a tutorial first.)
2. Use ls to find the files that are your project files. In this case, I have two: Barracuda and dys4ik 2006-02-28.
3. Install the Apple OSX Developer Tools if you don’t already have them.
4. Use the following command: SetFile -t PERF -c MOUP <project-file-name>substituting each project name in turn.

You’re not done. Your audio files may also be corrupted. Try loading the project into DP. Still problems? Getting a Resource file was not found (-193) error? Your DPII’s resource fork got lost, probably because you copied to a non-Mac system and back. Try these steps. Some guesswork may be required.

1. Download, install and run SoundHack.
2. Use File > Open Any (cmd-A) to open your first sound file from the Audio Files directory.
3. Use Hack > Header change (cmd-H) to assign the correct sample rate, number of channels and encoding. Most DP projects have single channel. You just have to know what the sample rate is (usually 44.1 or 48) and how many bits deep it is (8, 16, 24, 32 are most common). Press Save Info
4. Select File > Save a copy (cmd-S). Be sure to set the same bit depth here as you used in the file’s header, or SoundHack will do a conversion! Save a copy somewhere else, like your desktop. Be sure to save as the same name as the original file to prevent confusion later.
5. Navigate to where you saved the file and double-click to open in your favourite sound program. This could be QuickTime, AudioFinder, DSP-Quattro, or even DP itself. Play to make sure it sounds right. If not, you got the sample rate or number of bits wrong. Go back to SoundHack and try again.
6. Painstakingly repeat this for each of your sound files. This could take a while.
7. In the DP project folder, move your Audio Files folder aside. Place all of the newly patched files into a new folder called Audio Files.
8. Try re-opening the project in DP. You should be able to pick up where you left off.
9. Grab a cold one. You deserve it!

Originally published at An Atypical Life. Please leave any comments there.

Inspired by this stunning 1969 footage of Fred Rogers asking Congress for $20mil for PBS comes this Roland-y x0x electrofest, by just your being you (ogg, 3m47s, 3.27MB).

Lights a fire in the soul, doesn’t it?

Originally published at An Atypical Life. Please leave any comments there.

OK, I had to take a detour from my “all things music” thing-a-day for a special request. A house down the street from me had a building permit posted that I didn’t read clearly until 2 days ago. Seems they wanted to split the 30′ wide lot into 2 15′ wide lots, destroy the original 2-story 120-year old building with a gorgeous historic facade, and put up a 3-story semi-detached monstrosity. In fact, the historic facade (which is in surprisingly good shape!) is identical to a few others on the street! My response: “Ah, no.”

I spoke with the neighbours immediately to the north, who were also concerned, and a neighbor across the way from me who’s a good friend and fellow Victorian home owner/restorer. We jointly put together a letter of opposition and presented it in front of committee today.

We won.

The proposal was rejected on the grounds that it was not in keeping with the nature of the other homes in the area, and that the narrow proposed frontage was further unacceptable. The committee went so far as to impune the reputation of the current owner, who clearly hasn’t been keeping up with preventative maintenance. The implication was that, if he can’t maintain the current property, there’s no guarantee he won’t cut corners on the replacement structure as well.

Of course, the applicant is appealing the decision, so this isn’t over — but it’s exciting to get involved and to realize that just 30 minutes spent on a letter of objection, and 5 minutes speaking over at City Hall, is a great way to stay involved wtih your community.

If you want to read the letter I put together and had my neighbours sign, comment using your email address and I’ll send you a link.

Originally published at An Atypical Life. Please leave any comments there.

Too many computer crashes tonight caused this to get posted very late.

After Mr. Neilathotep made the suggestion, it had to come out. Here’s a short interpretation of The Rainbow Connection on banjo, double bass and analogue synthesized flute. This really was done in an hour, though computer crashes had me screwing around with getting the damn thing out of my DAW for about 3 more hours. :/

Originally published at An Atypical Life. Please leave any comments there.

thing-a-day #2: banjo. for an upcoming music track (hopefully later this month!) I wanted a realistic banjo strum. Here’s the best I could manage in 1 hour or less. Sample is from Sweetwater’s Stratus Sounds collection (Volume 3). Lots of tweaking, a bit of FX, and you’ve got the loop you hear here.

What do you think?

P.S. Euphonix update coming soon, I promise.