Name: Joseph
Email:
Posts by dreamreal:
- For each node, count the marbles the node contains, and return that count, along with the color of the first marble encountered.
- Collect each count and build a map based on the marble color and the count.
- Installation of powered USB hub with wireless dongle (currently in use with the wireless hotspot hardware, allocated for the Alcyone already)
- installation of MIDI breakout (ready for assembly and wiring, but not started yet; I have one already but the one I already have is less suitable)
- Setup of Alcyone’s control software in kiosk mode, so it starts when the Alcyone is powered up
- Create a screen layout for preferences.
- Define a menu option to invoke the preferences editor from the main Activity
- In the main Activity, load the preferences in onResume(). This will load the values for the initial invocation, as well as restore the values from the preferences Activity when it’s done.
- Also in the main Activity, store the preferences in a SharedPreferences object on onPause(). This is probably not necessary.
- In the menu handler in the main activity, invoke the Preferences Activity. (The
onPause()will take care of sharing the current values.) - In the preferences Activity, add a button handler that stores the new values in the SharedPreferences object.
Java Surgery: What is Map/Reduce?
May 22nd, 2013Map/Reduce is a strategy by which one uses a divide-and-conquer approach to handling data. The division is normally provided along natural lines, and it can provide some really impressive performance gains.
The way it does this is by limiting the selection of data for which processing applies. Here’s a good way to think of it, with an anecdote originally provided by Owen Taylor:
Imagine if you had two types of popcorn at a party; one type is savory, and the other is sweet. You could have both types in a single bowl, and partygoers who wanted sweet popcorn could dig through the bowl looking for what they wanted; those who preferred savory popcorn could do the same.
Apart from sanitary concerns, this is slow.
What you could do, however, is provide two bowls: one with savory popcorn, the other with sweet popcorn. Then people would just line up for the bowl that had the popcorn they liked. Speedy, and potentially far more sanitary. Just line up in front of the neckbeard with the cold who keeps sneezing into his hand.
To explain it in computing terms, let’s determine a requirement with some artificial constraints.
Our project is to count marbles; we have four colors (red, green, blue, white) and our data storage doesn’t provide easy collation… or we’ve terabytes’ worth of marbles, which might be a more logical actual requirement.
The lack of collation is important, for our artificial requirements; ordinarily one would create an index in a database for each marble, based on color, and issue a simple “select color, count(*) from marbles group by color” and be done. With a giant dataset the select could be very expensive; it might walk the entire dataset based on color.
Let’s be real: the best approach for this actual requirement (assuming a relational database) is to use a database trigger to update an actual count of the marbles as the marbles are added or removed.
So what’s another approach? Well, what you could do is provide a sharding datastore.
Imagine a datastore in which a write directed data to one of many nodes, based on the data itself. If we had four nodes, we’d shard based on the marble color; node one would get all red marbles, node two would get all green marbles, node three would get all blue marbles, and node four would get all white marbles.
If we have direct access to our data – embedded in the sharding mechanism, perhaps – we can then count the marbles very, very, very quickly. We don’t even have to consider what color a given marble is, only what the node is. Our structure would look like this:
If the shards have their own CPUs, then you can see how the runtime would end up taking only as long as it took to iterate over the largest dataset plus a touch of network communication (which, normally, won’t factor in by comparison to the first operation.)
The first step – where you send a request to each of the shards to collect data – is called the Map phase.
The second step – where you collate the data returned by the Map – is called the Reduction phase.
Thus: “Map/Reduce.” You map a request to many nodes, then reduce the results into a cohesive returned value.
Of course, I’ve simplified my requirements such that I only iterate over the marbles, gathering a simple count. Often, the real world is rarely so convenient; you’d be more likely to have marbles of multiple colors in each node.
In this case, you’re still iterating over each marble, doing a collation – definitely slower than a simple count, but your runtime is still going to only be as long as it takes to iterate over the largest node’s data.
This is really important, by the way. Your data distribution is critical. You’re not going to run in one fourth the time if you have four nodes; you’re going to run as long as it takes to process the node that takes the longest. If your nodes take 200ms, 220ms, 225ms, and then 1372ms… then your runtime is going to be 1372ms. Compare that to the 2017ms that it would take if you only had one node. If each one takes the same amount of time – let’s say 400ms – then your total runtime will be (roughly) 400ms.
The reduction would be a little more complicated if you’re looking at more than one color per shard, but not by much; you’d have a map of color and counts returned from the reduction already, so you’d simply be combining the results rather than building a map of the results in the first place.
Map/Reduce is a handy way to leverage multiple CPUs to gather data quickly – but it means you have to have your data sharded in the first place, in such a way that you can leverage the sharding. Hadoop and Gluster can do it, providing a filesystem as shards; in-memory products like Infinispan (produced by Red Hat, by whom I am employed), Gigaspaces, Coherence, Terracotta DSO, and even lowly GridGain can manage Map/Reduce, and the in-memory nature of the shards yields some truly impressive speed gains.
Map/Reduce is proven and useful, and one of the biggest logical drivers for “the cloud.”
A better audio interface for Linux
May 16th, 2013One of the things that’s been driving me crazy with respect to the Maker Faire this weekend is audio configuration.
I have an M-Audio FastTrack Pro, which is a midrange audio interface – not suited for an ideal recording environment, but certainly good enough for the stuff I do, which doesn’t require much real-world ambient recording.
When I’m on the Windows laptop – where I have done most of my recording, with Steinberg’s Cubase – the FastTrack works really well. Low latency (less than 7ms, respectable for USB), good feature set.
But on Linux, it’s proven to be problematic. It works, for some values of “working,” but it’s definitely not suited for actual recording. Too much fiddling, too much hoping things work.
This morning, I realized that I had another audio interface laying around, gathering dust: a Lexicon Lambda. Given that the FastTrack works but only so much, why wouldn’t I try the Lambda?
The worst that could happen is that it doesn’t work at all – definitely a concern. But the most likely result is that it works just as well as the FastTrack – meaning that it works, but only so much.
But the best result would be that it works correctly, which the FastTrack does not, for my purposes.
Why not take the shot?
Well, you know what happens next: I plug it in, and give it a try.
And what do you know – it actually fires up properly. Ardour3 records from it well. (I adjusted the sample rate from 48KHz to 44.1KHz, because I was getting some audio dropouts at 48K, but after that, everything cleared up.)
Now, I haven’t tried everything – I need to break out the Alcyone and make sure MIDI works over the Lambda, too. But it seems promising on first glances, and it was definitely easier to get it working than the FastTrack.
Alcyone Beta as a wifi hotspot: working
May 13th, 2013One of the challenges for the Alcyone is how to connect to it for control and configuration. Today I implemented a wireless access point on the Raspberry Pi, such that the Alcyone’s networking needs are addressable.
It already has a predefined output – the MIDI data – and a predefined input, the physical pedals themselves. However, MIDI also has a number of available parameters – namely, a transposition (an offset of octave and/or individual notes) and a MIDI channel.
I originally thought I’d put a set of buttons on the Alcyone for control of these, but that meant also hooking up some mechanism to display the current settings.
It wouldn’t have been difficult to do. Three or six switches, and an LCD display would have done it (and the Arduino version actually did exactly that).
Why three or six switches? Well, three switches implies a cycle: each button press would increment the value until it hit an upper bound, and then the value would wrap. For octaves and MIDI channels, that’s not a huge deal – you only have six octaves to worry about, and only sixteen MIDI channels – but transposition has twenty-three.
Yeesh, the more I think about it, even the MIDI channel cycling would be a pain.
Therefore, it just seemed easier to rely on a networked controller. The Pi has a network port on it already, but using that means either relying on pre-existing infrastructure (i.e., a local wired or wireless network, with wireless implying extra configuration for access) or installing a router or switch inside the Alcyone, with attendant extra power requirements.
I don’t like that. The more direct solution is one I’ve already mentioned: set up the Alcyone as a wireless access point itself, so an external controller can connect directly to the Alcyone’s wireless.
The way to do this is with two services: udhcpd and hostapd. I followed the Wireless Hotspot directions for the Pi fairly directly, except my network card isn’t supported by the default hostapd; for that, I used Jens Segers’ hostapd implementation and instructions.
Works like a charm; I haven’t installed this on the Alcyone Beta itself yet, but the test worked. I also have the case for the Alcyone; that means my milestones-to-be-crossed count is decreasing rapidly, with the biggest ones being the physical installation of all of the remaining bits into the case, and a system test.
Just as a note! The bits to be done are:
A system test will obviously involve the full installed hardware working with the software.
After those things, it’s bubble-wrapping this thing for shipping on a plane. At this point, that’s the thing I’m worried most about. 
Keeping my inbox clean
April 20th, 2013I decided late last week that seeing four-digit numbers in my unread email was a little too much. So I did the unthinkable, went through every freaking one of them and deleted not only the unread emails I didn’t need to worry about, but a ton of read mail that I didn’t need to keep, too.
That’s just my work email, though. My personal email inbox is probably even worse, although not filled with unread mail. 8,827 read mails in Gmail.
I think this says a lot about how I read mail and discard it – and how Gmail has (negatively) affected how I handle email.
I’ve been coding in Java too long – my C++ is affected negatively.
April 17th, 2013I realized during tests of the Alcyone Beta yesterday that I really have been coding in Java too long, in a lot of ways. My C++ might look okay for the casual reader, but I’m having to remember and test things about which I really shouldn’t have to think.
I’ve noticed a few issues, actually, and they’re interesting to fix, because the right way to fix them is immersion in C++; this project helps, but immersion isn’t instant. You have to live, breathe, and use something for a while to really immerse yourself in its zeitgeist.
So here’re the issues that stand out the most to me as I’m coding an actual polyglot, cross-platform application… hey, wait, I should probably clarify what that means, too.
Polyglot programming is fairly common. One writes part of an application in Java, then another part in Scala, then another part in Groovy, then another part in Ruby, maybe one more part in Python. Then you bind them all together with the JVM, and the parts that work best written in each language work together to yield a synergistic effect. (And programmers who’re better at one language than another get to leverage their skills.)
But this isn’t crossplatform – even if you’re using something like REST to connect over HTTP. The JVM is the common thread and platform.
However, what’s happening here is crossplatform. I’m using a controller application written in Java (sort of… remember, Android, and Dalvik isn’t Java although it uses the Java language) to connect to a C++ application on a completely different hardware platform.
Anyway, off to the actual concerns I have with my own C++…
Pointers
Ah, pointers. In Java, you don’t use pointers intentionally. The language always stuffs references on the stack when you invoke a method with an argument. This means the stack is underused (a good thing), but means you might accidentally alter an object as you pass a reference around.
In Alcyone, what was happening was that the server aspect (the REST facet) was showing that I was modifying the MIDI channel and octave and transposition settings (oh my!) – but the actual MIDI data wasn’t showing a change at all. The current status was even changing properly; it just was never using the new values.
Why?
Well, duh, since this section is called “Pointers,” it’s because I wasn’t using a pointer; I copied the object to the stack, and started the server. The server method altered the copy of the MIDI status that it had… conveniently leaving the original reference alone.
So I had two MIDI objects (one on the stack, the other in DSS, or heap) and the DSS (or heap) one was sending data to my instruments, and the other’s status was happily being reported and changed.
The fix was simple, of course; pass a pointer to the server method.
But this highlights a difference in paradigm, and it’s something about which I have to remember to think. A lot.
Coding Paradigms
This is where Java and its tooling has really spoiled me, I think. I’ve gotten used to a predictable set of inclusions about which I don’t have to think.
I need an IOException there? Oh, the development environment tells me I don’t have it in the current namespace, and offers to fix that for me. Bang! Zoom! Pow! No more worries about that.
In C++, I do have to think about what my current namespace is, and what’s imported… and I’m just not used to it yet. (Everything looks like keyboard vomit!) … and I have to look up where everything is declared and defined.
No convenient classpath to work with, etc.
It’s workable, sure, but C++ itself doesn’t have a monolithic culture like Java does. In Java, there are lots of libraries, many of which do things idiosyncratically, but there is an underlying idiomatic base behind all of them.
You might not immediately understand that idiomatic base as it applies to a given library – but idiom casts a very long shadow over everything. It’s hard not to conform to the idiom, and certainly not very useful.
Look at SWT, which is probably the most successful library that tries to buck the idiom – and there are a lot of Java coders, myself included, who reject it out of hand in part just because of the idiomatic differences.
In C++, though, the gunslingers rule the roost. What works for one library might be one idiom (namespaces, references rather than pointers, RTTI…) and for another library, a totally different idiom (pointers, a tendency for structures and exposed methods rather than encapsulation).
This, too, is a problem for which solutions exist; it’s not the end of the world, at all. In fact, in some ways, it’s freeing; no longer are you stuck to how everyone else might do it, just because that’s idiom and/or it’s all that’s available.
You can do not only what works – the Java way – but you can do what works best.
I’m not denigrating Java here, by the way. Java’s preference for what works is compensated with a just-in-time compiler that’s rather admirable; you can write run-of-the-mill, average-person Java code, and the runtime will apply magic performance pixie dust and it’ll scream. In a good way.
Meanwhile, in C++, you actually get to (or have to) think about what works best. If you want your code to fly at runtime, well, write it so that it does. It’s not especially challenging – especially in a project like the Alcyone, where you have very little code that’s actually performance-critical – but it’s definitely a change from Java’s mindset.
Library Ecosystem
Argh, the library ecosystem. For that matter, the ecosystem.
Along with the paradigm differences, there’s the awareness of the libraries themselves.
Alcyone uses an embedded web service, for example. How do you do that? Well, like the stars, let me count the ways… I chose WPP++, but I’ve looked at a dozen other possibilities, many with completely different approaches.
Most involve embedding a web server as a whole, and don’t provide easy access to actually responding to requests.
In their defense, HTTP ain’t so easy to do, and there’s a lot of work involved in the simplest requests.
What I wanted to find was a simple embedded HTTP server that would allow me to declare a REST endpoint, and serve static content as well.
I’m sure more-recently-experienced C++ coders are now thinking, “Why didn’t you use ____? It’s simple. Twenty lines of code.”
Well… because I didn’t know about it. (And still don’t, I think. I’m still using WPP++ and a semi-REST mechanism.) And of the libraries I do know about, well, it was far simpler for me to lower my goals and use WPP++ just like I’m doing.
That doesn’t mean that I’m under the illusion that I’m doing it correctly. I don’t think I am, actually; I’m just doing it, with the understanding that there’s an ideal out there of which I just haven’t yet learned.
And that’s a heck of a challenge.
I’m not even sure how to really address it.
A thought on platform
This morning, I thought a little about how I keep referring to the Alcyone Beta, referring to the Alcyone running on a Raspberry Pi as opposed to the Arduino Uno upon which I started the project.
In some ways, it’s still presenting challenges that maybe could be avoided through the Arduino. Yet the more I work with it, the more I think the Beta is, and was, the right path to follow.
For one thing, it’s more fun. Arduino is great, but …
Linux. Real multithreading. A real C++ application, for that matter. I designed much of the structure of the Alcyone code in the Arduino environment, and ported (and upgraded) for the Pi, but the development environment is definitely better for the Pi than it is for the Arduino.
That’s not to put down the Arduino; the Uno is a much smaller processor than the Broadcom stuck in the Pi. It’s just that the Pi provides many, many more services for the programmer than the Arduino can.
Sure, it presents extra challenges; a wifi access point, for example, and I have to eventually set up the Alcyone code to run as a service and not a console application.
But this is all doable, and the aspect of fun that the Pi brings in, along with the whole threading and memory and performance, more than makes up for the straight-forward implementation plan the Arduino makes available.
It also occurred to me that “Beta” is an interesting appellation for the project.
“Beta,” in coding, means an intermediate form, something that’s being tested.
But “Beta” in the Alcyone Beta doesn’t refer to testing or project stage; it refers to the nature of a binary star, in that this is the “other” and not the “first.”
I’m not even sure what to do about that; calling it the Beta has become ingrained in how I think of the project.
Adding Preferences Support in Android
April 10th, 2013Android supports a Preferences API, including the ability to render a preferences activity for you. Or, at least, it’s supposed to. I was unable to get it to work in any fashion that didn’t disappoint me, whether because it looked ugly or because it simply didn’t work.
So I looked, and looked, and looked. Here’s how I managed to get a preferences screen into the Android Alcyone Beta application.
The application has two configurable values: the hostname to which to connect, and the port for the Alcyone Beta server on that hostname. The default is to use “piui” and 8090, on the assumption that the Alcyone Beta will serve as its own access point and define its name as “piui.”
Naturally, that default name may change over time – it makes more sense for it to be “alcyone,” for example, or “beta.alcyone“. But for right now, it’s “piui” — and the thing is, I don’t have the Alcyone Beta set up as an access point yet.
So the hostname is premature – therefore I have to provide configuration, or else I have to hardcode the address. (And I don’t want to hardcode the address.)
Thus: preferences.
I created a preferences screen very simply, following these steps:
So let’s look at it some.
The screen layout is really pretty easy: it’s a couple of labels, two EditText objects, and three buttons. The buttons are: reset (discard current edits), reset to defaults (discard all edits and restore to “piui” and 8090), and submit (which preserves the values in the text edit boxes.)
The AndroidManifest.xml gets a very simple modification for the preferences screen:
<activity android:name=".EditPreferencesActivity"/>
With that, I can go about looking at the AlcyoneActivity code for managing preferences.
First, let’s look at the onResume() method, since that runs immediately after initialization. It’s the ideal place to make sure our values are current – i.e., what the preferences say they should be.
@Override
protected void onResume() {
super.onResume();
Log.d("alcyone", "onResume");
SharedPreferences preferences = getSharedPreferences("alcyone", 0);
host = preferences.getString("host", "piui");
port = preferences.getInt("port", 8090);
Log.d("alcyone", "Setting host to '" + host + "', port to " + port);
updateStatus();
}
This gets the SharedPreferences object associated with the key we provide – “alcyone” – and marks it as private for this application. (No other application needs the data, so it’s private here.)
There’s supposed to be a general way to get preferences such that we don’t have to provide a key or a mode, but I was unable to get it working; I’m sure it’s something I misunderstood, but I don’t know what that was, having misunderstood it.
Then I set the host and port based on the preferences, providing defaults for the values along the way. The updateStatus() method does some calls to the server to get the current status and update the UI to reflect that status.
With this method, I can start an activity that modifies the preferences, and when this activity resumes, it’ll get the new values properly.
So how do I start the other activity? Pretty simply, as it turns out. I have a menu defined for the AlcyoneActivity, and choosing a menu item invokes the onOptionsItemSelected(MenuItem item) method in the activity.
The menu contains three options: send a MIDI reset, reset the Alcyone to its startup condition (octave 3, channel 1, no transposition), and configuration, which is the menu item with which we’re concerned. Here’s that method:
@Override
public boolean onOptionsItemSelected(MenuItem item) {
switch (item.getItemId()) {
case R.id.mnuMIDIReset:
AlcyoneClientFactory.build(this, host, port).midiReset();
return true;
case R.id.mnuAlcyoneReset:
AlcyoneClientFactory.build(this, host, port).reset();
return true;
case R.id.mnuConfigure:
startActivity(new Intent(this, EditPreferencesActivity.class));
return true;
default:
return super.onOptionsItemSelected(item);
}
}
See how easy it is to crank up a new Activity? There’s nothing else to it. After this code runs, our EditPreferencesActivity will be in control.
This activity is ridiculously simple. It has its own onResume() that loads the current preferences from the SharedPreferences object; the submit button is handled with the following code.
submit.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View view) {
host = txtPrefHost.getText().toString();
port = Integer.parseInt(txtPrefPort.getText().toString().trim());
SharedPreferences preferences = getSharedPreferences("alcyone", 0);
SharedPreferences.Editor editor = preferences.edit();
editor.putString("host", host);
editor.putInt("port", port);
editor.commit();
Log.i("alcyonePrefs", "writing preferences host=" + host + " port=" + port + " " + editor);
startActivity(new Intent(EditPreferencesActivity.this, AlcyoneActivity.class));
}
});
This writes out the changes to the preferences, and then kicks off the AlcyoneActivity again. It’s the reverse of how this activity itself got invoked.
So there you have it: that’s how I managed to get the Alcyone Beta preferences working.
Alcyone Beta update: now embeds an HTTP server
April 8th, 2013I now have a working Alcyone Beta Android client. I had one before (i.e., earlier this week), but it was using sockets for the connection; now I’m using HTTP for the connection.
Why HTTP?
Well, the main reason is because it’s stateless. Using the original wire protocol meant retaining a socket for the life of the connection, with all of its state maintenance.
When the application paused? The socket needs to be released.
When the application resumes? The socket needs to be opened again.
The socket needs to be kept alive, too… and it eats battery.
The socket brings lots and lots of little problems. They’re all surmountable, of course, but… why? The application should be as light as possible, and that means light on the wire protocol, not light on application code.
And if we can shrink the application code while we’re preserving battery life? That’s even better.
So that’s what it does.
I did it by converting the C++ app away from the boost sockets, and over to WPP; this is an HTTP server in a single include file.
I basically exposed the wire protocol of the Alcyone via a single URL endpoint, with an HTTP parameter (“message”) – the message is the wire protocol in a single entity.
That left the Android client. I tried doing it with the Java HttpUrlConnection, but for some reason (I still don’t know why) it would block after the first request. Every time. No matter what I did.
So I converted it back to the Apache HTTP client code (similar to what I used in the sensor application)… et voíla! The Alcyone Beta application is working!
I still need to provide configuration of the server address and port, and there’s still a formatting issue here and there, but progress is being made.
Alcyone Beta at Maker Faire Bay Area, May 18-19
April 3rd, 2013
I’ll be taking the Alcyone Beta (and some other musical toys, yet to be created due to inventory) to the Bay Area Maker Faire this year.
Not only will you see the Alcyone Beta, but you will hear as I regale you with all of the interesting problems I had to solve, while I choose to oddly inflect all of my verbs!
Alcyone Android UI, first version
April 2nd, 2013
This is the Alcyone Android UI, as it stands right now. I plan on adding a sliding menu at the bottom for the reset commands the application configuration; I don’t know how to do that quite yet, but how hard can it be?
This should be a fairly simple application to build; once it’s done, I’ll get around to configuring the Alcyone Beta as an access point.
Then: off to build the case!
Alcyone and Communication Protocols
April 2nd, 2013First off, the desktop client for Alcyone Beta is working. It’s written in Java, and uses the wire protocol I designed a few posts back.
I was talking to my son about it last night (telling him about the progress, because he’s interested too, bwahahaha) and he brought up an interesting point: why am I using TCP/IP networking and not Bluetooth?
Bluetooth is, after all, far lighter; it’s designed for low-power devices, it’s got 1.0M/s bandwidth, and it’s serial; the connections for it are just as ubiquitous as for TCP/IP, and the USB connectors for it are actually far cheaper than for 802.11a/b/c.
It sounds ideal, honestly, and I put it on my list of nice-to-have features.
However, I’m not going to progress down the Bluetooth development path very much. Let me recount some of the reasons:
- I’ve already got the wireless connections.
- Bluetooth introduces explicit discovery and authorization phases that I can circumvent entirely by making the Alcyone its own access point.
- The power and bandwidth requirements are not being violated through the use of the current TCP/IP networking; TCP/IP is probably overkill, to be sure, but it’s not a resource that can be consumed such that I don’t have oodles to spare.
- The TCP/IP networking stuff is more familiar. I don’t mind learning new things (evidence: the Alcyone itself) but I’ve already got the TCP/IP stuff working as part of a polyglot platform.
- I’m nearing completion on the Alcyone Beta, to the point where it will be usable on stage within the next two weeks; adding a new protocol layer might push that back. It might not, but when there’s no need driving the new protocol…
I don’t want to sound like a stick-in-the-mud, mind. I think Bluetooth is an appropriate transport; it’s likely to be more appropriate than TCP/IP, quite honestly. That said, I have bigger fish to fry, and I’m going to shelve Bluetooth for the moment until I have more of an apparently need or desire for it.
