eyt* Blog

IBM Think 2023

Mon, 22 May 2023 03:08:50 GMT

In early May, I had the opportunity to attend IBM Think in Orlando. This is the first conference I attended in a while and wanted to share a couple thoughts. This blog post is mostly based on my Mastodon thread..

I think the three announcements that was showcased is:

Hybrid Cloud is the defacto standard now — you can no longer live in one cloud and they have tools to help, including Hybrid Cloud Mesh which was unvieled.
We have just crossed a new era in AI with generative models and watsonx.ai is IBM’s approach to this, separating out the Data, AI, and Governance.
And Quantum computers are coming — you need to start to prepare now!

For me personally, there are a few things I learned or was re-emphasized:

10% of all energy consumption is for data centers — I didn’t know this stat before, but definitely something we can work to bring down and and have more sustainable data centers and nnew laws in EU that will be similar to GDPR will require organizations to start reporting on this, both for their own consumption, but their partners. This seems like a huge undertaking, and while this is a great improve what gets measured, this will be a huge effort (and opportunity) for startups.
I have been hearing about quantum computers coming since I was a kid — IBM has about 20 of them now and they are expected to really be available for purchase in 2030. That’s cool — but the part that makes that scary is that cracking our current encryption codes from years to hours, and so in addition to threats of real-time attacks, the real threat is the store-and-decrypt-when-Quantum-is-ready. There’s some new Quantum safe encryption is coming — ready yourselves for the next Y2K-type rush!
I really enjoyed the talk by Megan Smith, particularly her comment on how people are getting coded out of society because of bad AI. There’s a lot of things she covered in this talk, but the bigger take aways for me was the DEI ones, particularly with AI models getting formed — we must teach them to not have biases and to not infer “facts” based on a lack of data.
And for my HPC friends, IBM also talked about IBM Vela — I still miss playing with the SGI Onyx!

And that’s my quick summary of what piqued my interests the most at IBM Think — hope you find some value in this post.

Digital Music Shops

Fri, 31 Mar 2023 22:18:31 GMT

I have been super quiet in the last several years on this blog and social media, but trying to make a little time this year to do a few blog posts. I have a backlog of things I want to do, but first, I thought I would start with a non-engineering topic.

I have played piano since I was 8 but took a break after my kids were born until a few years ago. While I love acoustic pianos, being a software engineer, I’ve always loved blending in technology, staying with my old Cakewalk and Yamaha PSR-500 configuration from the mid-90s.

Now that I’m back, there are a ton of things that have changed in technology, but the key one for me is sheet music — I no longer need to have a ton of books and flip through them like a madman or have a bunch of individual sheets and find creative ways to keep it together. Not only do you have tons of options for viewing sheet music, but you also have many options for organizing that music and annotating it.

Today, I’ll walk you through my current setup and how I get my sheet music. In case it matters, I play mostly classical and various music I like throughout the ages — it’s sometimes an “interesting” mix. In future blogs, I’ll discuss how I organize my music in my current setup. I’ve noticed in many of my piano circles that my setup is more tech-focused than the paper solutions, so hoping that my take will help with this.

Current Setup

I currently have an Apple iPad Pro 12.9-inch , which is a bit smaller than books, but not by much. It has a long battery life, is very stable (it won’t crash during your performances), and carries a lot of music in a small package.

I use the iPad in portrait mode, which provides a larger view of the music, but you could use landscape mode and show two sheets of music. I just find it’s too small to be useful. Some technology is being worked on to extend that iPad screen onto two iPads which can help this — but really, I think what is more valuable is a pedal which, while isn’t the same as having more music in front of you, does fit most of the same purpose. I use the AirTurn Duo, which is quick to configure, and you can quickly turn pages — assuming you aren’t using the Una Corda and sustain pedal concurrently!

I recommend the Apple Pencil so that you can annotate your scores just like you would with your paper copies, although I actually find the color and erasing options better than the paper alternatives.

In terms of software, I use forScore. This allows me to use the same library on my Mac, iPad, and iPhone — yes, you read that right, the iPhone. I only do organization on my iPhone, but you can play music from it. This is super powerful, very usable for free, and you can unlock some additional features for a super affordable $9.99 a year. If anything, I would like to give them more money…

Before forScore, I used the Musicnotes app for about a year, which is good if most of your music is purchased from it, but it has a lot of limitations beyond that discussed below.

Before moving on to the next section, I will say that picking an app is the most critical part here, especially once you start to have a lot of music (reorganizing it in the new app is a pain!), and the hardest one is really the annotations! Once you have started using annotations extensively in many pieces, it’s hard to switch. Newzik is one that I have considered trying a few times, but outside of the more expensive subscription and relatively few differentiating features, the time to redo the folders and, more importantly, the annotations is just not worth it.

Musicnotes

The vast majority of my music is purchased via Musicnotes. It has a good selection and is easy to use. When first using it, I would make sure that you look at many copies of the same piece of music before you buy one — for example, I am not a fan of their Singer Pro versions, but that is my preference.

They offer a subscription, which I recommend if you plan to make regular purchases with them. In addition to some other capabilities, it gives you 15% off purchases and you can purchase Pro Credits. When not on sale, Pro Credits are about $4 per piece, and whereas most of your pieces after your discount are just under $5, so a better value — this, of course, assumes you are purchasing them on the web; the app store is generally more as Apple has to take their cut…

In terms of the subscription, they have a Pro Premium plan, which for the first year makes sense, but after that, it feels expensive. Of note, in addition to 12 Pro Credits, it also gives you 12 free “Musicnote Editions”; this sounds great, but the selection of those is limited and, to me, not as interesting.

If you are into Concertos and longer pieces, I don’t find Musicnotes as great for this. They usually only offer the theme or the individual movements or something that will add up quickly.

I started using Musicnotes way before I had an iPad, and I want to mention this because most pieces you can print out. So your investment is still worthwhile even if you decide to return to paper later. I’d probably recommend saving the PDFs and printing a copy of the PDF, as some pieces have restrictions on how many times you can print them.

Lastly, they do offer a great IOS app. I used it for about a year and generally liked it. If most of your music is purchased from Musicnotes, it makes sense to use it. While it can import PDFs, it does not preserve those, so if you decide to change iPads in the future or use the Web app, you can’t get to these anymore — and even if you re-import them, they are not preserved in playlists. When I talk about my organization in a future blog post, this may make more sense.

That said, forScore has direct integration with Musicnotes, and once you’ve connected it, new purchases can automatically download.

Virtual Sheet Music

My next place to shop for music is Virtual Sheet Music (or VSM). It has a good selection, but you need to know how to find it, as its entire interface feels like a late ’90s website. They provide you with all kinds of pieces — it has the concertos and longer works that Musicnotes doesn’t have and is beautifully printed, and sometimes has versions of things that are better than Musicnotes in my experience, and pricing is better in many cases.

Virtual Sheet Music also has a subscription—like Musicnotes, it offers you discounts, and I would recommend it if you purchase things regularly from the site.

I wouldn’t recommend their app. Instead, forScore has a direct integration with it, and it is also super easy to export PDFs from VSM and get them to where you need them to be.

Sheet Music Direct

I find Sheet Music Direct very similar to the content of Musicnotes or Virtual Sheet Music; however, it does have some content that the other two don’t. For example, when I wanted the PDF version of Schirmer’s Chopin Ballades, Sheet Music Direct was my only choice for this particular copy.

While forScore does not have an integration with it, the PDFs exported from SheetMusicDirect slipped into forScore like a glove and required no metadata editing. I really loved this experience.

One of the things that SheetMusicDirect has is a subscription model where you can get all of the music you want without purchasing it. This sounds great, but I feel it is more expensive over time. I have no data here, but daily, I tend to play the same 4 things every day that I am working on for the last year, and if I still have time (that’s a big if), I’ll pick up a couple other things.

Henle Library

Another app and store I love is the Henle Verlag Music Library app. There are two things I love about this app. First, it is more like when you purchase the book; it will include the information around the piece, such as the reception of the work, notes, sources, editions, etc.

Second, I love the customization it provides. Some pieces will have a couple different options for fingerings, and you can pick which one you want or none if you are a power pianist. You can change the background color, customize the staff spacing, etc. And then, once you have it just like you like, you can also export it as a PDF into other apps like forScore.

Classical Music Repositories

Let’s face it: lots of classical music is (or should be) in the public domain, so some of it you can get for free or at super low prices.

The first one I’ll mention is IMSLP.org which has a bunch of recordings and PDFs of various pieces. I’m not a lawyer, so I will leave it to you to read the fine print to ensure it is legal, but I have used some copies from here. The quality generally varies; some are personally scanned versions with hand-written notes, while others are professional copies.

The next one is pianostreet.com which, if you have a subscription, you can access their sheet music library. I have found a couple pieces here, but I’ve not used them enough to justify the subscription.

Finding other Music

If you are not finding what you are looking for in one of the above places, take it to your favorite search engine and start looking for the piece. I have found a couple pieces on Noteflight.com or MuseScore.com, but your mileage will vary.

If you do not find it there, take a look at Google Play Books and see if they have a version. You can either use the Google Play Book App on your iPad or, with care, you can export the PDF from a desktop computer. Note that many of the copies of the books I’ve seen on Google Play have issues in their rendering; you can file a ticket with Google, and well, I’ll let you know if they resolve these eventually.

Importing the Physical

I much prefer importing a PDF from one of the other mechanisms in this article; however, if you have a book you love or can’t find it online, you can scan it.

ForScore can scan in music, but I’ve never used it. I use KeyScore instead. You point your camera at a piece of music, which will automatically line up most of your music, and you can even play a sample of the piece! Once you are happy with your importing, I export it to a PDF and bring it into ForScore.

If you subscribe to Pianist or similar publications with sheet music in their app, I take screenshots of the music and use Sheet Music Scanner to assemble those images into a PDF. Once you load the app, tap Import, select Photos, and import each page individually. When you are done, export it into a PDF.

Other forScore Integrations

ForScore integrates with a few other services that I do not use. To save you the time of searching for what types of music you love, I’ve put together this list:

BriLee Music	Choral Music
Presser	Classical Solos, similar to Musicnotes or VSM in my experience
Carl Fischer	Orchestral Music
Noteflight	I use it more for online notation (which doesn't integrate) but it has a music store that I have used in the past

Other options include:

Universal Edition	A variety of calssical pieces, but unfortunately, they do not yet have my favorite Chopin Preludes book as a digital download yet...
Donemus	Publisher of Dutch composer pieces

There are other providers such as Hal Leonard which also have digital libraries but these are not in PDF or formats that allow for general usage, so I would recommend against these.

Closing thoughts

This post was longer than I anticipated, but as with most things, there are countless apps, music stores, and opinions, so I’m just sharing my two cents. I hope some of you will find it helpful.

Moved off of bBlog

Mon, 6 Mar 2023 20:56:30 GMT

I have finally moved off of bBlog. It has been on my TODO list for years now, however, there has always been something else higher priority, but with my latest upgrade to Ubuntu 22.04 which comes with MySQL 8 by default, getting it to work again was more work than scrapping it. It seems strange that nothing in PHP has been updated to handle the “new” credentials added in MySQL 5.7.

As part of this project, I have removed a bunch of links which makes some of the posts a bit harder to read but with the dead link, it arguably is not much better than clicking on bad links. That said, it was a trip down memory lane with CUJ, Dr. DOBBS, and other great resources no longer being published or available! If you notice any other link problems, please let me know.

Anyhow, my current move is just to use static files directly extracted by from the database. If it helps anyone, just reach out.

Don’t return from within finally

Thu, 9 Aug 2012 03:26:34 GMT

Earlier today, I was debugging some seemed to be some pretty straight-forward code. The code effectively was the following function. What would you expect the output of this class to be?

public static void main( String [] args ) {
System.out.println( foo() );
}
public int foo() {
try {
throw new RuntimeException();
} catch ( RuntimeException e ) {
System.err.println( "Caught" );
throw e;
} finally {
System.err.println( "Finally" );
return 1;
}
}

If you expected an exception to be percolated out as I was, you are wrong. It actually turns out that the above returns 1 and ignores the exception that was just raised. Just something subtle to keep an eye out for.

How do you quickly sort 84 million unique Integers?

Sun, 8 Apr 2012 03:21:21 GMT

For the last few weeks, I’ve been working on increasing the performance of various areas at work and one of the areas that kept on hitting my profiles has been sorting integers. All of us with a Computer Science degree have sorted integers before and we know how to cost algorithms with Big-O notation, but most of the time, the default sort in Java, C#, and other languages is “good enough” that we don’t really think of it.

When it starts to show up on a profile though, sometimes that is time to start thinking about it. As with many things, algorithms based on N usually perform well when N is reasonably sized, so you do not always notice the issue until N becomes interesting or the number of times you call the algorithm increases.

Whenever diagnosing performance problems, it is important to characterize whether it is one or more big calls to the method that causes the entire time sink or whether it is the sheer number of calls to that method. In either of these problems, I think a reasonable first thing to ask is whether that call is really necessary or is it at the right location? More specifically, if you are not running binary searches or other algorithms or use-cases which require the values to be sorted, why sort them at all? If for no other reason, this causes you to think through how to code is or could be used and sometimes leads to a different solution.

In my particular case, there were a few large sorts that were taking all the time. Each of these sorts were sorting approximately 84 million integers via the JDK standard Arrays.sort() or the Colt IntArrayList.sort() method and taking approximately 10 seconds per sort. Depending on your background, 10 seconds may be an eternity or a dream, so suffice it to say that the entire processing required for this component had to be less than 3 seconds, approximately half of that had already been accounted for.

So assuming it is required, how do you sort faster? An obvious approach to server engineers is to multithread this operation, such as sorting chunks of that memory in parallel and then merging them together, or perhaps a multi-threaded version of quick-sort, and I am sure that my team expected me to go hide in a corner for a while and code up one of these approaches...

But I didn’t. You see, there’s something particularly interesting about the integers we were sorting. First, they are all positive, and second, there aren’t any duplicate integers. So instead of sorting them, can we do anything else with them that we can infer the sort and have more memory locality?

Well, one way to do it would be to use a set of bits. Say that the largest number in the array was 200 million, you can allocate 200 million bits and then sort the numbers using an algorithm like this:

for each N in array
set bit N
while more bits
find next set bit
set array[i] to that bit
increment i

Before we go further, you will note that the above algorithm goes through the array twice, and that may seem bad to you, however, there is another thing to keep in mind when assessing this algorithm, which is memory locality, and the fact that especially in the second loop, you are effectively iterating sequentially through two arrays which are probably in the L1 or L2 cache already, potentially making a big performance improvement if you have performance-grade processors.

The following sample code implements this in Java using the BitSet class. I’ve personally not gone through the code to see if it is the most optimal implementation for what I am doing with it, but after this change, the performance was in the acceptable range.

// Code available here
BitSet bits = new BitSet();
for ( int i = 0; i < array.length; ++ i )
bits.set( array[i] );
}
int lastBit = 0;
int i = 0;
while ( true ) {
lastBit = bits.nextSetBit( lastBit );
if ( lastBit == -1 ) {
break;
}
array[i] = lastBit;
++ lastBit;
++ i;
}
}

Performance across machines will vary. On my work servers with the latest Xeons, it averages around 800ms per sort, compared to an average of 10 seconds. On my home server, it is a little slower but averages around 9 seconds, verses 23 seconds for the QuickSort algorithm.

Now the above algorithm is not optimal for all sizes. In my tests, the magical point was somewhere around one million rows where the performance of the default Arrays.sort() and the above crossed over, so we do a quick size check and either go to the standard QuickSort or to the BitSet-based sorting.

Those familiar with the BitSet API may mention that I could gain better performance by telling the BitSet algorithm the number of bits I am interested in using and allowing it to pre-allocate the memory for that. Unfortunately though, I did not have that information available for most areas in the code and it was not feasible to guess in this area of the code. What was super interesting though is that doing a pre-scan of the entire array to find the maximum number did not change the runtime of the algorithm significantly. In fact, the TP99 of the two were off by around 100 milliseconds in my test. This could definitely be an area to improve the above algorithm going forward.

If the above algorithm was not fast enough for you still, there is still room for improvement. First, you can look at pre-allocating the array. Second, the second loop of the algorithm is easily parallelizable and perhaps the BitSet implementation for this use-case could be simplified.

As I mentioned above, the algorithm only works for unique, positive integers. If you have negative numbers, you could possibly use a second BitSet object for the absolute value of the negative numbers and iterate it in reverse (applying the sign obviously) which would also work. Duplicates obviously don’t work though, and if you have that problem, ask yourself if you really need the sort at all or whether you need the duplicates, and if you do, consider a parallel sorting algorithm.

Now off to another interesting problem. Wanna help?

Rounding numbers with Java’s NumberFormat...

Sat, 3 Dec 2011 22:16:29 GMT

Take a look at the following numbers and round them to the nearest whole number:

2
2.1
2.2
2.3
2.4
2.49
2.5
2.51
2.6
2.7
2.8
2.9
3

I’m pretty confident that you will get the same answers as my 4^th grader, which is that the numbers from 1-6 result in a 2 and 7 onwards results in 3, right? Well, given the following code though, you’d be surprised:

import java.text.DecimalFormat;
import java.text.NumberFormat;
public class test {
public static void main( String [] args ) {
double [] values = new double[] { 2, 2.1, 2.2, 2.3, 2.4, 2.49, 2.5, 2.51, 2.6, 2.7, 2.8, 2.9, 3.0 };
final NumberFormat format = new DecimalFormat( "#" );
for ( double d : values ) {
System.out.println( d + ": " + format.format( d ) );
}
}
}

Your surprise would come from the fact that the 2.5 rounds down to 2. Why? Well, reading the DecimalFormat javadocs closely, you will notice that the default rounding policy is HALF_EVEN, in which the above behaviour is explained. To change that, simply use format.setRoundingMode( RoundingMode.HALF_UP ); or one of the other policies available in the rounding mode.

Why this is the default behaviour is puzzling to me, but I’ll just add it to the list, which includes the Math.abs() handling of Integer.MIN_VALUE and non-thread-safe SimpleDateFormat [Link removed] (which, for the record, I do not believe that the code suggested in that blog post is guaranteed to be thread-safe either... since the DateFormat class is declared to be non-threadsafe also).

Internet Explorer 8 Compatibility Mode...

Tue, 14 Jun 2011 18:00:19 GMT

Earlier today, myscenicdrives.com released our road trip planner which I am really excited about. There are several more features that I want to build into it, but this was a really nice time to release it and get some feedback from real users.

For the last week, we have been doing some usability testing, optimizations, cross browser testing, and polishing it all up before the press release earlier today. With the final push going out the door late last night, we did our final round of testing and under Internet Explorer 8, I got that dreaded, “A problem displaying [site] caused internet explorer to refresh the webpage using compatibility view.” This resulted in a huge effort to figure out where that message from coming from.

The majority of the testing was done in the latest version of all browsers, which I was really pleasantly surprised at how standards compliant Internet Explorer 9 is. While I had to do a few tweaks specifically for it, it was relatively easy and no more than what I had to do for other browsers. When I cracked up Internet Explorer 8 (via Microsoft’s virtual machine image), it required a lot more tweaks but in the end, it seemed good.

This is why I was really surprised when I loaded up the page last night and saw the compatibility view message. Unfortunately, that message does not provide any insight into what the problem might be. The Microsoft documentation on the matter states that your pages must have a DOCTYPE and validate, but they also hint that there are other undocumented reasons — this makes finding the root of the problem orders of magnitude more frustrating.

After changing things I thought might be suspect, endless googles for any hints, and etc., I started commenting out various chunks of code trying to make it go. At the end, I had a simple empty page with nothing between the opening body tag and the closing body tag, and it was still happening! But this was illuminating because right above where the body tag came out was one little optimization we had done recently.

When usability testing, we noticed that there was a lag in images being displayed which was not user-friendly. To help prevent that problem, we looked into image prefetching techniques, and fell upon the CSS technique:

body:after {
content:url(image.png);
display:none;
}

This made a world of a difference in testing. According to my research, this technique is suppose to be ignored by browsers that do not support the :after clause and so it felt safe. Since the images in question were very specific to the application being developed and the page in question is not intended to be read by search engines, we decided to inline this code on the page.

Could this be the cause? Well, in trying to narrow it down further, I did comment out the entire style-tag and sure enough, my page did not go into compatibility mode.

This fixed my compatibility-mode problem and hopefully it may help someone else out there.

Math.abs has an edge case...

Thu, 18 Sep 2008 04:55:58 GMT

Consider the following Java code:

int ai = Math.abs( i );
assert ai >= 0;

Will the assert ever fire?

Yes, it turns out that it can fire if i = Integer.MIN_VALUE, and you have been warned about this in the Math.abs javadoc, which states “Note that if the argument is equal to the value of Integer.MIN_VALUE, the most negative representable int value, the result is that same value, which is negative.’

Keep this in mind when looking at such innocent looking code...

Java Mail: Make sure the mail.smtp.auth property's value is a String...

Fri, 12 Sep 2008 04:38:54 GMT

If you are using the JavaMail API to send out e-mail via an e-mail server that requires authentication and are getting an error like com.sun.mail.smtp.SMTPSendFailedException: 550 You must authenticate, make sure the mail.smtp.auth property is assigned as a String, like this:

properties.put( "mail.smtp.auth", "true" );

Not exactly the most obvious bug to find...

Using Ant with Cygwin

Sat, 24 May 2008 04:29:12 GMT

If you have ever tried building something with ant under Cygwin, you probably have noticed an error such as the following:

build.xml:35: c:\cygdrive\c\Progra~1\java/SunAppServer\jdk\lib\J2EE.jar

The current Ant documentation suggests that you should change your build.xml to wrap all such environment labels and class paths with code that executes cygpath, but this turns out to create some ugly code, especially if your build scripts use a lot of environment labels.

An alternative scheme that I have just come up with is to use a layer of indication. You can create an ant shell script that translates all of your necessary Cygwin environment labels using cygpath, and then call the ant.bat to do the actual build. The advantage of this approach is that more build.xml files are cross-platform out of the box and require very few, if any, tweaks.

The following file is an example of my approach:

#!/bin/sh
# OS-Specific Stuff
export PATH=`/usr/bin/cygpath --path --windows $PATH`
export TEMP=`/usr/bin/cygpath --path --windows $TEMP`
export TMP=`/usr/bin/cygpath --path --windows $TMP`

# Java Specific Stuff
export JAVA_HOME=`/usr/bin/cygpath --windows $JAVA_HOME`
export J2EE_HOME=`/usr/bin/cygpath --windows $J2EE_HOME`
export TOMCAT_HOME=`/usr/bin/cygpath --path --windows $TOMCAT_HOME`
export CLASSPATH=`/usr/bin/cygpath --path --windows $CLASSPATH`

# Ant Specific Stuff
ORIGINAL_ANT_HOME=$ANT_HOME
export ANT_HOME=`/usr/bin/cygpath --path --windows $ANT_HOME`

# Project-Specific Stuff
export WHATEVER_YOU_NEED=`/usr/bin/cygpath --windows $WHATEVER_YOU_NEED`

# Invoke the ANT.BAT via the original ANT_HOME value
$ORIGINAL_ANT_HOME/bin/ant.bat $1 $2 $3 $4 $5 $6 $7 $8 $9

Of course, this technique can be generally applied to other applications that use environment labels. Hope that helps.