How to do variable/conditional SQL sorting order of results

Posted on July 25, 2010 by patrick

I was jazzed today. I figured out how to conditional order of SQL results. Specifically being able to vary change the sorting order within the results based on a database field.

Specifically,

  1. Create two tables Main and Secondary
    • create table Main ( ID bigint, ordering varchar(3));
    • create table Secondary ( Main_ID bigint, Secondary_Value bigint);
  2. Insert Values:
    • insert into Main (ID, ordering) values (1, 'u'),(2,'d'),(3,'n');
    • insert into Secondary (Main_ID, Secondary_Value) values ( 1, 11),(1,21),(1,41),(1,31),(1,71),(1,61),(1,51),(2,62),(2,42),(2,82), ( 3,3), (3, 1), (3, 5);
  3. Create a query that orders the results by Main.ID, Secondary.Secondary_Value based on Main.ordering value. If the Main.ordering is:
    • ‘u’ then the corresponding Secondary_Value is ordered ascending,
    • ‘d’ then the corresponding Secondary_Value is ordered descending,
    • ‘n’ then the corresponding Secondary_Value is unordered

The solution is:

select * from Main join Secondary on ( Main.ID = Secondary.Main_ID ) order by Main.ID, (case MAIN.ordering when 'u' then 1 when 'd' then -1 else 0 end) * Secondary.Secondary_Value;

+------+----------+---------+-----------------+
| ID   | ordering | Main_ID | Secondary_Value |
+------+----------+---------+-----------------+
|    1 | u        |       1 |              11 |
|    1 | u        |       1 |              21 |
|    1 | u        |       1 |              31 |
|    1 | u        |       1 |              41 |
|    1 | u        |       1 |              51 |
|    1 | u        |       1 |              61 |
|    1 | u        |       1 |              71 |
|    2 | d        |       2 |              82 |
|    2 | d        |       2 |              62 |
|    2 | d        |       2 |              42 |
|    3 | n        |       3 |               1 |
|    3 | n        |       3 |               5 |
|    3 | n        |       3 |               3 |
+------+----------+---------+-----------------+
13 rows in set (0.00 sec)

Variable ordering within a single result set! woo-hoo!

Posted in help notes, technical | Leave a comment

User-Generated Content? Do not use BurstNET

Posted on July 20, 2010 by patrick

A hosting provider, BurstNet, shut down a wordpress hosting service, Blogetery.com, last week.

This is an example of egregious abuse of power by BurstNET against Blogetery.com. Blogetery.com hosts 70,000 wordpress blogs. Apparently, a few of those blogs had “an al-Qaeda hit list” and “bomb-making” instructions on them. So for the possible sins of a few, 69,999 innocent blogs are shutdown with no recourse.

You can read BurstNet’s lame press release here.

Any web service that allows user-generated content ( so pretty much any site that isn’t a static site) would be subject to a draconian response by BurstNET. What is especially egregious is that BurstNet is not letting Blogetery get access to any of Blogetery’s data!

Continuing, from the New York Times Bits blog,:

And Kurt Opsahl, a staff attorney for the Electronic Frontier Foundation, a digital rights group, said the “tragedy is that thousands of blogs will be taken offline for no good reason.”

Mr. Yusupov said he had backed up some of the blogging site’s data, but not all. And he said he was trying to negotiate with BurstNet to get the data so he could restart the blogging site with another hosting service. “This has been a big hassle for me,” he said.

In the New York Times Bits blog,:

Mr. Marr of BurstNet said the Blogetery server did “not get a lot of use or traffic,” suggesting the number of active users of the free blogging site was probably a tiny fraction of the 73,000 Blogetery claimed.

So basically, Blogetery is a small business and is therefore “not important”. Important news to any small business considering using BurstNet:

BurstNET thinks small businesses are not important

Stay away, stay away! You cannot run a business where at the arbitrary whim of the hosting provider you can be shutdown, especially if the hosting provider will not supply your data so you can go elsewhere!

For similar reason, I stayed away from GoGrid (and now BurstNet).

Update ( 28 July 2010 ) :

A last interesting point from CNET about how Burst.NET feels about their customers’ data (aka don’t use Burst.Net for mission-critical purposes):

Joe Marr, chief technology officer of Burst.net, stressed again that the reason for the service termination was that the materials the FBI alleges belonged to terrorists are a violation of Burst.net’s terms of service. He noted that typically, Burst.net does not return data to customers booted for TOS violations.

Violations of the Burst.NET Terms of Service result is losing all your data!

Excuse me????
Use Rackspace

Posted in management, starting a company | Leave a comment

Britian cancels runways because of global warming concerns

Posted on July 3, 2010 by patrick

Its time to give up “binge flying” says the Conservative government:

“The emissions were a significant factor” in the decision to cancel the runway-building plans, Teresa Villiers, Britain’s minister of state for transport, said in an interview. “The 220,000 or so flights that might well come with a third runway would make it difficult to meet the targets we’d set for ourselves.” She said that local environmental concerns like noise and pollution around Heathrow also weighed into the decision.

Posted in high-speed-rail, political, transportation | Leave a comment

If you don’t vote, your opinion doesn’t matter. And sometimes you shouldn’t vote

Posted on June 24, 2010 by patrick

(This post is related to management I promise! )

Robert Cruickshank over at the California HSR Blog whines about Palo Alto’s “undemocratic” democratic process:

In short, it is becoming increasingly clear that Palo Alto’s planning and citizen engagement process is a failure, distorting true public opinion by favoring a small, vocal elite at the expense of a silent majority whose opinions are much more supportive of new density and new transportation solutions – but whose voices are rarely ever included in the city’s planning process.

Of course, our democratic process requires energy and participation. There are lots of people who chose not to vote because their vote “will not be effective”.

Most people are uninformed about this issue, do not ride transit, or have no idea how to build transit effectively. Their opinion should not count as much as the people who are taking the time to inform themselves and to be involved.

If someone is not involved, their opinion is probably uninformed and negative.

Meetings to planning a company project can be just as bad.

Uninformed people should not be part of the process(see below). In
an old Inc. article, Joel:

When was the last time you scheduled a meeting and invited eight people instead of the three people who really needed to be there simply because you didn’t want anyone to feel left out?

When was the last time you sent a companywide e-mail that said something like, “Hey, attention coffee drinkers: If you finish the pot, make another!” even though there is actually only one person who violates this rule (and she’s your co-founder)?

When was the last time you got into a long discussion over the color palette for the new brochure with a programmer, who has nothing to do with the brochure but sure knows that he doesn’t like orange?

These are symptoms of a common illness: too much communication.

(“below”)
However, I disagree with Joel’s assertion that only people whose vote counts should be allowed to attend meetings. Decisions with no visible process result in no buy-in. While a company is not a democracy, and a city is not a company both should learn from each other.

What a company can learn from a city:

  1. Process does matter. Process means consistency and reliability in how decisions are being made. People know how to voice their opinion. They know that there is a means and method for voicing their opinion. Instead of voicing opinion in an adhoc, disruptive manner – they can wait until the allocated time.
  2. Only some people get a vote. Many people can show up to a city council to express their opinion, but only city council members get a vote. In company meeting, discussion can include everyone – but predecided ( and preannounce! ) who’s vote will be counted. For example, if a developer is trying to decide who to best implement a feature. Only his/her, the CTO’s, QA’s and customer service rep’s votes are counted. Others who are not involved, do not get to vote. They can express their opinion but they are not a decision maker (for this issue). Only people expending effort or where the decision has a material impact on their job should be counted.
  3. Representatives get “elected”. Allow some self-selection in the process. Try to allow the lead representative to be selected by people other than managers. If a developer selected to be the lead in a project makes a decision, this makes it easier for the decision to be respected.
  4. Make the discussion observable and inclusive While only some people get a vote, allowing others to learn from the process of making a decision prepares those observers to step into their own decision-making role. It also allows them to take knowledge from one decision-making group to another.

What a city can learn from a company:

  1. Require an energy expenditure to participate. A meeting should only be open to people who have attended the last 5 meetings.
  2. Allow adhoc representation. Allow people to represent adhoc groups. For example, allow a person to collect 10+ signatures of his neighbors giving him/her proxy authority to voice their collective opinion. Require that this adhoc representative keep the people she is representing informed of the progress and results. (With power should come responsibility).
  3. Allow weighted voting. In a company, the CTO’s vote counts more than a lowly developer. When voting for a company’s board of directors’, shareholders have a vote based on number of shares not a one vote per shareholder. In a city planning process, the “vocal” minority may represent no one other than themselves. Let the “vocal minority” collect proxy signatures to indicate how strongly their “silent” neighbors (who can’t participate) trust the “vocal” people to represent the “silent” majorities best interest. The more signatures, the more strongly a “vocal” representative’s vote/opinion should count. Allow certain signatures to be more valuable than others based on the issue. For example, distance to a housing project, transit user’s opinion on a transit project, etc.

Update:

Lastly, learn when you should not vote or participate.

  1. If you personally do not have any direct, meaningful, unique knowledge: don’t participate. Observing is o.k. – voicing a “I agree” content-free vocalization is not o.k.
  2. If you don’t have the time to stay involved: don’t sign up and then drop out.
  3. If an issue has no one who cares: then the decision can be made by a single person. Others should insist that that single person make the decision. The sole decision-maker should not need the CYA of a “group vote”.
  4. If you cannot expend effort on the solution, then don’t vote. Note that “effort” does not mean “coding” or “making”
  5. If the decision will not effect how hard your job is, then don’t vote. If the decision does meaningfully effect your job then you must participate and must vote.

There is this temptation to dismiss the concerns of Customer Service or QA people as being less important than that of the development team. This is ass-backwards.

A Customer Service rep will have to deal daily with a bad development decision. Their job satisfaction, their ability to deliver happy customers is daily determined by developers decisions. They must be allowed to participate and must be given a strong voice.

Posted in how to, management, political | Leave a comment

Where to start learning Java if you are a PHP person?

Posted on June 9, 2010 by patrick

Over at StackOverflow, the question was asked : Where to start with Java when coming from PHP?

PHP5 has the concept of objects, interfaces and exceptions. These are similar enough to Java’s version of objects, interfaces and exceptions for basic learning purposes.

Once you get the PHP5 equivalent understood then crossover to Java.

Since everyone starts with a Hello world program,

  1. Set up eclipse (J2EE version)
  2. Use the Java tooling to create a new class that is your Hello world.
  3. Figure out the basics of debugging with eclipse
  4. Figure out the basics of objects, interfaces, exceptions and inheritance.
  5. Understand the basic language differences PHP vs. Java’s
  6. Understand the differences between a static typed language and the dynamic-typed languages that you are used to.
  7. Learn the classes in java.lang.* and java.util.* packages.
Posted in how to, technical | Leave a comment

why people bother to interview here – if you can’t teach (Part 2)

Posted on June 3, 2010 by patrick

Part 1 was a technical question that I ask because so many “strong” developers cannot answer that technical Java question. Part 2, is more about the valuable soft skill of teaching.

One of my standard questions that candidates constantly struggle with is the most simple and yet the most subtle:

Teach me something you know.

If a candidate cannot teach “something”, then they have not completely thought through and learned the subject. Someone who claims to be “strong” in Java, databases, or cooking is only strong if they can articulate the core essence of their knowledge in a way that a beginner can understand.

When developing code at a startup, a good “teacher”:

  • understands that it is the teacher’s responsibility to reach the student. If the student doesn’t understand, then a good “teacher” patiently retries until the student does understand. In a startup, the “student” is a new hire, a sales person or the customer;
  • writes excellent code comments (for the student),
  • is patient and not egotistic when others don’t “get it”;
  • can articulate their own problems so that others can help;
  • can give presentations to non-technical people;
  • is empathic;
  • is motivated to do their own learning.
  • is enthusiastic

So what can you teach?

Posted in interviewing, starting a company | Leave a comment

Multi-threaded Homework Problem

Posted on June 1, 2010 by patrick

Occasionally I have been asked about learning multithreaded programming.

My standard suggested problem is to create a “simplistic” TCP/IP block ordering library. This is a variant of the standard producer/consumer problem that shows up in interviews. This problem is a standard multiple-producer/multiple-consumer problem with a twist and goes way beyond what can be handled in an interview.

Problem #1

Use case:

  • each block as a conversation Id and a sequence id within the conversation.
  • All blocks are received on a single queue.
  • Blocks are separated by conversation and then ordered by sequence number.
  • Blocks arrive semi-random order – the sequence id will be within +/-2 of the previous block (in the same conversation)’s sequence id. But there will be no gaps in the sequence id for a given conversation.
  • sequence id 0 starts the conversation, sequence id 99 ends the conversation

Sample sequence: (1,0), (1,2), (2,0), (1,3), (2,2), (1,1), (2,1), ….

Problem #2:

Extending Problem #1 with these additional requirements:

  • Conversations where the 0 block has not been seen are discarded.
  • Blocks may be repeatedly sent. Duplicates are discarded.
  • Conversations are now buffered when doing the ordering ( represents buffering within the TCP/IP stack )
  • Each conversation chunk can only hold 3 blocks.
  • Each buffer when complete is dispatched in order to a dummy thread ( represents the application ) – which has a random delay and then tests and discards the data.

Sample data (for single conversation):

(1,0) - buffer#1; (1, 5) – buffer#2; (1,6) - buffer#3; (1,1) – buffer #1; (1,2) -buffer #1 (dispatched); (1,1) – duplicate (discarded); (1,3) – buffer #2; (1,7) – buffer #3 ( cannot be dispatched because buffer #2 has not been dispatched); (1,4) – buffer #2 and #3 are both dispatched; (1,10) – buffer #1 is now reavailable; ….

Problem #3:

Additional requirements:

  • Each conversation now has the sequence variation by +/-10 (bell curve)
  • Each conversation (on the consumer side) has only 2 buffers available to it
  • Blocks that can not be sequentially placed in the conversation buffers are discarded.
  • When the consumer threads detect 3 failures to complete the buffer, the consumer notifies the producers that the missing block needs to be resent.
  • Conversations have an indeterminate length. Additional flag signals the last block in the conversation.
  • Conversations that have not sent data in the last 30 seconds are terminated. All uncompleted buffers are discarded. Any further TCP/IP blocks are discarded.
  • Producers have a 5 buffers per conversation available

Things to look for:

  • Starvation – conversation is not being processed in spite of buffers being available
  • Effective I/O rate
  • Retry rate because of dropped buffers

Update:

Some resource suggestions from Ronald Vermeij:

Simple very basic tutorial

Concurrency: State Models & Java Programs Book

Nice “Concurrency RefCard” at Devzone

Update 2

The homework problem is actually fiendishly difficult. The problem is similar to a real world problem I had constructing a network switch monitoring program. The switches run a http server on a low priority thread. The http server respond slowly, occasionally they just drop a request. In the test lab, the switches are not under high load but in the field they are. Under high load, the http server may not respond in a timely manner. Different switch models responded differently. Determining the monitoring program’s response to a switch’s behavior or (non) behavior was “interesting” – panic and tell the operator that the switch has crashed? Do we delay the failure notification (turning switch status to red)? How often to poll the switch ( we don’t want to add to the overloaded switch’s work ). We also had to process the incoming data as fast as possible so the switch doesn’t drop the connection because the response is not being processed.

Extra credit:

  • Separate out the producer and consumer portions into 2 different programs
  • Each conversation is on a separate Java NIO connection
  • 100+ producer threads (in the producer program), 5 consumer threads in the consumer program
  • each conversation’s producer produces data at a random rate (will be bursty).
  • The conversation’s consumer must consume the produced data fast enough so the producer always has an available buffer to write to ( producer has 2 buffers – but try reducing the producer’s available buffers ). Consumer’s per conversation buffers can be higher than previous problem.
  • Producer may arbitrarily stop sending data
  • Use Java NIO

Notice:

  • When setting up, the connection the consumer must quickly complete establishment of the connection otherwise the connection will be dropped. [Hint: Have a dedicated consumer thread do just the connection establishment. Many Java NIO sample programs try to have the same thread establish connections and do the read operations those connections.]
  • Consumer must be aggressive about processing incoming data.
  • Consumer needs to have a “give up and retry” mechanism
Posted in how to, technical | Leave a comment

Bad management advice from Jason Calacanis

Posted on May 28, 2010 by patrick

I am beginning to think that Jason Calacanis’ advice should be taken in the opposite. Jason Calacanis’s latest advice on when to fire people ignores human psychology:

Calacanis goes on to examine the three categories of that mistakes and employees can fall into and discusses his thoughts on who and when to fire:

  1. A great team member who makes a big mistake

    Verdict: Don’t fire them. Talk about the mistake, and brainstorm ways to fix it and to make sure it doesn’t happen again.

  2. An average team member who makes a big mistake

    Verdict: Fire them. In fact, Calacanis makes the argument that you shouldn’t hire average employees at your startup in the first place.

  3. A great team member who makes multiple mistakes

    Verdict: It’s complicated, and unfortunately, it’s all too common. Calacanis says that when faced with this, he does try to work through the person’s problems. And while he notes that founders might not have the time or the training to do this, he stresses its importance. “When you try to save a flawed, yet at other times effective, team member, you send the other members of your team a positive message: loyalty.”

Seems reasonable doesn’t it? But lets break this down from the perspective of the employee.

Flaw #1: Employee does not know where (s)he stands.

Does the employee know that he is an “average employee” or is a “great team member”? From earlier in the post, clearly in this case the employee responsible for the problem in question thought that he was an average employee living under the one strike rule:

And according to Calacanis, the individual responsible was apologetic, fearing that he’d be fired.

Flaw #2: “Great Team Member” Employee does not know when their chances are used up.

In the case, in question the sysadmin who screwed up and let people post the the jason-list discovered he was a “great team member”. This mistake did not mean his job. whew…. however, if that sysadmin makes another mistake – how much time has to elapse before he is in the category #3 – “great team member but multiple mistakes”? Is he allowed 1 per month? one per quarter?

Flaw #3: Inconsistent response

Human beings as a rule do not respond well to perceived inconsistently applied punishment. Having the “great team members” not get fired for a mistake that got a “average” employee fired does not create loyalty – just fear.

Flaw #4: Dependency on above average employees

A business owner should be creating a reproducible process that drives repeated high-quality execution. Relying on above average talent to sustain the business is simply bad practice that results in inconsistent and high-stress environments.

Flaw #5: Blame

Failures are always a sign of a systemic problem. When the B-17 (Flying Fortress) was developed there were several crashes traced back to pilot error. “Some newspapers had dubbed it[B-17] as ‘too much plane for one man to fly.’ . The Air Force’s solution was not to find better pilots but rather to develop the checklist.

My summarized counter suggestions:

  1. Look for and reward passion – passionate people will self-correct each other
  2. Define a fireable offense, in advance and apply consistently
  3. Expect better – Put everyone on a Performance Improvement Plan including the CEO – Performance improvement plans should be a plan to raise the organizational execution level.
  4. Expect Greatness – Drive a plan to get people to “hit above their weight class” – in other words exceed their personal best. Read the book Moneyball to understand this idea applied in baseball.
  5. Create Checklists – Use the above average people in the organization to create a process that does prevents mistakes. In other words, the creative great people should not necessarily be promoted to managers but rather be put in charge of formalizing their “above averageness”. Internally, I am working on having a great contributor (“Bill”) formalize his current responsibilities. This formalization will enable another contributor (“Peter”) with a significantly lower skill set to take over the tasks. Because of the formalization of greatness, “Peter” will operate at Bill’s higher skill level. Bill can now move on to other new challenges.

Update: Read ben horowitz’ post

Posted in management, starting a company | Leave a comment

Facebook indifference at work again

Posted on May 23, 2010 by patrick

Last week, Facebook made a change resulted in small businesses losing the ability to create custom landing pages. Facebook later reversed their decision. Dennis Yu at Blitzlocal comment on the custom landing page change noting:

this just underscores the risk of building your business on someone else’s website. You’re at the whim of their policy changes.

This should be in loud screaming letters. Having no Plan B, no alternatives is lunacy.

Having business alternatives is like keeping backups of your data. The Facebook change may have been inadvertent this time and reversed in part, but the larger reality is that it really doesn’t matter. There was a business impact on businesses wholly dependent on FB. If the *only* way a business has to communicate to their customers is FB – then what happens when:

  • Facebook decides that the business is a spammer?
  • Facebook has a bug that turns off access to sites starting with the letter ‘b’ ?
  • Facebook decides the account is violating the Facebook ToS?
  • Facebook suffers a data corruption that drops all the business’s fans?
  • Facebook enters into a court agreement requiring Facebook change the way businesses can interact with users?

Businesses dependent on Facebook need to realize :

  1. They have no enforceable contract with Facebook
  2. They have no recourse for any economic impacts of Facebook decisions or failures
  3. Blowback against Facebook impact their own business directly.

Right now Facebook shows little understanding of the larger issues around privacy. Read the “interaction” between Elliot Schrage VP at Facebook and the New York Times public. Elliot seems to think that Facebook’s problem is that there is just a lack of understanding. Elliot thinks that Facebook does not need to change their behavior, just explain better:

We’ve worked hard to educate our users about changes to, and innovations in, our products. Facebook users receive notices about our new products and whenever we propose a change to any policies governing the site, we have notified users and solicited feedback.

Clearly, this is not enough. We will soon ramp up our efforts to provide better guidance to those confused about how to control sharing and maintain privacy.

Facebook does not have to be malicious, to cause harm.

Facebook is indifferent. If a business is not big enough, can’t organize a massive response…..

Facebook.

does.

NOT.

care.

Posted in facebook, starting a company | Leave a comment

Net Neutrality is good for small businesses

Posted on April 6, 2010 by patrick

Take Action

The internet provider I use should ONLY provide the infrastructure to connect me with the Internet.

As a business owner, I depend on skype / chat to conduct all my business dealings. My internet provider should not be permitted to interfere with my business transactions.

For any internet provider to insert themselves into the conversation I am having with the world, is the same as if the phone company inserted themselves into my voice communication.

I do not want a operator breaking in on my voice conversation to add their two cents to my private voice conversations with my employees.

Nor do I want the internet provider to break into my private written conversations with my employees just because I happen to be using VoIP (skype) to talk with them.

Nor do I want the internet provider eavesdropping in on my chat messages with my employees!

To allow this eavesdropping in any manner is to allow wiretapping!

Bluntly, clearly – “deep packet inspection”, “traffic shaping”, whatever the buzz words used is SPYING.

Furthermore, my small business is internet-based. My small business cannot afford to pay for special treatment. If my customers have their traffic “shaped” so that my site appears to be unresponsive or slow, then my business will lose customers to the larger competitors that can pay off the Comcasts of the world.

Take Action

Posted in political | Leave a comment