Just wondering.... » technical

How to do variable/conditional SQL sorting order of results

patrick — Sun, 25 Jul 2010 18:48:49 +0000

I was jazzed today. I figured out how to conditional order of SQL results. Specifically being able to vary change the sorting order within the results based on a database field.

Specifically,

Create two tables Main and Secondary
- create table Main ( ID bigint, ordering varchar(3));
- create table Secondary ( Main_ID bigint, Secondary_Value bigint);
Insert Values:
- insert into Main (ID, ordering) values (1, 'u'),(2,'d'),(3,'n');
- insert into Secondary (Main_ID, Secondary_Value) values ( 1, 11),(1,21),(1,41),(1,31),(1,71),(1,61),(1,51),(2,62),(2,42),(2,82), ( 3,3), (3, 1), (3, 5);
Create a query that orders the results by Main.ID, Secondary.Secondary_Value based on Main.ordering value. If the Main.ordering is:
- ‘u’ then the corresponding Secondary_Value is ordered ascending,
- ‘d’ then the corresponding Secondary_Value is ordered descending,
- ‘n’ then the corresponding Secondary_Value is unordered

The solution is:

select * from Main join Secondary on ( Main.ID = Secondary.Main_ID ) order by Main.ID, (case MAIN.ordering when 'u' then 1 when 'd' then -1 else 0 end) * Secondary.Secondary_Value;

+------+----------+---------+-----------------+
| ID   | ordering | Main_ID | Secondary_Value |
+------+----------+---------+-----------------+
|    1 | u        |       1 |              11 |
|    1 | u        |       1 |              21 |
|    1 | u        |       1 |              31 |
|    1 | u        |       1 |              41 |
|    1 | u        |       1 |              51 |
|    1 | u        |       1 |              61 |
|    1 | u        |       1 |              71 |
|    2 | d        |       2 |              82 |
|    2 | d        |       2 |              62 |
|    2 | d        |       2 |              42 |
|    3 | n        |       3 |               1 |
|    3 | n        |       3 |               5 |
|    3 | n        |       3 |               3 |
+------+----------+---------+-----------------+
13 rows in set (0.00 sec)

Variable ordering within a single result set! woo-hoo!

Where to start learning Java if you are a PHP person?

patrick — Wed, 09 Jun 2010 20:07:37 +0000

Over at StackOverflow, the question was asked : Where to start with Java when coming from PHP?

PHP5 has the concept of objects, interfaces and exceptions. These are similar enough to Java’s version of objects, interfaces and exceptions for basic learning purposes.

Once you get the PHP5 equivalent understood then crossover to Java.

Since everyone starts with a Hello world program,

Set up eclipse (J2EE version)
Use the Java tooling to create a new class that is your Hello world.
Figure out the basics of debugging with eclipse
Figure out the basics of objects, interfaces, exceptions and inheritance.
Understand the basic language differences PHP vs. Java’s
Understand the differences between a static typed language and the dynamic-typed languages that you are used to.
Learn the classes in java.lang.* and java.util.* packages.

Multi-threaded Homework Problem

patrick — Tue, 01 Jun 2010 17:43:27 +0000

Occasionally I have been asked about learning multithreaded programming.

My standard suggested problem is to create a “simplistic” TCP/IP block ordering library. This is a variant of the standard producer/consumer problem that shows up in interviews. This problem is a standard multiple-producer/multiple-consumer problem with a twist and goes way beyond what can be handled in an interview.

Problem #1

Use case:

each block as a conversation Id and a sequence id within the conversation.
All blocks are received on a single queue.
Blocks are separated by conversation and then ordered by sequence number.
Blocks arrive semi-random order – the sequence id will be within +/-2 of the previous block (in the same conversation)’s sequence id. But there will be no gaps in the sequence id for a given conversation.
sequence id 0 starts the conversation, sequence id 99 ends the conversation

Sample sequence: (1,0), (1,2), (2,0), (1,3), (2,2), (1,1), (2,1), ….

Problem #2:

Extending Problem #1 with these additional requirements:

Conversations where the 0 block has not been seen are discarded.
Blocks may be repeatedly sent. Duplicates are discarded.
Conversations are now buffered when doing the ordering ( represents buffering within the TCP/IP stack )
Each conversation chunk can only hold 3 blocks.
Each buffer when complete is dispatched in order to a dummy thread ( represents the application ) – which has a random delay and then tests and discards the data.

Sample data (for single conversation):

(1,0) - buffer#1; (1, 5) – buffer#2; (1,6) - buffer#3; (1,1) – buffer #1; (1,2) -buffer #1 (dispatched); (1,1) – duplicate (discarded); (1,3) – buffer #2; (1,7) – buffer #3 ( cannot be dispatched because buffer #2 has not been dispatched); (1,4) – buffer #2 and #3 are both dispatched; (1,10) – buffer #1 is now reavailable; ….

Problem #3:

Additional requirements:

Each conversation now has the sequence variation by +/-10 (bell curve)
Each conversation (on the consumer side) has only 2 buffers available to it
Blocks that can not be sequentially placed in the conversation buffers are discarded.
When the consumer threads detect 3 failures to complete the buffer, the consumer notifies the producers that the missing block needs to be resent.
Conversations have an indeterminate length. Additional flag signals the last block in the conversation.
Conversations that have not sent data in the last 30 seconds are terminated. All uncompleted buffers are discarded. Any further TCP/IP blocks are discarded.
Producers have a 5 buffers per conversation available

Things to look for:

Starvation – conversation is not being processed in spite of buffers being available
Effective I/O rate
Retry rate because of dropped buffers

Update:

Some resource suggestions from Ronald Vermeij:

Simple very basic tutorial

Concurrency: State Models & Java Programs Book

slides, notes and lectures

demo programs with GUI interaction and source-codes

Nice “Concurrency RefCard” at Devzone

Update 2

The homework problem is actually fiendishly difficult. The problem is similar to a real world problem I had constructing a network switch monitoring program. The switches run a http server on a low priority thread. The http server respond slowly, occasionally they just drop a request. In the test lab, the switches are not under high load but in the field they are. Under high load, the http server may not respond in a timely manner. Different switch models responded differently. Determining the monitoring program’s response to a switch’s behavior or (non) behavior was “interesting” – panic and tell the operator that the switch has crashed? Do we delay the failure notification (turning switch status to red)? How often to poll the switch ( we don’t want to add to the overloaded switch’s work ). We also had to process the incoming data as fast as possible so the switch doesn’t drop the connection because the response is not being processed.

Extra credit:

Separate out the producer and consumer portions into 2 different programs
Each conversation is on a separate Java NIO connection
100+ producer threads (in the producer program), 5 consumer threads in the consumer program
each conversation’s producer produces data at a random rate (will be bursty).
The conversation’s consumer must consume the produced data fast enough so the producer always has an available buffer to write to ( producer has 2 buffers – but try reducing the producer’s available buffers ). Consumer’s per conversation buffers can be higher than previous problem.
Producer may arbitrarily stop sending data
Use Java NIO

Notice:

When setting up, the connection the consumer must quickly complete establishment of the connection otherwise the connection will be dropped. [Hint: Have a dedicated consumer thread do just the connection establishment. Many Java NIO sample programs try to have the same thread establish connections and do the read operations those connections.]
Consumer must be aggressive about processing incoming data.
Consumer needs to have a “give up and retry” mechanism

… and I nailed the javadoc

patrick — Thu, 07 Jan 2010 18:44:08 +0000

Yesterday was a red letter day. I found multiple bugs in the javac and javadoc. (1.6.0_17)

This javadoc issue occurred when there is a class that has no package line ( it is in the default package). Annoyingly enough it occurred when I was trying to create a test case for this javac bug. I was running maven 2.2.1 at the time.

java.lang.NullPointerException
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generatePackageUse(PackageUseWriter.java:180)
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generatePackageList(PackageUseWriter.java:124)
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generatePackageUse(PackageUseWriter.java:110)
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generatePackageUseFile(PackageUseWriter.java:99)
       at com.sun.tools.doclets.formats.html.PackageUseWriter.generate(PackageUseWriter.java:78)
       at com.sun.tools.doclets.formats.html.ClassUseWriter.generate(ClassUseWriter.java:116)
       at com.sun.tools.doclets.formats.html.HtmlDoclet.generateOtherFiles(HtmlDoclet.java:92)
       at com.sun.tools.doclets.internal.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:122)
       at com.sun.tools.doclets.internal.toolkit.AbstractDoclet.start(AbstractDoclet.java:64)
       at com.sun.tools.doclets.formats.html.HtmlDoclet.start(HtmlDoclet.java:42)
       at com.sun.tools.doclets.standard.Standard.start(Standard.java:23)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at com.sun.tools.javadoc.DocletInvoker.invoke(DocletInvoker.java:269)
       at com.sun.tools.javadoc.DocletInvoker.start(DocletInvoker.java:143)
       at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:340)
       at com.sun.tools.javadoc.Start.begin(Start.java:128)
       at com.sun.tools.javadoc.Main.execute(Main.java:41)
       at com.sun.tools.javadoc.Main.main(Main.java:31)

Solution was to put the java code into a package.

Woot! I crashed the javac

patrick — Thu, 07 Jan 2010 18:35:10 +0000

My javac crash:

An exception has occurred in the compiler (1.6.0_17). Please file a bug at the Java Developer Connection (http://java.sun.com/webapps/bugreport) after checking the Bug Parade for duplicates. Include your program and the following diagnostic in your report. Thank you.


java.lang.AssertionError: isSubtype 15 ( 15 = TypeTags.WILDCARD )
	at com.sun.tools.javac.code.Types$5.visitType(Types.java:347)
	at com.sun.tools.javac.code.Types$5.visitType(Types.java:328)
	at com.sun.tools.javac.code.Types$DefaultTypeVisitor.visitWildcardType(Types.java:3163)
	at com.sun.tools.javac.code.Type$WildcardType.accept(Type.java:416)
	at com.sun.tools.javac.code.Types$DefaultTypeVisitor.visit(Types.java:3161)
	at com.sun.tools.javac.code.Types.isSubtype(Types.java:324)
	at com.sun.tools.javac.code.Types.isSubtype(Types.java:308)
	at com.sun.tools.javac.code.Types.isSubtypeUnchecked(Types.java:288)
	at com.sun.tools.javac.code.Types.isSubtypeUnchecked(Types.java:460)
	at com.sun.tools.javac.comp.Infer.checkWithinBounds(Infer.java:388)
	at com.sun.tools.javac.comp.Infer.instantiateExpr(Infer.java:241)
	at com.sun.tools.javac.comp.Check.instantiatePoly(Check.java:356)
	at com.sun.tools.javac.comp.Check.checkType(Check.java:324)
	at com.sun.tools.javac.comp.Attr.check(Attr.java:160)
	at com.sun.tools.javac.comp.Attr.visitApply(Attr.java:1276)
	at com.sun.tools.javac.tree.JCTree$JCMethodInvocation.accept(JCTree.java:1210)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribExpr(Attr.java:377)
	at com.sun.tools.javac.comp.Attr.visitAssign(Attr.java:1550)
	at com.sun.tools.javac.tree.JCTree$JCAssign.accept(JCTree.java:1342)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribExpr(Attr.java:384)
	at com.sun.tools.javac.comp.Attr.visitExec(Attr.java:1017)
	at com.sun.tools.javac.tree.JCTree$JCExpressionStatement.accept(JCTree.java:1074)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitIf(Attr.java:1009)
	at com.sun.tools.javac.tree.JCTree$JCIf.accept(JCTree.java:1050)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitForLoop(Attr.java:740)
	at com.sun.tools.javac.tree.JCTree$JCForLoop.accept(JCTree.java:818)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitIf(Attr.java:1009)
	at com.sun.tools.javac.tree.JCTree$JCIf.accept(JCTree.java:1050)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitIf(Attr.java:1009)
	at com.sun.tools.javac.tree.JCTree$JCIf.accept(JCTree.java:1050)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribStats(Attr.java:413)
	at com.sun.tools.javac.comp.Attr.visitBlock(Attr.java:715)
	at com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:739)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.visitMethodDef(Attr.java:634)
	at com.sun.tools.javac.tree.JCTree$JCMethodDecl.accept(JCTree.java:639)
	at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
	at com.sun.tools.javac.comp.Attr.attribStat(Attr.java:397)
	at com.sun.tools.javac.comp.Attr.attribClassBody(Attr.java:2697)
	at com.sun.tools.javac.comp.Attr.attribClass(Attr.java:2628)
	at com.sun.tools.javac.comp.Attr.attribClass(Attr.java:2564)
	at com.sun.tools.javac.main.JavaCompiler.attribute(JavaCompiler.java:1036)
	at com.sun.tools.javac.main.JavaCompiler.compile2(JavaCompiler.java:765)
	at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:730)
	at com.sun.tools.javac.main.Main.compile(Main.java:353)
	at com.sun.tools.javac.main.Main.compile(Main.java:279)
	at com.sun.tools.javac.main.Main.compile(Main.java:270)
	at com.sun.tools.javac.Main.compile(Main.java:69)
	at com.sun.tools.javac.Main.main(Main.java:54)

I was not successful at reducing this crash to a simple test case.

However, the line that triggered the crash was:


ProxyMapperImplementor childProxy = this;

Some points:

Once again (and again) the eclipse compiler is more robust than the sun javac
ProxyMapperImplementor is defined as:
interface ProxyMapperImplementor
Removing the wildcards so the line looks like solved the problem:ProxyMapperImplementor childProxy = this;

Update (email from sun):

This issue is related to Bug ID: 6738538. You can review this bug at:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6738538

This problem has been fixed in Java SE 7.0. Consider downloading a free copy at http://download.java.net/jdk7/binaries/ and checking if the problem persists. If the problem persists do let us know and we shall investigate this furthur.

We greatly appreciate your efforts in identifying areas in the Java Standard Edition where we can improve upon and I would request you to continue doing so.

Generic annoyances in java

patrick — Thu, 07 Jan 2010 01:14:31 +0000

For some reason, the type variable cannot flow through from an external caller through a callee to another called method.

This compiles just fine in eclipse, but once again sun’s javac is temperamental.

The three solutions seem to be:

pass an object of type to the subcallee method (m0(), p0() )
use wildcard instead of – which is o.k. if O is not needed in the body of the method
assign to a variable with no generic information (universal backup solution to java generic wierdness)

I filed a bug with Sun.

(Technical details: javac 1.6.0_17 )


public class Foo {
    public > I m(I i) {
/*line3*/        R r = (R) p(i);
        return i;
    }
    public > R p(I i) {
        return null;
    }

    public > I m0(O o) {
        K r= (K) p0(o);
        return o;
    }
    public > K p0(O o) {
        return null;
    }

    public > I m1(I i) {
        K r= (K) p1(i);
        return i;
    }
    public > K p1(I i) {
        return null;
    }

    public > I m2(I i) {
/*line 27*/        K r= (K) p2(i);
        return i;
    }
    public > K p2(I i) {
        return null;
    }
    interface FooInterface {

    }
}

Foo.java:3: incompatible types; inferred type argument(s) I,java.lang.Object do not conform to bounds of type variable(s) O,R found : R required: java.lang.Object /*line3*/ R r = (R) p(i); ^ Foo.java:27: incompatible types; inferred type argument(s) I,java.lang.Object do not conform to bounds of type variable(s) O,K found : K required: java.lang.Object K r= (K) p2(i); ^

Gmail and the Electrical Grid: Looks the same

patrick — Wed, 02 Sep 2009 18:03:51 +0000

GMail had a large-scale cascading failure yesterday:

At about 12:30 pm Pacific a few of the request routers became overloaded and in effect told the rest of the system “stop sending us traffic, we’re too slow!”. This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

This cascading failure sounds exactly like that other universal network we all share: The electrical grid.

Large blackouts are cascading failures compounded by the failure of the “fuses” meant to isolate still functioning parts of the grid from the failed part.

Sounds a lot like what happened to gmail.

Interestingly the smart people at Google have recognized exactly that:

we have concluded that request routers don’t have sufficient failure isolation (i.e. if there’s a problem in one datacenter, it shouldn’t affect servers in another datacenter) and do not degrade gracefully (e.g. if many request routers are overloaded simultaneously, they all should just get slower instead of refusing to accept traffic and shifting their load).

Sounds like a good start. But what happens if a datacenter catestrofically fails. The routers stop accepting requests because they are gone ( or can’t respond ). Prepare for the next GMail failure.

As wikipedia notes:

Modern power systems are designed to be resistant to this sort of cascading failure, but it may be unavoidable (see below). Moreover, since there is no short-term economic benefit to preventing rare large-scale failures, some observers have expressed concern that there is a tendency to erode the resilience of the network over time, which is only corrected after a major failure occurs. It has been claimed that reducing the likelihood of small outages only increases the likelihood of larger ones. In that case, the short-term economic benefit of keeping the individual customer happy increases the likelihood of large-scale blackouts.

Mac OSX — the most insecure OS around

patrick — Fri, 07 Aug 2009 08:35:12 +0000

A year ago, I gave up on Windows. I was tired of the nervous eggshell feeling with anti-virus software, security patches, and a machine that would mysteriously be slower and slower no matter what I did.

Because of MacOSX Unix roots, I made the switch to MacOSX confident that my personal computer would be safe.

This faith has been severely shaken. I now regard MacOSX as one of the most INSECURE operation systems.

MacOSX has a dangerous default DNS/DHCP configuration. Even worse, this dangerous configuration can not be fixed from the UI. Even the command line fix is difficult. And worst of all Apple is aware of this and does nothing.

There are 5 bits of background you should know:

DNS is a fundamental part of the internet. DNS is the ‘name resolution’ service that converts ‘mail.google.com’ into the ipaddress: ’74.125.19.19′ which is what your computer really uses to contact GMail servers so you can read your email. This conversion from the human readable ‘mail.google.com’ to ’74.125.19.19′ is analogous to the post office converting the postal address on your snail mail envelope to a Zip+4 encoding that is printed at the bottom of the envelope. This encoding is what is actually read by the postal service mail sorting machines to determine where your snail mail goes. Now imagine that the postal service’s encoding machine was compromised. This compromised postal encoding machine was changed so that no matter what the Zip+4 code was supposed to be the machine always encoded the location of Dick Cheney’s house. As a result, all your mail that was processed by that compromised postal service would go first to Dick Cheney. Dick Cheney would get a chance to open all your snail mail, read it, copy it, etc. He could then reseal the original envelope, reencode the envelope with the correct barcode and put the envelope back in the postal service system to be delivered to the correct address. so that no matter what address you had actually printed on your envelope. All this would happen without you being aware of the problem nor able to stop it from happening.
Everything internet related depends on correct translation of ‘apple.com’ or ‘bankofamerica.com’ to the correct IP address, not some third party server. How does the your machine know that it is communicating with apple.com and not some evil server? Your computer relies on the DNS lookup being correct. If the DNS lookup is compromised then when your software update runs to check for the latest security patches it is really installing a virus from evilserver.com not apple.com.

There has been recent concern about DNS spoofing. ( links ). Being the cautious person that I am, I decided to explicitly listed opendns.org‘s DNS servers (208.67.222.222 and 208.67.220.220) as the DNS Servers to trust in my Network configuration. I felt pretty cocky and safe.
The second bit of the puzzle is DHCP. In order to talk to the world, computers need to have their own personal unique ipaddress (it’s very own ZIP+4 code). Every time you go into an internet cafe and pop open your laptop, your laptop uses the DHCP service to figure out what unique ipaddress (192.168.1.101) it should use while you are in that cafe. DHCP is nice because otherwise you would have to manually figure out and set an ipaddress for your computer that is different than everyone else’s laptop. And if someone else picks the same address as you did, all of a sudden your internet connection starts behaving odd. In addition to supplying a ipaddress, the DHCP server also supplies a DNS server that should be used. This is useful when you have your laptop at work and you need to go to an internal website such as http://go/wave Notice there is no ‘.org’, or ‘.com’ after ‘http://go/’ this means that ‘go’ is only visible when you are at work and can access the internal DNS server using the information that the corporate DHCP server supplied to your laptop.
So to summarize DHCP supplies your laptop with the information needed for the world to talk to your laptop ( by assigning an ipaddress to your computer) , and helps you find out about the world (by telling your computer about the corporate DNS server). DNS servers enables your laptop talk to the world by giving your laptop a ‘go-to-machine’ for all its addressing questions.
Third, DNS servers are usually big expensive computers secured by “smart people”. However, the DHCP server is really just a bit of software running on a Linksys router at your internet cafe. Your internet cafe’s Linksys router probably has the default password and no one ever checks on it. There are millions of this routers, with minimal security, no one checking on them and your laptop is trusting these unsecured routers with the keys to your kingdom. Your laptop is asking this router — “tell me which DNS server to trust?”
Fourth, Lets say that you are paranoid enough to say ” ohh this is bad. I am not going to trust such a router in a greasy, dark corner with telling my precious laptop which DNS server to trust.” So if you are like me you configure your laptop with an explicit list of DNS servers thinking that your laptop, especially your oh-so-secure Mac would never disobey you about something so critical as DNS.
Fifth, you would be wrong. The insecure MacOSX does disobey and it does trust that greasy spoon router over you. MacOsX doesn’t let the user (YOU!) say that only certain machines are allowed to be your laptop’s DNS servers! Furthermore even if you have supplied your own custom DNS servers that you trust, the insecure MacOSX trusts the greasy spoon DHCP server’s DNS servers over your trusted DNS servers. And there is NO way to convince MacOSX otherwise.

This means that if the DHCP server at your internet cafe has been compromised you are as well.

How I found out

So here I am feeling all cocky and safe. I type in my company’s web address, ‘amplafi.net’ and amplafi.net resolved to 113.29.236.168 which offered that the website was for sale!… I freaked out!

I discovered my MacOSX laptop was insisting on trusting these EVIL DNS servers 206.13.28.12 206.13.31.12 . Was the 10.5.8 OSX patch that was installed 6 hours ago really what it seemed? Who knows? When I installed updates to Firefox plugins was I really installing the correct versions or a compromised version that would report back to some site in Russia all my bank account information? I have no way of knowing.

This is the really scary part about everything. John Simpson reports:

Under 10.4 and earlier, when I specified a custom nameserver, the system would use only the nameserver(s) I specified. However, under 10.5 Apple has apparently changed that behavior, and uses my specified nameservers in addition to the DNS servers specified by the DHCP server. It shows the DHCP-provided server IP on the list, greyed out, so you can’t delete it.

For a while, I adopted a “grin and bear it” attitude — after all, the DHCP server at home is handing out the IP of my internal Linux server (also running djbdns) as the DNS server, so I was only unsafe when I used the laptop outside the house. However, with the recently announced vulnerability in the DNS protocol, the massive world-wide patch effort by major DNS vendors, and the fact that many networks haven’t applied the patches yet, I don’t really feel safe relying on anybody else’s nameservers.

I tried calling Apple about this, but it turns out that my AppleCare contract doesn’t cover technical support such as this.

My next approach was to just brute-force search the system for anything relating to DHCP. It took a while, but I was able to find the file which needed to be changed, and figure out the necessary changes. Basically, I found a file which controls which options are used by the DHCP client when handling a response from a DHCP server. I removed the DNS-related options from this list, and after rebooting the system, the laptop now ignores the DNS server options being sent by the DHCP server.

The file I found is named IPConfiguration.xml, and it’s buried in this folder: /System/Library/SystemConfiguration/IPConfiguration.bundle/Contents/Resources. You need to create a copy of that file, edit the copy, and remove a few entries in the DHCPRequestedParameterList key. (The entries to remove are those for 6, 15, and 119.) I have added full details on this process to my djbdns setup page, in the section titled Disabling DNS servers from DHCP.

Thankfully I found this web page from 2008(!) that showed how to fix this problem (thanks John M. Simpson):

It is possible to make the DHCP client ignore the “DNS server” options in the DHCP response. It’s not for the faint of heart, but if you’ve been able to handle the rest of the instructions on this page, you can handle this bit as well.

I have done this on my own laptop (a MacBook Pro) and it does work.

Be aware that this is a GLOBAL change. If you do this, your machine will not use the DNS servers specified by any DHCP server. This may affect your machine’s ability to easily work with corporate networks (especially those using Windows Active Directory) or other networks which use private DNS namespaces.
$ sudo -s
Password: You will not see your password as you enter it.
# cd /System/Library/SystemConfiguration/IPConfiguration.bundle/Contents/Resources
# vi IPConfiguration.xml

Find this block...

        DHCPRequestedParameterList
        
                1
                3
                6
                15
                119
                95
                252
                44
                46
                47
        
Comment out the 6, 15, and 119 entries. The result should look like this:
        DHCPRequestedParameterList
        
                1
                3
                
                95
                252
                44
                46
                47
        
Save your changes.

Be sure to flush the DNS cache.

On MacOsX:

dscacheutil -flushcache

I have tried John’s suggestion and those scary DNS servers are no longer present. But has my machine been compromised already? I will be visiting the apple store in a few hours asking for answers.

The story continues. I was sitting next to a customer. On her windows box she was picking up the same bad DNS servers. It wasn’t until later when I got home that I discovered all this information. I suspect (but am not completely certain) that windows will not override an explicitly specified DNS server.

Update: So after talking with some people, its pretty clear that MacOsX shares this issue with Windows XP because offering out internal DNS servers is part of what DHCP does. However with Windows XP, it is easy to explicitly lock down the DNS servers.

How to lock down a Windows XP box with safe DNS servers:

On Windows:

To clear Windows DNS cache:

ipconfig /flushdns

Code Review #8: When to comment

patrick — Mon, 16 Mar 2009 18:10:10 +0000

The last ? in a series of posts about commenting. See “the why”. See “not commenting is career threatening”. And the comment that started this off!

This post should have really been the second one I wrote. The first post was about who a developer needs to keep happy – the client or the manager.

But commenting everything is both hard, excessive, and wasteful of time. In “the why” post, I cover what should be in the comments – implicitly I was limiting comments to just the places where knowing “the why” is important.

However, still the question is “when” should a developer comment?

Here is my rule of thumb:

Any developer had to spend time figuring out what the code did. Refactor it and if still not clear comment the code with questions and assumptions. (Just in case the refactoring introduced a bug or caused an existing bug to appear.)
Another developer asks the implementing developer (or the current “owner”/responsible developer) about the code, interfaces, package. The developer asking the question copy the explanation into the code, cleaning up the chat log/email response and adding further edits and explanation as needed.
The code is not going to have further development for a while. Comment the code enough so that when development is resumed, the restart is faster.
The code is being handed off from one developer to another.
Key interfaces that are used by multiple groups.

And managers give developers time to wrap up and add these comments.

Conversely, what is not worth commenting:

Code that is actively, daily, being changed rearchitected. Every developer already knows what the comments would say and the structure is changing fast enough that the comments would become outdated.
Code that does not impact data integrity and end-user experience in a horrid way.
Utility classes/ methods.

Why and when to wrap external library classes

patrick — Mon, 16 Mar 2009 17:47:09 +0000

At some point, every developer starts using an external library. They then have to decide if that external library should be wrapped in their own custom interfaces and classes.

If the external library would be pervasively imported in throughout the code then the answer is empathically YES — the external library should be wrapped.

At Amplafi, hibernate is just such a library. Without wrapping every time the hibernate library changed its structure or behavior, the change would have a large impact. However, by wrapping the libraries hibernate became loosely coupled.

This also enabled us to solve other more major issues with Hibernate Query and Session.

Specifically:

control when a transaction could be committed. We wanted to count how many times a tx was “started” and only commit when the tx was “ended” the same number of times it was started. Useful for code that doesn’t know if it needs to start a transaction. Now any code that needs a tx just “starts” one and ends it when done.
Performance metrics gathering.
Delaying starting the transaction until it is known that something will actually be done.
More gentle behavior for query.uniqueResult() ( default Hibernate behavior is to throw an exception if there is more than one row returned. Amplafi’s wrapper logs the non-uniqueness + ids of the extra rows. )

How we did this:

Create an interface (AmplafiQuery) that extends Query
Create a class (AmplafiQueryImpl) that extends AmplafiQuery and wraps a org.hibernate.Query
Create a Txmanager that returns a Tx.
Tx has the various createQuery methods and returns AmplafiQuery

We also took the opportunity to add to AmplafiQuery:

a “asList()” that is a generic enabled version of Query.list()
a “unique()” that is a generic enabled version of Query.uniqueResult()