Itellity Blog

Yesterday, we had the privilege of having to create Hive tables on top of a HBase table with avro columns. Piece of cake, we thought, since you just have to specify the schema location in Hive (since Hive 1.0, it’s actually possible to map avro schema in HBase columns to Hive columns), map it to the column in HBase and voila, it works.

Except that it just doesn’t, at least if you have a large schema definition. In our specific case, we have a XSD Schema at the root – and like all things XML, it’s quite substantial. Converted to something decent like Avro, it is still quite big. That’s when we discovered that Hive has a limit for its Serializer of 4000 chars. That limit makes sense when you consider that Hive stores its metadata in a SQL database. So after googling a bit, we found this post:

http://gbif.blogspot.de/2014/03/lots-of-columns-with-hive-and-hbase.html

which basically pointed to the SERDE_PARAMS table. And unsurprisingly, the limit there is VARCHAR(4000).

Following that advice, we updated the schema of the database using this:

alter table SERDE_PARAMS MODIFY PARAM_VALUE TEXT;

That as such isn’t enough though, because Hive was still consistently failing. Since finding out the second cause was a slightly painful experience involving remote debugging Hive Server (thanks again, Roland), without further ado, here’s the second update command:

alter table SD_PARAMS MODIFY PARAM_VALUE TEXT;

That’s also necessary, because after deserializing, Hive stores the extracted column in StorageDefinition (hence the SD in the table name), and that has the same limit.

If you ever encounter this error while setting up Flume with HDFS Sink:

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

Just add the following JAR to your classpath in flume-env.sh:

HADOOP_HOME=your path 

FLUME_CLASSPATH="$HADOOP_HOME/share/hadoop/hdfs//hadoop-hdfs-2.5.0-cdh5.3.1.jar"

I’ve been digging around recently to see how to store avro records in HBase without “exploding” values to single columns; this being a viable alternative since Hive 0.14 with it’s support for Avro queries in HBase columns.

What you basically need to do is to serialize your avro record to a byte array and put that in your column:

ByteArrayOutputStream baos = new ByteArrayOutputStream();
DatumWriter writer = new SpecificDatumWriter(SpecifcRecord.SCHEMA$);
DataFileWriter dfw = new DataFileWriter(writer);
PutRequest req = null;
try {
    dfw.create(SpecifcRecord.SCHEMA$, baos);
    dfw.append(currentEvent);
    dfw.close();
    req = new PutRequest(table, currentRowKey, colFam,
        "record".getBytes(), baos.toByteArray());
} catch (IOException e) {
    e.printStackTrace();
}

That’s basically it.

Look no further, ditch Facebook.

In terms of battery usage, it’s the worst hog ever:

Data usage is quite massive too:

Once you terminate the app and switch to using safari for facebook, you’ll notice the difference.

Before we get started – just a quick note: this will only work for as long as your hob haven’t been submitted to a cluster or as long as your jobs run locally.

This is basically just the right thing to do if you want to debug configuration parameters or other set up relevant processes. I used this to debug a CLI call to a scalding job, for instance.

1) The first thing you need to do is to add the remote debugging facility to hadoop:

export HADOOP_OPTS="$HADOOP_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8999"

This need to be added to your conf/hadoop-env.sh or exported to your env variables.

2) Now eclipse:

Choose Run -> DebugConfigurations -> Remote Java Application

and add port 8999 to your connection settings. That’s what it should look like:

This post is basically the twin of an earlier post, which describes the same process for maven.

I had the questionable pleasure of having to convert my existing maven project to Gradle, which is basically almost as bad a maven and a lot slower, but hell, which build tool is perfect anyway?

So with not much further ado, here’s the basic structure:

apply plugin: 'scala'
apply plugin: 'eclipse'
sourceCompatibility = 1.7
version = '1.0'
configurations {
provided
}
configurations.all {
resolutionStrategy {
            force(
            'org.scala-lang:scala-library:2.10.4'
           )
    }
}
jar {
    from(configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }) {
        exclude "META-INF/*.SF"
        exclude "META-INF/*.DSA"
        exclude "META-INF/*.RSA"
        exclude "META-INF/license/*"
    }
    manifest {
    attributes( 'Implementation-Version': version,
                'Built-By': System.getProperty('user.name'),
                'Built-Date': new Date(),
                'Built-JDK': System.getProperty('java.version'))
    }
}
repositories {
    mavenCentral()
    maven { url "http://conjars.org/repo/" }
    maven { url "http://repo.typesafe.com/typesafe/releases/" }
}
dependencies {
    compile group: 'commons-collections', name: 'commons-collections', version: '3.2'
    compile "org.apache.flume.flume-ng-sinks:flume-ng-elasticsearch-sink:1.5.0.1"
    testCompile group: 'junit', name: 'junit', version: '4.+'
    
}
test {
    systemProperties 'property': 'value'
}

I must confess that this is actually a great deal shorter than the maven equivalent, mainly due to the simplicity of the scala plugin implementation, which also takes care of compiling java classes as well.

The project structure as such follows the basic pattern of maven projects:

src/main/scala
src/main/java
src/main/resources

and pretty much the same for test.