Skip to content

Prolonging An SSD’s Life By Mitigating Writes

Warning: The following details a process that is no longer supported. I do not yet have a replacement. (2012 November 13)

Last weekend, I updated my HTPC with all new hard drives and software. I had been putting it off because of an incompatibility between my IR receiver and the latest kernel drivers. But a couple weeks ago, I learned about the flirc device and immediately ordered one. Flirc doesn’t require any complicated setup or special drivers to work. Therefore, I was free to perform my long overdue HTPC upgrade.

Anyway, the core of my upgrade is a 64GB Solid State Disk (SSD) for the operating system. Since SSDs have a fixed lifespan (baring disaster), I want to limit the number of writes done to the disk. Given that my HTPC has 4GB of RAM, that is mostly unused (even when watching HD content), I am able to limit writes to my SSD by using RAM based disks. In particular, I have offloaded my /tmp and /var/log directories to RAM.

Putting /tmp in RAM is pretty much a no-brainer. The data doesn’t need to live across system boots so you can get away with using a tmpfs mount definition like so:

tmpfs /tmp tmpfs nodev,nosuid,size=1G 0 0

This definition limits my /tmp directory to 1GB in size and allows any user to read and write it.

The /var/log directory, however, is a bit trickier:

  1. General users should not be able to write to it.
  2. The data needs to be retained across system boots.
  3. It should be mounted beore anything writes to it.
  4. It should be unmounted at the last possible moment.

So, to solve problem #1, we can opt to use a ramfs mount (you can read about the difference between tmpfs and ramfs in the kernel documentation). Problems #2 — #4 are where things get complicated. That is, unless you are using Arch Linux.

Arch Linux doesn’t use the SysV init system. Arch has its own init system that is similar to BSD init systems. This init system includes “hooks” at key stages of the boot, and shutdown, process that can be used to solve problems 2, 3, and 4. You can read more about the hooks on the Arch Linux wiki. I’m merely going to explain my solution.

First, we need a place to store the data between boots. As another measure to prolong the life of my SSD, I opted to allocate a dedicated portion of my disk for the data store. I did this by creating a loopback image:

$ dd if=/dev/zero bs=1M count=512 of=/var/opt/var_log.img
$ chmod og-rw /var/opt/var_log.img # to prevent general users from snooping on it
$ mkfs.jfs /var/opt/var_log.img # fast recovery and restore

Having done this, I will always use the same blocks on my SSD for storing my /var/log data. Since they will only ever be written to at shutdown, these blocks will not be written to very often. And since the blocks are pre-allocated to this image, no other process will ever write to these blocks.

Next, I configured my mount definition:

none /var/log ramfs defaults,noatime,noauto 0 0

The key bit in this definition is the “noauto” option. This will prevent the init scripts from attempting to mount /var/log before I do.

Finally, I wrote two hooks to manage this ramdisk:

# mount /var/log to a ramdisk before starting any processes that use it

# Set to a disk image for storing the data when offline
IMG="/var/opt/var_log.img"

# A temporary mount point for saving and restoring (no trailing slash)
TMP_MOUNT="/var/tmp"

function restore_var {
  /bin/mount -o loop ${IMG} ${TMP_MOUNT}
  /bin/mount /var/log
  /usr/bin/rsync -a --delete ${TMP_MOUNT}/ /var/log/
  /bin/umount ${TMP_MOUNT}
}

add_hook sysinit_premount restore_var
# save /var/log to a loopback image before shutdown

# Set to a disk image for storing the data when offline
IMG="/var/opt/var_log.img"

# A temporary mount point for saving and restoring (no trailing slash)
TMP_MOUNT="/var/tmp"

function save_var {
  /bin/mount -o loop ${IMG} ${TMP_MOUNT}
  /usr/bin/rsync -a --delete /var/log/ ${TMP_MOUNT}
  /bin/umount ${TMP_MOUNT}
}

add_hook shutdown_preumount save_var

Technically, these hooks could reside in the same file. I opted to put them in separate files for clarity’s sake.

Important Note:

Before you implement this, you should understand something very important. By storing /var/log in RAM, you could potentially lose all information in /var/log since the last system boot. This could happen if your machine somehow enters a state (freezes) such that you cannot go through the shutdown process. Also, you could miss a few messages at either system boot or system shutdown (although, I think this solution catches all important ones).

For an HTPC, that is behind a firewall, I don’t see this as a problem. If I lose my logs, then I’m not really losing any important data. Sure, it’d be nice to have them to (hopefully) help diagnose the problem that cause the loss, but chances are it is some sort of failing hardware (e.g. the video card overheating). That, or an extended power loss.

If you really want to keep your logs. Then you should also send the messages to another syslog server on your network.

Addendum (10 May 2012):

Mentioned in a thread on the Arch Linux forums, there is a system daemon that does this. The daemon only uses tmpfs for RAM storage, and it syncs back to disk much more often, but it could make this much simpler if that’s what you need.

Returning A CLOB From A Java Stored Procedure

This took me way too long to figure out. All of my search results were coming back with questions about passing CLOBs or VARARRAYs to a Java stored procedure, and nothing about returning such datatypes. In the end, it turns out to be somewhat simple, but completely nonsensical.

First, know that you will need to use the OJDBC libraries. These should already be available in your Oracle installation if your database is configured to allow Java stored procedures. If not, you’ll have to talk to your DBA, because I don’t know anything about that. All I know is that the libraries are available in my instance.

So, let’s look at some example code:

package com.jrfom;

import java.io.*;
import java.sql.Connection;
import oracle.jdbc.driver.*;
import oracle.sql.CLOB;

public class ExampleProcedureClass {
    public static CLOB encodeFile(String file)
    throws java.io.FileNotFoundException,
      java.io.IOException,
      java.sql.SQLException
  {
    OracleDriver driver = new OracleDriver();
    Connection conn     = driver.defaultConnection();
    CLOB clob           = CLOB.createTemporary(conn, false, CLOB.DURATION_CALL);

    String sillyString = "This is a string that will be in the CLOB.";
    clob.setString(1, sillyString);

    return clob;
  }
}

What’s so “nonsensical” about this? The fact that you have to get a reference to the database connection. Without this connection reference, you cannot create a usable CLOB. There is the static CLOB.getEmptyClob(), but it “can not be read or written” (ref). The only other option is to construct and instance of the object, and every public constructor requires a Connection object. But, this is a stored procedure! Why should we have to do this? I have no idea. But the defaultConnection method of the OracleDriver object will get you a reference to the connection calling the procedure.

So there you go. Enjoy your ability to return CLOBs from your Java stored procedures.

A Simple Grails Plugin

Before I get into this, I want to make it clear that my Groovy knowledge is basic to moderate, at best. Also, my Grails knowledge is even less. In fact, things like Grails rub me the wrong way. But I haven’t seen any articles that clearly describe this process from a beginner’s perspective. So this one is for the people getting exasperated with the abysmal documentation and search results.

Without further ado, let’s write a simple Grails plugin that adds (and reads) its own configuration file, provides itself as a “bean” to the host Grails application, and is very simple to install and use. Our plugin is going to provide a simple object with some properties. We could provide a more complicated object, like a pooled database connection, using this same method. But for example’s sake, we’ll keep it simple. Here is the code for the object that will be provided by our plugin:

  package com.jrfom

  import grails.util.Environment

  class SpiffyObject {
    def foo
    def answer

    SpiffyObject() {
      def config = {}
      GroovyClassLoader classLoader = new GroovyClassLoader(getClass().getClassLoader())
      Environment env = Environment.getCurrent()
      String envName = env.getName()

      try {
        config = new ConfigSlurper(envName).parse(classLoader.loadClass('SpiffyObjectConfig'))
      } catch (Exception e) {
        // Do something good here
      }

      this.foo = config.foo
      this.answer = config.answer
    }
  }

So, let’s get started. First, we need to create our plugin and put our SpiffyObject in the correct location:

  $ cd ~/GrailsProjects/
  $ grails create-plugin spiffy-plugin
  $ cd spiffy-plugin
  $ mkdir -p src/groovy/com/jrfom
  $ touch src/groovy/com/jrfom/SpiffyObject.groovy # then open in your editor and paste in the code

Now we need to define our configuration file. This file will be added to the host Grails application’s grails-app/conf/ directory when our spiffy-plugin is installed. So:

  $ mkdir -p src/samples
  $ touch src/sample/SpiffyObjectConfig.groovy # and open in your editor

Now paste in this bit of code:

  environments {
    development {
      foo = "baz"
      answer = 66
    }

    test {
      foo = "bork"
      answer = 99
    }

    production {
      foo = "bar"
      answer = 42
    }
  }

Next we need to add some code to Grails’ special install script that gets run when a plugin is installed. So open scripts/_Install.groovy in your editor and add the following code to the end of the file:

  def configFile = new File('grails-app/conf/SpiffyObjectConfig.groovy')
  if (!configFile.exists()) {
    ant.copy(
      file: "${pluginBasedir}/src/samples/SpiffyObjectConfig.groovy",
      todir: "${basedir}/grails-app/conf")
  }

The above code will check to see if the config file already exists in the application’s config directory. If the config does not exist, it will copy our default configuration file (from src/samples) to the application’s config directory. Otherwise, it will leave the current configuration alone.

Finally, we need to add the code that will define our bean when the Grails application is started. So open SpiffyObjectGrailsPlugin.groovy (found in the plugin’s root directory) in your editor and adjust the doWithSpring method to look like:

  def doWithSpring = {
    // Define a bean for our plugin that can be used across the whole app
    spiffyObject(SpiffyObject) {}
  }

Additionally, you will need to add an import to the top of the file. So, before the “class SpiffyObjectGrailsPlugin {” line, add “import com.jrfom.SpiffyObject“.

You can now use the “
grails package-plugin

” command to package the plugin. Once you install the plugin into a Grails application, you can work with it like any other bean. For example, add the following code to grails-app/controllers/com/jrfom/SpiffyTestController.groovy:

  package com.jrfom

  import grails.converters.JSON

  class SpiffyTestController {
    def spiffyObject // indicate that we want to use the SpiffyObject bean

    def getAnswer = {
      def output = [:]

      output["answer"] = spiffyObject.answer

      render output as JSON
    }
}

Now run your Grails app and then navigate to this service. Assuming your Grails app is named “spiffy” and is running on port 8080 on your local machine, you’d navigate to
http://localhost:8080/spiffy/spiffyTest/getAnswer

. Doing so should return a JSON formatted string that reads `{"answer":42}`.

It’s definitely not intuitive, but that’s all there is to creating an easy to use, unobtrusive, plugin that provides an application wide bean. Of course, you can do a lot more than this simple, mostly static, plugin by following these steps. Also, don’t forget to modify the basic plugin properties in your SpiffyObjectGrailsPlugin.groovy file. These define the version number of your plugin, who wrote it, and its basic description. You should read over the official plugins documentation if none of this made any sense.

Oracle And Unix Timestamps Revisited

In my last post I provided some Oracle PL/SQL functions for dealing with Oracle DATEs and Unix timestamps. After using these functions in an application that is executed via Oracle Application Server I discovered a problem. In short, if the executing environment doesn’t supply any time zone information then the part of the functions that determines the local time zone would fail.

The following versions of the functions will work correctly no matter what time zone information is present in the executing environment:

create or replace
function unix_time
  (
    in_tz in varchar2 default 'America/New_York'
  )
  return integer
as
  ut      integer     := 0;
  tz      varchar2(8) := '';
  tz_date timestamp with time zone;
  tz_stmt varchar2(255);
begin
  /**
   * This function is for getting the current number of seconds since
   * 01 January 1970 in UTC, i.e. a Unix timestamp.
   *
   * @author James Sumners
   * @date 01 February 2012
   *
   * @param in_tz Specify the local time zone for converting to DST
   *
   * @return integer
   */
  
  -- Get the current timezone abbreviation (stupid DST)
  tz_stmt := 'select systimestamp at time zone ''' || in_tz || ''' from dual';
  execute immediate tz_stmt into tz_date;
  select
    extract(timezone_abbr from tz_date)
  into tz
  from dual;

  -- Get the Unix timestamp
  select
    (new_time(sysdate, tz, 'GMT') - to_date('01-JAN-1970', 'DD-MM-YYYY')) * (86400)
  into ut
  from dual;
  
  return ut;
end unix_time;
create or replace
function unix_time_to_date
  (
    unix_time   in integer,
    to_local_tz in integer default 0,
    local_tz    in varchar2 default 'America/New_York'
  )
  return date
as
  converted_date  date;
  tz              varchar2(8) := '';
  tz_date         timestamp with time zone;
  tz_stmt         varchar2(255);
begin
  /**
   * This function is used to convert a Unix timestamp (UTC) into an Oracle DATE. The converted DATE will
   * be in UTC unless the optional local_tz parameter is provided.
   *
   * @author James Sumners
   * @date 01 February 2012
   *
   * @param unix_time The Unix timestamp to convert
   * @param to_local_tz 0 = UTC, 1 = Local database time zone
   * @param local_tz The name of the local time zone
   *
   * @return date
   */
  select
    to_date('1970-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS') + numtodsinterval(unix_time, 'SECOND')
  into converted_date
  from dual;
  
  if to_local_tz <> 0 then
    -- Get the current timezone abbreviation (stupid DST)
    tz_stmt := 'select systimestamp at time zone ''' || local_tz || ''' from dual';
    execute immediate tz_stmt into tz_date;
    select
      extract(timezone_abbr from tz_date)
    into tz
    from dual;
    
    -- Convert the GMT time to the current local timezone
    converted_date := new_time(converted_date, 'GMT', tz);
  end if;
  
  return converted_date;
end unix_time_to_date;
create or replace
  function unix_time_from_date
      (
        in_date   in date,
        in_src_tz in varchar2 default 'America/New_York'
      )
    return integer
  as
    ut      integer       := 0;
    tz      varchar2(8)   := '';
    tz_date timestamp with time zone;
    tz_stmt varchar2(255);
  begin
    /**
     * This function is used to convert an Oracle DATE (local timezone) to a Unix timestamp (UTC).
     *
     * @author James Sumners
     * @date 01 February 2012
     *
     * @param in_date An Oracle DATE to convert. It is assumed that this date will be in the local timezone.
     * @param in_src_tz Indicates the time zone of the in_date parameter.
     *
     * @return integer
     */
  
    -- Get the current timezone abbreviation (stupid DST)
    tz_stmt := 'select systimestamp at time zone ''' || in_src_tz || ''' from dual';
    execute immediate tz_stmt into tz_date;
    select
      extract(timezone_abbr from tz_date)
    into tz
    from dual;
      
    -- Get the Unix timestamp
    select
      (new_time(in_date, tz, 'GMT') - to_date('01-JAN-1970', 'DD-MM-YYYY')) * (86400)
    into ut
    from dual;
  
    return ut;
end unix_time_from_date;

Oracle, Daylight Saving Time, And Unix Timestamps

Notice: the functions in this post are fragile. See Oracle And Unix Timestamps Revisited for a more robust implementation.

If ever there was a combination to make you hate developing web applications against an Oracle database, it’s the one defined in the title of this post. Let’s see:

  1. Unix timestamps are used in a lot of web applications because they’re easy to work with
  2. Oracle doesn’t have any native functions for working with Unix timestamps
  3. You have no way of knowing when a bored government will change the dates that DST starts and ends
  4. You shouldn’t include a timezone in a Unix timestamp

After much searching, and piecing together of different solutions to slightly different problems, I have written the following three functions that should cover all of your Unix timestamp handling needs within an Oracle database:

CREATE OR REPLACE
  FUNCTION unix_time
    RETURN INTEGER
  AS
    ut INTEGER     := 0;
    tz VARCHAR2(8) := '';
  BEGIN
    /**
     * This function is for getting the current number of seconds since
     * 01 January 1970 in UTC, i.e. a Unix timestamp.
     *
     * @author James Sumners
     * @date 01 February 2012
     *
     * @return integer
     */

    -- Get the current timezone abbreviation (stupid DST)
    SELECT
      extract(timezone_abbr FROM CURRENT_TIMESTAMP)
    INTO
      tz
    FROM
      dual;

    -- Get the Unix timestamp
    SELECT
      (new_time(sysdate, tz, 'GMT') - to_date('01-JAN-1970', 'DD-MM-YYYY')) * (
      86400)
    INTO
      ut
    FROM
      dual;

    RETURN ut;
END unix_time;
CREATE OR REPLACE
  FUNCTION unix_time_to_date(
      unix_time IN INTEGER,
      local_tz  IN INTEGER DEFAULT 0
    )
    RETURN DATE
  AS
    converted_date DATE;
    tz VARCHAR2(8) := '';
  BEGIN
    /**
     * This function is used to convert a Unix timestamp (UTC) into an Oracle DATE. The converted DATE will
     * be in UTC unless the optional local_tz parameter is provided.
     *
     * @author James Sumners
     * @date 01 February 2012
     *
     * @param unix_time The Unix timestamp to convert
     * @param local_tz 0 = UTC, 1 = Local database time zone
     *
     * @return date
     */
    SELECT
      to_date('1970-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS') + numtodsinterval(
      unix_time, 'SECOND')
    INTO
      converted_date
    FROM
      dual;

    IF local_tz <> 0 THEN
      -- Get the current timezone abbreviation (stupid DST)
      SELECT
        extract(timezone_abbr FROM CURRENT_TIMESTAMP)
      INTO
        tz
      FROM
        dual;
      -- Convert the GMT time to the current local timezone
      converted_date := new_time(converted_date, 'GMT', tz);
    END IF;

    RETURN converted_date;
END unix_time_to_date;
CREATE OR REPLACE
  FUNCTION unix_time_from_date(
      in_date IN DATE
    )
    RETURN INTEGER
  AS
    ut INTEGER     := 0;
    tz VARCHAR2(8) := '';
  BEGIN
    /**
     * This function is used to convert an Oracle DATE (local timezone) to a Unix timestamp (UTC).
     *
     * @author James Sumners
     * @date 01 February 2012
     *
     * @param in_date An Oracle DATE to convert. It is assumed that this date will be in the local timezone.
     *
     * @return integer
     */

    -- Get the local timezone from the passed in date
    SELECT
      extract(timezone_abbr FROM CAST(in_date AS TIMESTAMP
    WITH
      local TIME zone))
    INTO
      tz
    FROM
      dual;

    -- Get the Unix timestamp
    SELECT
      (new_time(in_date, tz, 'GMT') - to_date('01-JAN-1970', 'DD-MM-YYYY')) * (
      86400)
    INTO
      ut
    FROM
      dual;

    RETURN ut;
END unix_time_from_date;