Archive for the 'Programming' category

 | March 19, 2012 9:36 pm

XML Parsing with Python, Part 4In Part 2 of this tutorial, we looked at how libxml (through the use of the lxml Python bindings) can be used to parse XML documents. We covered how to load XML files from disk, parse them, look for specific tags and attributes, and generate custom datatypes from the results.

These are the operations that lxml is used most often for and are probably what most developers will need it for. It only scratches the surface of what it is capable of, however.

In Part 2, we talked about three common XML operations: parsing, validation, and transformation. Of these three, parsing is the most important. We touched upon validation and transformation, but didn’t look at how they work.

In this video, we’ll take a look at the last two operation. We’ll cover validation using DocType definitions and XML Schema and transformation, using XSLT style sheets.

Note: The example files used in the video can be downloaded from here.

Show me more… »

 | 1:59 pm

Parsing XML with Python, Part 2As we talked about in Part 1 of this series, XML is one of the most versatile structural languages, which has led it to be used in nearly everything every sort of application imaginable. It serves as the basis of file formats, a way to store financial records, and for communication between web servers. For this reason, programmers and designers should be familiar with not only how to write it, but also how to parse, validate, and transform it.

One of the most powerful tools for working with XML is libxml, an open source parser. Given its versatility, libxml is used in many open source projects, and even commercial products. It can be found at the base of the Gnome desktop and in banking software.

In this video, Part 2 of a series on working with XML from Python, we’ll look at how you can use libxml from Python through the lxml bindings. We’ll show how to load and parse an XML document, iterate through the elements and attributes, test for specific tags, and translate the document structure into a custom object.

Note: The example files used in the video can be downloaded from here.

Show me more… »

 | March 16, 2012 3:01 pm

Parsing XML with Python (Part 1, Installation)XML is one of the most versatile structural languages around. It is used extensively for file formats, storing miscellaneous data, and communicating between different programs. You can find it on the web, on the desktop, and places you might not expect, such as in interactive books.

Given its importance, most programming languages come with a parser that can be used to analyze and decode XML. Examples include MSXML, which is used in Microsoft’s .NET framework and libxml, a popular open source library.

In this series of videos, I’ll show you how to get up and running with libxml, through the lxml Python bindings. We’ll look at how to install the programs (Python, the Python package manager, setuptools, and lxml itself), read and parse an XML document from a file, inspect tags and attributes, validate XML structure, and perform XML transforms.

This first video will focus on the installation of Python, setuptools, and lxml.

Show me more… »

 | September 26, 2011 3:54 pm

The ability to create custom installation images (whether they be of servers, desktops, or more specialized devices) is a tremendously handy thing. In Windows Server 2008, Microsoft released a tool that makes the creation of these images much easier called ImageX.

ImageX works hand-in-hand with a second tool, called the Windows Preinstallation Environment (WinPE). In this series of videos, I will explain how you can use ImageX and Windows PE to create custom images and deploy them.

In this video, Part 2 of the series, we will cover how to create a custom WinPE image that includes ImageX and other useful tools. Part 1 talks about where to find and download WinPE/ImageX and how to install them.

The entire series is from a networking infrastructure course that I am currently teaching. I am posting it here in the hope that it will be helpful to a wider audience.

Show me more… »

 | September 22, 2011 9:50 pm

The ability to create custom installation images and deploy them to other machines is a very handy skill. If you are working in a large environment, you can create a master image with all of the tools needed for your work that could be used on all of your hardware. In this series of videos, I will explain how you can use ImageX and Windows PE to create custom images and deploy them.

Note: The following video is part 1 of a series that I have put together for a networking class that I am currently teaching. I have posted it here in the hope that it will be helpful to a slightly wider audience.

Show me more… »

 | September 21, 2011 7:57 pm

For the past year and a half, I’ve been using DreamHost, a company based out of California, to host this website. In most ways, it’s been a fantastic experience. The customer service is prompt and knowledgeable [1], they go out of their way to use renewable energy, and they give you all sorts of additional goodies when you sign up for an account [2].

But even though the experience has been mostly fantastic, I have one serious gripe: the actual hosting experience is terrible.

Let me see if I can explain why. With my previous host, Brinkster, it was rare to have a website outage. Things, for the most part, Just Worked ™. The website was up and people came to read it. That’s generally all you can ask from a hosting company. (To be clear, the hosting experience with Brinkster was great, everything else was middling to awful.)

With DreamHost, though, it’s another story. If you’ve come to this site lately (especially in the mornings or late evenings) it will be unresponsive. At other times, you might see a strange message in your browser, saying “Error 500: Internal Server Error.”

As the person paying for the hosting space, this is somewhat frightening.  HTTP errors (of which 500 is one) can be cryptic, and figuring out why your website (which was working five minutes ago and costing you money) is broken problem can require some serious time.

Wait, I've Almost Got It

500 Errors in General

The 500 error, in particular, is a true terror. It means that the web server encountered an unexpected condition and can’t return the web page to you. It’s a sort of catch-all error that gets generated when nothing more detailed is available. If you want additional information (something which might actually be helpful to fix the problem), you have to work your way through cryptic error logs and pray that some divinity is listening to your pleas. It’s not fun.

When hosting with Brinkster, I never had 500 errors. Not once. Occasionally there would be an outage, but these were repaired quickly and with little fuss. WIth Dreamhost, though, 500 errors happen all the time: at least daily, and sometimes hourly. They crop up when trying to do administrative tasks, when making posts, when trying to load the home page for the website, or when uploading files. The errors can be intermittent, or they might last for hours. It’s all very frustrating.

Lately, they’ve been so bad that I had essentially convinced myself to move away from DreamHost to another provider. Something I consider a true shame. It would mean giving up the goodies [3], probably paying more money, and losing shell access to the server. So, before I decided to move away forever, I decided to get in touch with DreamHost support (again) and work out the problems with my website.

Happily, after a protracted email message thread and a great deal of searching online, I’ve I’ve learned a few things about optimizing websites for DreamHost. I’ve also found a strategy (I think) for solving the dreaded “Internal server error” problem.

Show me more… »

 | September 20, 2011 8:46 pm

Note: Whenever I come across an interesting or potentially useful piece of source code, I like to make a copy and file it away. I post some of them here in the hope that they will be useful to others as well.

In my recent coding adventures, I needed to create a function to monitor the up-time of one of my Linux servers. A very simple thing (you would think), and yet, I wasn’t able to find a function in the standard library which would give me this piece of information. So … I decided to write my own.

The script below makes use of the “uptime” command (common in many Unix operating systems) to see how long the machine has been turned on. It works on both Linux and OS X computers. If I have time, I will add code for Windows as well.

FORMAT_SEC = 1
FORMAT_MIN = 2
FORMAT_HOUR = 3
FORMAT_DAY = 4

def uptime(format = FORMAT_SEC):
  """Determine how long the system has been running. Options:
  FORMAT_SEC returns uptime in seconds, FORMAT_MIN returns minutes,
  FORMAT_HOUR returns hours, FORMAT_DAY returns uptime in days."""

  def checkuptime():
    """OS specific ways of checking and returning
    the system uptime."""

    # For Linux and Mac, use "uptime" command and parse results
    if (sys.platform == 'linux2') | (sys.platform == 'darwin'):
      raw = subprocess.Popen('uptime',
        stdout=subprocess.PIPE).communicate()[0].replace(',','')
      up_days = int(raw.split()[2])

      if 'min' in raw:
        up_hours = 0
        up_min = int(raw[4])
      else:
        up_hours, up_min = map(int,raw.split()[4].split(':'))

      # Calculate the total seconds that the system has been working
      up_sec = up_days*24*60*60 + up_hours*60*60 + up_min*60
      return up_sec

  total_sec = checkuptime()

  # Get the days, hours, etc.
  if format == FORMAT_SEC: # Return seconds
    return total_sec
  if format == FORMAT_MIN: # Return minutes
    minutes = (total_sec)/(60)
    return minutes

  if format == FORMAT_HOUR: # Return hours
    hours = (total_sec)/(60*60)
    return hours

  if format == FORMAT_DAY: # Return days
    days = (total_sec)/(60*60*24)
    return days

  return -1 # If invalid option specified, return false.

 | 8:30 pm

Note: Whenever I come across an interesting or potentially useful piece of source code, I like to make a copy and file it away. I post some of them here in the hope that they will be useful to others as well.

For the past couple of days, I have been working on scripts to automate the administration of some of my computers. Because they’ve been running  big jobs that often take hours to complete, I’ve been leaving them unattended. But since I need to know as soon as the job is completed, it’s nice to receive some sort of alert. For that reason, I wrote a simple script that will send an email when the job is finished. It uses the string and smtplib modules from the standard library. Here’s the code:

import string, smtplib

# Email Settings
HOST = "smtp.gmail.com" # SMTP Server to Send Message From
PORT = 587 # Port you wish to communicate on
USER = "username@gmail.com" # Email Username
PASSWORD = "PASSWORD" # Email Password

FROM = "admin@oak-tree.us"
TO = "recipient@domain.com"

SUBJECT = "Server Notification from " + AMAZON_EC2
TEXT = "Server " + AMAZON_EC2 + " is shutting down.\n" + 'Timestamp: ' +
    str(CURRENT_TIME)

def sendmessage():
  """Send email notification to the specified email address."""
  server = smtplib.SMTP(HOST, PORT)
  server.ehlo()
  server.starttls() # Use Encryption
  server.ehlo
  server.login(USER, PASSWORD)
  BODY = string.join((
    "From: %s" % FROM,
    "To: %s" % TO,
    "Subject: %s" % SUBJECT ,
    "",
    TEXT
    ), "\r\n")
  server.sendmail(FROM, TO, BODY)
  server.quit()

In general, it’s pretty straightforward. You specify what host you want to connect to and the port, then you construct your message. Finally, you send it. In this example, I’ve added a few other things, like a line to log-on to the server and encryption.The actual connection and sending of the email is only two lines of code. The rest is setting up the message.

Pretty cool, huh?

 | June 20, 2011 11:13 pm

This video is the second in a series from a course (introductory and intensive) on web technologies that I am currently teaching. It shows how to validate an XML document using XML Schema. (The first entry in the series covers DocType definitions, DTDs.)

XML schema, like Document Type Definitions, are used to validate the structure of XML documents and web pages. XML Schema, though, is a much richer way to verify structure and data.

Like it’s older sibling, I am posting the video in the hope it will be useful to both students in addition to others that have lost themselves in the ether of XML Schema syntax.

Show me more… »

 | June 15, 2011 6:28 pm

The following video is from an intensive (albeit introductory) course that I’ve been teaching on web technologies — XML, XHTML and CSS. It discusses how to validate the structure of an XML file using DocType definitions (DTDs).

DocType definitions are sets of instructions that tell a document parser whether a particular document is well formed or not. This allows for the parser to recognize potential errors.

I’m posting the video here in the hope that it will be helpful to a larger audience than just those in the course.  After all, DocTypes are an extremely useful tool.

Show me more… »