Archive

Archive for February, 2009

Stubbing a Test Mail Server

February 17, 2009 1 comment

Django has built in support for sending email. We make use of this in our app, but when testing, we wanted to be able to access the emails sent so we could assert on their content, and pull data out of the body. In the unit tests, that’s easily solved by mocking the email client call, but we wanted to do this as a black-box regression test. That’s where the Python built in smtpd.SMTPServer comes in handy:

import smtpd, asyncore, threading

email_server = None

class FakeServer(smtpd.SMTPServer):

def __init__(self, localaddr, remoteaddr):
self.server = smtpd.SMTPServer.__init__(self, localaddr, remoteaddr)
self.emails = {}

def process_message(self, peer, mailfrom, rcpttos, data):
for recipient in rcpttos:
idx = recipient.replace(‘@’,’_at_’).replace(‘.’,’_dot_’)
existing_emails = emails.get(idx, [])
existing_emails.append(data)
emails[idx] = existing_emails

def stop(self):
self.server.close()

def main()
email_server = FakeServer((‘localhost’, 10920), None)
def start_email():
asyncore.loop()
thread = threading.Thread(target=start_email)
thread.setDaemon(True)
thread.start()

if __name__==’__main__’:
main()

This starts an email server on port 10910, which on receipt of a new email, stores it in a dictionary keyed by a modified version of the email address. Multiple emails to the same address get appended to the list. There is no reason in this context for replacing the ‘.’ and ‘@’ symbols from the email address in this context, but it was appropriate for our usage.

We run the stub mailserver on a separate box, and make the emails available via a get request on a test HTTP server, which allows us to run tests scripted entirely written in Selenium. I will blog again with details of how we manage our test HTTP server. You can use the this code as the basis for embedding the mail server in a unittest setup too.

By: Chris Tarttelin

Defaulting To JDK 1.6 In NetBeans 6.5 On OSX

February 17, 2009 19 comments

We try to stay current with our technologies at Point2. An example of this is using the most recent Java platform, Java 6 (JDK 1.6) and modern IDEs such as NetBeans 6.5. However, staying up to date with technology is not always as simple as it could be. Java 6 was released in December 2006 and NetBeans 6.5 was released almost 2 years later in November 2008 and yet it still wants to use Java 5 (JDK 1.5) as the default Java Platform, regardless of what you have set in the OSX Java preferences. Here is how you can set NetBeans 6.5 to use JDK 1.6 by default.

  • Ensure that JDK 1.6 and NetBeans 6.5 are installed
  • Locate the NetBeans config file, typically located at “/Applications/NetBeans/NetBeans 6.5.app/Contents/Resources/NetBeans/etc/netbeans.conf” and open the file in your favourite editor
  • Set the “netbeans_jdkhome” parameter to the JDK 1.6 home directory (ie. netbeans_jdkhome=
    /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home)
  • In NetBeans, right click on the project in the project window and select “Properties”
  • Ensure the “Source/Binary Format” is set to “1.6”
  • Click on “Tools” on the toolbar and select “Java Platforms”
  • Under “J2SE” the default platform should now be “JDK 1.6”

By: Damien Gabrielson

Web Site Speed – Best Practices for Achieving Good YSlow Scores

February 3, 2009 Comments off

Make Fewer HTTP Requests

  • 40-60% of daily visitors come with an empty cache – the key to making these first-time visitors happy is less HTTP requests
    • CSS sprites
    • Combined files (combine all scripts into a single file, same for CSS)
    • Inline images (not yet supported in all major browsers)

Gzip Components

  • Reduces page weight and accelerates user experience
  • Worthwhile to zip:
    • HTML docs
    • Scripts
    • Stylesheets
    • XML docs
    • JSON docs
  • Not worthwhile zipping (are already compressed – waste of CPU time and may actually increase file size)
    • Images
    • PDF files

Put Stylesheets at the Top, Scripts at the bottom

  • Allows for page to load progressively (display content as soon as possible instead of waiting until all data has arrived to display everything at once)
  • Scripts block parallel downloads – browser won’t begin another downloads (even on different hostnames) while a script is downloading
  • Minimize use of “document.write” (or equivalents to this functionality) ontil the bottom of the page
    • Use the DEFER attribute where possible, which indicates that the script does not contain a “document.write” line and the browser can continue rendering (firefox doesn’t support this)

Avoid CSS Expressions

  • Although powerful, these are evaluated more often than one would expect – they are evaluated when:
    • page is rendered
    • page is resized
    • page is scrolled
    • user moves the mouse over the page
  • Yahoo’s developers found (using counters in these expressions) that mobing the mouse around the page can easily generate more than 10,000 evaluations

Make JavaScript and CSS External

  • external CSS and JavaScript files are cached; JavaScript and CSS that are inlined in the HTML documents get downloaded every time the HTML document is requested.

Reduce DNS Lookups

  • Reducing the number of unique hostnames has the potential to reduce the amount of parallel downloading that takes place in the pagehowever;

    reducing parallel downloads may increase the response times

  • A balance must be struck between the number of unique hostnames and the amount of parallel downloading potential of each page – Yahoo recommends to split these components across at least 2, but no more than 4 hostnames; this generally results in a good compromise between reducing DNS lookups and allowing a high degree of parallel downloads.

Minify JavaScript and CSS

  • Minifying code is simply removing unnecessary whitespace and commenting to reduce file size.
  • 3rd party automated minifying tools exist such as JSMin

Make favicon.ico Small and Cacheable

  • “favicon.ico” – an image that stays on the root of your server
  • Even if you don’t care about it, the browser will still request it
  • it is better to not respond to this request with a 404 Not Found, and cookies are sent every time its requested (which will be every page if you are responding with 404 Not Found
  • To mitigate these problems:
    • Make sure it is small (< 1K)
    • If you don’t care about setting this to anything, give it a long expiry time

Minimize the Number of iframes

  • Pros of using iframes:
    • Helps with slow third-party content (ads, external pictures, etc.)
    • Can be used as a security sandbox
    • Allows for scripts to be downloaded in parallel
  • Cons of using iframes:
    • Costly, even if left blank
    • they block page onload
    • Non-semantic

Appropriately Preload/Post-load Components

  • Preload only what is absolutely necessary for the page to render
  • JavaScript libraries that do things like drag and drop animations can to be post-loaded

Optimize use of Ajax

  • Make Ajax Cacheable
  • Use GET for AJAX Requests
  • All rules and guidelines also apply to Ajax
    • Gzip Components
    • Reduce DNS Lookups
    • Minify JavaScript
    • Avoid 301s, 302s, and 404s
    • etc.

Other optimizations:

  • Avoid 301s, 302s, and 404s whenever possible
  • Don’t Scale Images in HTML (costs time to scale images – make the picture files themselves appropriately sized)
  • Keep Components under 25K
  • Remove Duplicate Scripts
  • Investigate whether or not ETags are a good idea for your website
  • Flush the Buffer Early
    • Ideally right between the end of the head tag and before the body tag
    • Yahoo pioneered research and real user testing to prove the benefits of using this technique (More Info)
  • Use a Content Delivery Network
    • Yahoo moved their static content from their web servers to a 3rd party CDN company and saw a 20% increase in user response times.
  • Add an Expires or a Cache-Control Header
    • Refers to an equivalent to our “Strong” and “Weak” cache policies, which we already implement heavily
  • Reduce DOM access and the number of DOM elements
    • Easy way to get the count of DOM objects on a page: in Firebug’s console, typedocument.getElementsByTagName('*').length
  • Reduce Cookie Size
  • Choose <link> over @import (Allows for progressive rendering)
  • Avoid Filters (such as the IE-proprietary AlphaImageLoader
    • blocks rendering and freezes browser while the image is being downloaded
    • Applied per element, rather than per image, which only multiplies the above-mentioned problem
  • Develop Smart Event Handlers
  • Split Components Across Domains where possible
  • Pack Components into a Multipart Document – allows you to fetch several components with one HTTP request (iPhone does not support this)

By: Brett McClelland

Creating Fixtures from Within Tests

February 3, 2009 Comments off

Django gives you a ‘dumpdata’ target which will create a fixture from all the records in your schema. For what we wanted, this was overkill. We had an existing unit test which was creating data in one test, and it was just the right amount of data for what we wanted. After searching through the Django codebase, it became clear that we could pass the objects we had in that test straight to the JSON serializer, and write the output to file. This ended up looking something like this:-

from django.core.serializers import json
serializer = json.Serializer()
objectsToSerialize =
MyModel.objects.filter(column='restriction')
with open('my_fixture.json','w') as f:
. f.write(serializer.serialize(objectsToSerialize, indent=4))

And that’s it! You can also spin up a shell, using:-

python manage.py shell

and load the data you want to create fixtures from if you don’t have any tests that create the data you want already.

By: Chris Tarttelin

Web Application Scalability

February 2, 2009 Comments off

For most small applications, scalability is usually not something that recieves much consideration. For applications that have potential to grow to tens of thousands of users and up, however; scalability may eventually become a concern.

Many web application success stories owe their scalability to the fact that they are written in Python/Django. It is extremely light-weight, as web application architectures go, and allows for much flexibility. Some of the things that we’ve kept in mind while coding that help with scaling are:

• Minimize external dependencies
• Replace/refactor/migrate components and modules as they become problematic (Python components help to streamline this process)
Emphasize ‘low coupling’ of code bases
• Attempt to localize and modularize failures (try and prevent them from spilling into other modules/applications)
• Limit the number of queries within loops (object.get(), object.filter(), etc.)
• It is much more efficient to fetch all records necessary in one query, and work with the retrieved dataset within a loop, rather than querying once per loop.
• Get rid of all obviously unnecessary leaf services.
• Customize reliable open-source software – bend it to your will.
• ‘psyco’ compiler – specialized Python compiler – extremely optimized
• Processor-heavy functions, or highly-executed functions, can be ‘psyco-ized’
• This compiler is something we have not yet tried using, however some development companies report as much as a 400% performance boost by using it properly.

A few rabbit-holes that developers should avoid running down if at all possible:

• Always be aware of the difference between “fast” and “fast enough”
• Python/Django has many scalability optimizations built right in; implement your functionality and test it under appropriate load-testing environments first, before spending too much time optimizing manually.
• Strive for hardware efficiency, but do not obsess over it
• Do not make the assumption that a technique for code optimization for one language will work for the language you are working in.
• Keep in mind that eventually you will have ‘no cards left to play’.

By: Brett McClelland