Another RIAForge update, and some good examples of my mistakes

I had a moment yesterday to take another look at RIAForge and it's ongoing issues. This time I focused on a problem with the project, Project Tracker. For some reason this project would take around 60 seconds to render the home page. Other projects were much faster. As far as I could tell, nothing was special about that project, but everything about it was just horribly slow, almost unusable. I did some digging and finally found a slew of things that I had screwed up. Here they are in no particular order.

The number one issue was a controller method named getProjectView. This method is used on the home page for a project (and other views as well) and is the primary "get the project crap" method. It does multiple operations that the front end will need when rendering the page. One of the operations was a bit unique. It would use the SVN interface to get a list of files. It would get all the files for the entire repository. It would then use a query of query to see if at least one entry was a file, not a directory.

The whole point of this operation was to just see if any real files existed in the repo. If so, on the front end I'd offer a link to get the latest as a zip.

Now I knew this was a bit slow, so I used Model-Glue's caching system to cache the result for 10 minutes, and here is where part of the problem lie. My code basically did:

if in cache, return value
else
get all files
check for a file
set result in cache
end if

And lastly, everything was wrapped in try/catch. But get this. If an error was thrown, I didn't cache anything. So if the operation timed out (which could happen with a large repo), then I never cached the result. So every time you hit the project it would try again to get the result, and time out again.

Ugh. Dumb. I should have cached a negative result in the cfcatch at least. And I probably should have rethought the whole idea in general. Getting the entire repo must have made sense at the time, but maybe I was on crack.

So for now what I did was simply comment out the entire block, essentially removing the feature. I'm fine with that (and I hope you are too) as I'd rather wait till the new CF/SVN code is put online.

So what else did I do stupid? In caching, the trusted cache was not turned on, even though I thought it was. Just goes to show you - it doesn't hurt to recheck the Admin settings even when you are "sure" you set things right.

Another issue - I use Reactor, although in a minor way, and for some reason I pushed the code up with "development" mode for Reactor instead of production.

There is nothing more humbling then finding a whole slew of issues like this in your own code. Hopefully this blog post will help others avoid the same mistake!

Comments

That's ballsy to point out your mistakes to everyone. :)

Thanks for doin' it though. It makes us all better programmers.

~Brad
# Posted By Brad Wood | 7/10/08 10:53 AM
Ray writes :"Another issue - I use Reactor, although in a minor way, and for some reason I pushed the code up with "development" mode for Reactor instead of production."

After a few mistakes like this I stick to the following: In SVN, in the trunk, I keep all settings equal to the production environment. All development is done in seperate branches. When development is done and I merge the branch with the trunk, I go through all changes that I've made one by one (subclipse is great for this). Often, I stumble upon a setting that I've changed in early development that I've forgotten about. This helps me fix it before the new release goes to production.

Hope this is an understandable comment. I could do a blog post where I explain this, with screenshots. Anyone interested?
# Posted By Martijn van der Woud | 7/11/08 2:33 AM
I think a blog entry would be great. When it comes to SVN, I'm still not yet comfortable with branches.
# Posted By Raymond Camden | 7/11/08 6:36 AM
Is it just me or does this page still take forever sometimes:

http://projecttracker.riaforge.org/index.cfm?event...

Many times it didn't load beyond the header or it loaded but then loaded the issues after another very healthy wait. (Why not just output a query rather than wait and load via js? Is that needed just to use the js filter?)

Few suggestions:

I noticed these two images are very heavy for what they are, especially the 2nd which should weigh nothing:

http://projecttracker.riaforge.org/images/hdr_logo...   23 KB
http://projecttracker.riaforge.org/images/hdr_shad...   21 KB

These two spry files are a little big too:

http://projecttracker.riaforge.org/js/SpryData.js   82 KB
http://projecttracker.riaforge.org/js/xpath.js   43 KB

Might consider jquery; doesn't it have your needed xpath/loading built in, and still weigh much less? (I have jquery/livequery/menu/custom code combined at 61kb, minified, not packed.) Or just add the two files together to speed loading time, and put it at the bottom of the page.

Most importantly, I don't see that anything at all is gzipped: pages, js, etc. Adding that would surely help a lot. Basically, any non-image will come in around 1/3 the normal weight. You can get that page down from "231 KB uncompressed" to about 54kb plus the images, then get those 2 noted above optimized and be fairly light.

Hope it helps speed it up a bit.

----------------

Btw, I still find any issues page with more than 1 comment almost unscannable with no differentiation via borders/space/text size to separate comments clearly, then duplicated info crowding it all, like the "Created by" and "name":

------------------
Project Tracker Issue: When Viewing a file source in the SVN browser
Name:    When Viewing a file source in the SVN browser

Creator:    David
Created:    07/27/08 1:45 AM
History:    Created by nzmehere (David) : 07/27/08 1:45 AM
------------------

I sent in code to clean up those pages a bit many months ago but really anything just to differentiate the comments would help.

Cheers.
# Posted By ziggy | 7/29/08 12:14 AM
The biggest issue with - well the issues - is that it uses XML still and not JSON. Changing to that would give a huge reduction in file size. I can also switch to the minified JS files for Spry as well. It's just a matter of finding the time.

And yep - I know you want to improve that page. I'm the barrier here, but there isn't much I can do right now.
# Posted By Raymond Camden | 7/29/08 6:32 AM
Ziggy:

Switched to packed JS files.
Switched to JSON.

I see significant size decreases now, especially for projecttracker.
# Posted By Raymond Camden | 7/29/08 11:15 AM
Some reason you don't gzip?
# Posted By ziggy | 8/1/08 1:53 AM
I didn't use the zipped ones because I assumed I needed to set something up in Apache first. I wasn't sure, so I just went with that solution.
# Posted By Raymond Camden | 8/1/08 6:31 AM
No, you just turn gzipping on in httpd.conf for Apache:

LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so

# deflate if available for the whole server
<Location />
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/x-js application/x-javascript application/xhtml+xml text/css
<IfModule mod_headers.c>
Header append Vary User-Agent
</IfModule>
</IfModule>
</Location>

Also, I noticed you don't have any expiry caching set on images either. See
http://www.websiteoptimization.com/speed/tweak/cac...

This is mine:

# speed optimization --------------------

FileETag none

<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "now"
ExpiresByType image/jpeg "access plus 2 month"
ExpiresByType image/gif "access plus 2 month"
ExpiresByType image/png "access plus 2 month"
ExpiresByType image/x-icon "access plus 2 month"
ExpiresByType application/x-shockwave-flash "access plus 2 month"
ExpiresByType text/css "access plus 2 month"
ExpiresByType text/javascript "access plus 2 month"
ExpiresByType application/x-javascript "access plus 2 month"
ExpiresByType application/xml "access plus 6 hours"
ExpiresByType text/xml "access plus 6 hours"
</IfModule>

The latter is for feeds.
# Posted By ziggy | 8/3/08 5:53 AM
Interesting. I did all of this, but I didn't yet switch to the gzipped JS files. However, in YSlow I went from a D to a B, so things seem to be a bit better already. I'll switch to the gzipped files on Monday.

I'm also going to see if I can do this to cfbloggers.org as well. That site should be zippier I think.
# Posted By Raymond Camden | 8/3/08 2:33 PM
>>I didn't use the zipped ones because I assumed I needed to set something up in Apache first.
>>I didn't yet switch to the gzipped JS files

Just to be perfectly clear in case anyone is reading this and wondering, you do NOT use zipped files after turning on gzipping. Just continue to use regular files.

I see gzipping is on for js/css now but you haven't set it on the cfm files yet.

But definitely much faster already!

Was "231 KB uncompressed"
Now "112 KB (153 KB uncompressed)"

So down over half already and then caching on top of that!

If you added cfm gzipping and had someone optimize those 2 big images, that's about as good as that aspect gets.

Thanks! Awesome!

--------------------------------

Documents (1 file)   13 KB
   http://projecttracker.riaforge.org/index.cfm?event...   13 KB
   Images (17 files)   68 KB
   http://projecttracker.riaforge.org/images/hdr_logo...   23 KB
   http://projecttracker.riaforge.org/images/hdr_shad...   21 KB
   http://projecttracker.riaforge.org/images/ftr01.gi...   3 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/navicons...   2 KB
   http://projecttracker.riaforge.org/images/divi03.g...   55 bytes
   http://projecttracker.riaforge.org/images/divi02.g...   54 bytes
   http://projecttracker.riaforge.org/images/spacer.g...   49 bytes
   Objects (0 files)   
   Scripts (3 files)   30 KB (70 KB uncompressed)
   http://projecttracker.riaforge.org/js/SpryData.js   18 KB (41 KB uncompressed)
   http://www.google-analytics.com/ga.js   9 KB (22 KB uncompressed)
   http://projecttracker.riaforge.org/js/SpryJSONData...   3 KB (6 KB uncompressed)
   Style Sheets (1 file)   410 bytes (1 KB uncompressed)
   http://www.riaforge.org/css/styles.css   410 bytes (1 KB uncompressed)
   Total   112 KB (153 KB uncompressed)
# Posted By ziggy | 8/4/08 10:39 PM
>>you do NOT use zipped files

Of course I should say you can pre-zip and set all that up, but does anyone bother? Creates a whole extra layer to worry about. Unless you're getting hammered on the cpu it is a waste of time.
# Posted By ziggy | 8/4/08 10:53 PM
@ziggy - I was having a brain fart. For some reason, I thought Spry included files that were zipped up and you would use it with gzip on the server.

Now - as for CFM zipping - I didn't do that on purpose. I was worried it might impact CFM pages that use cfcontent to push out non-CFM pages. Does it? If not, I'll add CFM as well.

FYI, I've done this to my server here as well, so all my sites (this blog, cflib, cfbloggers.org, etc) should be a bit zippier.

Thank you for your help!
# Posted By Raymond Camden | 8/5/08 6:41 AM
>>I was worried it might impact CFM pages that use cfcontent to push out non-CFM pages.

Well, it doesn't matter where it is launched from, it is still going through Apache, right? And it will gzip whatever you want or don't. For example, your json ajax request will be coming through gzipped because you have that set on js files. If you mean a cfm page launches an html page, you should gzip the html too (just don't setup to cache html if it is actually dynamic like cfm).

If you mean some download that is opened in another application, like an Excel sheet, it won't accept the gzip handshake on the Excel file (if you've even setup to gzip Excel) and thus be sent normally.

I don't see a problem there, but never actually tried the latter. I'd just turn it on and try your specific case right away. Only takes 1 minute. (You can check response headers with the firefox webdeveloper addon, or liveheaders.)

Setting up gzipping and expiry headers is awesome. It's like finding a bag of money on the road :-)
# Posted By ziggy | 8/6/08 12:36 AM