Lessons learned
With the past two years, there’s been an incredible development in my experience and knowledge as a software developer. Still, it’s important to look back at ones mistakes and learn from them.
I still have the complete source code and database of my very first large scale program. The year was 2007 (I think) when I first discovered the Dragon Knight open source MMORPG project. In no time at all I had the system up and running in localhost and the next summer (or two) went into learning web technologies, programming and CS in general.
Now, as part of a software testing course assignment, I had the incentive to look back and write down some notes.
- Don’t use PHP closing tags in php-only files
- If you have a single entry point (framework index.php), do a check in the first line of every file that the entry point is used. Kohana does that with a constant defined check that dies on failure.
- DRY: put config into one place
- Use InnoDB relational tables
- Don’t mix Models, Views, Controllers in a single file – use arhitecture. It’s also DRY!
- I18n mostly everything. You don’t have to actually translate it, just make it translatable in the future. In Kohana: use the __() function. For javascript, find a library.
- Keep font sizes and color contrasts readable
- Always let the user visually distinguish where (s)he is in terms of navigation. Breadcrumbs are a good idea. Back links should be present in multi-staged navigation. Logo should link to the index page. The user shouldn’t feel lost.
- Main navigation should be visible on all pages (there are justified exceptions)
- Be consistent: an UI element that act the same should always look so, too. Don’t make one link blue, the other green, if they behave the same. DO make a link red if it’s dangerous, but not simply for fun.
- Avoid opening multiple popup windows, use modal dialogs, tooltips, javascript plugins to present the information. When you do open a popup, make sure it’s sized so that the user doesn’t have to resize it to see the whole content.
- Don’t hardcode system messages via a query string (…&messageID=1) (nor ?message=Please+Try+Again)
- Test against security vulnerabilities! Malicious input (injection, XSS) in particular.
Opinions: Be a Developer… Part III
Ando Roots
Just a picking note to begin with: WordPress is more of a CMS than a pure framework…but never mind that. I wholeheartedly agree with you on the point of Kohana. It’s the fastest and most comfortable to work with PHP framework I’ve used, to date.
Indeed – before writing your first class or function, a developer should know the system he’s building. That’s a fact I’ve come to realize since I started working as a programmer. I used to begin with an idea and then just see where it goes, often tearing down an hours worth of progress. College and the real world taught me to do some initial planning (hey, at least I’m trying), but it’s still difficult to foresee the possible problems in the future. I guess it comes with experience.
Much of the initial planning is usually done by the project manager: collecting client requirements, making technical decisions (if qualified) etc. Still, modelling the database, the system and dreaming up the architecture is the job of a developer or an architect.
I still strongly object to the idea of labelling writing code as a mechanical process. The job of a System Architect is undeniably creative, but so is writing code or even mere functions with fixed signatures. There are many similarities with Mathematics, but even in Math one has to have certain creativity to see the possible paths to the solution.
When speaking of a typical end user (that’s another topic entirely), it’s true that he or she doesn’t really care about how the functions are written nor whether the product is analog or digital… as long as it fills it’s intended role. However, it’s wrong to say that the customer doesn’t benefit from a well-written codebase. Notice, I’m using the phrase well-written instead of beautiful. A painting is beautiful, the code can look like that with proper spacing and indention (and it should), but the characters and comments are what count. A documented, structurally solid and thought trough codebase makes later development and/or maintenance cost effective and ultimately, saves the customer money. If the company has a contract that says they’ll fix any and all bugs for free and the codebase is unmaintainable or undocumented, the cost of maintenance skyrockets.
I’m ashamed to admit it, but by your definition, I’m a programmer. I get an assignment and start to work on it, keeping in mind in-code documentation, a solid design and TDD. Sometimes I screw up (again, the inexperience) and have to start a particular piece of functionality using a different approach (and that’s only natural). I do think about usability, but the priority is to get the system up and running…and when that happens, the next project flies in. Not to mention the lack of feedback so I’d know how to improve the system from the user’s point of view.
You’re right in that I haven’t done any Agile development before. Nevertheless, I’d like to visit conventions like Tampere goes Agile and find out exactly what’s true. I for one don’t believe that Agile consumes huge loads of money for bureaucracy. The biggest investment on the client’s part is to iron out the specifics of the application with the help of the team and that’s only for their own benefit because if they don’t know what it is they want, they usually don’t get it. With that point, I’d like to copy the Agile Manifest from http://agilemanifesto.org/.
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
We (you and I) build different systems. I work on a small company and know the possible expansion opportunities for my applications are virtually nonexistent since they are for internal, specific usage, not for a public service like Twitter. Therefore, frameworks always win in my case (I don’t want to reinvent strlen only to see if I could squeeze an extra millisecond out of it). If my system indeed needed to respond to thousands of queries per second, I’d ditch everything I could… but at the moment, it’s not a problem: the client’s company hires 100 new people, we increase the memory of the VPS by another GB. Different tools for different situations.
I believe most of us collect code snippets, samples and libraries like photographers: always ready with a camera. They’re extremely useful… but they don’t make the man. If your external HDD is stolen, you have to continue as normal. It’s the knowledge, experience and the ability to reason and envision what makes a worthy developer.
Time and Cost in Favor Of Security
Beginners dilemma: do it properly and actually learn something new or do it so it gets done fast. Never mind the flexibility, security and the feelings of future developers as long as the task at hand gets done with minimal time consumption and makes one look like a pro, for the moment.
That kind of attitude reminds me a mix of The Office and The Simpsons. Yes – you can hack together something that barely manages to fulfill the current requirements… but then someone exploits the broken system or it collapses because a change in the environment or another developer has to do it over again, maybe from his own spare time because the business side can’t justify the time spent on refactoring a kind-of working system.
I’m in full agreement that we (the developers) can’t follow the golden rules all the time in the real world and compromises must be made to lower the costs… and then I wonder what the clients would say if the company told them point-blank from where the money is/was saved. The fact that the developers think the website/app will never be under any sort of attack, trust the user to input the right kind of data or guard their cookies is naive.
A situation like that is even more passionate when the team is split: one of the developers believes and is willing to do extra work to have some confidence in his creation, the other just does like it’s always been done.
A Real-world Example
Imagine now that you had to set up a WYSIWYG editor with file upload capability – like CKEditor. CKEditor has 2 crucial parts for the server-side. One is the config file, the other the actual (PHP) uploader script. Below is a section of the config.php file.
/**
* This function must check the user session to be sure that he/she is
* authorized to upload and access files in the File Browser.
*
* @return boolean
*/
function CheckAuthentication()
{
// WARNING : DO NOT simply return "true". By doing so, you are allowing
// "anyone" to upload and list the files in your server. You must implement
// some kind of session validation here. Even something very simple as...
// return isset($_SESSION['IsAuthorized']) && $_SESSION['IsAuthorized'];
// ... where $_SESSION['IsAuthorized'] is set to "true" as soon as the
// user logs in your system. To be able to use session variables don't
// forget to add session_start() at the top of this file.
return false;
}
As you can see, CKEdit doesn’t really do anything until true is returned from the CheckAuthentication() function. It’s really tempting to do just that and skip the whole auth-thing to win those crucial minutes. The red (figuratively) warnings above don’t really mean anything because surely, who’d ever want to attack our site OR who’d ever think to exploit the file upload script? We have one login page in front of the uploading script anyway.
Yeah. So what really happens is that the user doesn’t notice anything unless (s)he has some technical knowledge and knows what to look for: an unprotected HTML page that accepts POST data. You’ve solved the current situation and don’t want to go back to deal with the problem. The customer is (wrongly) happy and you get paid.
You should never neglect the most basic security to win time – not on anything that eventually ends up in production. The problem above would be solved by including the framework/code that manages the main site and calling the auth check function. On Kohana 3 it would look like this:
define('SUPPRESS_REQUEST', TRUE);
require('../../index.php');
return Auth::instance()->logged_in();
Done! Problem solved. Did it really take that long? Assuming you knew something about Kohana and it’s modules.
If no-one specifically forbids you from thinking about security, do think about it. If you’re not allowed to write code you’re confident in, tell the managers they get what they pay for or find a new job.
Opinions: Be a Developer… Part II
?This is a continuation of the previous post, "Opinions: Be a Developer, Not a Programmer Part I".
Ando Roots
It's true that teachers always try to teach things as they should be… at least the good ones. That's because, as with every learning, we learn things in details that later wear off. If we're told to use design pattern X then there's hope that as years go by, we fall back to design pattern M instead of all the way to A. Also consider that most professors work in the field they teach so they know of the 'real world'… at least in my applied sciences college.
Even though I haven't done any real agile development before, I don't think it’s "extremely slow" because in it's heart, Scrum (as an example) is made up of iterations that allow the team quick feedback and the customer control over what to change in the next iteration. So if Twitter invents video chat and FB uses SCRUM, there's hope Facebook's own feature will be out the next week.
There are no developers who don't test their own code (or they aren't developers). Any real testing however involves writing automated test cases. Which is notoriously difficult when you work with a person who thinks it's pointless and a waste of time and breaks (my) tests constantly by modifying the code, but not the tests. Do I even dare to dream of version control when "cp folder1 folder2" suffices?
?Framework restrictions
There ARE restrictive frameworks out there. Some might find the rules comfortable since then it doesn't matter whose code you write/modify – you know exactly where to look for something and even if you happen upon an undocumented piece of code, you have a general idea of what's it for.
On the other hand, there are frameworks that implement a general structure such as how controllers/models are structured and then leave the rest up to you. You can and are encouraged to use the particular conventions of the framework, but you may also deviate and create your own rules. It's the same as with laws: you don't like Estonian, go east – maybe it's just right for your style. In general however, you wouldn't ever want to go all the way to the North Pole where there are no (maybe an exaggeration) rules whatsoever – rules such as "thou shall not kill" make life easier.
I approve of the style of using classes and libraries because that strips away the need to invent a triangular wheel…but on the other hand, doesn't give (for granted) you the separation and all other goodies a truer HMVC approach does.
My boss and I have long argued over placing functions in classes, his argument being that loading classes takes far more memory than including a function collection. I've never proven it, but my train of thought is that if the rest of the world – including some opinion leaders – do it and succeed in earning their monthly paycheck, it's the way to go.
What is more, computing power on small-to-medium scale web applications is not an issue anymore. I can allow for some more MB-s of memory on scalable hosting such as the Amazon cloud or Google's own Appengine. And, speaking of the real world again from an (local) Estonian company perspective, most developed application never get such an amount of traffic/usage that the issue of critical resource depletion comes up.
?Convention over Configuration
I disagree with you on the point that complicated tasks should be simplified instead of the simple/common ones. It's far more difficult to guess the particular needs of a complicated task, thus creating the restrictive rules you mentioned. Think for a second: what are the two most common actions a web application does?
Nope, think some more…
Yes! The answer is user auth (signup/login) and CRUD. If I have Kohana with it's Auth and Formo modules, I can present the user with a working prototype of a password protected memo/tweet/warehouse system within the span of half an hour – that I quite possibly whipped up while (s)he was still going over the details with the company salesman.
With Frameworks, this kind of a task is trivial. With libraries, it's not: you have to write some code (don't forget your validations!) and HTML forms, instead of creating the database and routing info and leaving the rest up to Convention over Configuration. That's great for prototyping, but also during and after the main development. Adding a new field is as simple as creating the database field and specifying validation rules.
It's important to go for speed from the start if you're building the next Twitter. It doesn't matter if you're doing a custom-built app for a company of 100 employees. This may be another trap for the impatient ones, too. The client wants results and fast. If none are given, well, there are always other willing parties, especially if you don't do Waterfall, where the contract is "signed with blood" ( – P. Leis).
A word in favor of ditching frameworks tough: you are right that if you indeed work at NASA, you DO reinvent the space shuttle, just so you could have the absolute and total knowledge and control.
?To use a framework…
Frameworks are equally important when you're building a front-end for other services, in fact I'd say that's a case where you’d want to use a Framework even more. Why? If you're not doing any complex calculations on the same machine, you have free memory; memory that the framework can use and justify it's existence. I'd say you should pass on the framework when you're building an API. THEN you need all the juice you can get since API-s usually have far bigger loads.
Speaking of reality (yeah, but we must)… You'd want to use a framework by default, unless special requirements – huge load, knowledge of the future, framework restrictions – point you the other way. You want to develop fast and present the client with a prototype early, so you can get feedback faster, so you can ship a better product. To do that, you use a framework so you don't have to tell the boss that you wrote HTML forms and created validations in the form of a series of if statements for the whole 1st day. You'd also do things the Agile way to adopt to changes fast. And when that doesn't work out, you can always Kanban and be done with it.
And no, I wouldn't stamp a developer bad just because he's not fast enough. I've seen no scalar attribute by which to grade a developer, with the exception of WTF / minute… and the opinions of developers who have to maintain the other's code in the future.
?Development – a Creative Process ?
Now comes the part I dread to write, because your last point is the one I disagree with most strongly.
Writing software IS (with double underlining) a creative process. [big fat period] It's the typical way of thinking for functional programmers who look at the world like this:
Okay, to solve this problem, I write function X and I can see that function X needs function Y and so I write function Y. Clearly, function Y must calculate the tax return so I have to look up the percentages and just code the function to return the value Z. I don't really fancy writing function Y because I don't like rinse-and-repeat, but I have to do this so I could do the same with function X.
I'm clearly exaggerating again, but without personal experience, my opinion stands.
On the other hand, I'm like this:
Hmm…what an interesting problem! I already see three possible solutions for it. What if I asked the framework to do the calculations for X? Hmm, no, that would expose vulnerability A so I don't do that. What about using an external API? Ahh, damn, that could work for a short while, but I know for a fact the company is going to need B on the next month so I better do this right. Oh! But really, the problem boils down to calculating the tax return rate and I can do this easily by implementing Z! Okay, so it will take 5 minutes longer, but the return value will be far greater than doing X or Y.
Another exaggeration. But the point is that by thinking things over, by being aware of conventions, of what's happening out there, experimenting and, most importantly, using your own head to seek for solutions – not just going by the book with the rinse-and-repeat method – is so much more than writing software following the user manual or the only “right” way.
Think of the life of a small town physics professor and his colleague at MIT. They do the exact same thing, but one of them takes and gives far more to the job.
Code or programming in itself might not be creative in the strict sense of the word, but neither is a blank canvas. It's the idea of the artist that matters. Innovation born of experimentation is what thrives us forward. If you take the act of coding as a mundane task that can never offer you such self-satisfaction as say for an example, an architect seeing his building come to life… what good are you to the society as a whole? I could only see myself as a depressed person forced to write character after character, day after day. None of that when I have the creative freedom to dream up solutions. Then you'll see an excited geek hammering away at the keyboard.
I'll take your quote
Software serves creative needs, development it's not creative itself, it's just functional.
- K. Vaher
and use it. So when some of my fellow Developers cross me, I'll tell them to shut it, because they're "just functional", like a cogwheel in a big machine called the World.
?Parting words
It's a passionate topic of discussion. I tried to argument my reasons as best as I could and if it appeared that I only ever argued for the sake of arguing – it's not so.
There are merits in using no frameworks, there are compromises to be made between idealism and getting things done. Different people have different understandings of what's good and why – and it should be so for if there were no disagreement, we'd be a hive.
Thank you for being a worthy opponent on the bloody arena of one of the most violent wars since Linux vs. Windows.
Kristo Vaher
Alright, for starters, frameworks vs no frameworks is not the bloodiest war since Linux vs Windows. It really is not. I absolutely love web development frameworks. I both like the heavy-handed things like Zend, which I studied when I was a young developer because everybody said it was 'the thing'. I later on loved the speed of Codeigniter and then Kohana, which I think is by far the best framework for website development out there. And I love WordPress since even a beginner can set up quality website with custom needs there with little problems without sacrificing potential.
Creativity Before Coding
But writing code is not a creative process. You misunderstood my point there. You should never write code out of thin air. You need to have the idea of the system and its components already in your head before writing first lines of code. You should know what is the workflow of registration system, what needs the client has and what the end system has to be able to do while at the same time staying within development budget, which is very restricted in majority of projects I've taken part of.
The creative process of software development happens before writing a single line of code. Being able to write beautiful functions and classes or having source code that looks so pretty that you'd want to make babies with it is not the goal. A developer has a duty to realize the development plan as well as possible, but 'writing code' itself is not creative. You can solve the same problem multiple ways and since the end user is unable to tell the difference, it's not a creative process since it does not benefit from being one.
This is the big difference between a software developer and a programmer. Most of the developers that have gone through in my company have been just programmers. They are not creative, they are functional. They know how to solve problems or build modules and they are working on something that has already been designed. Some programmers dabble more into designing the whole system and paying attention to usability, but it is rare. Those that do, dabble into software design, which is indeed a creative process.
?Development in Reality
As for agile development, let's speak again in a few years time when you've tried to implement it in a small company in numerous client projects without wanting to blow your brains out with a shotgun loaded with dynamite. In the end of the day it is all about the budget and agile development costs more time and money 90% of the time. There are projects where this is possible, but very few companies are willing to invest in it and even fewer clients are willing to pay for it. You have to compromise. If you have to test out and confirm every feature as it's being implemented you either have to do multiple projects at the same time or just have a client who is willing to pay a lot of money, which is very rare in Estonia in current economy.
I totally agree with your idealized vision of software development, but it is vision of software evangelists. In reality it is just not possible. There's not a single system developed that has not compromised bigtime. Facebook is being refactored even today and it is very slow to release features. Twitter refactored large parts of itself last year and is – in general – a really simplistic system. Even Google is known for refactoring (and sometimes breaking) even its own systems and they are known for writing some of the most 'beautiful' code out there. Hacks are nature of software development since getting the feature out in working condition is more important than how good it looks inside. And all of those systems share one very important need for well defined and documented API.
This returns to the point of if you are making something large, you have to design it without putting much weight on frameworks unless you are able to completely strip it to its bones and only use the essentials that are not hindrance on the end product. In general, if you begin designing something that might have thousands of active users at the same time, you really have to start from the clean sheet and then speed up the process of development by refactoring classes and methods from your pre-existing libraries. It's not the framework that developer uses that makes them valuable, but the tools they have gathered over the years. In my own example, I have an external encrypted hard drive that contains hundreds of code examples, classes and open source packages for any need that might arise without needing me to implement an entire framework.
Speed and computing power is of critical importance no matter how much computing power you have available. Worst thing that can happen to a system is to become popular and not be able to keep things running. Facebook won the battle of social networking because it implemented gradual releases in different parts of America and then later on in the world, while the big competitor Friendster attempted to bite too much at the same time and suffered with downtimes and slow service. Twitter would never have become what it is today had it suddenly developed problems with performance. While it is true that just buying more computing power could solve that problem, with large systems a more optimized and lean code can save a lot of money.
When to Use What
And since API's (which are a must for any public system that isn't just a website or e-shop these days) benefit strongly from being just as lean, frameworks just are not good enough.
And people speaking of MVC or other software development patterns in a way as if they need a framework to be implemented perhaps have not designed MVC system themselves, because the concept behind MVC is relatively simple and can be set up by less than couple hundred lines of code with all the basics a system might need.
I strongly recommend frameworks for developing websites and e-shops and simpler info systems and I most certainly recommend studying how frameworks function since that may give better insight into MVC and other patterns than tutorials. But if speed is important you are better off implementing refactored functions and classes and build the system from ground up.
As a sidenote, in Apple vs Android battles this is also very evident. While Android is a much more flexible system than iOS, iOS implements better native code simply because of being written in Objective-C. iOS app's are simply more smooth and faster. Java is also compiled, but it is quite bloated (and there's a reason why Java frameworks are really not that well known), if not the most bloated system out there.
Parting Thoughts
So, to summarize.. There are compromises to both sides, but the choice of going one way or another should depend entirely on the project. Of course once that decision is made, the entire team should work in unison and if project implements a framework, its standards should be followed by the entire team.
There is no right or wrong answer, but I know that if I am given a choice between two systems that are functionally exactly the same, I will prefer the faster one.
Still more to come…
Opinions: Be a Developer, Not a Programmer Part I
The following is a transcript of a debate between me an a friend of mine – Kristo Vaher – about different ways of developing software. Published with his permission.
Ando Roots
Developing PHP is messy on its own…but when you have different views of the words 'elegance', 'simplicity' or DRY with your colleague…
Kristo Vaher
PHP is very elegant. When you write a wrapper for everything. It's like having PHP wear makeup.
Ando Roots
I spent a day writing everything in ORM, using built-in modules of the framework, CRUD generation etc… then my co-worker slash project manager told me he'd do it himself and basically dumped most of what I did.
About The Author
Ando “David” Roots is a college student and a software developer from Kunda, Estonia. Living, working and studying in Tallinn, he hopes to get his bachelor degree from the Estonian Information Technology College on IT Systems Development. 
