Software Engineering Practices, Part 1
A Summary of lecture in AWS ML Foundations Course, via Udacity
- Clean and Modular Code
- Refactoring Code
- Writing Clean Code
- Writing Modular Code
- Refactor: Wine Quality Analysis
Clean and Modular Code
- PRODUCTION CODE: software running on production servers to handle live users and data of the intended audience. Note,this is different from production quality code, which describes code that meets expectations in reliability, efficiency, etc., for production. Ideally, all code in production meets these expectations, but this is not always the case.
- CLEAN: readable, simple, and concise. A characteristic of production quality code that is crucial for collaboration and maintainability in software development.
- MODULAR:logically broken up into functions and modules. Also an important characteristics of production quality code that makes your code more organized, efficient, and reusable.
- MODULE: a file. Modules allow code to be resued by encapsulating them into files that can be into other files
Refactoring Code
- REFACTORING: restructuring your code to improve its internal structure, without changing its external functionality. This gives you a chance to clean and modularize your program after you've got it working.
- Since it isn't easy to write your best code while you're still trying to just get it working, allocating time to do this is essential to producing high quality code. Despite the initial time and effort required, this really pays off by speeding up your development time in the long run.
- You become a much stronger programmer when you're constantly looking to improve your code. The more you refactor, the easier it will be to structure and write good code the first time.
Writing Clean Code
Meaningful Names
Tip:Use meaningful names
- Be descriptive and imply type : E.g. for booleans, you can prefix with
is_orhas_to make it clear it is a condition. You can also use part of speech to imply types, like verbs for functions and nouns for variables.
-
Be consistent but clearly differentiate : E.g.
age_listandageis easier to differentiate thanagesandage. -
Avoid abbreviations and especially single letters : (Exception: counters and common math variables) Choosing when these exceptions can be made can be determined based on the audience for your code. If you work with other data scientists, certain variables may be common knowledge. While if you work with full stack engineers, it might be necessary to provide more descriptive names in these cases as well.
-
Long names != descriptive names : You should be descriptive, but only with relevant information. E.g. good functions names describe what they do well without including details about implementation or highly specific uses.
Try testing how effective your names are by asking a fellow programmer to guess the purpose of a functino or variable based on its name, without looking at your code. Coming up with meaningful names often requires effort to get right.
Nice Whitespace
Tip:Use whitespace properly
- Organize your code with consistent indentation - the standard is to use 4 spaces for each indent. You can make this a default in your text editor.
- Separate sections with blank lines to keep your code well organized and readable.
- Try to limit your lines to around 79 characters, which is the guideline given in the PEP 8 style guide. In many good text editors, there is a setting to display a subtle line that indicates where the 79 character limit is.
For more guidelines, check out the code layout section of PEP 8 in the notes below.
Writing Modular Code
Tip:DRY (Don't Repeat Yourself)
- Don't repeat yourself! Modularization allows you to reuse parts of your code. Generalize and consolidate repeated code in functions or loops. Tip:Abstract out logic to improve readability
- Abstracting out code into a function not only makes it less repetitive, but also improves readability with descriptive function names. Although your code can become more readable when you abstract out logic into functions, it is possible to over-engineer this and have way too many modules, so use your judgement. Tip:Minimize the number of entities (functions, classes, modules, etc.)
- There are tradeoffs to having function calls instead of inline logic. If you have broken up your code into an unnecessary amount of functions and modules, you'll have to jump around everywhere if you want to view the implementation details for something that may be too small to be worth it. Creating more modules doesn't necessarily result in effective modularization. Tip:Functions should do one thing
- Each function you write should be focused on doing one thing. If a function is doing multiple things, it becomes more difficult to generalize and reuse. Generally, if there's an "and" in your function name, consider refactoring. Tip:Arbitrary variable names can be more effective in certain functions
- Arbitrary variable names in general functions can actually make the code more readable. Tip:Try to use fewer than three arguments per function
- Try to use no more than three arguments when possible. This is not a hard rule and there are times it is more appropriate to use many parameters. But in many cases, it's more effective to use fewer arguments. Remember we are modularizing to simplify our code and make it more efficient to work with. If your function has a lot of parameters, you may want to rethink how you are splitting this up.
In this exercise, you'll refactor code that analyzes a wine quality dataset taken from the UCI Machine Learning Repository here. Each row contains data on a wine sample, including several physicochemical properties gathered from tests, as well as a quality rating evaluated by wine experts.
The code in this notebook first renames the columns of the dataset and then calculates some statistics on how some features may be related to quality ratings. Can you refactor this code to make it more clean and modular?
import pandas as pd
df = pd.read_csv('./dataset/winequality-red.csv', sep=';')
df.head()
new_df = df.rename(columns={'fixed acidity': 'fixed_acidity',
'volatile acidity': 'volatile_acidity',
'citric acid': 'citric_acid',
'residual sugar': 'residual_sugar',
'free sulfur dioxide': 'free_sulfur_dioxide',
'total sulfur dioxide': 'total_sulfur_dioxide'
})
new_df.head()
And here's a slightly better way you could do it. You can avoid making naming errors due to typos caused by manual typing. However, this looks a little repetitive. Can you make it better?
labels = list(df.columns)
labels[0] = labels[0].replace(' ', '_')
labels[1] = labels[1].replace(' ', '_')
labels[2] = labels[2].replace(' ', '_')
labels[3] = labels[3].replace(' ', '_')
labels[5] = labels[5].replace(' ', '_')
labels[6] = labels[6].replace(' ', '_')
df.columns = labels
df.head()
def replace_labels(columns):
labels = list(columns)
for i in range(len(labels)):
labels[i] = labels[i].replace(' ', '_')
return labels
df.columns = replace_labels(df.columns)
df.head()
Analyzing Features
Now that your columns are ready, you want to see how different features of this dataset relate to the quality rating of the wine. A very simple way you could do this is by observing the mean quality rating for the top and bottom half of each feature. The code below does this for four features. It looks pretty repetitive right now. Can you make this more concise?
You might challenge yourself to figure out how to make this code more efficient! But you don't need to worry too much about efficiency right now - we will cover that more in the next section.
df = pd.read_csv('./dataset/winequality-red.csv', sep=';')
median_alcohol = df.alcohol.median()
for i, alcohol in enumerate(df.alcohol):
if alcohol >= median_alcohol:
df.loc[i, 'alcohol'] = 'high'
else:
df.loc[i, 'alcohol'] = 'low'
print(df.groupby('alcohol').quality.mean())
median_pH = df.pH.median()
for i, pH in enumerate(df.pH):
if pH >= median_pH:
df.loc[i, 'pH'] = 'high'
else:
df.loc[i, 'pH'] = 'low'
print(df.groupby('pH').quality.mean())
median_sugar = df['residual sugar'].median()
for i, sugar in enumerate(df['residual sugar']):
if sugar >= median_sugar:
df.loc[i, 'residual sugar'] = 'high'
else:
df.loc[i, 'residual sugar'] = 'low'
print(df.groupby('residual sugar').quality.mean())
median_citric_acid = df['citric acid'].median()
for i, citric_acid in enumerate(df['citric acid']):
if citric_acid >= median_citric_acid:
df.loc[i, 'citric acid'] = 'high'
else:
df.loc[i, 'citric acid'] = 'low'
print(df.groupby('citric acid').quality.mean())
def wine_quality(column):
median = df[column].median()
for i, col in enumerate(df[column]):
if col >= median:
df.loc[i, column] = 'high'
else:
df.loc[i, column] = 'low'
return df.groupby(column).quality.mean()
features = ['alcohol', 'pH', 'residual_sugar', 'citric_acid']
for feature in features:
print(wine_quality(feature))