code


When building a new project, it's a smart move to be very strict right from the
start. It is much harder to add more linting/typing checks once you have 1000+
lines of code.

That's why I'm providing an opinionated list of libraries for your new Python
project. I might write a more in-depth article on the best practices when
building a web app with Python. For now, this is mostly a checklist with some
obvious recommendations.

**Why not a template repo instead of a checklist?** Template repositories (e.g.
built with [cookiecutter](https://github.com/cookiecutter/cookiecutter)) go
quickly out of date and discourage learning about the ins and outs of all those
best practices. They might make sense for your organization, but they're not
the point of this article.

## Summary

- Development tasks: `Makefile`
- Typechecking: [mypy](http://mypy-lang.org/)
- Dependency and virtualenv management: [poetry](https://python-poetry.org/)
- Linting: [flake8](https://flake8.pycqa.org/en/latest/)
- Code autoformatting: [black](https://github.com/psf/black) and [isort](https://pycqa.github.io/isort/)
- Tests: [pytest](https://docs.pytest.org/en/stable/) with plugins such as
  [pytest-cov](https://github.com/pytest-dev/pytest-cov), [pytest-factoryboy](https://pytest-factoryboy.readthedocs.io/en/latest/)
- Docstring: [pydocstyle](http://www.pydocstyle.org/en/stable/)
- Logging: [structlog](https://www.structlog.org/en/stable/)
- Configuration: [Pydantic BaseSettings](https://pydantic-docs.helpmanual.io/usage/settings/) with [dotenv support](https://pypi.org/project/python-dotenv/)
- Error reporting: [Sentry](https://sentry.io/welcome/)
- Documentation: [Docusaurus](https://docusaurus.io/) or [Sphinx](https://www.sphinx-doc.org/en/master/)
- Profiling and performance: [profile](https://docs.python.org/3/library/profile.html), [pyinstrument](https://github.com/joerick/pyinstrument), [sqltap](https://github.com/inconshreveable/sqltap)

Other topics:

- Data validation: [Pydantic Models](https://pydantic-docs.helpmanual.io/usage/models/)
- DB ORM: [sqlalchemy](https://www.sqlalchemy.org/docs/latest/) and [alembic](https://alembic.sqlalchemy.org/) for migrations
- Web framework: [fastapi](https://fastapi.tiangolo.com/)
- FP utilities: [toolz](https://toolz.readthedocs.io/)
- CLI framework: [typer](https://typer.tiangolo.com/)

Not detailed below:

- [pre-commit](https://pre-commit.com/)

## Running development tasks: Makefile

Makefiles are well understood, work almost everywhere, and shorten the ramp-up
time for fellow developers who might not have much experience with Python.

Here's an example:

```makefile
SHELL := bash
.ONESHELL:
.SHELLFLAGS := -eu -o pipefail -c
.DELETE_ON_ERROR:
MAKEFLAGS += --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules

install:  # Install the app locally
	poetry install
.PHONY: install

ci: typecheck lint test ## Run all checks (test, lint, typecheck)
.PHONY: ci

test:  ## Run tests
	poetry run pytest .
.PHONY: test

lint:  ## Run linting
	poetry run black --check .
	poetry run isort -c .
	poetry run flake8 .
	poetry run pydocstyle .
.PHONY: lint

lint-fix:  ## Run autoformatters
	poetry run black .
	poetry run isort .
.PHONY: lint-fix

typecheck:  ## Run typechecking
	poetry run mypy --show-error-codes --pretty .
.PHONY: typecheck

.DEFAULT_GOAL := help
help: Makefile
	@grep -E '(^[a-zA-Z_-]+:.*?##.*$$)|(^##)' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[32m%-30s\033[0m %s\n", $$1, $$2}' | sed -e 's/\[32m##/[33m/'
```

This is a great article about Makefile: [Your Makefiles are wrong](https://tech.davis-hansson.com/p/make/)

## Typechecking: mypy

Type annotations in Python libraries are not yet pervasive, but getting better
every day.

Supported by Python's founder, Guido van Rossum, [mypy](http://mypy-lang.org/)
is the de facto standard. The [cheat sheet](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html)
is a very helpful resource.

You can configure mypy inside `setup.cfg`:

```
[mypy]
strict = true

# I prefer to be explicit about ignoring packages which do not yet have types:
[mypy-psycopg2.*]
ignore_missing_imports = True
```

## Dependency and virtualenv management: poetry

Unfortunately because of the way `pip` installs dependencies, you have to deal
with virtualenv in most cases (although [things might change rapidly with pdm](https://pdm.fming.dev/)).

Nowadays I'd recommend using [poetry](https://python-poetry.org/). It is not
yet absolutely perfect, but it provides a very elegant CLI API.

It's super easy to start a project:

```bash
poetry new service-name
cd service-name
$EDITOR pyproject.toml
rm -Rf tests
mv README.rst README.md

poetry add sqlalchemy  # for instance
```

Then either go into a virtualenv-enabled shell with `poetry shell` or prefix
your commands with `poetry run ...`.

## Linting: flake8

There are two main linters:

- [flake8](https://flake8.pycqa.org/en/latest/)
- [pylint](https://www.pylint.org/)

I usually rely mostly on flake8 because it has fewer false positives. While
pylint is super configurable, it includes too many checks to my taste.

I use the following configuration (in `setup.cfg`):

```
[flake8]
max-line-length = 99
extend-ignore =
    # See https://github.com/PyCQA/pycodestyle/issues/373
    E203,
```

## Code autoformatting: black and isort

[black](https://github.com/psf/black) autoformats your code so that you don't
have to think about it.

[isort](https://pycqa.github.io/isort/) is a nice addition to black, it will
sort your imports to comply with [PEP 8](https://www.python.org/dev/peps/pep-0008/#imports):

- Sort alphabetically
- Group into standard imports, third-party imports, app imports

Here's my isort config to comply with flake8 and black (in
`pyproject.toml`):

```toml
[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
ensure_newline_before_comments = true
line_length = 88
```

## Test: pytest with coverage

[pytest](https://docs.pytest.org/en/stable/) has a lot of magical features but
it makes writing tests so efficient. The fixture system is brilliant and
super powerful. Using plain `assert` instead of having to learn an `assertEqual`
metalanguage makes your life more meaningful.

Here's the config I use (in `pyproject.toml`):

```toml
[tool.pytest.ini_options]
# Personal preference: I am too used to native traceback
addopts = "--tb=short"

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "if __name__ == .__main__.:",
    "nocov",
    "if TYPE_CHECKING:",
]

[tool.coverage.run]
# Activating branch coverage is super important
branch = true
omit = [
  # add your files to omit here
	]
```

I usually use the following plugins and lib:

- [pytest-factoryboy](https://pytest-factoryboy.readthedocs.io/en/latest/) use
  factories to create your fixture. Super powerful, and avoids having a single
  file where all your reusable fixtures are defined a thousand times with
  different variations.
- [pytest-mock](https://github.com/pytest-dev/pytest-mock/) makes it easier to
  work with `unittest.mock`.
- [pytest-cov](https://github.com/pytest-dev/pytest-cov) provides coverage
  reports for your tests.
- [doubles](https://github.com/uber/doubles): sadly not maintained anymore (but
  it still works!), it provides a much simpler and stricter mocking experience
  than `unittest.mock`.
- [requests-mock](https://requests-mock.readthedocs.io/en/latest/) to check
  integration with HTTP services called with [requests](https://requests.readthedocs.io/en/master/). It automatically integrates with pytest and provides a fixture named `requests_mock`.

## Checking docstring: pydocstyle

[pydocstyle](http://www.pydocstyle.org/en/stable/) enforces [PEP
257](https://www.python.org/dev/peps/pep-0257/) for docstring styling.

Here's my config (in `setup.cfg`):

```toml
[pydocstyle]
# Do not require any docstring
ignore = D100,D101,D102,D103,D104,D105,D106,D107,D213,D203
```

## Logging: structlog

[structlog](https://www.structlog.org/en/stable/) is a must-have for all your
logging needs.

```python
from structlog import get_logger


logger = get_logger(__name__)

def hello(name: str):
		logger.info("saying hello", name=name)
		# instead of :
		# logger.info("saying hello to %s", name)
```

Structuring your logs has numerous advantages:

- Immediately parsable by automated tools (kibana, m/r jobs, etc.)
- Easier to write: you don't have to think about the order of your logging
  message
- Flexible: can be further manipulated since all log messages are dicts
  until they're displayed

You can create `yourapp.lib.log`:

```python
import json
import logging
from typing import Any, Dict
from uuid import UUID

import structlog

from yourapp.config import config


def default(obj: Any) -> Any:
    if isinstance(obj, UUID):
        return str(obj)

    raise TypeError(f"Can't serialize {type(obj)}")


def dumps(*args: Any, **kwargs: Any) -> str:
    kwargs.pop("default", None)
    return json.dumps(*args, **kwargs, default=default)


def add_version(
    logger: logging.Logger, method_name: str, event_dict: Dict[str, Any]
) -> Dict[str, Any]:
    """Add version to log message."""
    event_dict["version"] = config.git_commit_short
    return event_dict


class ConsoleRenderer(structlog.dev.ConsoleRenderer):
    def _repr(self, val: Any) -> str:
        # Display shorter uuid
        # https://www.structlog.org/en/stable/_modules/structlog/dev.html#ConsoleRenderer
        if isinstance(val, UUID):
            return str(val)
        return super()._repr(val)


def configure_logger(level: str = "INFO", *, console: bool = False) -> None:
    """Configure logging.

    console should be True for console (dev) environment.
    """
    # see https://stackoverflow.com/questions/37703609/using-python-logging-with-aws-lambda
    root = logging.getLogger()
    if root.handlers:
        for handler in root.handlers:
            root.removeHandler(handler)
    logging.basicConfig(format="%(message)s", level=level)

    if not console:
        processors = [
            add_version,
            structlog.stdlib.filter_by_level,
            structlog.stdlib.add_logger_name,
            structlog.stdlib.add_log_level,
            structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M.%S"),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            structlog.processors.JSONRenderer(serializer=dumps),
        ]
    else:  # nocov
        processors = [
            structlog.stdlib.add_logger_name,
            structlog.stdlib.add_log_level,
            structlog.stdlib.PositionalArgumentsFormatter(),
            structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M.%S"),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            ConsoleRenderer(),
        ]

    structlog.configure(
        processors=processors,  # type: ignore
        wrapper_class=structlog.stdlib.BoundLogger,
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )
```

## Configuration: Pydantic's BaseSettings

Like most people, I usually ended up having my own mechanism for handling
configuration. Thanks to the web framework fastapi, I've discovered that
[pydantic](https://pydantic-docs.helpmanual.io/) provides a very handy
[BaseSettings](https://pydantic-docs.helpmanual.io/usage/settings/) class that
relies on environment variable for its configuration. `BaseSettings` provide
many things that would be annoying to implement from scratch:

- Type hints
- Read from environment variables
- Validate configuration values
- `.env` support with [python-dotenv](https://pypi.org/project/python-dotenv/)
- Secrets support

```python
import os
from pathlib import Path
from typing import List, Optional

from dotenv import load_dotenv
from pydantic import BaseSettings

ENV_FILENAME = os.environ.get("DOTENV", ".env")


class MisconfiguredException(Exception):
    pass


class Config(BaseSettings):
    # Please use env_name ONLY for informational purpose (see docs)
    env_name: str
    git_commit_short: str = "unknown"

    # Activate this to get profiling - see documentation.
    is_db_enabled: bool = False

    db_user: str = "unconfigured"
    db_password: str = "unconfigured"
    db_name: str = "unconfigured"
    db_port: str = "5432"
    db_host: str = "localhost"

    sentry_dsn: Optional[str]


def get_config() -> Config:
    """Get the config."""
    # We follow serverless's dotenv plugin's behavior here:
    # https://www.npmjs.com/package/serverless-dotenv-plugin

    # First load .env
    load_dotenv(dotenv_path=".env")

    if not Path(ENV_FILENAME).exists():
        raise ValueError(f"Config file {ENV_FILENAME} does not exist.")

    if ENV_FILENAME.endswith(".local"):
        raise ValueError(
            "Expected env filename like '.env.dev', "
            f"got override ending with .local instead: {ENV_FILENAME!r}. "
            f" Try with {ENV_FILENAME.replace('.local', '')!r}"
        )

    # Then load .env.{env}
    load_dotenv(dotenv_path=ENV_FILENAME)

    # Then load .env.{env}.local if it exists
    override = ENV_FILENAME + ".local"
    if Path(override).exists():
        load_dotenv(dotenv_path=override)

    return Config()


config = get_config()
```

Now you just have to run your commands like this:

```bash
DOTENV=.env.test poetry run pytest .
```

## Error reporting: Sentry

[Sentry](https://sentry.io/welcome/) is a service that provides exception
monitoring. Its SDK is very simple to integrate.

Usually, I use the following pattern:

```python
# app.py
from app.lib.log import configure_logger
from app.lib.sentry import configure_sentry

configure_logger(config.log_level)
configure_sentry()
```

```python
# app.lib.sentry
from typing import Any

import sentry_sdk
from structlog import get_logger

from app.config import config

logger = get_logger(__name__)


def configure_sentry(**kwargs: Any) -> None:  # nocov
    if not config.sentry_dsn:
        logger.info("not configuring sentry")
        return

    sentry_sdk.init(
        debug=False,
        dsn=config.sentry_dsn,
        environment=config.env_name,
        traces_sample_rate=1.0,
        release=config.git_commit_short,
        **kwargs,
    )
```

## Documentation: Docusaurus

[Sphinx](https://www.sphinx-doc.org/en/master/) is another great choice
(especially if you want to get Python code auto-documentation), but at my
current company Gens de Confiance we use [Docusaurus](https://docusaurus.io/),
a powerful yet simple documentation management tool.

## Domain models, data validation: Pydantic

[Pydantic Models](https://pydantic-docs.helpmanual.io/usage/models/) is
a flexible way to create your domain model objects:

- Type hinting
- Validators
- Export to json-schema

You can also use Python's standard lib
[dataclass](https://docs.python.org/3/library/dataclasses.html) together with
something like [marshmallow](https://marshmallow.readthedocs.io/en/stable/).

## ORM: sqlalchemy

If you need to interact with the DB,
[sqlalchemy](https://www.sqlalchemy.org/docs/latest/) is a very safe choice. It
comes with loads of features and is the most used non-Django Python ORM, which
means that you'll find Stack Overflow solution for all your problems. Using
[alembic](https://alembic.sqlalchemy.org/) for DB migrations is the next
logical move.

Both libraries are written by the insanely productive [Mike
Bayer](https://github.com/zzzeek).

## Web framework: fastapi?

You have a lot of excellent choices when it comes to Python web framework.
I usually prefer microframework and am currently developing with
[fastapi](https://fastapi.tiangolo.com/), which is a lot of fun to work with.

I usually refrain from using any plugins that come with the framework, because
they are usually too coupled to the framework and the context of an HTTP
request. I might write an article about my preferred setup.

## Utility functions for functional programming: toolz

While Python is not a strict functional programming language, it is possible to
write FP-styled code with it. [toolz](https://toolz.readthedocs.io/) is
a great companion and provides many utility functions that make writing code
easier. It has a curried-by-default namespace (`from toolz.curried import take`). Checkout its [cheat
sheet](https://toolz.readthedocs.io/en/latest/api.html).

## CLI framework: typer

[typer](https://typer.tiangolo.com/) (same author as fastapi and pydantic)
leverages type annotations to make it super easy to write powerful scripts with
a command line interface:

```python
#!/usr/bin/env python3

"""Say hello.
"""

import typer


def main(name: str) -> int:
    typer.echo(f"Hello {name}")

    password = typer.prompt("password")
    assert len(password) > 8

    return 0


if __name__ == "__main__":
    typer.run(main)
```

## Performance profiling: pyinstrument and sqltap

- [pyinstrument](https://github.com/joerick/pyinstrument) is a recent Python
  profiler which can export to HTML.
- [sqltap](https://github.com/inconshreveable/sqltap) integrates with
  sqlalchemy to allow you to introspect SQL queries and also exports to HTML.

Some services (like Sentry) can also profile code thanks to their SDK.
Otherwise, you can also rely on the standard library module [profile](https://docs.python.org/3/library/profile.html).

## Wishlist

- Lots of inconsistencies for where to put configuration: `setup.cfg`,
  `pyproject.toml`, specific files, etc.
- The more complex the library, the more useful it would be to have type
  annotations and... the less probable it is to have those annotations.
  Libraries such as sqlalchemy (coming in 2.0) and toolz don't have official types for now.
- mypy encourages nominal subtyping (see [this FAQ](https://mypy.readthedocs.io/en/stable/faq.html#can-i-use-duck-typing-with-mypy)) which is a bit sad because it discourages using simple dicts. Fortunately something like [PEP 589 TypeDict](https://www.python.org/dev/peps/pep-0589/) will improve things.
- It would be so nice to avoid having to use virtualenv (even through
  something like poetry or pipenv). First-time Python developers get so
  confused about them (compared to Node's simpler `node_modules` setup). I'm really looking forward to seeing [PEP
  582](https://www.python.org/dev/peps/pep-0582/) deployed.

## Missing something?

Drop me an email at `charles at dein.fr` if you think I'm missing something!

For more resources related to Python, check out my repo
[charlax/python-education](https://github.com/charlax/python-education).


All the best practices for setting up a new app in Python

A Python project checklist


**Summary**: all lines of code should be covered by a test or _explicitly_
excluded from test coverage.

What's your test coverage? What do you consider a good test coverage
percentage?

Those questions often get asked during engineering interviews.
It gives the interviewee a sense of how much the company cares about quality.
It's also a good question for the candidate because you can't have a cargo
cult answer. You need to argue. It gives useful insights into one's quality
mindset.

I believe the only right value to track for a test coverage metric is 100% of
the _lines that should be covered_ (_not_ the lines that are written), which
ensures that the lines that you don't want to cover are explicit, and not
implicit.

Please don't skim the articles as you'll find that the point I'm making is pretty common sense. If you still disagree have a look at the response
to common objections at the end.

_Note: I'm using Python for all the examples below, but the principles are
applicable regardless of the programming language._

## Why you should measure test coverage

If you're not already monitoring test coverage, you should probably start now
(if you don't have tests, that's for another article). Some programming
languages have better support than others. Python has excellent support for it,
through `pytest` and `pytest-cov` (using `coverage` under the hood).

### Test all functionality

Test coverage allows you to check which lines of code are run while the tests
are executed. If some lines are missing coverage, it means you're not testing
("covering") the underlying behavior. You could introduce a bug, and no test
would fail, which is not desirable. Introducing a bug should ideally trigger a
test to fail.

Let's see a contrived example:

```python
def toast(bread: str) -> str:
    if bread == "brioche":
        ...
        1 / 0  # let's say a bug is hidden here
        return "ok"

    return bread

def test_toast():
    assert toast("baguette") == "ok"
```

The test will pass, even though `toast("brioche")` should fail. Adding
a test for this case will let you find the `ZeroDivisionError`.

### Ensure all tests are run

Another case for checking is that you want to make sure that your test actually
run. In this contrived example, the tests will pass, even though one of
those test does not do anything:

```python
def identity(bread: str) -> str:
    return bread

def test_identity():
    assert identity("brioche") == "brioche"

def test_identity_noop():
    if 1 == 1:  # this is a contrived example, there shouldn't be any 'if' in tests
        return
    assert identity("baguette") == "baguette"
```

Once you've agreed that you need test coverage, the question remains: what is the
right level of coverage? It is very costly to ensure that absolutely every line
of code is tested. Should it be 80%? 90%?

## Why you should check for 100% coverage

```
TOTAL                                      18435     16   1742      0    99%

FAIL Required test coverage of 100% not reached. Total coverage: 99.86%
```

100% is clear cut.

Enforcing 100% coverage requires you to be explicit about the lines that don't
have to be covered ("explicit is better than implicit"). By including the `no cover` marker (or whatever your language/coverage lib supports) next to the
code that is excluded from coverage, you allow code reviewers to see your
coverage decisions, and challenge them.

The problem with having a goal that is not 100% is that it leaves room for
interpretation and negotiation. If we're currently at 80% and I bring coverage
at 79.9% because I'm too lazy to test my code (or exclude it from coverage),
then can't we say that it's ok? Explicit markers in the code make the
discussion more factual and objective.

The other problem with something different from 100% is that you might be
adding coverage in one area, and reducing it in another, without knowing it. It
might still lead to 80% coverage - but the reduction in coverage might have
appeared in a crucial area of your code.

## Will 100% coverage prevent any bug from being introduced?

Of course not. Just because a line is run in a test does not mean it is 100%
correct. You might not be testing all variations. The test might assert the
wrong thing.

Some lines of code should be tested twice to ensure they're correct. See this
contrived example:

```python
def can_toast(bread: str) -> bool:
    return bread in ["croissant"]  # this line is covered

def test_can_toast():
    assert can_toast("baguette") is True
```

All the lines are covered, but I'm not testing that you can't toast `"brioche"`
and `"baguette"`. Let's say this behavior is critical (toasting a croissant is
a crime!). Without another test, somebody could change the code like this, and
nothing would break.

```python
def can_toast(bread: str) -> bool:
    return True  # this line is still covered

def test_can_toast():
    assert can_toast("baguette") is True
```

Therefore sometime you definitely should test the same line more than once,
testing the failure case and the success case. I'd strongly recommend this for
permission checking for instance.

```python
def test_can_toast():
    assert can_toast("baguette") is True

def test_can_toast_cannot():
    assert can_toast("croissant") is False
```

### Is this realistic?

You don't need to cover 100% of the lines you write. Some are not worth
covering as you'll see in the examples below. However, you need to cover 100% of the
lines **you want to be covered**. You need to be explicit about excluding lines
from coverage. Tools such as Python's `coverage` allow you to specify which
line should be excluded from the coverage.

Using `pytest-cov` you can achieve this pretty simply:

```python
PLUGGED = True

def toast(bread: str) -> str:
    if not PLUGGED:  # pragma: no cover
        return "ok"

    ...
    return "ok"

def test_toast():
    assert toast("bread") == "ok"
```

With this setup, test coverage will be at 100% even though we did not test the
case where the toaster is not plugged because it is too simple. Writing a test
for this "functionality" would not be worth it. Therefore we explicitly exclude
it from coverage. You should revisit this decision if it becomes more complex
and needs to be tested.

## What can/should be excluded from coverage?

It depends on your codebase and your quality expectations. Also, as
for any rules, there are exceptions. Examples are writing using Python, but this
is not language-dependent.

### Simple error handling

```python
def toast(bread: str) -> str:
    if not bread:  # pragma: no cover - the handling is too obvious
        raise ValueError("no bread provided")
    ...
```

Anything more complicated than that should probably be covered.

### Proxy functions

```python
from external_lib import Toaster

def toast(bread: str) -> str:  # pragma: no cover - we're just proxying
    return Toaster.toast(bread)
```

It also applies to thin wrappers you might have in front of your
dependencies.

### Code as configuration

```python
def get_temperature(bread: str) -> int:  # pragma: no cover - no complexity there
    if bread == "brioche":
        return 1

    elif bread == "baguette":
        return 3

    ...
    return 2
```

### Simple early return/continue

```python
def toast(breads: List[str]) -> str:
    for bread in breads:
        if not bread:  # pragma: no cover - simple early continue
            continue

        ...

    return "ok"
```

If the early return/continue is part of the logic, or leads to complicated side
effects, it should be tested.

### Code that is too difficult to test (rare)

In rare cases, it is a perfectly fine decision to exclude some code from
coverage if the tests are too expensive to write (because of the required
setup, or the use of an external library, for instance). Again - this should
only be in rare cases, for instance when you're using an external library that
is not easy to mock. In most cases, if your code is complicated to test that
probably means it needs to be refactored - easy-to-test code is usually better
designed.

```python
from packagea import A
from packageb import B

def toast(bread: str) -> str:  # pragma: no cover - too complicated to test
    a = A.expensive_function_that_needs_to_be_mocked()
    b = B.expensive_function_that_needs_to_be_mocked()
    ...  # lots of code
    return "ok"
```

Make sure that this code is easy to test manually, and that the code calling it
is well tested.

### Debugging/manual test code

```python
import os

DEBUG = os.environ.get("DEBUGGING")
p
def toast(bread: str) -> str:
    if DEBUG:  # pragma: no cover - debugging code
        print(f"toasting {bread}")
        ...
    return "ok"
```

## What should be covered

Here's a list of things that should be covered, ideally more than
once.

### Branches

If your language supports it, make sure you test all branches:

```python
def toast(bread: str) -> str:        # 1
    if bread == "baguette":          # 2
        bread = "toasted_baguette"   # 3
    return bread                     # 4


def test_toast_baguette():
    assert toast("baguette") == "toasted_baguette"
```

You're technically at 100% code coverage, but the case where `bread` is not
"baguette" and the `if` is never evaluated to False: line 2 never jumps to line 4. If you activate branch coverage, it will be flagged as a partial branch, and
coverage won't be at 100%. To fully test it, you'll need:

```python
def toast(bread: str) -> str:        # 1
    if bread == "baguette":          # 2
        bread = "toasted_baguette"   # 3
    return bread                     # 4


def test_toast_baguette():
    assert toast("baguette") == "toasted_baguette"  # 2 -> 3 -> 4

def test_toast_other():
    assert toast("brioche") == "brioche"  # 2 -> 4
```

As shown in this example, this lets you test more functionality, thus reducing
the probability of bugs.

### Security: permission check, input sanitization...

Anything that relates to security should be heavily tested, with all sorts of
inputs.

```python
import re

import pytest

REGEX = re.compile("[a-z]+")

def toast(bread: str) -> str:
    match = REGEX.match(bread)
    if not match:
        return ValueError("Invalid bread variant")
    ...


def test_toast_fail():  # note: you can use pytest's parametrize feature here.
    with pytest.raises(ValueError):
        toast("12")
    with pytest.raises(ValueError):
        toast("@#*$")
    with pytest.raises(ValueError):
        toast("<script>window.alert('hello world!')</script>")
    with pytest.raises(ValueError):
        toast("'; DROP TABLE users;")

```

### Complex error handling

If error handling is sophisticated enough and can be considered a feature (e.g.
recovering gracefully from some error) - it needs to be covered.

```python
import pytest
from doubles import expect

def toast(bread: str) -> None:
    if bread == "brioche":
        toaster.unplug()
        toaster.cleanup()
        raise ValueError()
    ...

def test_toast_fail():
    expect(toaster).unplug()
    expect(toaster).cleanup()

    with pytest.raises(ValueError):
        toaster("brioche")
```

### New functionality (even for proof-of-concept)

While it might be tempting to exclude new prototyping code from coverage, I
think it's a bad idea because that's one more thing to remember and track.
Moreover, once a feature is shipped, we usually move on to the next feature and
never clean up our tests. So, it's safer to get into the habit of covering all
new code.

## How to get to 100%

1. Start tracking and displaying the test coverage. Make it easy to find
   uncovered lines (for instance with an HTML report). Have an open discussion
   with your team.
2. Ensure all new code is fully covered, and that coverage can only strictly
   increase. Some Github bots let you enforce this.
3. Before writing new code, check the coverage of the existing functionality,
   and bring it to 100%. Try to structure your tests so that running a single
   test file (e.g. `test_toaster.py`) will cover 100% of its associated code
   file (`toaster.py`). It allows for quicker iterations.
4. Once you're at 100%, use your preferred tool's configuration to fail the
   tests if coverage is not 100%.

```
Coverage HTML written to dir htmlcov

Required test coverage of 100% reached. Total coverage: 100.00%
```

## Answer to common objections

### 100% test coverage does not mean you're testing everything right.

Absolutely - this point is explicitly stated in this article. I even give an
example situation showing how just checking the test coverage leads to missing
an important test.

100% test coverage does not make your code invulnerable, and it should
evidently not be your only metric. This article is only about the test coverage
metric.

### A test suite that covers 80% is pretty good

Absolutely. It is a good number and a good goal for any
codebase.

However, what about that remaining 20 %? Why are they not tested? Will it be clear
in 2 months why they were not tested? In 6 months? In a year? While it may make
perfect sense not to test them, you should be explicit about that decision and
keep the reason in the code.

If you don't keep the test coverage _metric_ at 100%, then you leave it up to the
code reviewer to challenge your test coverage assumption.

### 100% is a blanket rule that leaves no room for negotiation

Once again, the goal is not to cover 100% of the lines of code - it would be
almost impossible. Thanks to `no cover` markers, you can still decide to
exclude code from coverage. It actually makes this negotiation explicit in the
code, as opposed to implicit during the code review phase coverage.

Consider the example below:

```python
def toast(bread: str) -> str:
    if toast == "brioche":
        raise ValueError("...")
```

Let's say you're fixing a bug in this code, and you find out that the `if`
branch is not covered. You're left to wonder why. The developer did not have
enough time? They decided it was too trivial to need a test?

With an explicit marker, the intent is clear and gives you insight with what is
considered appropriate coverage for _this_ codebase:

```python
def toast(bread: str) -> str:
    if toast == "brioche":  # pragma: no cover, trivial error case
        raise ValueError("...")
```

### It is not feasible in an old codebase

Right, that's probably the case. However, this is not different from proper
testing, monitoring, logging practices. Decide if it's important, then start
with something attainable that brings you closer to the goal, and iterate.

Also, if it is not a desirable goal for the codebase, then for sure don't monitor
test coverage!

### Enforcing 100% test coverage leads to bad tests

It bears repeating: this is not about testing 100% of the lines; this is about
keeping the code coverage metric at 100%. I am not sure how that would lead to
bad tests.

Putting too much focus on one particular metric might lead to gaming behavior.
That does not mean that no metric you should be enforced. I believe that there
isn't enough material and training about what it takes to write good tests.
This is beneficial regardless of whether you enforce test coverage. Also, the
more you talk about test coverage, the more you iterate and learn about
what it takes to write good tests in your language and codebase.

What's sure is that it is straightforward to write bad tests. It takes a lot of
skills, experience, and hygiene to write great, maintainable tests. That's a
topic for another article.

### 100% coverage only matters for languages without a type checker

No. Sure, a type checker might catch some classes of bugs that would require a
test in a language without type checking, but you still need to test
correctness (among others).

### You're creating blind spots

Unless you're reviewing your test coverage report after every single commit,
leaving explicit markers in the code and keeping the metric at 100% is actually
a much safer practice.

When working in code that has been excluded, you'll immediately see the `no cover` marker, perhaps with a comment explaining why the code is excluded. This lets you reconsider the coverage decision.

Any regression in coverage will break the tests.

### You should not exclude code from coverage because of setup cost

This article is not about end-to-end vs. unit testing. I have provided some
examples of code that I sometimes exclude from testing, but your mileage may
vary.

## Changelog

- 09/07/2019: added answer to common objections
- 09/06/2019: added summary, clarified that not testing complicated should only
  apply to extreme cases and a few other points.


All lines of code should be covered by a test or explicitly excluded from test coverage.

Test coverage only matters if it's at 100%


How did Uber go from a 450,000 lines monolithic Python application to more than 1,000 microservices? This short presentation focuses on the technical aspects of this 5-year migration, and concludes with its cultural and management challenges.

I did this talk at the [DevFest](https://devfest.gdgnantes.com/) 2017 in Nantes, France.

<iframe width="560" height="315" src="https://www.youtube.com/embed/O25xbCq_LFU?rel=0" frameborder="0" allowfullscreen></iframe>

<script async class="speakerdeck-embed" data-id="61e701c03a7440b680367ef80c6c5ac5" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>

Feel free to also checkout my [professional-programming repo on Github](https://github.com/charlax/professional-programming) which compiles a list of useful resources for programmers.


How did Uber go from a 450,000 lines monolithic Python application to more than 1,000 microservices?

The great microservices migration


It’s always interesting to apply ideas, principles and notions from one field to another. Especially when those two fields are very different.

For instance, software and product design look very different on the surface. While the former is supposedly about pure mathematical rationality, the latter is, at least apparently, about emotions, psychology i.e. about fathoming the messiness of the human mind and the physical world.

Is that true? I don’t think so. Code is written primarily for human to read [1]. Code also usually fixes problems that are coming from the physical world, and thus inherits some of its irrationality.

So, what happens if we apply one of the most regarded designers’ principles to code? Namely, how can Dieter Rams’ [ten principles for good design](https://en.wikipedia.org/wiki/Dieter_Rams) [2] enable us to write better code?

<img src="http://40.media.tumblr.com/9022d63c7546866dc64aaa05b35275ae/tumblr_nhjmijf3EF1qjfk8po1_1280.jpg" width="70%" />

## Good code is innovative

> The possibilities for progression are not, by any means, exhausted.
> Technological development is always offering new opportunities for original
> designs. But imaginative design always develops in tandem with improving
> technology, and can never be an end in itself.

Some problems our code is solving are very old (e.g. moving people across a city). How can we use software, and the machine's capabilities, to better solve them? Even when we’re already using computers (e.g. to scan signed paperwork), how can we use them to go one step further (e.g. ask for an electronic signature), delivering a better experience to the end user?

Even if you have a well established product that is deemed very innovative today, you’re still at the mercy of a new competitor out-innovating you. The tech sector is full of such examples. Companies that stay successful are the ones which don’t fear to constantly disrupt themselves through technological innovation.

## Good code makes a product useful

> A product is bought to be used. It has to satisfy not only functional, but also
> psychological and aesthetic criteria. Good design emphasizes the usefulness of
> a product whilst disregarding anything that could detract from it.

Code is written to be used. As I wrote in [We should not ship code](/posts/2013-08-16-we-should-not-ship-code), we don’t code for the sake of coding. We code because that’s how a lot of problems can be solved.

Because code is an intellectual product, its usability needs to be even more an area of focus. Front-end interfaces and back-end API need to provide good affordance, i.e. their capabilities needs to be self-evident for the user.

Using the pareto principle can be useful here: the codebase should stay focused on the problem at stake, with more than 80% of its LOC being about the business problem. Otherwise, those other functionalities need to be put in libraries, or other services.

## Good code is aesthetic

> The aesthetic quality of a product is integral to its usefulness because
> products are used every day and have an effect on people and their well-being.
> Only well-executed objects can be beautiful.

Note that while this obviously applies to the web interface or the mobile app’s interface, it also applies to a back-end API.

Since code is written to be primarily read by human beings, there’s no reason it can’t take inspiration from literature. Describing how code can be made to be aesthetically pleasing would take a much longer blog post, but here’s some ideas:

- Consistency of code style
- Structure showing clarity of thoughts
- Absence of cruft (`TODO`, `FIXME`, etc.)
- Orthography and grammar for comments
- Appropriate naming
- Conciseness of code, comments, naming, function
- Reusability of concepts and routines

## Good code makes a product understandable

> It clarifies the product’s structure. Better still, it can make the product
> clearly express its function by making use of the user's intuition. At best, it
> is self-explanatory.

Code, more than anything, has a tendency to drift away from the business’ reality. This is one kind of technical debt. When the abstractions that are used in code don’t map the physical world’s reality, they need to be refactored.

One good way to identify when code is impeding a product’s understandability is to document it. If you find yourself having to give too much context about how code and reality differ, then it means code can be improved.

Documentation has a tendency to become out of date, so if you need to invest some time, it’s probably safer to invest it in refactoring the code (taking the opportunity of improving the product as well, refactoring for the sake of refactoring rarely brings enough value) to make it more self-documenting.

## Good code is unobtrusive

> Products fulfilling a purpose are like tools. They are neither decorative
> objects nor works of art. Their design should therefore be both neutral and
> restrained, to leave room for the user’s self-expression.

Code can go too far. Overusing abstraction (here’s an interesting article about how [Go is an anti-abstraction language](http://jeremymikkola.com/posts/2015_08_28_warming_up_to_go.html)), using fancy data structures, overly complex libraries (for instance, using an ORM where it’s not needed), reinventing the wheel: all of those get in the way of having code that is actually maintainable.

This is another point I’ve made in [Maslow’s pyramid of code review](/posts/2015-02-18-maslows-pyramid-of-code-review). Code that is elegant for the sake of being elegant and at the expense of being correct is just getting in the way, and should be refactored or removed.

## Good code is honest

> It does not make a product more innovative, powerful or valuable than it really
> is. It does not attempt to manipulate the consumer with promises that cannot be
> kept.

Over-engineering a solution is a very powerful temptation. Even when a company is moving really fast, and seems to be very far from over-engineering, it can still invest in the wrong area: invest in the wrong tooling too soon, build something from scratch when there are some better off-the-shelf alternatives, etc.

## Good code is lasting

> It avoids being fashionable and therefore never appears antiquated. Unlike
> fashionable design, it lasts many years – even in today’s throwaway society.

More than in any other domain, software development is plagued with short-lived programming libraries, frameworks, patterns. Pick solutions that are battle-tested, and be conservative in your technical choices.

Write tests so that functionality can’t be removed and changed without the future maintainer getting explicit feedback about it.

## Good code is thorough down to the last detail

> Nothing must be arbitrary or left to chance. Care and accuracy in the design
> process show respect towards the user.

Edge cases needs to be accounted for, without going into over-engineering. While it’s fine to sacrifice certain use cases when building an MVP, it’s not what makes the end product delightful. When code elegantly accounts for all edge cases (for instance, through using patterns such as [strategy](https://en.wikipedia.org/wiki/Strategy_pattern)), maintenance is easy and edge cases can be fixed as they come, leading to an amazing user experience.

Also, code is not enough. There’s a lot of context around code that needs to be there:

- Documentation
- Monitoring (operational and business monitoring)
- Alerting
- Testing (unit, integration, end-to-end, capacity)
- Logging and ability to be introspected and debugged

## Good code is environmentally-friendly

> Design makes an important contribution to the preservation of the environment. > It conserves resources and minimizes physical and visual pollution throughout
> the lifecycle of the product.

Code consumes electricity, which is finite resource. When code is made more performant, it not only has an impact on the customer, but also on the environment.

Good code also makes efficient use of data structure and algorithm, and promote reusability. Too often the same piece of functionality is copy pasted between codebases, leading to huge loss in developer efficiency. They should be put in libraries, and open sourced when possible.

## Good code is as little code as possible

<img src="https://upload.wikimedia.org/wikipedia/commons/d/dc/606-Universal-Shelving-System.jpg" width="70%" />

> Less, but better – because it concentrates on the essential aspects, and the
> products are not burdened with non-essentials.
> Back to purity, back to simplicity.

This is my preferred principle. I already talked about it in [We should not ship code](/posts/2013-08-16-we-should-not-ship-code), but the basic idea is that software engineering is not primarily about writing code. It’s about solving problems. It just so happens that a lot of problems can be solved through code - but that does not mean that code solves everything.

How do we write less code?

- By finding existing, off-the-shelf solution and avoiding the NIH (not invented here) syndrome.
- By focusing on the problem at stake, escaping the YAGNI (you ain’t gonna need it) syndrome.
- By spending enough time thinking about the problem, without writing code, so as to design the simplest solution possible (but not any simpler).

Why having less code is better in the long term is pretty obvious: less maintenance, (usually) more performance, less cognitive load for new developers, etc.

## Conclusion

Because code is as much a literary experience as it is a rational undertaking, using principles coming from product design can be useful to think and talk about code.

References:

- [1]: Harold Abelson, Structure and Interpretation of Computer Programs
- [2]: Wikipedia contributors, "Dieter Rams," Wikipedia, The Free Encyclopedia, https://en.wikipedia.org/w/index.php?title=Dieter_Rams&oldid=682540168 (accessed October 1, 2015).
- [3]: Vitsoe at en.wikipedia [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia Commons


Taking inspiration from Dieter Rams' ten principles for good design, here are the ten principles of good code.

10 principles for good code


As in Maslow's pyramid, each layer requires the previous one. It is useless for code that is charging the wrong customer to be readable.

Code should be:

- **Correct**: does the code do what it's supposed to? Does it handle edge cases? Is it adequately tested to make sure that it stays correct even when other engineers modify it? Is it performant enough for this use case?
- **Secure**: does the code have vulnerabilities? Is the data stored safely? Is personal identification information (PII) handled correctly? Could the code be used to induce a DOS? Is input validation comprehensive enough?
- **Readable**: is the code easy to read and comprehend? Does it make clear what the business requirements are (code is written to be read by a human, not by a computer)? Are tests concise enough? Are variables, functions and classes named appropriately? Do the domain models cleanly map the real world to reduce cognitive load? Does it use consistent coding convention?
- **Elegant**: does the code leverage well-known patterns? Does it achieve what it needs to do without sacrificing simplicity and conciseness? Would you be excited to work in this code? Would you be proud of this code?
- **Altruist**: does the code leave the codebase better than what it was? Does it inspire other engineers to improve their code as well? Is it cleaning up unused code, improving documentation, introducing better patterns through small-scale refactoring?

![](https://36.media.tumblr.com/72e7200921159a4374b7fc163fe0f6f2/tumblr_njwlh7rZui1qgj0nao1_400.png)

_Update Feb 18th_: there's some great comments on [reddit](https://www.reddit.com/r/programming/comments/2wau2x/maslows_pyramid_of_code_review/) about this. I removed the notion of large refactor because I think small refactors have a higher probability of success. I also added a comment about why layering matters.


Maslow's pyramid can help us think about what great code reviews look like.

Maslow's pyramid of code review


> To win a race, the swiftness of a dart
>
> Availeth not without a timely start.

[The Hare and the Tortoise](<http://en.wikisource.org/wiki/The_Hare_and_the_Tortoise_(de_La_Fontaine)>)

Etsy's engineering blog "[Code as Craft](http://codeascraft.com/about/)" can
easily be considered one of the best ones in the industry.

In their definition of "craftsmanship", they make this long quote from [The Pragmatic
Programmer](http://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X):

> The construction of software should be an engineering discipline. However, this doesn’t preclude individual craftsmanship. Think about the large cathedrals built in Europe during the Middle Ages. Each took thousands of person-years of effort, spread over many decades. Lessons learned were passed down to the next set of builders, who advanced the state of structural engineering with their accomplishments. But the carpenters, stonecutters, carvers, and glass workers were all craftspeople, interpreting the engineering requirements to produce a whole that transcended the purely mechanical side of the construction. It was their belief in their individual contributions that sustained the projects: We who cut mere stones must always be envisioning cathedrals. (Quarry worker’s creed).

There are two parts about craftsmanship: one that is intrinsic, and one that is
extrinsic.

The extrinsic part is perhaps the most obvious one: craftspeople **think
holistically** about what they're building. In some sense, what they aim to
accomplish is much bigger than their work. A great woodworker experiences
empathy about her customer and aspires to build something that will delight the
end-user. They not only support a use case, but go much further and let the
produced artifact transcends it. Well-crafted code not only fixes
a problem, it fixes it elegantly. It does not show any rough patch and
feels a natural solution. The consequence of that is when a problem is not
well-defined, or when the timeline set to fix it is not feasible, engineers
should push back - or at least openly share their concerns, just like a good woodworker would
do. Nobody would ever ask a woodworker to take shortcuts when he's building
the house's framework.

The intrinsic part is about **taking pride in one's work and reflecting about
it**. It's about sharpening one's tools and adding new ones. A great
woodworker would talk for hours about his tools. It's highly probable that his
workshop will be tidy, following the 5S methodology (more about that below).
There's no reason software engineers should feel differently. On the contrary,
they should always be learning, and a great way to learn is to strive for code
quality. When you optimize for speed, you usually don't take time to learn
about your tools and end up misusing them. In the long term, this strategy will
ultimately backfire.

Engineers who take the time to learn about tools, who take time to write
testable & quality code, who take time to carefully think about the
architecture of a service or of an implementation (e.g. following design
patterns) will perhaps be a bit slower in the short term, but this will pay out
ten times in the long term (in quality but also in speed). Striving for quality
in every project, in every task, even mundane (e.g. by automating a test that
could totally be done manually) will enable you to reach the same amount of
quality much faster. Always caring about quality enables continuous
improvement: it might take one hour to learn about how to properly use
a complex but powerful feature of a Python library, but the next time you'll
have 1- probably a much more reliable and fast solution 2- trained your brain
to be constantly learning. Our industry is moving so fast that this is in my
opinion the only durable strategy.

Craftsmanship is ultimately about the worker as a person. Craftspeople are
those who learn and have fun doing it!

![](http://media.tumblr.com/e9919eb6a23489be91c11635608d3763/tumblr_inline_mtcwn0g54e1qfkyvc.jpg)

Here are the [5S](<http://en.wikipedia.org/wiki/5S_(methodology)>) (seiri,
seiton, seiso, seiketsu, and shitsuke) applied to software engineering:

- Sorting: remove code that is never run.
- Set in order: streamline deployment and management through scripts and
  automation.
- Systematic cleaning: remove all TODOs, FIXMEs and temporary code.
- Standardising: remove hacks by improving patterns, abstractions and
  architecture.
- Sustaining: use continuous integration tools to prevent backsliding.

(there's a similar take about 5S applied to software development in [this
article](http://blogs.versionone.com/agile_management/2013/07/25/5s-applied-to-agile-software-development/))

_Pictures from RaidersLight (CC BY-NC-SA) and Alan Cleaver (CC BY)_


Let's take pride in our software engineering craft.

Code as Craft


I'm often asked about resources to learn programming. Here are a few of them.

- [charlax/professional-programming](https://github.com/charlax/professional-programming): a collection of full-stack resources for programmers
- [Coursera](https://www.coursera.org/) (the best IMO)
- [Udacity](https://www.udacity.com/)
- [Udemy](https://www.udemy.com/)
- [Khan Academy](https://www.khanacademy.org/)
- [Learn Python the Hard Way](http://learnpythonthehardway.org/)
- [CodeAcademy](http://www.codecademy.com/)
- [iTunesU](http://www.apple.com/education/itunes-u/) has some good courses.

There are numerous of them, and this does not even count the books. There are also numerous articles about the best resources to learn coding, and the best method of learning.

Here's my only advice: **nothing beats learning by doing**. Try to build something meaningful and exciting to apply what you learn.


A list of resources to learn programming.

Resources to learn coding


It easy to be mistaken into thinking that a software developer's job is to ship code.

It is not.

Our job is to solve intellectual problems.

It so happens that out of all the tools humans created, computers are best at solving those, because software is great at dealing with abstractions. However, we will never get recognition from the business if we just ship code. We get recognition if we solve problems and if we do it fast.

That is why the best companies focus on problem-solving in their interviews, and not on fundamental knowledge. A quick learner can learn whatever she wants to solve things. We are in the learning business. We are always learning new programming languages, databases, libraries, services, patterns... to be faster at solving problems. We also need to learn about the problems itself, and that is why waterfall is rarely a good thing: how can developers fix business problems if they cannot build their understanding of it?

Yesterday, during our show & tell session one developer demonstrated a remarkable tool that would fix one of our long-standing problems. This tool requires minimal integration. A problem was solved without code.

We are in the business of solving intellectual problems. The business is never going to measure our ROI with the number of lines of code we write. The only thing it is concerned with is whether or not the problem was solved efficiently.

It does not mean that everything has to be simple, nor this is a good excuse for fast and loose solutions (ultimately those will make the code less maintainable, and it will be very costly to solve new problems, as requirements tend to grow over time). We should take pride in elegantly solving complex problems. Some complex problems might require a complex solution. That is fine. Just like woodworkers prides themselves in their mastery of their tools, we should be proud of how much we master our tools. However, our skills are not the end goal. The solution is.


An engineer's job is not to write code - it is to solve business problems.

We should not ship code


## Dependencies

`graphviz` is required for the call graph.

```bash
$ brew install qcachegrind graphviz
$ pip install pyprof2calltree
```

## Profiling

Profile your code with `cProfile` or `Profile`. `pyprof2calltree` won't work with `hotshot`. Store the profile data in a `.pstats` file.

```bash
$ python -m cProfile -o profile.pstats yourscript.py
```

Then run:

```bash
$ pyprof2calltree -i profile.pstats -o profile.kgrind
```

Then open this `kgrind` file with `qcachegrind`:

```bash
$ qcachegrind profile.kgrind
```

![](http://media.tumblr.com/8faf28c1b1d9227330b4f185c48982cd/tumblr_inline_mspsujDXvH1qz4rgp.png)


How to visualize your code's profile with qcachegrind.

Using qcachegrind to visualize Python profiling data


Disclaimer: this article only conveys my personal point of view. Please excuse my English!

## Introduction

I tried numerous text editors: Komodo, Emacs, Nano, Textwrangler, gedit, SPE, [Sublime](http://www.sublimetext.com/), Coda, [Vico](http://www.vicoapp.com/)...

When I discovered TextMate, I really loved it because it was both flexible and
Mac-friendly. Even today, I think that it's a great tool that can greatly
improve productivity.

Yesterday, MacroMates [released](http://blog.macromates.com/2011/textmate-2-0-alpha/) the first public
Textmate 2.0 alpha version. Its authors acknowledged the long wait that preceded
this release by stating that it would "provide some relief to those who have been questioning TextMate’s future".

I began using Vim two weeks ago, and right now I don't see any reason
why I would get back to TextMate. I'll try to explain in details how I got
to use Vim and what are its (numerous) advantages.

## Vim: [Je t'aime moi non plus](http://en.wikipedia.org/wiki/Je_t'aime..._moi_non_plus)

Maybe some other hackers will recognize themselves in this
behavior: even though I've been using Vim for a long time, **I never
took the time to really learn it**. I used to think that Vim was too old to be
interesting: the modal behavior looked really old-fashioned and
counter-intuitive, to say the least.

What's paradoxical is that I kept on using Vim, primarily because it's on all
computers and it's really efficient when modifying system files. Even when
I tried Emacs, I kept on using Vim because it felt so much faster!

With the benefit of hindsight, it was really strange because for all the other
tools I was always trying to improve my productivity (learning keyboard
shortcuts, tuning configuration files, looking for ways to avoid repetition)
but I never took the time to really learn Vim: for instance, two weeks ago
I learned that entering `a` in normal mode would go into insert mode _after_
the cursor…

Why did I suddenly became aware of this paradox?

- I read numerous articles about Vim on Hacker News (I guess it's **useful
  procrastination**).
- TextMate was missing key features like split windows.
- I felt that Vim offered so much flexibility and modularity that it
  would be great to test it.

## Vim's numerous advantages

**Vim is free and open source.** Well-known open source projects are
superior in the long term because they are not dependent on a company's
objectives.

**Vim is superior for text manipulation.** Most of our
work in an editor consists in text manipulation. Contrary to Emacs'
painful shortcuts (from my point of view), Vim's modal mode provides an
_elegant_ and efficient interface. What is more, most of the shortcuts are
easily memorable. TextMate does not provide such a great combination of
ease of use and effectiveness.

**Vim is empowering.** As Bram Moolenaar (Vim's main author) [puts
it](http://www.moolenaar.net/habits.html), "if you spend a lot of time
typing plain text, writing programs or HTML, you can save much of that
time by using a good editor and using it effectively." Vim is great
because it really puts you in charge. The learning curve is quite steep
but it's what's exciting!

**Vim has a vibrant community.** I agree that it's also the case for
Emacs and Textmate. For those who still thinks Vim is old, have a look
at all those repositories on Github!

**Vim's scripting language is straightfoward.** Again, this is really
personal, but I find Lisp not very well adapted to scripting. I really
like Vim's streamlined scripting and configuration files. And you can
also use Python or Ruby.

**Vim's vision is compelling.** As Jeff Bezos says, "maintain a firm grasp of
the obvious at all times." To me it is common sense that a text editor should
focus on efficiency and text manipulation, as Brian Moolenaar puts it:

> It has always been the spirit of Unix to have separate programs that
> do their job well, and work together to perform a bigger task.
> Unfortunately, most editors don't work too well together with other
> programs - you can't replace the e-mail editor in Netscape with
> another one, for example. You end up using a crippled editor. Another
> tendency is to include all kinds of functionality inside the editor;
> Emacs is a good example of where this can end up. (Some call it an
> operating system that can also be used to edit text.)

Vim follows the Unix philosophy, which makes it extremely easy to use it
with other programs. It is not the case for Textmate.

**An editor's configuration files should be put under version control.**
A text editor is arguably the single most important tool for
a developer. This is why it should be possible to constantly improve one's
productivity. TextMate configuration files are not as easy to tweak.

**Vim is everywhere.** Vim is great for sysadmin because it's present on
most computers.

## Stuff I don't like in Vim

- We are in 2012, the user should not have to download syntax files and activate them. It should be the default.
- It should use Perl's regex syntax by default.
- Help files are quite difficult to follow.

More generally, defaults could be improved. For instance, plugins management
could be more automated, instead of having to install Pathogen and using `git submodules`. This could happen outside Vim, with a kind of universal repository
for plugins and a text file that would list all the plugins (like
`.gitmodules`). Plugin would be automatically updated regularly.

Here is another idea to have great defaults: why analyzing all the `vimrc` that
have been posted on the web? This methodology has its flaws, but we would get
a sense of what are the best defaults.

## How to try Vim

If you want to try Vim, just follow these two steps:

- Do **not** use any plugin or anyone else's vimrc. [Really](http://mislav.uniqpath.com/2011/12/vim-revisited/).
- Use `vimtutor` to learn the basics.

## Other articles

- [Checkout my vim configuration](https://github.com/charlax/dotvim)
- [Seven habits of effective text editing](http://www.moolenaar.net/habits.html)
- [Everyone Who Tried to Convince Me to use Vim was Wrong](http://yehudakatz.com/2010/07/29/everyone-who-tried-to-convince-me-to-use-vim-was-wrong/)
- [Vim: revisited](http://mislav.uniqpath.com/2011/12/vim-revisited/):
  a must-read.
- [Vimcasts](http://vimcasts.org/)

![](http://media.tumblr.com/tumblr_lw7fsqf9ge1qfkyvc.png)


Why taking the time to learn Vim is a smart investment.

Why I keep on using Vim instead of going back to TextMate


These articles were written in 2000, but are still relevant today:

- [Painless Functional Specifications - Part 1: Why Bother?](http://www.joelonsoftware.com/articles/fog0000000036.html)
- [Painless Functional Specifications - Part 2: What's a Spec?](http://www.joelonsoftware.com/articles/fog0000000035.html)
- [Painless Functional Specifications - Part 3: But... How?](http://www.joelonsoftware.com/articles/fog0000000034.html)
- [Painless Functional Specifications - Part 4: Tips](http://www.joelonsoftware.com/articles/fog0000000033.html) (if you have time for only one article, read this one)

Key excerpts:

## Part 1

> So that's giant reason number one to write a spec. Giant reason number two is
> to save time communicating. When you write a spec, you only have to
> communicate how the program is supposed to work once. Everybody on the team
> can just read the spec. The QA people read it so that they know how the
> program is supposed to work and they know what to test for. The marketing
> people use it to write their vague vaporware white papers to throw up on the
> web site about products that haven't been created yet. The business
> development people misread it to spin weird fantasies about how the product
> will cure baldness and warts and stuff, but it gets investors, so that's OK.
> The developers read it so that they know what code to write. The customers
> read it to make sure the developers are building a product that they would
> want to pay for. The technical writers read it and write a nice manual (that
> gets lost or thrown away, but that's a different story). The managers read it
> so that they can look like they know what's going on in management meetings.
> And so on.

## Part 2

> When you're building a product with a team, everybody tends to have their
> favorite, real or imagined pet features that they just can't live without. If
> you do them all, it will take infinite time and cost too much money. You have
> to start culling features right away, and the best way to do this is with a
> "nongoals" section of the spec. Things we are just not going to do. A nongoal
> might be a feature you won't have ("no telepathic user interface!") or it
> might be something more general ("We don't care about performance in this
> release. The product can be slow, as long as it works. If we have time in
> version 2, we'll optimize the slow bits.") These nongoals are likely to cause
> some debate, but it's important to get it out in the open as soon as possible.
> "Not gonna do it!" as George Sr. puts it.

## Part 3

> Program managers are invaluable. If you've ever complained about how
> programmers are more concerned with technical elegance than with
> marketability, you need a program manager. If you've ever complained
> about how people who can write good code never do a good job of writing
> good English, you need a program manager. If you've ever complained about
> how your product seems to drift without any clear direction, you need a
> program manager.

## Part 4

> So. Specs are good, but not if nobody reads them. As a spec-writer, you
> have to trick people into reading your stuff, and you should also probably
> make an effort not to cause any already-too-small brains to leak out
> through eye-sockets.


Some quotable quotes about functional specs.

dein.fr

Category: code