Category

DevOps

Upgrade Python to support Django 2.0

Upgrading Python To Support Django 2.0

By Agile, Development, DevOps, Django, Python

Across the land, there are many developers, dev ops, and software delivery managers are terrified of the big move from Python 2.7 to Python 3.6 (at the time of writing this).  You can see all of the versions from the beginning of time.    I am going to walk you through why it is happening, how to plan, and more importantly, how to upgrade your infrastructure in a systematic manner.

Assumptions:

  • There are multiple developers on the team
  • There are multiple Django instances in your organization

Why move? Django

The answer is pretty easy… Django, the core web platform programming framework associated with the Python  programming language won’t support future releases in Python 2.7 any more. In fact, the Django 1.11.x series is the last to support Python 2.7.

Django 2.0 supports Python 3.4, 3.5, and 3.6. We highly recommend and only officially support the latest release of each series.

Note according to PEP 373, Python 2.7 is currently expected to remain supported by the core development team (receiving security updates and other bug fixes) until at least 2020 (10 years after its initial release, compared to the more typical support period of 18–24 months).

How to Plan?

When it comes time to plan for an event like this, it is important that you identify the critical components of your infrastructure.

Sample User Story

As a developer, would like to have a complete list of technical assets that use Python so I can generate an upgrade plan that reduces risk to the company.

Acceptance Criteria: Generate a list of the following assets which include:

  • Your Django platforms to include, dev, test, and production instances.
  • Other internal platforms ( AWS lambdas, small flask instances)
  • External/Internal libraries
  • Cronjob or schedule tasks running Python
  • Continuous Integration systems that build  (ie Jenkins)
  • Unit testing
  • QA regression testing scenarios
  • Code repositories
  • Your development team list (each one of them will have to complete upgrades)

Step 1 – Select Your First System

Pick your first system to do an end to end test.  If you only have one single platform of Django, then… you are done.  We recommend that you clone your dev instance for your first test.  If you don’t have a dev instance, stop reading this and make sure that you do have a dev, test and production version of your platform!

If you have multiple systems, we recommend that you select one that is much smaller in scope and will have as little impact on your operations as possible.

Sample User Story:

As a developer, I would like to select my first computing environment to upgrade Python on so I can minimize the impact on our operations.

Acceptance Criteria:

  • You selected one of the least impactful systems in your eco-systems
  • You clone a working environment that it identical to this instance.
  • You notify the team of what you are doing and discuss any impacts that your testing might have with them.

Step 2 – Start your documentation

If you have other people on the team, you will want to make sure that you can guide them along the path of upgrading their computing environments and be able to discuss problems that you had.

Sample User Story:

As a developer, I would like to document the upgrade process steps that  I took so I can help others on the team upgrade their computing environments to Python 3.6 with Django.

Acceptance Criteria:

  • Document your finding in confluence.

Step 3 – Review external dependencies

Not all pypi libraries are Python 3.6.3 compliant.  Many are only built on Python 2.7.  The good news however, is that most major libraries have been ported to 3.6.x.  The complete list of PyPi libraries per version is listed here.

Individual Library Inspections

As a developer, you can see the details of any pip installed library by calling this command: pip show LIBRARYNAME –verbose

Example:

(bytelion_env)~>pip show requests –verbose
Name: requests
Version: 2.13.0
Summary: Python HTTP for Humans.
Home-page: https://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: Apache 2.0
Location: /Users/terrancemacgregor/.virtualenvs/bytelion_env/lib/python2.7/site-packages
Requires:
Metadata-Version: 2.0
Installer: pip
Classifiers:
Development Status :: 5 – Production/Stable
Intended Audience :: Developers
Natural Language :: English
License :: OSI Approved :: Apache Software License
Programming Language :: Python
Programming Language :: Python :: 2.6
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Programming Language :: Python :: 3.6
Programming Language :: Python :: Implementation :: CPython
Programming Language :: Python :: Implementation :: PyPy

Sample User Story

As a developer, I would like to know which PyPi libraries are not supported by my current system so I can determine a migration strategy.

Acceptance Criteria

  • List is generated and shared with the team
  • For each library that is not supported, identify an alternative.

 

Remember the 80/20 rule?

Many of the upgrades from 2.7 to 3.6 can be safely automated, some other changes (notably those associated with Unicode handling) may require careful consideration, and preferably robust automated regression test suites, to migrate effectively.

 

Upgrading Server Code

 

Programming Language :: Python

Here is a list of what some other people experienced:

https://www.calazan.com/upgrading-to-ubuntu-1604-python-36-and-django-111/

https://blog.thezerobit.com/2014/05/25/python-3-is-killing-python.html

Lean Smoke Testing

By DevOps, Software Lifecycle, Testing

Summary

You are trying to develop a new application and want to ensure high developer output while installing quality control.  Before installing build servers, setting up testing scripts, etc., make sure that you have proven your market first.  The easiest and cheapest way to create quality in your application is via the developer run smoke test. A lean smoke test is a basic test of the system to ensure that 80% of the application works.  The goal of smoke tests are to ensure a high-quality routine deliverable at very low cost.  This is not a substitute for more complex testing which requires more infrastructure.

Setup Time

5 minutes to set up a template at the start of a project. 5 minutes to update at the end of every sprint with either new tests or to reduce tests. If the tests take more than 10 minutes, then they are no longer smoke tests, they have moved into the realm of regression testing.

Who

  • Software Developers  You do not need nor should you have a tester or PM run your smoke tests.  The dev team needs to be in the repeatable habit of ensuring quality without the laborious burden of detailed regression testing.
  • Scrum Master /PM Ensure that the Smoke tests are accessible via the web to every team.

Actions

For example, if the team is building a ticket sales system, a simple action or a set of simple actions like:

  1. User Looking For Events
  2. User buys tickets
  3. User prints tickets

Ideally, if you have more than one person, you will assign them a different platform, so they are routinely testing it against different operating environments to find problems before they arise.  For web development, this could be IE or Firefox.  For mobile, you could have them alternate between iOS and Android.

Frequency

Like all things, it depends.  Ideally, any time a team conducts development, a smoke test would be completed.  This, however, isn’t always a good idea and introduces waste, especially if you are building your MVP.  A more realistic scenario would be time/effort based.  As a rule of thumb, every 80 hours of development and deployment cycle time should be followed by a smoke test.  A team may elect however to run smoke tests more often if they are trying to debug a set of problems.  Additionally, if you enter production, the product owners may require a zero bug atmosphere.  If this is the case, you need to have the cost vs. benefit conversation for smoke testing.

Time Ratio

80 hours x 60 minutes per hour = 4800 minutes = 10/4800 =  0.2% of time for devs to test

 

Client Discussion Template

Client:  I tried to do XYZ and found a bug.  Your team screwed up!  I AM NOT PAYING YOU FOR BUGS.  And if you can’t tell, I am angry.

You:  We take quality very seriously. The developers, like you, want to do a great job and make you happy.  There is a balance however between having bugs and building software.

Client: You mean you are delivering shoddy work and I just have to deal with it?

You:  No, that isn’t the case.  We are building functionality into your product to make it more attractive to users and provide value to them.  We can spend more money to reduce errors.  Those efforts include building improved test features, conducting software practices like Test Driven Development, build suites of automated UI, backend testing and would most likely require us to bring another person on the team to achieve those results.   That work requires additional money, time, and management.

Client: That is crazy.  I don’t want all of that.  I just want bug free software.  You should be building software that doesn’t have any bugs.

You:  That is fine, we can do that, but it is going to cost you more money.  You control how we spend our time and what is important to you.  Using our experience, we select software libraries and techniques that will reduce bugs as much as possible.

Client:  Hmmm, well, I guess we can live with minor bugs.  What about the major ones?

You:  Agree.  We cannot have major bugs.  We will spend as little as possible to make sure that major bugs don’t happen and will start conducting small tests every sprint called smoke tests.  That should resolve most of the issues and cost very little on your part.

Client:  Ok, as long as it doesn’t cost me more money.

You:  It is a very lean option.  If we build more complexity, then we will need to invest in more testing capabilities.

Client:  If our product is flying off the shelf, then the investment will make more sense.

You:  Now you are thinking lean!