Monthly Archives

December 2017

Upgrade Python to support Django 2.0

Upgrading Python To Support Django 2.0

By Agile, Development, DevOps, Django, Python

Across the land, there are many developers, dev ops, and software delivery managers are terrified of the big move from Python 2.7 to Python 3.6 (at the time of writing this).  You can see all of the versions from the beginning of time.    I am going to walk you through why it is happening, how to plan, and more importantly, how to upgrade your infrastructure in a systematic manner.

Assumptions:

  • There are multiple developers on the team
  • There are multiple Django instances in your organization

Why move? Django

The answer is pretty easy… Django, the core web platform programming framework associated with the Python  programming language won’t support future releases in Python 2.7 any more. In fact, the Django 1.11.x series is the last to support Python 2.7.

Django 2.0 supports Python 3.4, 3.5, and 3.6. We highly recommend and only officially support the latest release of each series.

Note according to PEP 373, Python 2.7 is currently expected to remain supported by the core development team (receiving security updates and other bug fixes) until at least 2020 (10 years after its initial release, compared to the more typical support period of 18–24 months).

How to Plan?

When it comes time to plan for an event like this, it is important that you identify the critical components of your infrastructure.

Sample User Story

As a developer, would like to have a complete list of technical assets that use Python so I can generate an upgrade plan that reduces risk to the company.

Acceptance Criteria: Generate a list of the following assets which include:

  • Your Django platforms to include, dev, test, and production instances.
  • Other internal platforms ( AWS lambdas, small flask instances)
  • External/Internal libraries
  • Cronjob or schedule tasks running Python
  • Continuous Integration systems that build  (ie Jenkins)
  • Unit testing
  • QA regression testing scenarios
  • Code repositories
  • Your development team list (each one of them will have to complete upgrades)

Step 1 – Select Your First System

Pick your first system to do an end to end test.  If you only have one single platform of Django, then… you are done.  We recommend that you clone your dev instance for your first test.  If you don’t have a dev instance, stop reading this and make sure that you do have a dev, test and production version of your platform!

If you have multiple systems, we recommend that you select one that is much smaller in scope and will have as little impact on your operations as possible.

Sample User Story:

As a developer, I would like to select my first computing environment to upgrade Python on so I can minimize the impact on our operations.

Acceptance Criteria:

  • You selected one of the least impactful systems in your eco-systems
  • You clone a working environment that it identical to this instance.
  • You notify the team of what you are doing and discuss any impacts that your testing might have with them.

Step 2 – Start your documentation

If you have other people on the team, you will want to make sure that you can guide them along the path of upgrading their computing environments and be able to discuss problems that you had.

Sample User Story:

As a developer, I would like to document the upgrade process steps that  I took so I can help others on the team upgrade their computing environments to Python 3.6 with Django.

Acceptance Criteria:

  • Document your finding in confluence.

Step 3 – Review external dependencies

Not all pypi libraries are Python 3.6.3 compliant.  Many are only built on Python 2.7.  The good news however, is that most major libraries have been ported to 3.6.x.  The complete list of PyPi libraries per version is listed here.

Individual Library Inspections

As a developer, you can see the details of any pip installed library by calling this command: pip show LIBRARYNAME –verbose

Example:

(bytelion_env)~>pip show requests –verbose
Name: requests
Version: 2.13.0
Summary: Python HTTP for Humans.
Home-page: https://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: Apache 2.0
Location: /Users/terrancemacgregor/.virtualenvs/bytelion_env/lib/python2.7/site-packages
Requires:
Metadata-Version: 2.0
Installer: pip
Classifiers:
Development Status :: 5 – Production/Stable
Intended Audience :: Developers
Natural Language :: English
License :: OSI Approved :: Apache Software License
Programming Language :: Python
Programming Language :: Python :: 2.6
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Programming Language :: Python :: 3.6
Programming Language :: Python :: Implementation :: CPython
Programming Language :: Python :: Implementation :: PyPy

Sample User Story

As a developer, I would like to know which PyPi libraries are not supported by my current system so I can determine a migration strategy.

Acceptance Criteria

  • List is generated and shared with the team
  • For each library that is not supported, identify an alternative.

 

Remember the 80/20 rule?

Many of the upgrades from 2.7 to 3.6 can be safely automated, some other changes (notably those associated with Unicode handling) may require careful consideration, and preferably robust automated regression test suites, to migrate effectively.

 

Upgrading Server Code

 

Programming Language :: Python

Here is a list of what some other people experienced:

https://www.calazan.com/upgrading-to-ubuntu-1604-python-36-and-django-111/

https://blog.thezerobit.com/2014/05/25/python-3-is-killing-python.html

image recognition

Image Recognition of Basic Shapes | Google Vs Amazon

By Tips & Tutorials, Tools

This article was written for an individual or company that is looking to conduct image recognition of basic shapes in their operations. 

The Problem

Bytelion was recently approached by a client seeking to ID basic shapes via image recognition. The premise being: A computer takes a snapshot of a shape, validates its ID and provides validation. There are many options and tools out there, so we wanted to see how Google and Amazon stacked up each other.

How Does Image Recognition Work?

Image recognition is a form of machine learning designed to recognize patterns in data. Once an image is digitized it is just another form of data. Therefore, attributes that are present in multiple sets of images of one thing can inform us that an element of those images always remains consistent.

For instance, multiple pictures taken of a dog from the same angle but in different lighting conditions would still retain the basic ‘outline’ of the dog, if not the same coloring/shading etc. Labeling this set of graphics ‘dog’ informs the computer that these are images of dogs and that similar images presented are also ‘dogs’. The more images of dogs you provide initially, the better the computer is at guessing whether other images have dogs in them.

Google vs Amazon

There are numerous services available for image recognition, but we decided to test the two leading options: Amazon’s ‘Image Rekognition’ and Google’s ‘Vision API’.

Note:
Each services has its own pros and cons. It is best to fully flesh out your use cases before choosing which service to use.

Testing Conditions

Studio

Bytelion’s high-tech photography studio

We conducted rounds of testing in both Amazon and Google’s platforms using each of their default image comparison libraries.  All of our records are stored in AWS in this s3 bucket.

For our ‘photo studio’ we set up a camera mounted and angled above several books (6 inches high). We took images of 7 different shapes, each image taken against a blank background twice in different positions and twice again against a mesh background. We kept consistent light conditions throughout the process (average Lux count of 360, using this meter).

Definitions:

ConfidenceThe degree/percentage of probability the image recognition system associates to its given result.

NAThe resulting output does not match the shape presented. That is, the input image did return other results but they are not relevant and therefore have been redacted from this article.

Test Results:

Circle

circles

4 variants of the same circle

 

Image Google Vision Confidence Amazon Rekognition Confidence
Circle Against White Background 1 72% NA
Circle Against White Background 2 66% NA
Circle Against Mesh Background 1 52% NA
Circle Against Mesh Background 2 NA NA

As seen by the data presented above, Google Vision is the clear winner. While the confidence wasn’t strictly high, Amazon Rekognition was unable to determine that a circle was present in any of the images.

Note: NA means the results didn’t contain a circle in the listed guesses. Also, just because there is a match, does not mean it was the only match, or most likely match.

Triangle

4 variants of the same triangle

 

Image Google Vision Confidence Amazon Rekognition Confidence
Triangle Against White Background 1 69% 63%
Triangle Against White Background 2 63% 96%
Triangle Against Mesh Background 1 NA 95%
Triangle Against Mesh Background 2 NA NA

This round went to Amazon Rekognition. Not only did it identify the triangle in one more image, it also detected a triangle with greater confidence.

All Other Shapes

The following shapes were also tested in both systems in the same format as above:

  • Oval
  • Ring
  • Semi-Circle
  • Octagon
  • Bracket

Unfortunately, neither platform was able to find comparable imagery within their libraries for these shapes. Ranged from having no matches at all to having some matches but none of them the correct shape.

Summary & Conclusion

We cannot say with much confidence that the standard services provided by image recognition services are suitable for detection of basic shapes with out of the box configurations. Default libraries have proven better at identifying abstract imagery (e.g scenery, mountains, animals, people etc) for categorization rather than explicit identification.

Even with the best recognition models in imaging AI, at best there is a 96.54% chance of having a correct match in the top 5 guesses (ref: benchmarking). Maintaining this level of accuracy requires consistent training. Large data sets may require further training time as well as multi-thousand dollar machines. Re-training the same (base) system for a specialized use (such as our case above) can be done in far less time with regular computing power.

Moving Forward

If you really want to identify basic shapes, you will need to conduct your own machine training.  A five minute video on how to do this is here.  Note, it will take you a little longer than 5 minutes.  :- )

Bytelion is a Software as a Service Company.
Have questions? Feel free to contact us at info@bytelion.com

Enabling PostGIS on PostgreSQL with Amazon Web Services(AWS) | Relational Database Service(RDS)

By AWS, Databases, GIS

Part 1 of our PostGIS series is to help developers enable PostGIS for a PostgreSQL database in AWS.

Amazon’s Relational Database Service (RDS) allows users to launch PostgreSQL databases easily and quickly. PostgreSQL allows developers to extend the functionality of the core database via extensions.  For this reason, extensions loaded into the database can function just like features that are built in.

Because this database is on Amazon’s servers, they have limitations on what extensions the will allow to be hosted on their service.  These limitations are mostly for performance reasons.

PostGIS Version

By default, the database has PostGIS installed on it. This isn’t critical to run, but it may be helpful in your troubleshooting efforts.

To get the current version, use this:

SELECT PostGIS_full_version();

Result:

POSTGIS=”2.3.2 r15302″ GEOS=”3.5.1-CAPI-1.9.1 r4246″ PROJ=”Rel. 4.9.3, 15 August 2016″ GDAL=”GDAL 2.1.3, released 2017/20/01″ LIBXML=”2.9.1″ LIBJSON=”0.12″ RASTER

Normal Installation Guidance

If you follow PostGIS’s guidance here, you will get the following error.

When you run this command:
CREATE EXTENSION postgis_sfcgal;

ERROR: Extension “postgis_sfcgal” is not supported by Amazon RDS
DETAIL: Installing the extension “postgis_sfcgal” failed, because it is not on the list of extensions supported by Amazon RDS.
HINT: Amazon RDS allows users with rds_superuser role to install supported extensions. See: SHOW rds.extensions;

AWS RDS PosGIS Installation Guidance

AWS covers enabling PosGIS via their common database administrator (DBA) tasks here.

In about 3 minutes, you can run through these commands and enable PosGIS tables in your database.

AWS Supported Extensions

This is a pretty nifty command to see what extensions are displayed on the device.

SHOW rds.extensions;

Here is a list of the Posgresql Extentions that RDS does support:

address_standardizer
address_standardizer_data_us
bloom
btree_gin
btree_gist
chkpass
citext
cube
dblink
dict_int
dict_xsyn
earthdistance
fuzzystrmatch
hll
hstore

hstore_plperl
intagg
intarray

ip4r
isn
log_fdw
ltree
pgaudit

pgcrypto
pgrouting
pgrowlocks
pgstattuple
pg_buffercache
pg_freespacemap
pg_hint_plan
pg_prewarm
pg_repack
pg_stat_statements
pg_trgm
pg_visibility
plcoffee

plls
plperl
plpgsql

pltcl
plv8
postgis
postgis_tiger_geocoder
postgis_topology
postgres_fdw
sslinfo
tablefunc
test_parser
tsearch2
tsm_system_rows
tsm_system_time
unaccent
uuid-ossp